Can we quantify the complexity of learning? This paper introduces the concept of predictive information, Ipred(T), defined as the mutual information between the past and future of a time series, to characterize the complexity of learning. It reveals three distinct behaviors in the limit of large observation times: finite, logarithmic, or fractional power-law growth of Ipred(T). The goal is to find ways to measure complexity. The study finds that logarithmic growth occurs when learning a model with a finite number of parameters, with the coefficient representing the model space dimensionality. Power-law growth, conversely, is linked to learning infinite-parameter models, such as continuous functions with smoothness constraints. The research connects predictive information to complexity measures defined in learning theory and statistical mechanics. This framework could be applied to a variety of problems in physics, statistics, and biology, helping to quantify the complexity of underlying dynamics in diverse systems. The authors have argued that the divergent part of Ipred(T) is an accurate measurement of a times series complexity.
Published in Neural Computation, this paper fits the journal's scope by addressing computational aspects of learning and information theory. The research, connecting predictive information with complexity measures, aligns with the interdisciplinary nature of neural computation, drawing from mathematics, physics, and computer science. The content relates to computer science, technology, and neural networks.