# Frequency principle/spectral bias The frequency principle (spectral bias) describes how deep neural networks learn low-frequency components of data first, then gradually higher frequencies, a phenomenon observed across architectures and tasks. The frequency principle, also known as spectral bias, is an empirically observed property of deep neural networks. It describes the tendency of these networks, during training, to first learn the low-frequency components of the target function before progressively capturing higher-frequency details. This behavior has been documented across a wide range of network architectures, including fully connected networks, [convolutional neural networks](https://www.wikiprompt.org/wiki/convolutional-neural-network), and [residual networks](https://www.wikiprompt.org/wiki/residual-network), and across various tasks such as image generation, regression, and classification. The phenomenon is often visualized by plotting the error of the network's output against frequency, showing that the error decreases more rapidly for lower frequencies early in training. This principle is closely related to the concept of spectral bias, a term used to describe the same preferential learning of low frequencies. While the two terms are often used interchangeably, the frequency principle is sometimes framed as a broader empirical observation, while spectral bias can refer to the underlying theoretical explanations. The effect has significant implications for understanding the generalization capabilities and limitations of deep learning models, particularly in tasks involving high-frequency details, such as image super-resolution or generating sharp textures. ## Theoretical Explanations Several theoretical frameworks have been proposed to explain the frequency principle. One prominent line of work, developed by researchers including Zhiqin Xu and Tao Luo, analyzes the training dynamics of neural networks in the Fourier domain. They showed that for two-layer networks, the convergence rate of the gradient descent algorithm is inversely proportional to the frequency of the target component. This means that low-frequency components, which have larger eigenvalues in the neural tangent kernel, are learned faster. The neural tangent kernel theory, which approximates a wide network's training as a linear system, provides a mathematical basis for this frequency-dependent convergence. Another explanation involves the concept of the "F-principle" (frequency principle) in the context of the loss landscape. The optimization trajectory of gradient descent tends to move in directions that reduce the loss most quickly, and for many loss functions, these directions correspond to low-frequency changes. This is analogous to the behavior of partial differential equations, where low-frequency modes decay more slowly, but in the context of gradient descent, they are learned first because they contribute more to the initial loss reduction. ## Empirical Observations The frequency principle has been observed in numerous experiments. For instance, in image generation tasks using generative adversarial networks, early training epochs produce blurry images that lack high-frequency details, with sharp edges and textures appearing only in later epochs. Similarly, in regression tasks, networks initially fit smooth approximations of the target function, then refine the fit with higher-frequency oscillations. This behavior is consistent across different activation functions, such as ReLU and sigmoid, and across different optimization algorithms, including [Adam](https://www.wikiprompt.org/wiki/adam-optimizer) and stochastic gradient descent. A notable empirical finding is that the frequency principle holds even when the training data is not uniformly distributed. For example, in cases where data is concentrated in certain regions, the network still learns low-frequency components globally first, but the local frequency content can vary. This has led to research on how data distribution and sampling strategies can influence the learning of high-frequency features. ## Implications for Deep Learning The frequency principle has several practical implications. First, it explains why deep networks often struggle with high-frequency details, such as sharp edges in images or rapidly varying signals, unless the architecture is specifically designed to handle them. This has motivated the development of architectures like [U-Net](https://www.wikiprompt.org/wiki/u-net) for image segmentation, which uses skip connections to preserve high-frequency information, and the use of positional encoding in [transformers](https://www.wikiprompt.org/wiki/transformer) to represent high-frequency features in natural language processing. Second, the principle informs training strategies. For example, curriculum learning, where models are trained on simpler (low-frequency) examples first, aligns with the natural learning order. Additionally, techniques like learning rate scheduling can be adjusted to account for the slower convergence of high-frequency components. The principle also has implications for understanding the generalization gap, as models that fail to learn high-frequency components may underfit on tasks requiring fine details. ## Relation to Other Concepts The frequency principle is connected to several other concepts in deep learning. It is related to the idea of implicit regularization, where gradient descent inherently favors simpler solutions, which often correspond to low-frequency functions. It also interacts with the phenomenon of double descent, where model performance can improve with overparameterization, partly because larger models can capture higher frequencies more effectively. Furthermore, the principle has been linked to the success of techniques like [batch normalization](https://www.wikiprompt.org/wiki/batch-normalization) and [layer normalization](https://www.wikiprompt.org/wiki/layer-normalization), which can alter the effective learning dynamics and potentially mitigate spectral bias in some cases. ## Current Research and Open Questions Research on the frequency principle is ongoing, with several open questions. One key area is understanding how the principle scales to very deep and very wide networks, and how it interacts with modern architectures like [large language models](https://www.wikiprompt.org/wiki/large-language-model) and [transformers](https://www.wikiprompt.org/wiki/transformer). While the principle has been primarily studied in the context of fully connected and convolutional networks, its applicability to attention-based models is an active area of investigation. Another question is whether the frequency principle can be leveraged to design more efficient training algorithms, for example by explicitly weighting loss contributions by frequency. Some researchers have also explored the idea of "frequency-aware" loss functions that penalize errors at higher frequencies more heavily, potentially accelerating convergence. As of the early 2020s, the frequency principle remains a robust empirical observation with a growing theoretical foundation. It provides a unifying lens through which to understand the learning dynamics of neural networks, and it continues to inspire new research in both theory and application. --- Source: https://www.wikiprompt.org/wiki/frequency-principle-spectral-bias License: CC BY-SA 4.0 (https://creativecommons.org/licenses/by-sa/4.0/) Last updated: 2026-09-14T06:28:32.655956+00:00