The full manuscript is available in the TU Delft repository. This page is the shorter version: the same argument and the same results, with the proofs left in the thesis.

The question

Neural networks trained with gradient descent do not learn every part of a target function at the same speed. They fit the smooth, slowly varying part first and the oscillatory part later. This is called spectral bias, and it is easy to observe and hard to state precisely, because “the operator that governs training” is usually not something you can write down.

My thesis picks a setting where you can write it down. Inputs live on the unit circle \(S^1\), the network is a one-hidden-layer ReLU network, and the target is a finite Fourier mixture. In the infinite-width limit the training dynamics are governed by the Neural Tangent Kernel (NTK), and on the uniform circle that kernel is a convolution. A convolution operator is diagonalized by the Fourier basis. So each frequency gets its own eigenvalue, each residual component decays at its own rate, and low frequencies are learned first because their eigenvalues are larger. That is spectral bias, derived rather than observed.

The whole thesis then asks one thing:

How much of that exact Fourier description survives once you leave the idealised limit?

Three things break the ideal picture, and I treat them separately:

1. Finite sampling. The uniform measure is replaced by \(n\) sampled points.
2. Finite width, frozen kernel. The infinite-width kernel is replaced by the random tangent kernel of a width-\(m\) network, held fixed at initialization.
3. Finite width, evolving kernel. The tangent kernel is allowed to move during training, which is what actually happens.

The short answer: (1) and (2) are controlled perturbations, and I can prove bounds on them. (3) is where the fixed-kernel account stops, and there my results are empirical.

Setup

The network is a one-hidden-layer ReLU network in NTK parameterization,

with \(a_i\sim\mathcal N(0,1)\), \(w_i\sim\mathcal N(0,I_2)\), and hidden biases \(b_i\) initialized at zero but trainable. That last detail matters more than it looks; see below.

Inputs are embedded as \(x(\varphi)=(\cos\varphi,\sin\varphi)\) with the uniform measure \(d\mu = d\varphi/2\pi\). The real Fourier basis of \(L^2(\mu)\) is

Since the full operator acts on an infinite-dimensional function space, all comparisons are made on a fixed low-frequency block

Under gradient flow with least-squares loss, the residual \(r_t = f_t - y\) obeys

where \(T_t^{(m)}\) is the integral operator of the empirical tangent kernel. Every regime in the thesis is a different choice of what to put in place of \(T_t^{(m)}\): the continuum limit \(T_\infty\), its sampled version, the frozen finite-width operator \(T_0^{(m)}\), or the full time-dependent one.

Before any of the analysis, a set of preliminary experiments checked that the NTK baseline behaves as advertised at finite width: the kernel profile concentrates as width grows, and residual projections onto the leading kernel eigendirections decay in the expected order.

The continuum reference model

The first result is the explicit kernel and its spectrum. Writing \(\delta\) for the angular distance between two inputs, the limiting NTK is

which depends only on \(\delta\). The associated operator is therefore a convolution and the Fourier modes are exactly its eigenfunctions. Their eigenvalues are

and for \(k\ge2\),

Numerically: \(\lambda_0 \approx 0.55\), \(\lambda_1 \approx 0.35\), \(\lambda_2 \approx 0.079\), \(\lambda_3 \approx 0.011\). Both branches decay like \(O(k^{-2})\). In the continuum fixed-kernel model each Fourier coefficient of the residual then evolves independently,

so the eigenvalue ordering is the learning-speed ordering. This is the baseline against which everything else is measured.

Most closely related two-layer analyses simplify the NTK by training only the first-layer weights and keeping the output weights fixed. Here the output weights stay trainable and their contribution is kept in the kernel, which is why the eigenvalues above differ from the usual ones.

The trainable bias is not cosmetic

Removing the hidden-bias contribution gives a different kernel, \(\Theta^{(\mathrm{nb})}(\delta) = \frac{1}{2\pi}(\sin\delta + 2(\pi-\delta)\cos\delta)\), whose spectrum is

The even frequencies are unchanged. The odd frequencies \(k\ge3\) go to zero. Without a trainable bias those modes are null directions of the fixed-kernel dynamics: they are not learned slowly, they are not learned at all. The architecture, not the data, decides which frequencies are reachable.

Perturbation 1: finite sampling

Replace the uniform measure with \(n\) i.i.d. points \(\varphi_1,\dots,\varphi_n\). Exact Fourier diagonalization is gone immediately, since the sampled Fourier basis is not even orthonormal. The question is how much is lost on \(\mathcal H_K\).

Write \(\Phi\in\mathbb R^{n\times d}\) for the sampled Fourier feature matrix and \(A_{ij} = \tfrac1n\Theta(\varphi_i-\varphi_j)\) for the sampled kernel operator, and form

Three quantities have to be controlled at once, each written as “sampled value minus its ideal value”:

The reference \(\Lambda^{(n)}\) is not the continuum spectrum. A short calculation gives \(\mathbb E[H] = \Lambda^{(n)}\) exactly, with the finite-sample corrected eigenvalues

So sampling shifts the eigenvalues by an explicit \(O(1/n)\) diagonal term before it introduces any noise.

Result (finite-sample concentration). With \(\kappa=\sup_\theta|\Theta(\theta)|\) and \(d = 2K+1\), for every \(\varepsilon>0\):

Union-bounding these gives a single high-probability event on which all three errors are

The same sampled points simultaneously make the Fourier basis nearly orthonormal, the compressed operator nearly diagonal, and the Fourier modes nearly eigenvectors. The Fourier description degrades at rate \(n^{-1/2}\); it does not break.

When does the leftover error become visible?

The bound is a statement about the operator, not about the trajectory. Pushing it through the dynamics gives a two-stage picture. The predicted motion of component \(p\) is \(\lambda_p^{(n)}e^{-\lambda_p^{(n)}t}\), while the extra motion caused by the sampling error is bounded by \(\|A\Phi-\Phi\Lambda^{(n)}\|_{\max}\sum_q e^{-\lambda_q^{(n)}t}\). Early in training the first term dominates and the sampled curves follow the corrected eigenvalue prediction. Once a component has decayed close to zero, its own predicted motion is tiny and the residual sampling error, which depends on all retained components including the slow ones, becomes visible. This is exactly what the mode-decay curves show: clean agreement early, deviations late.

Preconditioning: a direct test of the eigenvalues

If the frequency-dependent rate separation really comes from the spread of the predicted eigenvalues, then dividing each Fourier coefficient by its own eigenvalue should erase it. That is a falsifiable claim, so I tested it.

Let \(C_n^{(K)} = G^{-1}H\) be what the sampled operator actually does to Fourier coefficients. Two preconditioners follow:

On the retained block \(P_{\mathrm{emp}}A\) reduces to the identity by construction, so it flattens the rates exactly. \(P_{\mathrm{th}}A\) leaves the factor \((\Lambda^{(n)})^{-1}C_n^{(K)}\), which is the identity only if the theory is right.

This is also the one place where the analysis suggests something usable. If the target is known to sit in a prescribed low-frequency range, the theory preconditioner needs no estimation of the sampled block at all, only the closed-form \(\lambda_p^{(n)}\), and it still equalizes the rates.

Perturbation 2: finite width, kernel frozen

Now keep the continuum measure and shrink the width instead. The object is the low-frequency block of the tangent operator at initialization, \(B_m^{(K)} = P_K T_0^{(m)} P_K\), compared against the continuum block \(\Lambda_K = \operatorname{diag}(\lambda_0,\lambda_1,\lambda_1,\dots,\lambda_K,\lambda_K)\).

The useful structural fact is that \(B_m^{(K)}\) is an average of \(m\) i.i.d. one-neuron matrices, so it should concentrate.

Result (frozen finite-width concentration). \(\mathbb E[B_m^{(K)}] = \Lambda_K\), and there is a universal \(c>0\) with

so that for fixed \(K\),

Same story as before, one perturbation over: an \(m^{-1/2}\) rate instead of an \(n^{-1/2}\) one.

The block bound controls entries; it says nothing directly about eigenspaces. So I also measured alignment geometrically. Because \(\phi_{k,c}\) and \(\phi_{k,s}\) share an eigenvalue, the continuum operator does not single out either one, only the plane \(\mathcal F_k\) they span. The right comparison is therefore between subspaces, using the projection error \(\|\widehat P_k - P_k\|_F\) and the principal angles between \(\mathcal F_k\) and its matched empirical eigenspace.

Alignment improves with width at every frequency, and higher frequencies are worse at fixed width. The likely reason is spectral separation rather than perturbation size: high-frequency eigenvalues are both small and close together, so the same absolute perturbation rotates their eigenspaces much more. There is one clean exception, \(k=2\), which aligns better than \(k=0\) and \(k=1\) do. I flag it as an empirical observation and not a claim, though it is suggestive that \(k=0\) and \(k=1\) are precisely the frequencies the architecture represents most directly — through the bias and the input embedding.

Perturbation 3: finite width, kernel evolving

This is where the theory stops and the experiments take over. Freeze nothing: let the tangent kernel move during training and track

which decomposes as

The frozen theorem controls the second term. The third is the open one.

Small width: the evolving kernel wins

At widths 128 and 256 the evolving kernel reaches a lower final loss than the frozen baseline. At 512 the gap narrows. At 1024 it reverses. So the natural question is what the evolving kernel is doing that helps, and only at small width.

There are two candidate explanations, and they make opposite predictions:

• Geometry. The evolving kernel keeps its eigenspaces better aligned with the Fourier basis, so the frequency decomposition stays accurate.
• Strength. The evolving kernel redistributes eigenvalue mass toward the low frequencies, so those modes get pushed down harder, regardless of alignment.

The measurements say it is the second one.

So the improvement is not bought with better Fourier geometry. It is bought with extra operator strength concentrated exactly on the low-frequency blocks that were already well aligned. Read next to the fixed-kernel result, this looks like a second-order version of the same bias: gradient descent learns low frequencies faster under a fixed spectrum, and it moves the spectrum to favour them further.

Why it fades with width

The evolving kernel helps precisely where it moves. At large width the frozen kernel is already close to the continuum operator, there is little drift available, and nothing is gained.

Checking the obvious confound

The original target puts larger amplitudes on lower frequencies, so the low-frequency strength increase could just be the kernel amplifying wherever the residual happens to be biggest. I re-ran the experiment with an equal-amplitude target and a reversed-amplitude target (largest amplitudes on the highest frequencies).

The effect survives all three targets, so it is not simply tracking where the target puts its mass.

A guess, and how to test it

Why \(k=0\) and \(k=1\) specifically? The pre-activation of hidden neuron \(j\) is

which contains a constant and a frequency-one term and nothing else. Every higher harmonic exists only indirectly, through the ReLU acting on that. The frequencies the network strengthens are the ones its parameterisation represents most directly, and which therefore need the least movement to amplify.

That is a hypothesis, not a result, and it is separable from the alternative with the same machinery: train on a target with no constant and no frequency-one component. If the strength increase still lands on \(k=0\) and \(k=1\), the parameterisation account wins. If it follows the lowest frequency actually present in the target, then gradient descent is just amplifying wherever the residual is largest.

What holds, and what does not

The Fourier basis is exact in the continuum model, approximately valid on the low-frequency block under finite sampling and frozen finite width, and a measurement instrument for the evolving kernel. Turning that last row into theory is the main thing this work leaves open.

Some honest limitations:

• The bounds are fixed-block results. They hold for a prescribed \(K\) and are not designed to stay sharp when \(K\) grows with \(n\) or \(m\) — which is the regime where they would matter most, since high frequencies have small, tightly spaced eigenvalues.
• The proofs use entrywise concentration and union bounds, so the constants are conservative. Operator-norm or variance-sensitive tools would tighten them substantially.
• The evolving-kernel findings are a correlation with lower loss, not an established cause, from one architecture on one domain.
• The whole picture depends on two things lining up: the kernel is rotation-invariant and the input measure is uniform. Break the second and the operator stops being a convolution; the fast-learned components become the leading eigenfunctions of a density-weighted operator, which need not be low frequencies at all. On a Gaussian input measure the eigenfunctions are Hermite polynomials and degree replaces frequency. The mechanism should carry over — eigenvalue size sets the rates in any basis — but the proofs would not, since they rely on the circle being compact and the kernel bounded.
• A smoother activation should keep the framework intact and only change the spectrum. The \(O(k^{-2})\) decay traces back to the non-differentiability of ReLU at the origin, so \(\tanh\) or sigmoid would give faster-decaying eigenvalues and a stronger low-frequency bias. Whether the evolving-kernel behaviour carries over is untested.

Why it matters outside the circle

The same spectrum shows up wherever networks have to represent fine detail. Physics-informed neural networks fail on the high-frequency part of PDE solutions because of a gap in the convergence rates of the loss terms, and the fix is to rescale the dynamics by the kernel eigenvalues — the same correction as the preconditioner above. Coordinate-based networks in vision and graphics blur fine detail for the same reason, and Fourier-feature mappings and periodic activations work by reshaping the effective kernel spectrum so high frequencies stop being suppressed. The mechanism this thesis pins down on the circle is the one those methods are modifying in practice.

A result on the uniform circle is still a long way from a guarantee about a deployed model. But knowing why a network learns some components before others makes its behaviour predictable, and that is worth having.

The proofs, the derivation of the continuum kernel, and the full set of experiments are in the manuscript: Training Dynamics of Overparameterized Neural Networks, TU Delft repository.