ARC White-Box Estimation Challenge 2026 - Phase 1

The goal

You get the weights of a random MLP. You have to report, for every neuron, the mean post-ReLU activation when the input is standard Gaussian. They score the last layer only, and they bill FLOPs.¹

Phase 1 is width 256, depth 32, He-Gaussian weights, no biases.² Small enough to hold in RAM, and deep enough that a Gaussian story for layer 1 does not automatically extend to layer 32, which is the whole reason this is a contest and not a homework problem.³⁴

I wanted a problem where using the structure of the net has a number attached. If a method is wrong, the score says so.

Phase 2’s width, depth, and budget were still unset when I wrote this. A trick that only works at 256×32 is not a result I would bet on.

Layer 1 is closed form

I start with the simplest claim that could be true.

Layer 1 pre-activations are Gaussian: h = W x with x ~ N(0,I) and iid Gaussian W, so for a zero-mean coordinate with width σ you get E[ReLU(h)] = σ / √(2π) and there is nothing left to estimate on that layer.⁵

If I implement that and last-layer MSE barely moves, I was not losing the score on layer 1. Then I should stop touching layer 1.

Click Before / After / The mean on one neuron. I needed to see that the mean is a formula, not a sample, before I spent FLOPs on layer 1.

That is 256 neurons of free accuracy. I don’t get extra credit for Monte Carlo on a formula.

The kit as a baseline

The starter kit is covariance propagation: keep a mean vector and a covariance, push them through ReLU with a gain, which is the finite-width cousin of Cho and Saul’s arcsine kernel and the thing the kit already ships, so it is the right dumb baseline.⁵⁶

Two things could be true at once.

Hypothesis A: the chain is “Gaussian enough” at width 256, so better second-order matching will cut last-layer MSE.

Hypothesis B: after ReLU, h_{ℓ+1} = W_{ℓ+1} ReLU(h_ℓ) is a sum of truncated, correlated Gaussians — skewed, not Gaussian — and that mismatch grows with depth, so a two-moment state is the wrong object to carry for thirty-two layers.³⁴

Layer 2 is already the second object. Width 256 makes A look tempting. Depth 32 is why B might win.

Click Layer 1, then The rest. I wanted a picture of where the Gaussian story stops, because that is the line between homework and the contest.

I ran the kit. Local last-layer MSE 8.93×10⁻⁵, adjusted 8.93×10⁻⁶ (the 0.1 floor). The grader came back 6.62×10⁻⁶ on the same config. Direction matches. I treat the local three-net split as a kill-switch, not a leaderboard.

That is the baseline. Next experiment has to beat it, or die.

Better second moments did not help

I thought the leak was the gain map cov_ij ≈ Φ(α_i)Φ(α_j) cov_ij.

If A is right, replacing that with exact pairwise E[ReLU_i ReLU_j] should move chained last-layer MSE by more than the extra FLOPs cost, because otherwise I am paying for digits that the score never sees.⁷ I checked the pairs offline against the zero-mean arcsine formula and against Monte Carlo. Accuracy around 10⁻¹⁰.

Chained, last-layer MSE went from 8.93×10⁻⁵ to 8.86×10⁻⁵. Eight tenths of a percent. FLOPs went from 3.26×10⁹ to 1.84×10¹⁰ per MLP. 5.6×.

A died. That is a finding. The pairs were never the score.

Click Pairs, then Chained, then FLOPs, in that order. The pair test and the score are different numbers, and the third view is why I refused to keep paying for the first.

Why it failed, as far as I can tell: even if μ is perfect, E[ReLU] = σ φ(α) + μ Φ(α) with α = μ/σ, so a wrong σ still poisons the mean, and correlations you drop at layer 6 come back as variance error at layer 12.⁵ The state is missing skewness and kurtosis. Wu et al. write the companion paper for this problem in cumulants and Hermite expansions of ReLU, which is the natural next object if you want a better closure, and I did not implement a K-cumulant propagator for Phase 1 because I wanted a cheaper test first.⁸

A correction trained on true incoming moments will also fail for the usual reason: at test time you are rolling out your chain, not the true one, so a one-step formula that looks perfect on oracle inputs is still allowed to rot when you feed it its own outputs.⁹

I stopped looking for a prettier one-step map.

Then I read the score

A better closure that ignores the meter is a result about a different problem.

The formula, as implemented:¹⁰

adjusted = final_layer_mse × max(0.1, C/B)

C = F + 10¹¹ × residual_wall_s

B = 2.72×10¹¹ FLOPs per MLP in Phase 1.

The website still said 0.5 in places. The installed scorer says 0.1. I trusted the code.

The first 10% of the budget is free, and between 10% and 100% extra compute is a linear tax on MSE, so spending 2× FLOPs for a 1.5× MSE cut is a net loss even before you count the Python you forgot to bill. 0.1 s of unmetered Python is 10¹⁰ FLOPs, about 3.7% of B, even if you never wrote a matmul.

Hold the raw MSE fixed and click the four tax rates. Under the floor the adjusted bar does not move, which is why sitting at C/B ≈ 0.074 was leaving free accuracy on the table, and above it the bar grows linearly so extra FLOPs have to buy more MSE than they cost.

Plain Monte Carlo at the floor is about 6,500 full forwards. MSE around 2.5×10⁻⁵. Full budget, still unbiased: maybe 2.5×10⁻⁶. Everything I shipped that beat that was biased on purpose.

The next experiments had to sit at C/B ≈ 0.1, or prove the raw MSE fell faster than the multiplier rose.

A last-layer scale, then a blend

Cov-prop overestimates the final mean. I tried a per-layer scale in the chain. Last-layer MSE went to 3×10⁻². Dead. That matches the compounding-error picture: you cannot correct a trajectory using a multiplier fit on the true states.⁹

I tried the same idea on the readout only. Offline, across 13 seeds, ×0.99187 on the last layer cut MSE by about 3.5×, which is an ugly constant I would rather derive than fit, and I still don’t have a derivation. Local adjusted went to 2.18×10⁻⁶. It survived the in-chain kill-test. I kept it.

The sampler was still sitting at C/B ≈ 0.074. Free floor I wasn’t using. I added antithetic Gaussian Monte Carlo and blended it with the analytic, first at a frozen λ=0.45.¹¹ Local adjusted 7.83×10⁻⁷. Then I let λ_i = b² / (b² + Var[MC_i]) with b² = 4×10⁻⁵, and raised N from 4,000 to 5,500 so I actually spent the 10%, because sitting at C/B ≈ 0.074 was leaving free accuracy on the table.¹² Local adjusted 3.42×10⁻⁷ at C/B ≈ 0.098.

Adaptive beat a frozen 0.65. The blend also stops a bad sampler from replacing a decent analytic. That is the champion I would rerun tomorrow: exact layer 1, last-layer ×0.99187, 5,500 antithetic Gaussians, adaptive blend.

In-chain scaling and a last-layer constant look like the same multiplier. They are not. Click them in order, then the two blends. Log scale, because 3×10⁻² would eat the plot.

Is the sphere doing the work?

ReLU is homogeneous: ReLU(c z) = c ReLU(z) for c > 0. A standard Gaussian is a radius times a direction, and E[χ_256] ≈ 15.984 is a ratio of gammas, so the radial integral is free and whatever is left of the estimator is an integral on the sphere.¹³

If cubature is doing the work, a spherical design at the same row count as my Gaussian sampler should beat it. If it doesn’t, I have not tested cubature. I have tested a small set of directions.

It does not make a 5-design exact for a depth-32 ReLU net. The integrand is piecewise linear and the cells depend on the particular weights.

Click Radius, then Angles, then Both. If the radius is free, the remaining work is the directions, and that is the only reason a spherical design was even a candidate.

First test: Sylvester Hadamard, identity, a few column-sign flips, scaled to E[χ_256], antipodes, N ≈ 5,500 so the comparison was fair.¹⁴ Adjusted MSE went from 3.42×10⁻⁷ to 1.64×10⁻⁶. Five times worse. The scaling was fine. Those directions are not a mutually unbiased basis.

Second test, same N, same blend: real Kerdock chirps (Carlet’s quadratic on GF(2⁷)×GF(2)), nested prefix of 10 bases plus the coordinate basis plus antipodes, 5,632 points, first layer an unnormalized FWHT, local 3.27×10⁻⁷ against 3.42×10⁻⁷.¹⁵¹⁶ A 4% tick, inside the noise of local GT at N = 2.56×10⁶. I am not promoting it. A fake bank and a real prefix are not the same object, and a tie at this N is not a finding that cubature wins.

Click the three carriers. Same row count is the only fair test I could afford, because a prettier set of directions at a different N is a different bill.

I did not drop bases at random to save FLOPs. A prefix of 11 was never going to reproduce a 66,048-point design. The FWHT saves work on layer 1 and almost nothing on the bill, because 31 of 32 layers are still X @ W, and at 5,632 rows I sat at C/B ≈ 0.098, same as Gaussian. You cannot 12× the sample count at the floor without deleting work.

The next honest test is the full design, billed, at the 0.1 floor — or it isn’t a test of cubature.

What I actually ran

Each row is the experiment that the previous result forced. The ones that died are findings too.

Scientific notation hides a 26× drop from the kit to v4.2 and a 5× collapse on signed Hadamards, so click a step: the bars are log-scale local adjusted MSE, and the table under it is the reason I moved.

Local whest run --seed 42 --n-mlps 3.

I have not shown that 66k directions × 32 layers fit in 10% of B, and I doubt it does without dropping dead neurons and folding always-on last layers into linear maps, which is a next experiment I have not run.

What would kill v4.2: a fresh-seed worst-decile 2× worse than local, or a carrier at the same C/B that beats it by more than GT noise. If I can’t name that, I don’t have a next idea yet.

The work that moved the score was the work that matched an error I could point at — layer 1, the meter, the readout bias — and then refused to keep paying FLOPs to the other ones, including the ones that looked more like research.