PCG backend: the complete experiment record (2026-08-25 to 2026-09-10)¶
This is history, frozen on 2026-09-10. It is the verbatim record of every experiment, measurement, decision and reversal of the PCG reconstruction backend and its priors, kept so that no experiment is re-run and no number has to be re-measured. It is NOT the description of the backend as it stands and it is not maintained as one. For that, read:
pcg_backend_overview.md-- what the PCG backend IS today, in four pages, present tense only;pcg_decision_log.md-- one dated paragraph per decision, in order, with what forced it and what it superseded; new decisions go there;- the binding contracts:
doc/policies/3D/reconstruct3D_pcg_policy.md, the PCG sections ofrefine3D_policy.mdandrefine3D_auto_policy.md, anddoc/policies/3D/automasking_policy.md.Reading guide to what follows. The design sections (1-11) were written in late August around estimators that no longer exist: the binary-envelope solvent prior
Q_s(removed 2026-08-27), the Wilson priorQ_W(removed 2026-08-29) and the NU-evidence replay precisionQ_NUwith its auto-lambda/auto-target controllers (removed 2026-09-06). The "Status" text they carry is the status at the time of writing. The dated records under "Staged development plan" and the "Active dev list" (a numbered list that became a journal; items 1-12 with nested records through 2026-09-10) are the primary sources the decision log cites by date. Where an entry is superseded, the decision log says by what; this file was not edited to say so.
Status (as written 2026-08-29; superseded, see the banner)¶
This is the single reference for regularization of the PCG reconstruction
backend: what exists on master, what was tried and removed (with the
measurements that killed it), and the approved development order for a direct
NU-evidence prior followed by a Wilson molecular prior. It consolidates and
replaces pcg_euclid_crash_investigation.md,
pcg_reconstruction_production_readiness.md,
pcg_refine3D_integration_plan.md, and
pcg_real_space_solvent_flatness_prior_proposal.md. Current implemented
contracts live in doc/policies/reconstruct3D_pcg_policy.md and the PCG
section of doc/policies/refine3D_policy.md.
Decision (2026-08-27): the direct NU-evidence prior is priority 1. When NU
regularization is active it replaces the FSC/SSNR P_tau replay precision; it
is never added to P_tau. The binary NU-envelope solvent prior is scientifically
retired and must be removed from the target workflow. Its implementation and
measurements remain recorded below only as experiment history until the code is
removed. Wilson is priority 2, after the NU prior has passed its gates.
Outcome (2026-08-29): the Wilson question is answered and Stage 7 is
CLOSED. Q_W was implemented, corrected twice by truth-judged failures,
validated, adjudicated against Q_NU on the 1WCM fixture, and REMOVED from
the code base the same day: Q_NU dominated at every shell even with Q_W
handed the ground-truth spectrum (S5.5 hypothesis confirmed). Q_NU is the
regularized estimator of this document; the Stage 7 sections below are the
complete experiment record. All forward-looking Wilson language elsewhere in
this document predates this outcome and is retained as plan history.
1. Current regularization inventory (master)¶
The production PCG path performs, per state and half, one base solve and one
ML replay solve from the same raw (B,D) statistics:
- base solve:
(H_data + lambda_0 I) x = b; its half pair (*_unfil) produces the FSC. Each half warm-starts from the previous iteration's own base half when that artifact is explicit (*_unfil) or the primary pair is proven base-only by its sidecar. First solves and ineligible/mixed inputs retain the zero start. - ML replay:
(H_data + P_tau + lambda_0 I) x = b, whereP_tauis the FSC/SSNR shell-diagonal Fourier precision (ml_prior, present in both the operator and the preconditioner). Its output is the shipped map. It warm-starts from the previous refinement iteration's ML half map when one exists (own half only, support re-masked after constant-FOV resampling, first-iterationstartvolexcluded), falling back to the base solution. - Solvent experiment REMOVED (2026-08-27): the binary-envelope solvent
precision
Q_s, its envelope loader,pcg_solvent_lambda_rel, the suppression/inflation convergence readout, and the harness strength-ladder sweep were removed from the codebase after the direct NU-evidence replay passed its first Gate C ablations (truth-judged win on 1WCM at both budgets; truth-free observables equal-or-better on bgal with the ML-replay stall fixed). §4 and the completed Stage 2--5 entries retain the algebra and measurements as the experiment record;test=pcg_priorswas repurposed as the Q_NU Gate A suite. The abinitio3D automsk staging that the solvent bootstrap introduced is KEPT for its reference-masking registration benefit (Gate B streptavidin: base-pair 3.812 -> 3.191 A). lambda_0is the fixed absolute base TikhonovPCG_LAMBDA = 1e-3.- the soft spherical support
Pconstrains the solve asP H P u = P b. - maps are stored at the data-quotient amplitude convention
(
drop_legacy_box_division.md); deapodization is inside the solver. report_beyond_band_excesslogs when post-band RMS exceeds 10x the band-edge shell; it is the regression signal for beyond-band defects.
2. Record of prior experiments (tried and removed)¶
Kept so no experiment is re-run. Measurements in
drop_legacy_box_division.md §11–12.
| Experiment | Outcome | Lesson |
|---|---|---|
pcg_lambda_lap — biharmonic (nu^4) smoothness prior, operator + preconditioner |
Removed. On the synthetic fixture it suppressed beyond-band noise 10–80x and closed the 2-iteration in-band deficit (ratio 0.84 → 1.00). On real data (bgal, box 256) no usable strength exists: rho falls ~3 orders across the band while nu^4 spans ~1.5 from half-band to edge, so any lambda that suppresses beyond-band also attenuates the band edge (−25% at 5.4 A even at 0.01). | A data-scale-anchored spectral prior cannot separate in-band from beyond-band; band-selective suppression is the FSC-informed P_tau's job. The real benefit was conditioning the ML replay (RESID 6.5 → 0.86 at 2 its) — captured instead by the cross-iteration warm start. |
pcg_lambda_rel — relative base-lambda CLI (the "P0 lambda contract") |
CLI removed as unused; the internal set_lambda_relative mechanism and the deterministic s_data(D) data-scale functional remain (test commanders, diagnostics). |
Relative anchoring machinery is retained where priors need it; it is not user-facing. |
| Kernel-derived preconditioner | Removed. Did not improve runtime at one iteration and degraded reconstruction metrics. | The sampling-density diagonal with a shell-relative floor is the preconditioner. |
| Whole-volume-L2 ML-vs-base rescale guard | Removed. Whole-volume norms are solvent-dominated and never fired on the failing solves. | Compare shell profiles, not whole-map statistics. |
x0 = P b right-hand-side base warm start (proposed, never built) |
Superseded by the previous-iteration base solution. | Reusing the prior compatible same-half solution exploits actual iteration-to-iteration continuity without constructing a differently scaled surrogate from b. |
ref_taper — legacy KB matching-reference taper |
Removed after a master reliability control (1/10 failures on master vs 2/10 on the branch) showed no regression to fix. | Measure the baseline before attributing a failure rate to a change. |
Two historical defects, both fixed, whose lessons are contracts now:
- Beyond-band excess (defect A). PCG base solves can leave hot spectral
content just above the matching band; a stage transition that extends
ktothen exposes it to euclid matching. The diagnostic survives (PCG BEYOND-BAND EXCESS); the structural mitigation is ML-replay convergence (warm start + adequate iterations), not a smoothness prior. - Amplitude-convention discontinuity (defect B). The euclid/sigma2 equilibrium survives any stable reference amplitude but not an abrupt scale change between consecutive iterations. Retired by the shared data-quotient convention; any future backend or prior must preserve scale continuity at handoffs.
Prior priority (decision 2026-08-27):
| Idea | Priority | Recommendation |
|---|---|---|
| direct NU-evidence-conditioned precision | 1 — immediate | replace P_tau in NU mode; §3 and §5 |
| Wilson molecular precision | CLOSED 2026-08-29 | adjudicated and removed; Q_NU dominates; §5.5 and the Stage 7 records |
| binary-envelope solvent precision | retired | remove from the target workflow; preserve §4 as experiment history |
| soft state weights | integration semantics | weight both B and D |
| symmetry projection/permutation | performance only | after distributed profiling |
| total variation / non-negativity / wavelet L1 | deferred | separate nonlinear/proximal solver project |
| joint pose/shift refinement | deferred | separate program (continuous_3D_refinement_on_pcg_operator.md) |
Any prior must preserve half independence, positive-semidefinite CG structure, conditioning, data-mass scaling, and separate reporting of fit and prior energy, and must be evaluated against a strength-zero control.
3. Approved architecture: mode-exclusive regularized replay¶
The regularized replay has one precision source selected by filtering mode. There is no additive stack of global ML, solvent, Wilson, and NU priors.
3.1 Ordinary PCG mode¶
Outside NU mode, keep the production estimator unchanged:
base: (H_data + lambda_0 I) x = b
replay: (H_data + P_tau + lambda_0 I) x = b
P_tau remains the global FSC/SSNR shell precision derived from the current
unregularized half pair and raw D. Nothing in the NU experiment changes or
reinterprets this path.
3.2 NU mode¶
When the NU machinery is active, replace P_tau with one NU-evidence-
conditioned precision Q_NU(E):
base: (H_data + lambda_0 I) x = b
evidence E = NU analysis(base_even, base_odd)
replay: (H_data + Q_NU(E) + lambda_0 I) x = b
The sequence is current-iteration empirical Bayes, exactly as the existing ML
replay derives P_tau from the current base half pair. The NU evidence state is
built after both base solves, then frozen before either replay. It contains no
phase-bearing target and changes only the precision, never the RHS.
Fixed rules:
- Derive
Eonly from the unregularized_unfileven/odd pair. Do not let an ML-priored auxiliary replacement candidate validate or parameterize its own prior. - Keep the base maps, FSC/cFAR, raw
(B,D), and resolution authority unchanged. - Attach exactly one replay precision.
P_tauandQ_NUare mutually exclusive; a hard assertion must reject simultaneous attachment. - Generate no binary molecular envelope for PCG and take no complement to
manufacture solvent confidence. There is no
Q_sin the NU target path. - Preserve the spherical
mskdiamsupport needed by the NU objective and its noise/null estimation. Removing envelope generation does not make the NU evidence domain self-defining. - Keep the NU selected-cutoff/local-resolution products as evidence
diagnostics and, where required, LP-set bandwidth handoffs. Initially write
post-hoc
_nu_filtmaps only as controls; the in-solve NU replay is the candidate matching reference. - Preserve one base solve plus one replay per half and reuse the same raw accumulation. No particle reread or third solve is introduced.
- Wilson is not mixed into this first experiment. It begins only after the direct NU estimator passes Gate D.
Two verification prerequisites apply to the NU operator:
- The
P H Pcontract. The algebra below assumes the support operator is applied on both sides of the normal operator and once on the RHS (reconstruct3D_pcg_policy.md§4 documents this as the contract); verify the sharedapply_normalwrapper and RHS construction actually satisfy it before building on it. - The deapodization domain. Any spatially varying NU precision is defined
on the deapodized map domain. Restrict it to deapodized kernel mode (hard
error otherwise), as the solvent experiment and
pcg_lambda_lapdid.
4. Retired binary-envelope solvent experiment¶
This section is an experiment record, not an implementation proposal. The operator was implemented and calibrated successfully, but the 2026-08-27 decision superseded its binary molecular/solvent partition with the direct NU precision of §5, and the code was REMOVED the same day once the NU replacement passed its first Gate C ablations (see §1). The algebra, gates, and real-data measurements below are retained so the experiment is not repeated.
4.1 Motivation and literature position¶
LocScale and solvent flattening solve related but different problems. LocScale-1.0 performs local radial Fourier-amplitude scaling against a model map [2]. The physics-informed route in LocScale-2.0 constructs a pseudoatomic or hybrid reference and scales local Fourier amplitudes while preserving the observed local phases [1]; it lists solvent flattening as a possible future real-space prior. Wilson statistics [3–6] underlies the molecular prior of §5. Solvent flattening proper starts with Wang [7], is combined with reciprocal-space constraints by Cowtan and Main [8], expressed as an explicit likelihood by Terwilliger [9], and carried into cryo-EM density modification [10]. That likelihood line is the direct ancestor of this penalty. The proposal keeps three ideas separate:
- Wilson prior: a molecular mean/covariance model.
- LocScale operation: local, phase-preserving amplitude rescaling against an expected or reference spectrum (not PCG-compatible as-is; Stage 8).
- Solvent prior: expected low real-space variance in the solvent.
4.2 Graded solvent confidence¶
Let m_v be the NU molecular envelope at voxel v (0 <= m_v <= 1,
1 = confident molecular region), and p_v the existing broad support value.
Define
w_v = p_v (1 - m_v), s_v = w_v^2, S = sum_v s_v
Including p_v prevents the many fixed zero voxels outside the broad support
from dominating the solvent mean, and grades the prior smoothly at both the
NU-envelope boundary and the spherical-support boundary.
4.3 Weighted solvent mean and penalty¶
For a trial map x, the weighted solvent mean and penalty are
mu_s(x) = (sum_v s_v x_v) / S
R_s(x) = (lambda_s/2) sum_v s_v [x_v - mu_s(x)]^2
The generating operator is L = W C_s with W = diag(w) and
C_s = I - 1 s^T / S (weighted centering). The normal operator must add
L^T L, not L: L is not symmetric because weighted centering is not an
orthogonal projection in the ordinary inner product, and adding it directly
would invalidate CG's symmetry assumption. The required precision is
Q_s = L^T L = D_s - s s^T / S, D_s = diag(s)
Q_s x = s .* [x - mu_s(x)]
No matrix is stored: one deterministic weighted reduction and two volume passes per application.
4.4 Per-half ML-replay normal equation¶
The ML replay for each half becomes
P (H_data + P_tau + lambda_0 I + lambda_s Q_s) P u = P b
lambda_s = pcg_solvent_lambda_rel * s_data(D)
The base solves are unchanged. The data-scale anchoring uses the retained
internal s_data(D) functional. Note this anchoring is defensible here where
it was not for the removed nu^4 prior (§2): the solvent penalty competes with
the data term as a real-space aggregate over a fixed region, not per Fourier
shell, so the orders-of-magnitude shell-wise decline of rho does not create an
empty strength window. Before applying the scale, normalize the precision to a
declared mean diagonal on its effective support so the two relative inputs of
§5 have comparable meaning.
4.5 Algebraic properties¶
For every real x: x^T Q_s x = sum_v s_v [x_v - mu_s(x)]^2 >= 0, so Q_s
is symmetric positive semidefinite, and Q_s (c 1) = 0 — it penalizes solvent
variation, not the solvent offset, which cryo-EM data do not determine. The
broad support P remains: the solvent operator has an unpenalized constant
mode and the interpolation/deapodize path is least reliable near the padded
corners, so removing P would add weakly constrained unknowns while the new
prior is being evaluated.
A historical solvent-experiment hypothesis was that the known PCG low-shell
anomalies — the k=1–2 backend amplitude excess (~1.3–2.0) and the centre-bin
radial deficit — are plausibly solvent-offset/low-k noise behavior that Q_s
addresses directly. The gated rec3D_backends shell and centre-bin
diagnostics measure this for free. Re-baseline those numbers after the ML
warm start is validated, since it moves them too.
5. Direct NU-evidence-conditioned precision — priority 1¶
5.1 Why the full NU state is the prior input¶
The NU engine already evaluates a noise-whitened cross-half prediction cost
C_c(v) for every retained local low-pass candidate. That full curve contains
two kinds of information required by a reconstruction prior:
- whether any reproducible signal is supported at voxel
v; - the finest scale to which that signal remains supported, including the confidence/ambiguity of the scale assignment.
The current envelope path throws most of this information away. It reduces the
bank to C_20A - min_c C_c, thresholds the scalar against a whole-support
median/MAD null, solves a binary MRF, filters components, grows the result, adds
a soft edge, and finally takes a complement for Q_s. The direct prior must
consume a compact representation of the full candidate curve before that
binary collapse.
The existing bank still lacks one state needed for this use: the coarsest 20-A label absorbs both solvent and genuinely coarse ordered density. Add an explicit no-reproducible-signal candidate or equivalent calibrated null score. The resulting latent state is
z(v) in {null, 20 A, 15 A, 12 A, 10 A, 8 A, 6 A, 5 A, 4 A, accepted extensions}
Its null-versus-signal evidence must remain cross-half predictive rather than a plain density threshold. Candidate likelihoods/confidences must be calibrated against the exact bank, whitening profile, and smoothing scales; raw Huber costs are not automatically normalized posterior probabilities.
5.2 Compact evidence state¶
Do not persist dmats_mask(n_vox,n_candidates) between reconstruction phases.
Before releasing it, reduce it to the sufficient state needed by the solver:
a_b(v): monotone support/confidence that detail bandbis reproducible;- the selected local cutoff for diagnostics and LP-set handoff;
- an uncertainty measure such as best-versus-next margin or label entropy;
- the null calibration/provenance needed to reproduce the mapping.
Use the existing ordered-label spatial model to regularize this state, but do not force it through binary topology, component removal, dilation, or a mask edge. Confidence must remain graded through molecular boundaries and ambiguous domains.
5.3 Quadratic precision¶
Let B_b be a normalized detail/band-pass analysis operator and let W_b be a
nonnegative spatial weight derived monotonically from lack of evidence for
band b. Define
R_NU(x | E) = 1/2 sum_b || sqrt(W_b) B_b x ||^2
Q_NU(E) = sum_b B_b^T W_b B_b
Every summand is positive semidefinite, so the fixed replay operator remains
symmetric positive definite after lambda_0 I and the existing P H P
support sandwich. The detail bank should exclude the global DC mode, making a
constant field a null mode of the NU precision without constructing a solvent
mean explicitly.
The one field has the required limiting behavior:
- null evidence: penalize all non-DC detail, which yields solvent flattening;
- coarse ordered density: preserve supported coarse bands and suppress finer unsupported bands;
- strong ordered density: preserve detail through the evidenced local cutoff;
- uncertain boundaries: grade the precision continuously instead of choosing a molecular/solvent side.
The initial implementation should use three or four broad, normalized detail bands rather than one operator per NU label. Record the frame normalization so adding bands does not silently increase total strength. A full 8--16-label operator is justified only after the compact form demonstrates scientific gain and acceptable replay cost. The preconditioner needs a declared nonnegative approximation to the NU precision; scientific comparisons must still be made at matched convergence, not matched iteration count.
5.4 Relationship to global ML regularization¶
Q_NU is the nonstationary empirical-Bayes alternative to P_tau, not a
correction layered on top. The existing global ML path maps shell FSC to SSNR
and scales its precision by shell-mean raw D. The NU development must define
and validate the corresponding mapping from calibrated local evidence to
bandwise precision. Until that mapping is established, the evidence field is
a detection/selection statistic, not yet a quantitatively normalized prior.
The mode assertion is load-bearing:
ordinary mode: P_tau present, Q_NU absent
NU mode: P_tau absent, Q_NU present
5.5 Wilson molecular prior — priority 2¶
Wilson supplies population information about molecular Fourier covariance, especially where the experiment is uninformative. NU evidence cannot contain information the data never observed, but the first question is whether such extrapolation is needed at all: the conservative NU estimator regularizes only the degrees of freedom the half-map evidence does not support.
Only after the direct NU estimator passes Gate D should a Wilson experiment
begin. Keep it separate from the NU acceptance experiment. The first Wilson
form remains a zero-mean, shell-diagonal precision
Q_W = F* diag(q_W(k)) F, implemented through one declared spectrum-source
path rather than a parallel Fourier array. Compare Wilson against the accepted
NU estimator; do not combine them until each has an independently established
benefit and a separate combination experiment is approved.
Singer derives the non-diagonal covariance of Fourier coefficients from the random bag-of-atoms model [5]; Gilles and Singer use the corresponding mean and covariance as a Bayesian molecular prior [6]. Off-diagonal structure, nonzero means, and LocScale-informed local covariances remain later experiments. A nonzero or phase-bearing mean is outside the first Wilson scope.
OUTCOME (2026-08-29): this experiment was run and its central question
answered in Q_NU's favor -- see the Stage 7.1 records and the removal
decision. The hypothesis stated above (that the conservative NU estimator
already regularizes the unsupported degrees of freedom, leaving Wilson
extrapolation little to add) was CONFIRMED with Q_W at its best case.
6. Workflow, evidence contract, and bias discipline¶
6.1 Per-state sequence (one particle accumulation, two solve phases)¶
- Reduce the raw, data-only even and odd statistics as today.
- Base-solve each half unchanged, with neither
P_taunorQ_NU; retain and write the_unfilpair. - Compute FSC/cFAR from that pair. It remains the unregularized resolution authority and is not used as a second NU replay precision.
- If NU mode is inactive, build
P_tauand run the existing ML replay. - If NU mode is active, run the NU candidate/null analysis on the current
base pair, compact and freeze the evidence state, build
Q_NU, assert thatP_tauis absent, and replay both halves with the same fixed precision. - Write the standard replay maps. Retain selected-cutoff/local-resolution and post-hoc NU products as diagnostics during validation; generate no PCG solvent envelope.
6.2 Evidence contract¶
The direct prior accepts only evidence derived from the current state's unregularized even/odd base pair. Validate before replay:
- identical dimensions, sampling, and physical extent for the base halves;
- finite, nonnegative candidate costs and a valid radial whitening profile;
- a nonempty spherical
mskdiamsupport with an adequate null population; - exact candidate-bank, smoothing, extension, and null-calibration provenance;
- monotone band-support confidences in
[0,1]and finite nonnegative precision weights; - one immutable evidence identity used for both half replays.
Emit at least:
pcg_replay_prior_mode=global_ml|nu_evidence
pcg_nu_prior_enabled= pcg_nu_candidate_count= pcg_nu_null_fraction=
pcg_nu_supported_fraction_bandNN= pcg_nu_uncertain_fraction=
pcg_nu_prior_energy_final= pcg_nu_preconditioner_mode=
pcg_nu_evidence_source=base_unfil pcg_nu_evidence_provenance=
There is no mask loader, missing-envelope skip, mask resampling, morphology, or silent fallback in the NU prior path. Failure to construct valid evidence must be explicit; fallback policy, if any, is a workflow decision rather than an inferred substitute prior.
6.3 Half-set and bias discipline¶
Both halves use the same fixed scalar confidence/bandwidth field, so their
errors can still become correlated even though the prior has no phase-bearing
mean. The reported FSC must therefore remain that of the unregularized base
pair. Keep raw accumulators half-specific and unchanged; freeze the evidence
before either replay; report shipped-pair FSC only as a correlation-inflation
diagnostic; compare against the ordinary P_tau estimator using map-to-truth,
independent map-to-model, or held-out predictive evidence; and track local-
cutoff/confidence overlap across refinement iterations. Deriving the field
from the current base pair removes lagged envelope feedback, but matching
against the NU-regularized replay map remains an outer refinement feedback
loop and weak omitted domains must be tested explicitly.
6.4 Ownership¶
| Concern | Owner |
|---|---|
| Mode selection/defaults/validation | parameters, dictionaries, UI, and refinement policy; ordinary global ML versus NU replay is explicit and mutually exclusive |
| Numerical precision | src/main/volume/simple_reconstructor_pcg.f90 (fixed NU evidence state behind the shared normal-operator wrapper and inside the support sandwich) |
| Evidence construction | src/main/nu_filt/ (null candidate, full unary reduction, spatial regularization, compact band-confidence state) |
| PCG reconstruction orchestration | simple_rec3D_pcg_strategy.f90 and assembly owner (base pair -> FSC -> evidence -> one replay; shared/distributed parity) |
| NU diagnostics/LP handoff | simple_commanders_rec_distr.f90 and NU modules; avoid recomputing an inconsistent second evidence state |
| Policy record | reconstruct3D_pcg_policy.md, nonuniform_filtering_policy.md |
The raw (B,D) format does not change. NU evidence and precision are
solve-time state, never accumulated into D; constructing the prior triggers
no particle or raw-artifact reread.
7. Hypotheses¶
- H1: A single NU-conditioned replay suppresses unsupported solvent and high-resolution variation without a binary molecular/solvent partition.
- H2 (architectural invariant): Selecting NU replay changes neither raw
(B,D), the base solves, nor FSC/cFAR, and preserves one base solve plus one replay per half. - H3: The explicit null state preserves genuinely coarse ordered density better than treating the 20-A saturating label as solvent.
- H4:
Q_NUproduces at least the scientific benefit of post-hoc NU filtering while participating in deconvolution and improving or preserving replay conditioning. - H5: NU replay outperforms the ordinary global
P_taureplay on heterogeneous/local-resolution data without degrading uniform high-SNR cases materially. - H6 (performance gate): The compact band operator and its preconditioner reach matched residual accuracy within an operationally acceptable budget.
Record numerical thresholds for "material loss", parity, convergence, and acceptable overhead before the real-data sweep.
8. Staged validation¶
Gate A: algebra and operator identity¶
Small synthetic volumes and fixed evidence fields: linearity of Q_NU; the
dot-product identity <x,Q_NU y>=<Q_NU x,y>; nonnegative quadratic form;
constant/DC null behavior; zero action for a fully supported band; monotone
action as evidence is withdrawn; continuity under small confidence changes;
penalty gradient versus finite differences; correct P(H+Q_NU)P
composition; hard failure when P_tau and Q_NU are both requested; and
unchanged ordinary-mode numerical identity. Mutation tests must break the
adjoint factorization, band ordering, mode exclusion, and evidence freeze.
Gate B: workflow and artifact invariants¶
Selecting NU replay does not change raw (B,D) checksums; leaves the base
solves and FSC bit-identical; performs exactly one replay per populated half;
derives evidence from _unfil maps without an ML auxiliary candidate; freezes
one evidence identity for even and odd; never attaches P_tau; generates no
PCG solvent envelope; keeps shared/distributed parity under the existing
deterministic-reduction tolerance; and leaves ordinary global-ML mode
bit-identical.
(Fractional/trailing reconstruction is hard-errored on the PCG path today,
which keeps this gate's surface small; revisit when that guard lifts.)
Gate C: controlled scientific tests — with convergence isolation¶
The direct lesson of the removed nu^4 prior is that, at production iteration
budgets, measured "prior effects" are confounded with replay under-convergence.
The spatially varying NU precision is not representable exactly by the current
Fourier-diagonal preconditioner, so convergence cost must be measured.
Scientific comparisons run at converged settings (high maxits_pcg /
tight rtol); the production-budget behavior is measured separately as H4.
Harness: the neutral-phantom fixture and the gated rec3D_backends
ground-truth mode (map-to-truth FSC, radial LS profiles, background handling,
centre-bin and shell diagnostics) already provide the measurement
infrastructure. Measure map-to-truth FSC (not only half-map FSC), local
resolution calibration, background variation, boundary ringing/leakage,
recovery of deliberately weak and coarse peripheral domains, PCG residual
history/stop reason, alignment overlap, and shipped-versus-base half-map
correlation inflation.
The first ablation is deliberately two-way:
| Run | Replay precision | Purpose |
|---|---|---|
| A | existing global FSC/SSNR P_tau |
Production estimator/control. |
| B | direct Q_NU, with P_tau absent |
Isolate the complete NU estimator. |
Use identical particles, poses, raw accumulators, base maps, candidate bank, and converged solver settings. Within B, sweep only one declared NU precision temperature/strength if calibration has not eliminated it. Do not tune from half-map FSC alone; prefer held-out predictive likelihood, map-to-truth FSC, or independent map-to-model comparison. Wilson is not part of this gate.
Gate D: acceptance for an experimental release¶
All algebraic and workflow invariants pass; ordinary global-ML outputs are
unchanged; improvements are supported by independent truth/model or held-out
evidence; coarse and weak-domain recovery is not systematically worse;
background suppression does not require a binary envelope; the NU replay
outperforms or sits on a useful quality/cost frontier versus P_tau; per-half
convergence stays within the operational budget or a measured new budget is
approved; and shared/distributed routes agree. Only then does Wilson become
the active development priority.
9. Risks and mitigations¶
| Risk | Consequence | Mitigation |
|---|---|---|
| Null state fails to separate solvent from coarse density | Real coarse signal is over-regularized. | Explicit no-signal competitor; coarse-domain synthetic gate; calibrate against the exact bank. |
| Huber costs treated as posterior probabilities | Arbitrary and nonportable precision scale. | Calibrated null/temperature with recorded bank, whitening, and smoothing provenance. |
| Evidence-derived replay map drives later alignment | Self-reinforcing local-bandwidth loss. | Current base-pair evidence, full spherical support, weak/omitted-domain recovery tests, field-overlap tracking. |
| Common evidence precision inflates half-map correlation | Misleading shipped-pair FSC. | Resolution from base pair only; shipped FSC diagnostic only; independent truth/model evidence. |
P_tau accidentally remains active |
Uninterpretable combined estimator. | Hard mutual-exclusion assertion and mutation test. |
| Wilson is introduced before NU is resolved | Multiple moving scientific variables. | Wilson starts only after NU Gate D. |
| LocScale amplitude target treated as linear | Operator depends on x; PCG assumptions fail. |
Fixed Wilson covariance or a lagged surrogate; never rescale inside apply_normal. |
| Common LocScale target carries phases | Half errors directly correlated. | Share only phase-free profiles; lagged, half-specific otherwise. |
| Overlapping detail bands change total strength | Candidate-bank size silently changes regularization. | Declared frame/band normalization; parity test when refining the bank. |
| Prior effect confounded with under-convergence | Wrong scientific conclusions (the nu^4 lesson). | Gate C convergence isolation; H4 measured separately. |
Preconditioner cannot represent spatial Q_NU exactly |
Iteration-count regression at production budgets. | Nonnegative declared approximation; matched-convergence science; cost gate. |
| A second NU pass builds different evidence | Replay and postprocess disagree about local support. | Construct once from base halves; share the compact state with diagnostics/LP handoff. |
10. Staged development plan¶
Each active stage ends at a gate; a later stage does not begin until the
preceding gate passes and the stage's changes are committed. New NU behavior
must be explicit and protected while the existing ordinary global-ML path
remains unchanged. "Fixture" means the neutral-phantom project with ground
truth plus the gated test=rec3D_backends. Stages 2--5 below are retained as
the completed/superseded solvent experiment record, not as active work.
Cross-cutting rules (the distilled foot-gun list)¶
- R1 — one variable per experiment. Never change solver behavior and a baseline in the same measurement.
- R2 — no moving baselines. Re-record every reference number after any solver change lands (the warm start moves the centre-bin, k=1–3, and RESID numbers the priors will be judged by).
- R3 — converged-settings science, production-budget cost. Scientific
claims at high
maxits_pcg/tightrtol; iteration-budget impact measured separately (the nu^4 lesson: 2-iteration results inverted conclusions). - R4 — every gate gets a mutation test where feasible. A gate that has never failed on a deliberate defect is not known to gate anything (the box-factor/deapodization mutations are the model).
- R5 — strength-zero bit-identity at every stage boundary. Raw
(B,D)checksums,_unfilmaps, and FSC unchanged with the feature off. - R6 — never tune from half-map FSC; truth/model comparisons or held-out evidence decide.
- R7 — reliability changes require a measured control at matched n (the
ref_taperlesson: master's own failure rate was 1/10). - R8 — evidence plumbing before operator math. First expose the null
candidate and compact immutable evidence state, then attach
Q_NU. - R9 — record acceptance thresholds in this document before running the experiment that is judged by them.
- R10 — replay precisions are mode-exclusive. Ordinary PCG attaches
P_tau; NU PCG attachesQ_NU. No command-line combination may activate both. Wilson is a later estimator choice, not an additive knob in the NU acceptance experiment.
Stage 0 — historical warm-start foundation¶
The cross-iteration ML warm start changed the baselines against which the solvent experiment was measured (R2). These entries are retained as history; they do not block the active Stage 6 NU design.
- 0.1 VALIDATED (2026-08-25): the cross-iteration warm start produces a clean bgal map with a sensible resolution estimate at production settings.
- 0.2 Re-baseline and record here: fixture, streptavidin, and bgal
rec3D_backendsnumbers (in-band ratio, k=1–3, centre-bin, RESID) at both 2 and converged iterations. - 0.3 Commit the consolidation + warm start.
Stage 1 — verification prerequisites (no behavior change)¶
- 1.1 DONE (2026-08-25). Audit confirmed:
apply_normalapplies the support on both sides and the RHS carries exactly onePin bothend_accumandsolve.test=pcg_reconStage 3b now asserts the masked operator's dot-product symmetry and strict positivity (the soft edge makesPnon-idempotent, so a one-sided mask application fails this gate where the unmasked check cannot see it). Full suite green. - 1.2 DONE (2026-08-25). Production never calls
set_deapod(default on) and always setsPCG_OP_KERNEL.assert_prior_attachment_mode(hard error unless kernel + deapodized) is called at both ML-replay attach sites — sharedregularize_state_halfand distributedprepare_distributed_half_job. - 1.3 DONE, refine3D diagnostics check pending (2026-08-25). Inert
report_solvent_envelope_statusin the strategy runs the former mask contract (presence, cubic lattice, physical-extent identity, constant-FOVread_and_cropresample on lattice mismatch, finiteness,[0,1]re-clip, nonzero solvent evidence) and emits thepcg_solvent_*block withprior_enabled=F, once per state before either half's ML replay, both execution paths. Fixture gate PASS unchanged (median ratio .8428, identical to the pre-change runs). NOTE: the fixture runsobjfun=cc, andl_ml_regrequiresobjfun=euclid, so the diagnostic block cannot fire there — the envelope-absent skip and the resample demo must be read from the next production refine3D/abinitio3D run.
Stage 2 — historical solvent Q_s operator record¶
DONE (2026-08-25). Q_s lives in simple_reconstructor_pcg
(set_solvent_prior / apply_solvent_precision / get_solvent_stats):
matrix-free s.*(x - mu_s(x)) with s = [p(1-m)]^2 normalized to unit mean
diagonal on its effective support; effective strength
lambda_solvent = pcg_solvent_lambda_rel * s_data(D) derived alongside the
relative ridge lambda in update_lambda_from_density; attached in both
concrete operators (deliberately NOT in the Fourier-shell preconditioner — a
real-space diagonal cannot be fused there; conditioning, not correctness).
pcg_solvent_lambda_rel (default 0.0) is registered and exposed on
reconstruct3D; the strategy attaches the prior to both halves' ML replays
when the envelope loader (resolve_solvent_envelope, the Stage-1.3 loader
gone live) validates it AND the strength is positive, with a per-half
PCG SOLVENT PRIOR line (lambda_eff, mean/rms/penalty_final) after each
priored solve, and a loud THROW_WARN skip when a positive strength has no
usable envelope.
- Gate results:
test=pcg_priors(8 stages: normalization contract, adjoint, PSD,Q_s 1 = 0, zero action on them=1plateau, graded-edge continuity at eps=1e-3, FD gradient ofR_s, composition with the masked normal operator) ALL PASS; mutationLinstead ofL^T L(outer weightsqrt(s)) fails the adjoint, gradient, and composition stages; mutation dropping the rank-one mean term fails exactly the null-space stage (and only it —s.*xis still symmetric PSD, the correct signature);test=pcg_reconfull suite unchanged; R5 fixture bit-identity holds (gated median .8428, identical to baseline, strength defaulting to 0). - Remaining for Stage 3: shared vs
nparts=2parity at a positive strength, abinitio3D/refine3D forwarding,rec3D_backendsstrength registration.
Stage 3 — historical solvent workflow-integration record¶
Implementation DONE (2026-08-25); real-data gate items pending.
pcg_solvent_lambda_rel is registered on refine3D and abinitio3D
(activation-gated on rec_backend=pcg, like maxits_pcg) and on
test=rec3D_backends; abinitio3D forwards it through the inherited
cline_refine3D plus apply_refine3D_reconstruction_controls (R10, exactly
the maxits_pcg/rtol route). No further forwarding was needed: the PCG
master always runs inside the process whose own command line carries the key
(simple_refine3D_strategy and simple_rec3D_strategy both pass params),
and workers only accumulate raw statistics.
- Gate results so far:
test=pcg_priorsStage 9 solves the same priored synthetic problem through monolithic streaming AND two-part raw reduction — the exact seam separating shared fromnparts=2execution — with identical effective strengths and rel_err(x) = 4.2e-5 (PASS); the fixturerec3D_backendswithpcg_solvent_lambda_rel=0explicit on the command line is bit-identical (.8428) — R5 strength-zero identity end-to-end. - Harness-validated (2026-08-25, neutral fixture, see the alignment-free
loop below): prior-on at strength 1e-3 —
prior_enabled=T,skip_reason=none, per-halflambda_eff/mean/rms/penaltylines, gates PASS (in-band .9904 vs .9927 unpriored; centre-bin ratio 1.16); strength-zero control —skip_reason=strength_zero, .9927 identical to the no-envelope run (R5); envelope-absent at positive strength — THROW_WARN withskip_reason=envelope_absent, run completes and PASSES. - Pending on real data (user-side run): one full abinitio3D with
rec_backend=pcg automsk=yesand a positive strength completing, showing the prior activating exactly when the envelope first exists lag-one and exercising the stage-handoff resample.
DECISION (2026-08-26): automasking auto-enabled for pcg in abinitio3D¶
With the prior on by default, abinitio3D closes the loop itself: when
rec_backend=pcg and NU filtering is active (the default), the stage policy
forces automsk=yes from the first NU stage (NU_FILTER_STAGE) onward — the
first NU stage's assembly generates the state-specific evidence envelope and
every later stage's ML replays are solvent-priored lag-one, with no user
flags. An explicit automsk=no vetoes; a user-supplied mode (tight) is
respected; gridding runs keep the last-stage-only automasking behavior.
Verification (pending, user-side): one abinitio3D run with rec_backend=pcg
showing NU ENVELOPE OCCUPANCY at stage 6 and pcg_solvent_prior_enabled=T
from stage 7 — this now doubles as the outstanding Gate B lag-one activation
item.
DECISION (2026-08-26): solvent suppression readout in the convergence report¶
With the prior default-on and self-bootstrapping, refinement needs an in-run answer to "is the prior firing, and is the strength right?" that requires no harness and no ground truth. The readout is the solvent suppression percentage, per half:
supp% = 100 * (1 - rms_s(x_ML) / rms_s(shrink(x_base)))
where shrink(x_base) is the base solution passed through the closed-form
per-shell FSC/Wiener shrinkage (fsc2shrink_filter in simple_estimate_ssnr,
shared with the regularized ML initialization). The shrunk base is the
reference rather than the raw base because the Wiener component of the ML
replay reduces solvent variation on its own; referencing it out isolates the
solvent prior's contribution, so an inert prior reads ~0% regardless of tau.
Plumbing: report_solvent_solve_stats prints pcg_solvent_rms_ref and
pcg_solvent_suppression_pct per half; both PCG execution paths average
even/odd per state and persist simple_pcg_solvent_stats.txt
(delete-then-rewrite each ML volassemble, so a skipped prior leaves no stale
values); check_conv3D prints % SOLVENT SUPPRESSED (PCG SOLVENT PRIOR)
with the other iteration stats, writes it to simple_stats.txt
(PCG_SOLVENT_SUPPRESSION_PCT + per-state keys), and advises:
- < 5%: prior inert — increase
pcg_solvent_lambda_rel(~3x) - 5-60%: nominal — keep the strength
- > 60%: over-flattening risk — decrease (~3x)
Thresholds are provisional constants in simple_convergence
(PCG_SOLVENT_SUPP_INERT/OVER_PCT), anchored to the bgal ladder verdicts;
recalibrate by rereading the ladder now that every run prints the readout.
Automatic sweep (2026-08-26): test=rec3D_backends now sweeps the
default ladder {0, 1e-3, 1e-2, 3e-2, 1e-1, 3e-1, 1} automatically whenever a
prior-capable invocation (ml_reg + NU envelope in the cwd) leaves
pcg_solvent_lambda_rel unset -- one execution directory per rung, collated
Stage-5 observables printed and written to rec3D_backends_sweep_summary.txt
(base/shipped pair FSC, suppression %, solvent RMS, gated-band ratio/FSC,
radial min/max, truth FSC, gates), with the base-pair-spread negative control
asserted across the ladder. An explicit strength runs a single comparison --
the follow-up mechanism for extra rungs.
Fixture sweep record (its=5, rtol=1e-3, lp=10, reg-init; base pair pinned 4.036/3.347 across the ladder, spread 0.0000):
| lambda | supp% | sol_rms | ship 0.5/0.143 | band_ratio | band_fsc | rad_min/max | truth_fsc_pcg |
|---|---|---|---|---|---|---|---|
| 0 | -- | -- | 4.036/3.347 | 1.006 | 1.000 | 0.98/1.01 | 0.971 |
| 1e-3 | 5.0 | 12.2 | 4.036/3.347 | 1.002 | 1.000 | 0.99/1.01 | 0.970 |
| 1e-2 | 14.7 | 11. | 3.920/3.347 | 0.987 | 0.999 | 0.97/1.06 | 0.970 |
| 3e-2 | 29.1 | 8.8 | 3.920/3.347 | 0.950 | 0.997 | 0.92/1.12 | 0.966 |
| 1e-1 | 51.7 | 6.0 | 3.709/3.267 | 0.872 | 0.981 | 0.86/1.23 | 0.948 |
| 3e-1 | 71.2 | 3.6 | 3.430/2.744 | 0.798 | 0.930 | 0.81/1.42 | 0.898 |
| 1 | 86.1 | 1.7 | 2.144/2.144 | 0.759 | 0.848 | 0.75/1.85 | 0.810 (1 gate) |
Fixture read (R6: judged by truth, not half-map FSC): truth agreement is flat to 1e-2, mildly down at 1e-1 (-0.023) and clearly damaged from 3e-1; first material truth loss coincides with supp ~52% and clear damage with ~71%, bracketing the provisional 60% over-flattening bound from both sides. The fixture's truth-optimal strength (~1e-2) is below bgal's visually-chosen 1e-1 -- dataset-dependent, which is exactly what the in-run suppression readout is for. The 5% inert bound is confirmed: the 1e-3 rung reads 5.0% with every other observable at the zero-control value.
DECISION (2026-08-26): solvent prior on by default at the calibrated strength¶
Following the bgal calibration the solvent prior graduated from experimental
toggle to default behavior: pcg_solvent_lambda_rel now defaults to 0.1
(the calibrated operating strength), so whenever rec_backend=pcg runs an ML
replay and a valid state-specific NU evidence envelope exists, the prior is
applied — no flag required. 0 remains the explicit off-switch, preserving
the A/B capability R10 requires for the remaining Gate C envelope-variant
tests and for calibration on new dataset classes. With default-on, an ABSENT
envelope is the normal lag-one state (early iterations, gridding-era
projects) and produces only the diagnostic block; a PRESENT-but-unusable
envelope (extent mismatch, invalid values) still warns loudly. The former
never-substitute-a-mask rule remained in force for that experiment.
Stage 5 record: streptavidin sweep (2026-08-27, minimal-invocation harness)¶
First bare-invocation sweep on real data (10335 streptavidin, d2, box 128, smpd 1.072, its=5): base pair pinned 3.430/3.119 across the ladder, spread 0.0000, all gates PASS.
| lambda | supp% | shipped 0.5/0.143 | band_fsc | rad_min | verdict |
|---|---|---|---|---|---|
| 0 | -- | 3.347/3.049 | 0.999 | 1.000 | control |
| 1e-3 | 6.6 | 3.347/3.049 | 0.999 | 1.000 | inert |
| 1e-2 | 21.8 | 3.267/3.049 | 0.996 | 1.000 | clean |
| 3e-2 | 36.1 | 3.191/3.049 | 0.991 | 1.000 | clean -- operating point |
| 1e-1 | 51.2 | 3.119/2.800 | 0.981 | 0.998 | shipped inflation fires |
| 3e-1 | 63.5 | 3.049/2.144 | 0.965 | 0.912 | inflation + rim erosion |
| 1 | 78.7 | 2.144/2.144 | 0.932 | 0.741 | over-flattened |
Cross-dataset findings: (a) the suppression-vs-lambda curve nearly overlays the fixture's (6.6/22/36/51/64/79 vs 5/15/29/52/71/86 per decade) -- the data-scale anchoring makes the readout portable; (b) the FIRST over-regularization signature fires at ~51% suppression on both datasets (fixture truth-FSC loss at 51.7%, streptavidin shipped inflation at 51.2%), independent observables agreeing that trouble onset is a property of the suppression coordinate. OPEN DECISION (superseded 2026-08-29, see the CLOSED entry under Stage 6): lower PCG_SOLVENT_SUPP_OVER_PCT from 60 to ~45-50 (at 60 the default 0.1 reads "nominal" on streptavidin while its inflation diagnostic has fired), and whether the default strength should move 0.1 -> 3e-2 (clean everywhere measured) with the convergence guidance steering upward toward the ~50% line.
Gate B lag-one activation: CLOSED (2026-08-27, streptavidin abinitio3D)¶
The warm-started rec_backend=pcg abinitio3D run verified the full
self-bootstrapping staging on real data with zero user flags: stage 5
automasking off + envelope_absent quiet skip; stage 6 first iteration
forces automasking on, NU engages with the spherical reference ("envelope
not available yet" — correct first-iteration state) and the volassemble
generates the envelope (occupancy 7.4%); the next iteration consumes it
lag-one (prior_enabled=T, skip_reason=none) and every later iteration
re-generates/re-consumes. Steady-state warm-started suppression: ~28-35%,
stable (third regime datum — cold its=5 box-128: 51%; converged bgal
box-256: 14%; warm box-88: ~32% — triply confirming the removal of
absolute-% upper thresholds). The prior-free base-pair 0.143 improved
3.812 -> 3.191 A through stage 6 (crop Nyquist) — the loop demonstrably
improves registration. WATCH ITEMS (not acted on): at prior/NU engagement
the promoted matching lp (6.2 -> 3.5 A), the switch to envelope-masked
references, and the prior all hit the matcher in one iteration ->
orientation overlap collapses 0.70 -> 0.04 and recovers slowly; stage 6
exits unconverged on its budget. Envelope occupancy oscillates 7 -> 23 ->
14-18% (jitter, not runaway shrinkage). Candidate mitigations: stagger the
reference-mask switch one iteration after lp promotion; cap the promoted
lp a margin below the crop Nyquist.
DECISION (2026-08-27): inflation-based over-flattening guidance (implemented)¶
The convergence guidance is restructured per the anchor verdict below:
- Inert: suppression < 2% (
PCG_SOLVENT_SUPP_INERT_PCT) -> increasepcg_solvent_lambda_rel~3x. Dataset-robust at the low end. - Over-flattening: shipped-pair FSC=0.143 pulling > 5% finer than the
base-pair 0.143 (
PCG_SOLVENT_INFL_PCT) -> decrease ~3x. The portable signal (fired at 1e-1 on streptavidin, quiet through 3e-1 on bgal, matching both ladder verdicts). Absolute suppression thresholds are NOT used for the upper bound. - Suppression % is retained as the monotone within-run trend readout.
Plumbing: the PCG strategy now measures the shipped (regularized) pair's
FSC crossings in-run after the ML replays (shipped_pair_res, same soft-
masked procedure as the harness diagnostic; computed only when the prior
fired) and persists PCG_BASE_FSC0143_STATEXX/PCG_SHIP_FSC0143_STATEXX
in simple_pcg_solvent_stats.txt; check_conv3D prints
% SHIPPED-PAIR FSC INFLATION (PCG PRIOR) (per-state lines for
nstates>1), applies the two-test guidance, and records
PCG_SHIP_INFLATION_PCT in simple_stats.txt. The shipped-pair crossing
shares regularization between halves and is never a resolution claim.
Stage 5 record: bgal its=30 anchor at 1e-1 (2026-08-27) — portability verdict¶
Base solves converge (RESID 2.7e-2 @ 30 its); ML replays stop on XTOL at 18/19 its with RESID still 0.12 (CG stalls by step size on the box-256 ML system — preconditioner item). Suppression at 1e-1: 12.8/14.0% (up from 4.7% at its=5; rms 0.675 -> 0.638; reference rose to ~0.73 because the 30-it base carries more high-frequency noise). VERDICT: convergence explains ~3x of the bgal deficit but NOT the gap to streptavidin's 51% — lambda_eff at identical lambda_rel=0.1 is 1.98e4 (bgal) vs 1.53e6 (strep), i.e. the s_data(D) anchoring does not equalize prior-vs-data balance across box sizes; bgal's dose-response is genuinely right-shifted (~lambda_rel>=1 for 50%-class suppression). Side findings: converged 1e-1 rim is pristine (bin-9 ratio 1.067 vs 0.851 at its=5 — better peripheral preservation); Nyquist-edge shells k=126-128 blow up (amp ratio up to 17x) over 30 ML its — S6 beyond-band growth, outside the gated band, irrelevant at production budget. GUIDANCE REDESIGN (proposed, pending decision): absolute suppression thresholds cannot be universal; the observable that landed at the right per-dataset operating point on BOTH datasets is shipped-vs-base pair inflation (fired at 1e-1 strep, 3e-1 bgal). Restructure convergence guidance as: inert = suppression ~<2% (dataset-robust); over-flattening = shipped-pair 0.143 materially finer than base-pair 0.143 (portable, both already computed in-run); suppression % retained as the monotone within-run trend readout.
Stage 5 record: bgal harness sweep + suppression-portability caveat (2026-08-27)¶
Bare-invocation sweep on bgal (d2, box 256, its=5, shared): base pinned 4.132/3.709, spread 0.0000, all PASS; the rms ladder and shipped-inflation verdicts reproduce the 2026-08-26 manual ladder exactly (rms 0.716/0.712/0.703/0.675/0.61/0.46; inflation first at 3e-1). BUT the suppression scale does NOT overlay streptavidin/fixture: bgal reads -1/-0.5/+0.7/4.7/14/35 % per decade vs streptavidin's 6.6/22/36/51/64/79 %. Diagnosis: convergence state — bgal ML RESID ~0.17-0.19 at its=5 (box 256, STOP=maxits) vs streptavidin ~0.03 (box 128); an unconverged iterate has not expressed the prior, so cold-solve suppression under-reports on large boxes. The negative values at inert strengths expose a smaller per-dataset reference bias (shrunk base is not exactly the lambda=0 ML solution; +6.6% strep, -1% bgal at inert rungs). CONSEQUENCES: (a) the ~50%-onset universality claim below is RETRACTED as an absolute-percentage rule — suppression is monotone and useful within a ladder, comparable across datasets only at matched convergence; (b) threshold/default retuning is ON HOLD pending the its=30 bgal anchor at 1e-1 (decisive: convergence vs data-scale anchoring) and the warm-started abinitio3D suppression trajectory (the accumulated steady-state number is the one the convergence guidance actually sees); (c) the harness should report/flag RESID next to suppression when the ML solve stops on maxits far from rtol.
Stage 5 record: bgal solvent-prior strength calibration (2026-08-26)¶
Full dose-response on beta-gal (box 256, smpd 1.275, d2, ~4.6k ptcls,
maxits_pcg=5 with the regularized ML initialization; every run's base-pair
FSC pinned at 4.132/3.709 A -- the negative control across the 1000x sweep):
| lambda_rel | solvent RMS | rim ratio (64-72 px) | shipped FSC 0.5/0.143 | verdict |
|---|---|---|---|---|
| 1e-3 | 0.7156 | 0.862 | 4.132/3.709 | inert |
| 1e-2 | 0.7117 | 0.861 | 4.132/3.709 | inert |
| 3e-2 | 0.7032 | 0.859 | 4.132/3.709 | whisper |
| 1e-1 | 0.6752 | 0.851 | 4.132/3.709 | active, clean |
| 3e-1 | 0.6075 | 0.827 | 4.080/3.709 | first inflation + mild rim loss |
| 1 | 0.4616 | 0.741 | 4.030/3.667 | over-flattening |
Findings: (1) the active regime on real data starts ~1e-1, roughly 30x above
the neutral fixture's (noise-dependent, as the aliasing argument predicts);
(2) the two over-regularization diagnostics -- envelope-boundary rim erosion
in the radial table and shipped-pair FSC inflation -- fire TOGETHER at 3e-1
and grow monotonically, while the base-pair FSC never moves; (3) backend
agreement (gridding vs pcg) improves monotonically with lambda through the
whole range (band k=2-103 -> 104, high-shell excess shrinking), evidence the
prior removes error rather than adding structure (independent algorithms,
shared data). Recommended default pcg_solvent_lambda_rel=1e-1; 3e-1 is
the aggressive bound. In refinement, prefer the conservative default: the
shipped map feeds next-iteration alignment and the lag-one envelope, so rim
erosion is the entry point of the self-reinforcing support-shrinkage feedback
(S9) that the standalone harness cannot observe. Earlier note: the first
its=5 sweep (pre-initialization) was transient-dominated and inconclusive --
strength conclusions require the regularized init or converged solves (R3).
Alignment-free prior validation harness (no new abinitio3D runs)¶
Prior validation and strength sweeps need only a project file with a previous
abinitio3D registration (poses in ptcl3D), the last iteration's half maps,
and that run's sigma2_it_*.star files — no new alignments. nu_filt3D is
the envelope generator: with nu_envmsk=yes it builds the identical NU
evidence envelope object volassemble publishes
(envmask3D_from_lmask on the NU evidence margin), just under a different
name. The loop, run in a scratch dir containing the sigma2 star files (or in
the final refine3D stage dir of the abinitio3D run):
simple_exec prg=nu_filt3D vol1=<prev_odd.mrc> vol2=<prev_even.mrc> \
smpd=<smpd> mskdiam=<D> nu_envmsk=yes mkdir=no nthr=<N>
cp outvol_nu_envmask.mrc nu_envmask3D_state01.mrc
simple_test_exec test=rec3D_backends projfile=<proj.simple> pgrp=<pgrp> \
mskdiam=<D> objfun=euclid ml_reg=yes pcg_solvent_lambda_rel=<X> nthr=<N> \
[vol1=<truth.mrc> lp=<lp>]
Notes: vol1 to nu_filt3D is the ODD half, vol2 the EVEN half (nu_filt3D
convention); the envelope is written as <outvol-basename>_nu_envmask.mrc
(default outvol_nu_envmask.mrc); mkdir=no keeps it in the cwd; a
native-box envelope against a cropped reconstruction is handled by the
loader's constant-FOV resample (and demonstrates that path); X=0 is the R5
control and must be bit-identical; with ml_reg=yes the truth LS-profile
check is reported as a diagnostic rather than gated (the shipped maps are
ML-regularized; the gate is calibrated on unregularized maps). The
envelope-tuning knobs (nu_msk_sig/beta/dens, amsklp) let the same loop
drive the former Gate C eroded/dilated envelope variants without touching the
reconstruction inputs. Every historical sweep value of pcg_solvent_lambda_rel
reuses the same envelope file — one nu_filt3D call per envelope variant,
one rec3D_backends call per strength. Validated on the neutral fixture
(2026-08-25): a 20-second single-iteration refine3D seeds
sigma2_it_1.star and the half pair when a project has poses but no sigma2
files; the loop then activates the prior end-to-end
(pcg_solvent_prior_enabled=T, per-half lambda_eff/mean/rms/penalty lines).
Stage 4 — retired solvent Gate C plan (do not execute)¶
- Record thresholds first (R9): "material molecular loss", acceptable weighted-solvent-RMS reduction, omitted-domain damage bound.
- Fixture sweep at converged settings; envelope variants: true, eroded, dilated, omitted-domain (the bias test); measure the former solvent hypotheses plus map-to-truth FSC + boundary ringing + residual histories; the former cheap locres- diagonal control was planned for the same sweep.
- Abort criterion: if omitted-domain damage exceeds its bound at every useful strength, stop; the former aux-competition validator would have become a prerequisite rather than an option.
Stage 5 — retired solvent Gate C plan and completed measurements¶
- Primary real-scenario harness: abinitio3D with
automsk=yesin the late stages, prior flags on vs off, everything else identical. Because the toggles ride the existing refine3D plumbing (R10), the A/B is two command lines differing only in prior strengths — a direct comparison in the exact configuration applications will use. Read: final map quality and resolution, the DIAG trajectory, the historical low-shell/centre diagnostics from a post-runrec3D_backends, and the solvent RMS diagnostics. - Standalone sweeps on bgal and streptavidin at converged settings against matched strength-zero runs (fixed poses, isolates the estimator); then H4 at the production budget, including the warm-start interplay (the prior changes the map the next iteration warm starts from).
- Reliability control (R7): a 10x abinitio3D batch at the chosen strength vs the recorded intrinsic ~1/10 rate; treat small-n differences as noise unless they replicate.
Stage 6 — direct NU-evidence replay — active priority 1¶
6.1 IMPLEMENTED, build/runtime gate pending (2026-08-27). Evidence contract, no solver behavior change:
setup_nu_dmats(..., evidence_source='base_unfil')opts into replay-evidence construction. The tagged path fingerprints the exact spherical-support values of both input halves, rejects auxiliary replacement candidates, andbuild_nu_evidence_staterechecks the fingerprint before compaction. Existing callers omit the tag and are unchanged; the current solvent prior remains in place until Stage 6.3 replaces it in the NU target workflow.- The explicit null candidate is zero cross-half prediction: raw even is scored
against zero odd prediction and raw odd against zero even prediction with the
same radial whitening profile and Huber loss as the signal bank. It is
smoothed at the discretized nominal-20-A scale, exactly matching its adjacent
coarsest candidate. The raw zero loss has a systematic offset from smoothed
predictors even under independent noise, and choosing the best signal label
introduces a multiple-comparison advantage. The calibrated null therefore
subtracts the robust median-plus-three-MAD offset of
C_zero-min(C_signal bank)over the generous spherical support. This preserves the explicit cross-half null and lets truly coarse shared signal win whenever the coarse candidate's improvement exceeds the exact-bank null distribution. Null plus the full retained signal bank are regularized by a separate copy of the established ordered-label model, so the production NU filter map is not changed. - The frozen
nu_evidence_statehas private packed storage and a copy-out API. It retains four nested coarse-to-fine support confidences (detail supported through 20/12/8/5 A), selected cutoff including cutoff zero for the null, normalized label entropy, spherical-support geometry, and scalar summaries. The geometry is sufficient to recreate the lexicographic packed-voxel order after mutable NU state is released. Confidence is a spatial-model softmax whose temperature is the median final best-versus-next energy gap for the exact bank. Provenance records the bank, ordering, whitening checksum, smoothing, temperature, and Potts scale; a content-derived FNV-1a identity is carried with the state. The null occupies one packed vector and reads the signal bank in place during compaction; it does not duplicate the full unary matrix. Validation enforces finiteness,[0,1]bounds, and monotone coarse-to-fine support. simple_test_nu_envmasknow gates the null competitor on its independent-noise solvent/common-signal molecule: molecular coarse-band support must exceed solvent support, solvent null selection must exceed molecular null selection, all packed fields must satisfy the immutable-state contract, and the state must remain valid after mutable NU unary storage is released. Candidate-order mutation is guarded by a hard strict-order check; null calibration and exact provenance are emitted byprint_nu_evidence_summary.- Lightweight source validation completed:
git diff --checkand the Fortran source-index parser pass. The user-side build succeeded; runtime validation remains open before 6.1 is marked gate-complete.
First user-side execution reached compact-state construction after the NU bank
and ordered-label passes, then tripped the [0,1] immutable-state check. The
coarse support field sums all signal-label softmax probabilities and can exceed
one by single-precision accumulation roundoff. The derived support fields and
their summaries are now clipped at their mathematical probability bounds.
Softmax terms more than 80 calibrated energy units above the best state are set
to zero explicitly, avoiding the otherwise harmless IEEE underflow flag. Rerun
then showed that the uncalibrated zero predictor never won: molecular and
solvent null fractions were both zero, and coarse support saturated throughout
the support. This was a scientific null-model failure, not a test tolerance.
The first bias-calibration rerun still selected no null voxels: calibrating only
against the coarsest candidate did not account for another signal-bank member
winning solvent by chance. The calibration now uses the best exact-bank signal
cost, including that multiple-comparison advantage, without redefining the
coarsest signal label as solvent. Rerun of simple_test_nu_envmask is pending.
6.2 Operator Gate A:
- implement the normalized three- or four-band
Q_NUfactorization; - prove adjoint identity, PSD, DC null, monotonicity, finite-difference
gradient, and
P(H+Q_NU)Pcomposition; - add the hard
P_tau/Q_NUmutual-exclusion assertion; - implement and identify a nonnegative preconditioner approximation.
6.2 operator IMPLEMENTED, Gate A algebra tests pending (2026-08-27;
review-corrected the same day). set_nu_prior/apply_nu_precision in
simple_reconstructor_pcg: Q_NU = C (sum_b B_b^T W_b B_b) C with
B_b = crop o IFFT o M_b o FFT o pad, M_b the disjoint radial 0/1 masks on
the padded Toeplitz lattice cut at the NU_EVIDENCE_BAND_LIMITS boundaries
(20/12/8/5 A; band 1 also absorbs everything coarser, band 4 everything
finer, padded DC and physically unaddressed points belong to no band), and
C = I - 11^T/N the native-box mean-centering projector applied on BOTH
sides. Review corrections to the first cut, recorded so the claims stay
honest:
- Constant null mode. Padded-DC exclusion alone is NOT sufficient: a
native constant zero-pads to a box window with substantial non-DC padded
content, so the first cut had
Q_NU(c 1) /= 0. The explicit symmetric centeringCon both sides now provides the EXACT constant null mode (Gate A must verifyQ_NU(c 1) = 0to roundoff, not merely small). - Not a tight frame. The pad/crop sandwich means the
B_bare not orthogonal projectors andsum_b B_b^T B_b < Iwith cross-band leakage; refining or merging the band partition CHANGES the operator. The invariance claim is retracted. What survives, and is the declared normalization: eachB_bis a contraction (restriction o projection o extension) and theM_bare disjoint, sosum_b ||B_b x||^2 <= ||x||^2and withW in [0,1],||Q_NU|| <= 1. Gate A gains a partition-change test: uniform weights, two different band partitions, MEASURE the deviation and record it as the bank-refinement sensitivity rather than asserting zero.
W_b = [p*(1-a_b)]^2 grades the lack-of-evidence weight by the soft support
exactly as the solvent weight did; each B_b is symmetric (restriction
adjoint to zero-extension, real even diagonal) and C is symmetric
idempotent, so Q_NU is symmetric PSD. Per-band results accumulate in real
space (two padded complex workspaces live). Strength:
lambda_nu = pcg_nu_lambda_rel * s_data(D), derived in
update_lambda_from_density beside the ridge and solvent lambdas. Mutual
exclusion (R10) is enforced bidirectionally at set time in ALL pairs:
set_ml_prior rejects an attached Q_NU and vice versa, and set_nu_prior
and set_solvent_prior each reject the other (both attachment orders must be
mutated in Gate A). The declared nonnegative preconditioner approximation is
the support-mean band weight fused as a shell diagonal in
finalize_density_accum, mirroring the ML-prior fusion
(nu_precond_shell_diag). Deapodized-kernel-only attachment rides the
existing assert_prior_attachment_mode. Gate A algebra/mutation tests
(planned as a test=pcg_priors extension) are NOT yet written; the list now
includes the exact-constant-null check, the partition-change measurement, and
both mutual-exclusion attachment orders.
6.3 workflow wiring IMPLEMENTED for both PCG paths, runtime gate pending
(2026-08-27). pcg_nu_lambda_rel (default 0 = ordinary global-ML replay;
registered on reconstruct3D and test=rec3D_backends) selects the NU
replay: after both base solves and the FSC, build_nu_replay_evidence runs
setup_nu_dmats(..., evidence_source='base_unfil') ->
optimize_nu_cutoff_finds -> build_nu_evidence_state -> cleanup_nu_filter
on the current base half pair, prints the evidence block
(pcg_replay_prior_mode=nu_evidence + pcg_nu_* summary), and
expand_nu_evidence_band_weights (new, recreates the packed lexicographic
order from the frozen geometry alone; w_b = 1 - a_b inside the spherical
evidence support, 1 outside it) hands the solver its weights. Both half
replays share the one frozen state; set_ml_prior/set_solvent_prior are
never called in NU mode and no envelope is resolved, read, or written.
Review-added contracts (2026-08-27):
- Readiness contract. A valid compact state is necessary but not
sufficient:
assert_nu_evidence_replay_readyhard-errors before either replay when the explicit null wins less thanNU_EVIDENCE_MIN_NULL_FRAC(provisional 1%, R9) of the generous spherical support -- the observed zero-null calibration failure must never attach silently -- or, after the first streptavidin run exposed the opposite failure, more thanNU_EVIDENCE_MAX_NULL_FRAC(provisional 90%): a saturated null is equally a calibration failure, never a specimen property. The same assert now gatessimple_test_nu_envmask, and that test must pass before the reconstruction harness results are trusted. - Explicit activation, no silent fallback.
validate_nu_replay_request(called from both PCG execution entries) hard-errors on a non-finite or negative strength and on a positive strength without the euclid ML replay (ml_reg=yes);reconstruct3Dhard-errors on a positive strength with any non-pcgrec_backend. The harness deletes the key on its gridding leg. - Timing honesty. The post-solve
get_nu_prior_statscosts one fullQ_NUapplication (~13 padded FFTs, material at small iteration budgets); it is timed separately, printed asstats_overhead_son thePCG NU REPLAYline, and included in the shared path's replay total.
The
replay warm-starts from the previous-iteration shipped half when one exists
and otherwise cold-starts from the base solution (the closed-form shrinkage
init encodes the P_tau/Q_s optimum and is skipped). Per-half
PCG NU REPLAY lines report lambda_eff and pcg_nu_prior_energy_final;
the shipped-pair inflation crossings are measured in NU mode too. Distributed
parity: the same attach path runs in prepare_distributed_half_job; trailing
reconstruction and the trailing bootstrap are hard-errored with the NU replay
(the evidence contract requires the plain current-cohort base pair). The
solvent prior, its envelope bootstrap, and its default remain untouched
outside NU mode — removal from the target workflow is still owed once the NU
estimator passes its gates.
Testable harness invocation (test=rec3D_backends). A NU-replay run is a
single measurement (gates soft, as for any prior-active run), needs NO
envelope artifact, and forces pcg_solvent_lambda_rel=0 explicitly (R10):
simple_test_exec test=rec3D_backends projfile=<proj.simple> pgrp=<pgrp> \
mskdiam=<D> pcg_nu_lambda_rel=<X> nthr=<N> [vol1=<truth.mrc> lp=<lp>]
against the strength-zero control (omit the key and use
pcg_solvent_lambda_rel=0 for the pure P_tau reference, R5/Gate C run A)
and the solvent rungs already recorded. The execution directory carries a
_nu<X> token.
Stage 6 first run: streptavidin NU replay at 0.1 (2026-08-27) — saturated-null calibration failure¶
First end-to-end harness run (10335 streptavidin, d2, box 128, mskdiam 80,
its=5, pcg_nu_lambda_rel=0.1). Workflow invariants all held: base pair
bit-identical between backends (3.518/3.119 A both legs, H2); one frozen
evidence identity for both half replays; solvent forced 0, no envelope
touched; lambda_eff 1.73e6 (vs the solvent prior's 1.53e6 on the same data
-- s_data(D) anchoring consistent); soft gates reported FAIL without
aborting, as designed. First H6 cost data: NU replay 26-31 s/half vs 6 s
base (~5x) plus 3-4 s stats_overhead_s.
Scientific failure, opposite sign to the zero-null one:
pcg_nu_null_fraction=1.000000 with uncertain_fraction=0 -- confidently
null everywhere including the molecule core, while the softmax still carried
28.6% coarse-signal mass (band01=0.286). Q_NU therefore degenerated into
a global detail penalty: shell amplitude ratio pcg/gridding declining
monotonically 0.9 (20 A) -> 0.33 (4 A), gated-band median 0.63 (FAIL at
<0.67). Diagnosis: the null offset median + 3*MAD of
C_zero - min(C_signal) over the whole support (median 0.37, MAD 0.18,
threshold 0.92) is a DETECTION threshold -- a signal label could only win
where its advantage exceeded the null population's 3-sigma -- and on a
molecule-dominated support the median itself is signal-contaminated. Right
shape for the retired binary envelope, wrong shape for a likelihood offset:
with overlapping components at 3.1 A it makes the null unbeatable. (It passed
the envmask fixture only because that molecule's gaps are hugely separated.)
Calibration redesign (implemented same day): the subtracted offset is now
the null component's CENTER only, estimated by the lower quartile of the gap
mixture (signal gaps sit strictly higher, so Q25 tracks the null component
even on a majority-molecule support); median/MAD stay recorded as
diagnostics, null_offset=lower_quartile_center in the provenance, and the
+3*MAD term is gone -- graded competition belongs to the softmax and
spatial consolidation to the ordered-label model. The readiness contract
gained the matching ceiling (NU_EVIDENCE_MAX_NULL_FRAC=0.90, provisional
R9): this run would now hard-error at evidence construction instead of
attaching a degenerate precision. Envmask rerun under the center-only
offset: PASS (2026-08-27). null_fraction 0.123 (inside the readiness
window); molecule coarse support 1.000 with zero null selections; solvent
support 0.829 with 18% null selections; band support monotone
0.884/0.884/0.804/0.291; envelope path untouched and passing. Watch item,
recorded not acted on: solvent coarse-band support 0.83 on an
independent-noise fixture means the Q25 offset is deliberately permissive
(~25% of gaps below it) -- weak coarse-band solvent penalty (W_1~0.17),
stronger fine-band penalty (W_4~0.7). Conservative direction for H3;
whether it regularizes ENOUGH is the S5.4 evidence-to-precision mapping
question the harness measures.
Stage 6 first PASSING run: streptavidin NU replay at 0.1 (2026-08-27)¶
Identical invocation to the failed run; center-only offset (Q25 0.252 vs the old detection threshold 0.92 on the same gap distribution: median 0.369, MAD 0.184). PASS, all gates. Evidence now graded and plausible: null_fraction 0.189, band support 0.828/0.755/0.670/0.373 (monotone), uncertain 3.1%. Observables:
- base pair pinned 3.518/3.119 (bit-identical to gridding and to the failed run -- H2/R5 hold);
- gated-band median amplitude ratio 1.066 (was 0.63 saturated-null), radial profile flat 1.00-1.09 inside 0.85x mask radius, centre-bin 0.79;
- shipped pair 3.267/3.049 -- 0.143 inflation vs base 2.2%, under the 5% bound, and equal to the ordinary P_tau shipped pair's 3.049 on this data;
- prior energy 7.1e9 (10x below the saturated run), ML RESID 1.7e-2 vs the P_tau-less base 4.0e-2 -- the precision also conditions the replay;
- known low-shell excess reappears mildly (k=1-2 ratio 1.75/1.50, inside the recorded historical 1.3-2.0 range);
- WATCH: near/beyond the band edge the NU replay carries more amplitude than the FSC-weighted gridding product (ratio rising 1.3 -> 7 over k=35-45, large beyond-band excess above k=46, outside the gated band). Expected structurally -- Q_NU has no per-shell Wiener rolloff, it suppresses only what local evidence disclaims (band-4 mean weight ~0.63) -- but whether that retained edge amplitude is signal or noise is precisely an R6 question: only truth/held-out comparison decides it, never the half-map tables.
Next per Gate C: neutral-fixture run with ground truth (vol1= + lp=) and
the two-way P_tau-vs-Q_NU ablation at converged settings; then bgal (large
box, right-shifted dose-response) as the second dataset.
Gate C first two-way ablation: 1WCM phantom, truth-judged — Q_NU wins (2026-08-27)¶
New user-built fixture (1WCM/RNA-Pol-II phantom, box 256, smpd 1.0, 2500
ptcls, c1, mskdiam 200, euclid+ml_reg, its=30 rtol=1e-4, lp=10 truth
comparison; an earlier B attempt at mskdiam=60 amputated the molecule inside
the support sphere -- negative low-k truth FSC, discarded, mask now matched).
Run A = P_tau control (pcg_solvent_lambda_rel=0), run B = Q_NU at 0.1.
Both PASS all gates; base pair bit-identical across A/B/gridding
(3.368/2.943 -- H2/R5). Evidence in B: null 0.126, bands
0.924/0.637/0.513/0.173, uncertain 0.
Map-to-truth FSC (the R6 decision variable; gridding column identical in both runs):
| shell (A) | gridding | A: P_tau pcg | B: Q_NU pcg |
|---|---|---|---|
| 4.00 | 0.9427 | 0.9408 | 0.9543 |
| 3.20 | 0.7397 | 0.7317 | 0.7968 |
| 3.01 | 0.5931 | 0.5824 | 0.6748 |
| 2.91 | 0.5014 | 0.4951 | 0.5818 |
| 2.56 | 0.2079 | 0.1977 | 0.2844 |
Truth FSC=0.5 crossing: ~2.91 A (gridding), ~2.91 A (P_tau), ~2.84 A (Q_NU). Q_NU matches everyone at low k (>=0.9988 through 5 A) and is uniformly better than BOTH the P_tau replay and gridding from ~4 A outward -- H1/H4/H5 supported on this fixture. Truth LS radial profile flat for Q_NU (0.94-1.04 through bin 13) where gridding drifts to 1.3-1.5. Shipped pair 3.241/2.844: 0.143 inflation vs base 3.4%, under the 5% bound.
Costs and watch items:
- Convergence cost (H6). The Q_NU replay ran 29-30 its at ~300 s/half
(one half hit xtol at 29); the P_tau replay converged in 2 its / 6 s
(warm shrinkage init ~= its optimum). ~50x converged-cost gap =
Q_NU FFT stack x no closed-form initialization. Candidate mitigation for
the production budget: a bandwise-shrinkage initial guess (scale each
band by ~1/(1+lambda*W_b/rho) locally or shell-mean), the Q_NU analogue
of
regularized_ml_initial_guess. Production-budget (its=5) A/B still to be measured before any refinement integration. - Rim amplitude. Q_NU map carries 1.5-2.0x gridding's |rho| in radial bins 10-13 (72-104 px) where P_tau's map declined; truth LS stays ~1 and truth FSC is better, so it reads as retained real signal at the periphery rather than halo -- but verify against eroded/omitted-domain variants before trusting it generally.
- Beyond-band retention. amp_pcg/truth grows 20-80x for k>105 where truth FSC < 0.2: unshrunk beyond-band content survives at converged settings (S6 signature, outside the gated band). The evidence-mapped band-4 weight suppresses by lack of evidence, not by SSNR, so a Wiener-like rolloff is absent by design; postprocessing/FSC filtering owns the shipped-map rolloff as it does for the base maps.
Gate C bgal ablation (no truth), its=30 rtol=1e-4 (2026-08-27)¶
Three-way on a fresh bgal registration (bench.simple, d2, box 256,
mskdiam 180, ~4.5k ptcls; base pair pins at 4.295/3.752 across ALL runs --
different registration than the recorded 4.132/3.709 ladder, internally
consistent): P_tau control, solvent Q_s at 0.1 (command-line slip, kept as a
free reference), Q_NU at 0.1. All three PASS all gates. Truth-free
observables (R6):
| observable | P_tau | Q_s 0.1 | Q_NU 0.1 |
|---|---|---|---|
| shipped 0.5/0.143 (A) | 4.185/3.667 | 4.132/3.667 | 4.132/3.627 |
| 0.143 inflation vs base | 2.3% | 2.3% | 3.3% (bound 5%) |
| gated-band ratio | 1.024 | 1.026 | 1.043 |
| agreement band | k=2-94 | k=2-94 | k=2-94 |
| radial norm min/max (in 0.85 mask) | 0.82/1.21 | 0.80/1.22 | 0.94/1.06 |
| centre-bin ratio | 0.76 | 0.76 | 0.85 |
| ML replay RESID / stop | 8.6e-2 xtol@12 | 1.4e-1 xtol@15 | 2.0e-2 maxits@30 |
Findings: (a) the recorded bgal ML-replay stall is gone under Q_NU --
the box-256 P_tau system stalls on step size at RESID ~0.09-0.14 while the
NU system reaches 2.0e-2, the second dataset where the shell-mean
preconditioner fusion conditions the NU replay BETTER than P_tau's; (b) the
Q_NU map has the flattest radial profile of the three and the best
centre-bin -- the rim erosion that dogged the solvent prior on bgal is
absent, consistent with the 1WCM rim-retention finding; (c) evidence sane
and mid-window (null 0.220, bands 0.763/0.664/0.576/0.169, uncertain ~0) --
the 0.90 ceiling was not approached despite the solvent-dominated support;
(d) shipped-pair inflation 3.3%, inside the bound. OPEN QUESTION (DEFERRED
2026-08-29, see the note at the end of this record) (the one
number that cannot be adjudicated without truth): the in-band amplitude
ratio pcg/gridding runs 1.5-2.0x through k=44-75 and higher toward the band
edge (in-band median 1.59 vs the control's 1.19) -- the expected signature
of no per-shell Wiener shrinkage on a low-SNR dataset (supported bands keep
base-level amplitude), and on 1WCM the same signature was truth-confirmed as
signal, but on bgal only an independent map-to-model comparison against the
deposited structure can decide it (R6). The postprocessing Wiener layer of
nu_evidence_local_sharpening.md is the designed consumer of exactly this
retained amplitude. Cost: 163 s/half at 30 its (production budget ~2 its
after warm start).
DEFERRED (2026-08-29): the map-to-model adjudication is parked by user decision. It is not trivial in practice -- the obtained maps are not docked to the deposited structure, so an honest comparison would require docking and model building, which is not being undertaken now. The question stays on record, not open-for-action. Confidence in reading the retained amplitude as signal rests, for now, on the 1WCM truth-judged evidence (where the identical signature was truth-confirmed, and where the 2026-08-29 Q_NU run was simultaneously amplitude-faithful and all-gates-passing); this is supportive but is not a substitute for the real-data adjudication. If the question is ever reopened, docking + model building against the deposited bgal structure is the prerequisite; the LocScale-style Wiener postprocessing layer remains parked behind it.
Production-budget A/B (its=5, rtol=1e-3), same fixture (2026-08-27) --
H4 PASSES. The Q_NU truth advantage survives the 5-iteration budget
essentially undiminished (truth FSC at 3.01 A: gridding 0.593, P_tau 0.580,
Q_NU 0.670 -- vs converged 0.593/0.582/0.675; at 2.91 A: 0.501/0.491/0.578;
at 2.44 A: 0.139/0.082/0.206). Both runs PASS all gates; base pair pinned
3.368/2.943 in all four runs; shipped-pair crossings identical to the
converged runs (Q_NU 3.241/2.844, inflation 3.4%). Notably the Q_NU replay
reaches RESID 8.4e-3 in 5 its (vs P_tau's 2.2e-2): the shell-mean
preconditioner fusion conditions the NU system well, and the converged-cost
gap was initialization distance, not ill-conditioning -- production budgets
do not suffer it. Cost at its=5: NU replay 56 s/half vs P_tau 13 s/half
(4.3x per replay; whole harness run 199 s vs 92 s including evidence build
and the 7 s stats overhead). Evidence stable across budgets (null 0.134 vs
0.126, bands 0.928/0.657/0.539/0.180). The bandwise-shrinkage init remains
worthwhile for converged/offline solves only.
Fallback if the quantile estimator proves fragile: calibrate the null on
noise-matched surrogates built from the half-map difference (even-odd)
(right noise spectrum, signal cancelled) -- costs a second bank pass and has
an anti-correlation subtlety (one independent difference realization only),
so it is recorded, not built.
6.3 Workflow Gate B:
- sequence base pair -> FSC/cFAR -> NU evidence -> NU replay from one raw accumulation in both shared and distributed paths;
- remove the binary-envelope solvent prior and its automatic-mask bootstrap from the NU target path;
- reuse the compact evidence state for cutoff/local-resolution diagnostics and LP-set handoff rather than recomputing it after replay;
- leave the ordinary global-ML path bit-identical.
6.3 refinement integration + solvent removal DONE (2026-08-27), runtime
gate pending. pcg_nu_lambda_rel is registered on refine3D and
abinitio3D (activation-gated on rec_backend=pcg) and forwarded through
apply_refine3D_reconstruction_controls, exactly the maxits_pcg/rtol
route the solvent key used -- an abinitio3D run opts in with
rec_backend=pcg pcg_nu_lambda_rel=0.1 and every stage's regularized
replays run the NU precision with the established cross-iteration warm
start. The solvent prior is fully removed: Q_s and its stats from
simple_reconstructor_pcg; the envelope loader, suppression reference,
stats reporter/persistence, and the envelope-flattening part of the
regularized init from the strategy (both paths); the suppression/inflation
readout, guidance thresholds, and stats reader from simple_convergence;
pcg_solvent_lambda_rel from parameters/parse and all four UIs;
PCG_SOLVENT_STATS_FILE from the fname defs; the strength-ladder sweep,
envelope requirement, envelope symlinking, and solvent summary fields from
test=rec3D_backends (now always a single comparison; NU runs get soft
gates). test=pcg_priors is repurposed as the Gate A Q_NU suite: adjoint,
PSD, EXACT constant null (the centering mutation target),
zero-action-under-full-support, monotonicity under evidence withdrawal, FD
gradient, the MEASURED 4-band-vs-2-band partition sensitivity, masked
composition, and priored solve parity across the shared-vs-nparts reduction
seam. The abinitio3D automsk-from-first-NU-stage staging is kept for the
reference-masking registration benefit; the shipped-pair crossing
measurement is kept as the NU over-regularization diagnostic (in-strategy
print only; the solvent stats file and convergence guidance are gone).
Build + test=pcg_priors + test=rec3D_backends reruns pending user-side;
first abinitio3D run with the NU prior is the outstanding refinement gate.
First NU-replay abinitio3D: bgal (2026-08-28) — refinement gate observations¶
rec_backend=pcg pcg_nu_lambda_rel=0.1, distributed, stages 4-6+ observed
(box crop 130 -> 140, lp 10.2 -> 9.3 -> 4.66 promoted by the NU filter):
- End-to-end NU refinement loop works. Every iteration: base solves (2 its, ~1 s/half) -> fresh evidence from the current base pair (distinct identity per iteration, readiness passing) -> warm start from the previous shipped half -> 2-it Q_NU replay (~4.5-7 s/half + ~1.1-1.8 s stats overhead). Production-viable cost: reconstruction remains a small fraction of the alignment wall time.
- Evidence trajectory is stable and physically sensible (the S6.3 field-overlap tracking item, answered by inspection): null pinned at 0.229-0.232 across every iteration and stage; coarse bands constant (band01 ~0.764); band04 support 0.000 through stage 5 (crop Nyquist above 5 A -- correctly no fine evidence), then 0.086 -> 0.146 -> 0.161 as stage 6 extends the band and resolution genuinely develops. No oscillation, no runaway. Warm-started ML RESID 0.056-0.075 at 2 its (base 0.13).
- Resolution progression: 5.44 -> 5.02 -> 4.66 A at 0.143 through stages 4-6, cFAR 0.64 -> 0.75.
- The solvent-era stage-6 overlap collapse is largely CURED: at NU/lp engagement (lp 9.3 -> 4.66 + switch to envelope-masked references) the orientation overlap dips 0.92 -> 0.80 -> 0.60 and recovers to 0.86 within two iterations -- versus the recorded 0.70 -> 0.04 collapse with the solvent prior. The NU replay map is evidently a much better matching reference at promotion.
- Kept automsk staging coexists correctly: stage 6 first iteration generates the envelope (occupancy 29.6%, one component), later iterations consume it for reference masking only; no solvent-prior lines anywhere.
- Remaining S6.3 sub-item now visible as measured cost: the NU analysis runs TWICE per stage-6 iteration -- once for the replay evidence, once for the volassemble NU filter/envelope postprocess (identical whitening and ordered-label logs back to back). Sharing the compact evidence state with the postprocess remains the outstanding dedup, now with a concrete per-iteration price attached.
DECISION (2026-08-28): Q_NU default-on in NU mode; post-hoc NU filter and pcg automasking retired from the NU-replay path¶
Following the first NU-replay abinitio3D (above) and the user-side observation that the Q_NU shipped map and the post-hoc NU-filtered map are nearly indistinguishable, three policy changes landed together:
- Default-on.
pcg_nu_lambda_reldynamically defaults to 0.1 wheneverrec_backend=pcg, the NU machinery is active (l_nonuniform), and the euclid ML replay runs -- i.e. the NU-filtered stages of abinitio3D/refine3D run the Q_NU replay with no flags. An explicit 0 restores the P_tau replay (the R10 A/B control); the plain-reconstruct3D/harness default (nofilt_mode) stays 0, preserving R5 for the recorded baselines. - No post-hoc NU filtering with Q_NU in the solve.
filter_pcg_nonuniform_mapsreturns early when the NU replay is active: no second NU analysis (S6.3 dedup closed), no_nu_filt/_nu_locresproducts, no evidence envelope (envelope clause amended 2026-08-29, see the automsk decision below). The LP-set matching handoff survives, now derived from the frozen replay evidence itself (finest evidenced local cutoff per state, threadedbuild_nu_replay_evidence -> execute_rec3D_pcg_distributed_master -> filter_pcg_nonuniform_maps, sameset_all2single('lp',...)contract). Matchers fall back to the regular (Q_NU-regularized) references, which is the point. The full postprocess remains for the P_tau/gridding paths. - No forced automasking in the pcg path of abinitio3D. The staging
existed to bootstrap the solvent envelope and switch to envelope-masked
references; with the prior gone and the Q_NU map already locally
regularized, automasking follows the same explicit user control as
everywhere else. (The Gate B registration-benefit observation attributed
to envelope-masked references is superseded by the user's read that the
NU-replay reference makes it unnecessary; if registration regresses on a
future run, this is the first knob to revisit -- and since 2026-08-29 that
knob works: explicit
automsk=yesregenerates the envelope from the frozen replay evidence without resurrecting the post-hoc filter, see the automsk decision below.)
Verification pending: an abinitio3D rerun with zero prior flags showing the dynamic default engaging at the first NU stage, single NU analysis per iteration, evidence-derived lp promotion, and no envelope/automask lines. CLOSED 2026-08-28/29 by the three-dataset run record below.
Stage 6 record: zero-prior-flag abinitio3D verification runs (2026-08-28/29, msp1 + embb + exp_gate)¶
Three abinitio3D runs with rec_backend=pcg and NO prior flags
(commit 6d06a1fa; logs under Processing/pcg_integration/), closing the
default-on decision's pending verification. Observed on the completed runs
(msp1, 5947 s; embb, 4228 s; exp_gate consistent while still in flight):
- dynamic default engaged at every NU stage (
Q_NU on, LAMBDA_REL 0.1), early non-NU stages correctlyoff; evidence built once per state per volassemble from the FSC half pair (source=base_unfil), no second NU analysis, no_nu_filt/_nu_locres/envelope/automask lines; - evidence-derived matching low-pass promoted every cycle
(
set_all2single('lp',...)values tracking the base FSC=0.5 region); - no over-regularization: shipped-pair FSC=0.143 crossings track the base crossings essentially exactly (msp1 3.91-3.96 vs 3.86-3.96 A; embb both 4.00 A), never pulling finer;
- every solve stopped on
fixed_iterations(2 its) with residuals of a few percent and near-identical even/odd values; halfset cohorts balanced; trailingF == Uwith the expected stage fraction ramps; ML warm starts firing throughout; - stable, slowly rising evidence support (band01 ~0.77 -> 0.78); final resolutions Nyquist-limited at the stage crops (msp1 3.86 A, embb 4.00 A); refinement cFAR ~0.80 (msp1) with a 0.51 final full-crop bootstrap value (real directional anisotropy, not a mechanism issue); B-factors -103/-58.
- convergence contrast worth keeping an eye on, not a gate item: embb converged classically (orientation overlap 0.35 -> 0.87, ~1.6 deg mean angular distance) while msp1/exp_gate plateaued at overlap ~0.10-0.22 with within-stage climbs and no collapse signature (the periodic dips align with fill-in sampling iterations).
6.4 Science/cost Gates C and D:
- run only the two-way
P_tauversusQ_NUablation of §8; - include uniform, heterogeneous, coarse-domain, weak-domain, and background- disorder fixtures plus real-data fixed-pose and refinement tests;
- judge at matched convergence, then measure the production-budget cost;
- do not begin Wilson work until Gate D is closed.
Stage 7 — Wilson molecular precision — priority 2¶
Sequential only after Stage 6 Gate D. Start with a zero-mean shell-diagonal Wilson spectrum behind one declared spectrum-source mechanism. Establish its benefit against the accepted NU estimator as a separate experiment before any combination is considered. Off-diagonal covariance, nonzero means, and local Wilson/LocScale variants remain outside the first Wilson stage.
DECISION (2026-08-29): Wilson development STARTS now, ahead of formal Gate D closure, by user direction -- the Stage 6 verification runs and the Gate C ablations already on record justify overlapping the engineering. The sequencing constraint that matters is preserved unchanged: Wilson is compared against the NU estimator as a separate experiment, is never combined with it (or with P_tau) in this stage, is off by default behind an explicit opt-in, and cannot be considered for any default until Stage 6 Gate D and its own gates close. See the Stage 7 implementation plan below.
Stage 7.1 — Q_W operator and workflow wiring (historical record: IMPLEMENTED, VALIDATED, ADJUDICATED, and REMOVED 2026-08-29 — Q_NU dominates, see the Gate C record and the removal decision)¶
Form. Q_W = F* diag(q_W(k)) F, zero-mean and shell-diagonal, exactly
the first form S5.5 prescribes (anchoring corrected twice by the first two
runs, see the S7.1 run records below):
q_W(sh) = lambda_rel * Dbar(sh) * min(WILSON_PREC_CAP, s(k_hp) / s(sh))
WILSON_PREC_CAP = 1e2
s is the Wilson spectrum SHAPE from the declared source, anchored at the
first prior-active shell k_hp; Dbar(sh) is the shell-mean raw data-only
density -- the same per-shell statistic P_tau divides by tau*SSNR. The
per-shell D anchoring makes the prior/data ratio exactly
lambda_rel * min(CAP, s_ref/s(sh)) at EVERY shell, bounded in
[lambda_rel, CAP*lambda_rel] and independent of the CTF/sampling decay
of D -- the bounded-dynamic-range discipline Q_NU established (bounded
operator, one strength knob), expressed in P_tau's per-shell data
convention. The spectrum supplies shape only. Shells coarser than the hp
limit get no prior, mirroring the P_tau low-frequency no-prior guard.
No parallel Fourier array. The builder
(build_wilson_prior_from_spectrum) fills the SAME calibrated ml_prior
diagonal P_tau uses, with the identical padded-radius -> native-shell
mapping. Application in the replay operator, kernel-diagonal fusion, the
preconditioner contribution, and the get_ml_prior_stats readout are
therefore shared verbatim -- Q_W is a different builder behind the one
diagonal, and three-way mode exclusion (P_tau | Q_NU | Q_W) is both
structural (one diagonal) and asserted in all three setters
(set_ml_prior/set_nu_prior/set_wilson_prior, R10 convention).
Declared spectrum source (7.1). An explicit reference volume,
pcg_wilson_vol: read via read_and_crop to the solve grid, rotationally
averaged power spectrum (image%spectrum('power')), loaded once per
execution and shared by both halves like the P_tau FSC prior. For the
phantom gates this is the ground-truth map -- the best-case Wilson prior,
so the first experiment answers "does a CORRECT molecular spectrum help"
before any estimation machinery is built. Estimated sources (Wilson-line
fit of the supported band with extrapolation) and analytic
composition-based spectra are later sub-stages (7.2+), each behind the same
single declared-source mechanism.
Activation contract. Explicit opt-in only: pcg_wilson_lambda_rel > 0
(no dynamic default) requires rec_backend=pcg, the euclid ML replay,
pcg_wilson_vol, and ordinary (non-NU) mode -- Q_W + NU filtering
hard-errors, which keeps the first-stage comparison clean (no post-hoc NU
filtering of Wilson maps, no dynamic Q_NU default interference) and makes
Q_NU/Q_W mutual exclusion follow from the mode split; both exclusions
are ALSO hard-asserted at parameter validation and in the reconstructor.
A positive strength on any other backend hard-errors (explicit-activation
contract, S6.2). Both halves cold-start from the base solution (the
shrinkage initial guess encodes the P_tau optimum; no Wilson closed form
yet); the own-half warm start applies as everywhere.
Wiring inventory. set_wilson_prior/build_wilson_prior_from_spectrum
+ state in simple_reconstructor_pcg; load_wilson_spectrum + replay
branches in both PCG execution paths of simple_rec3D_pcg_strategy;
pcg_wilson_lambda_rel/pcg_wilson_vol in parameters (declaration, parse,
validation phase); UI registration for reconstruct3D, refine3D, and the
rec3D_backends test harness; backend guard in the reconstruct3D
commander. The rec3D_backends harness handles the Wilson keys like the
NU key: validated up front (ml_reg required, Q_NU exclusion), soft gates
when the prior is active (it legitimately moves the pcg leg away from
gridding), _wil<strength> execution-directory tag, pcg_wilson_vol
made absolute before the chdir, and both keys deleted from the gridding
leg.
Stage 7.1 gates (as planned; overtaken by events -- Gate C was run first and concluded the stage, so the Gate A/B formalization below was never executed and is moot after the removal).
- Gate A algebra:
Q_Wdiagonal positivity and shell-constancy on the padded lattice; the three-way exclusion mutation tests (each setter pair rejected in both attachment orders); prior-stats readout sanity (prior_to_khatratios finite, hp shells zero). - Gate B invariants: artifact set identical to the
P_taureplay (Q_Wchanges only the diagonal); FSC/resolution reporting unchanged; the spectrum-source log line present exactly once per execution. - Gate C: truth-judged 1WCM two-way ablations
Q_WvsP_tauandQ_WvsQ_NUat matched convergence, with the truth spectrum as source (best case); then bgal with the deposited-model spectrum. Judge on the same criteria as the Stage 6 ablations; the S5.5 hypothesis to test is that the conservative NU estimator already captures most of the benefit and Wilson extrapolation adds little on data the experiment observed.
Stage 7.1 first run: 1WCM truth-judged (2026-08-29) — spectrum-peak anchoring failure¶
First rec3D_backends run with the truth map as spectrum source
(pcg_wilson_lambda_rel=0.1 pcg_wilson_vol=1WCM.mrc, euclid+mlreg, its=5,
box 256 smpd 1.0). The harness worked as intended: both legs ran, soft
gates reported, and the truth tables adjudicated. Result: FAILURE of the
initial Q_W anchoring, cleanly diagnosed.
- The base pcg solves were healthy (rel residual ~1.2e-2, base-pair
FSC=0.143 at 2.94 vs gridding 2.91 A, cFAR 0.89 vs 0.90) -- the failure
was confined to the
Q_Wreplay, as the mode split predicts. - The
Q_Wreplay solves left rel residual ~42-45 after 5 iterations (vs ~3e-2 forQ_NU/P_taureplays): the system was unsolvably stiff. The shipped map was a transient-dominated CG iterate: fsc(truth,pcg) NEGATIVE through the mid band (to -0.74), amplitude collapsed to a flat floor mid-band and rising junk toward Nyquist, shipped-pair FSC crossings far coarser than base. - Cause: the initial form max-normalized the spectrum shape and floored it at 1e-4. A real map's power spectrum is dominated by the lowest shells by orders of magnitude, so ESSENTIALLY EVERY prior-active shell sat at the floor and received precision ~1e3-1e4 x data_scale: the prior crushed the data term band-wide.
- Fix (implemented): anchor the shape at the first prior-active shell
(
k_hp) and cap the precision dynamic range atWILSON_PREC_CAP = 1e2--q_W in [lambda_w, 1e2*lambda_w]over the active band, preserving the Wilson decay shape where it is resolved and saturating beyond. This is the same bounded-operator discipline asQ_NU. Rerun below.
Stage 7.1 second run: 1WCM truth-judged (2026-08-29) — anchoring fix verified; global-data_scale over-shrinkage identified¶
Identical invocation after the hp-anchoring fix. Solvability restored and
the first genuine Q_W measurement obtained:
Q_Wreplay residuals 1.2e-1 (from 42-45), truth-FSC positive across the band, shipped map a real regularized estimate.- LOW-BAND WIN against the ML-regularized gridding leg: fsc(truth,pcg) >= fsc(truth,gridding) through k=2-10, decisively at k=3-5 (0.998 vs 0.956 at k=3) -- the molecular spectrum is informative exactly where Wilson statistics say it should be.
- Progressive high-band over-shrinkage: pcg truth-FSC falls behind gridding from mid-band (k=80: 0.59 vs 0.74), amp_pcg/truth decays to ~0.29 by k~75, and the shipped-pair crossing moved COARSER than base (3.08 vs 2.94 A) -- the over-regularization direction.
- Diagnosis: anchoring
lambda_wto the GLOBAL data_scale while the data termDbar(sh)decays with k (CTF + sampling) inflates the effective prior/data ratio k-dependently bydata_scale/Dbar(sh)on top of the spectrum ratio -- a mis-anchoring no single lambda can compensate. - Fix (implemented): per-shell D anchoring,
q_W(sh) = lambda_rel * Dbar(sh) * min(CAP, s_ref/s(sh)), mirroringP_tau's convention exactly; the prior/data ratio is now bounded in[lambda_rel, CAP*lambda_rel]at every shell. Rerun below.
Stage 7.1 third run: 1WCM truth-judged (2026-08-29) — per-shell anchoring VALIDATED; Q_W wins both band ends¶
Identical invocation after the per-shell D anchoring. The operator form is now measured-correct and the first Wilson science result is on record:
- Solve health at
Q_NUclass: replay residuals 3.0e-2 (from 1.2e-1); shipped-pair FSC crossings IDENTICAL to base (3.368/2.909 A) -- zero over-regularization signature at lambda=0.1. - LOW-BAND WIN retained: fsc(truth,pcg) >= gridding through k=2-9 (0.9993 vs 0.9908 at k=2; 0.9981 vs 0.9556 at k=3).
- HIGH-BAND WIN gained: from k~79 to the band edge pcg matches or beats the ML-regularized gridding reference against truth (k=80: 0.743 vs 0.740; k=90: 0.455 vs 0.440; k=100: 0.225 vs 0.209; k=110: 0.100 vs 0.058) -- the molecular prior paying off exactly where Wilson statistics extrapolate into weakly measured shells. Run 2 lost this entire region.
- Remaining: a small systematic mid-band truth-FSC deficit (k~30-75, 0.005-0.02, e.g. 0.964 vs 0.974 at k=50) with the amplitude ratio plateauing ~0.86 -- honest lambda=0.1 shrinkage, the lambda sweep's target. High-k amplitude rise beyond the resolved band (rim shells) persists but with better truth-FSC than gridding there.
Next: the lambda sweep (0.01/0.03/0.1) to see whether a weaker prior
closes the mid-band deficit while keeping both band-end wins, and the
Q_NU two-way on the same project (pcg_nu_lambda_rel=0.1 in place of
the Wilson keys) -- the first direct Q_W vs Q_NU adjudication. These
are the Gate C entry points. Both executed below.
Gate C: Q_W lambda sweep + first Q_W vs Q_NU two-way, 1WCM truth-judged (2026-08-29) — Q_NU DOMINATES; S5.5 hypothesis CONFIRMED¶
Same fixture and budget throughout (euclid+mlreg, its=5, truth spectrum as the Q_W source -- Wilson's best case).
Q_W lambda sweep (0.1 / 0.03 / 0.01):
- The mid-band truth-FSC deficit against the gridding reference shrinks monotonically with lambda (k=50: 0.964 / 0.967 / 0.972 vs gridding 0.974) while the high-band win is essentially lambda-INSENSITIVE (k=110: ~0.10 at all three strengths vs gridding 0.058), and the low-band win is common to all pcg runs. Within the family, lambda~0.01 dominates: near-zero mid-band cost, full high-band gain, replay residuals 1.0e-2. The high-band gain evidently comes from suppressing noise in weakly measured shells at all, not from the precise Wilson strength -- a first hint that the spectrum SHAPE is not the operative ingredient.
- Amplitude: mid-band amp/truth ~0.86 / ~2.3 / ~4.6 (vs gridding ~9) -- the single global knob trades amplitude fidelity against noise suppression band-wide, as a shell-diagonal must.
Q_W vs Q_NU two-way (pcg_nu_lambda_rel=0.1):
Q_NU beats the gridding reference AND every Q_W run at EVERY shell,
including the high band that was Wilson's presumptive niche (truth-FSC
k=50: 0.978; k=70: 0.925; k=80: 0.793; k=90: 0.523; k=100: 0.284; k=110:
0.137 -- vs gridding 0.974/0.905/0.740/0.440/0.209/0.058 and best-Q_W
0.972/0.901/0.738/0.445/0.219/0.099). It does so while remaining
amplitude-faithful: amp/truth ~9 band-wide (no shrinkage), the flattest
radial LS profile of any run including gridding (0.89-1.02 vs gridding's
1.14-1.22 over-amplification), and the harness passed ALL gates --
including the gridding-parity amplitude gates every other prior run
soft-failed. Shipped-pair inflation modest (3.24/2.84 vs base 3.37/2.94),
suppression ~36%, replay residual 8.4e-3. Cost is the one Q_W
advantage: ~60 s/half (15 s evidence overhead) vs ~13 s.
Verdict. The S5.5 hypothesis is CONFIRMED on this fixture: the
conservative NU estimator captures more than the full Wilson benefit --
even with Q_W handed the ground-truth spectrum -- because the evidence
field regularizes locally and anisotropically exactly the degrees of
freedom the half-map evidence does not support, which no shell-diagonal
can express. Q_W is NOT a candidate to displace Q_NU and no further
solo Q_W development is warranted on this evidence (removal decision
below). Remaining bookkeeping if ever needed: a bare pcg run (no prior
keys) on this fixture for the clean Q_W vs P_tau two-way; not
required for the verdict above.
DECISION (2026-08-29): Q_W REMOVED from the code base¶
Following the Gate C verdict, the Wilson estimator was removed the same
day it was implemented, by user direction and in keeping with this
document's discipline: retired estimators live on as experiment records,
not as code. Removed: set_wilson_prior/build_wilson_prior_from_spectrum
and all Wilson state/constants from simple_reconstructor_pcg; the
spectrum loader and replay branches from both PCG execution paths;
pcg_wilson_lambda_rel/pcg_wilson_vol from parameters, parsing,
validation, and the reconstruct3D/refine3D/rec3D_backends UI; the
backend guard from the reconstruct3D commander; and the Wilson key
handling from the rec3D_backends harness. The three-way mode assertion
reverts to the two-way P_tau/Q_NU exclusion. The S7.1 records above
(operator form, both anchoring corrections, the sweep, and the two-way)
are the complete archaeological record; if a Stage 8 combination
experiment is ever approved, the validated per-shell-D-anchored form
documented here is the reference implementation to resurrect. Wilson
remains formally priority 2 in name only; in practice Stage 7 is CLOSED.
Stage 8 — explicitly out of first scope¶
Singer/Gilles off-diagonal covariance; nonzero or phase-bearing prior means; lagged LocScale surrogates; nonlinear/proximal priors; and any combined NU/Wilson product-of-experts model. Each requires its own approved experiment after the preceding estimator is independently understood.
Related but separate (POSTPROCESSING, not a solver prior, so the in-solve
LocScale risks above do not apply): model-free LocScale-style local
sharpening driven by the same frozen NU evidence state is specified in
nu_evidence_local_sharpening.md — parked until this document's Gate C/D
program completes.
DECISION (2026-08-28): NU-replay firing readout in the convergence report¶
With Q_NU default-on in NU mode, refinement needs the same in-run answer the
retired solvent prior had: "is the prior firing, and is
pcg_nu_lambda_rel right?" — with no harness and no ground truth. The readout
is the NU prior-energy suppression percentage, per half:
supp% = 100 * (1 - sqrt(E_NU(x_ML) / E_NU(x_base)))
where E_NU(x) = (lambda_nu/2) x^T Q_NU x (get_nu_prior_stats; the
lambda_nu factor cancels in the ratio) and x_base is the unregularized
base solution of the SAME half — the replay's own reference: with P_tau
absent in NU mode and the replay cold-started from the base solution, a
vanishing pcg_nu_lambda_rel reproduces x_base, so an inert prior reads
~0% with no shrinkage referencing needed (unlike the solvent readout, whose
reference had to factor out the Wiener component of the P_tau replay). The
amplitude-domain square root mirrors the solvent readout's rms ratio. Cost:
one extra full Q_NU application per half (~13 padded FFTs), timed into the
existing stats_overhead_s diagnostic.
Plumbing mirrors the retired solvent readout: report_nu_solve_stats prints
pcg_nu_prior_energy_final/base and pcg_nu_suppression_pct per half; both
PCG execution paths average even/odd per state and persist
simple_pcg_nu_stats.txt (delete-then-rewrite each volassemble, so a
skipped replay leaves no stale values), including the shipped-pair FSC=0.143
crossing per state (the over-regularization diagnostic, never a resolution
claim); check_conv3D prints % PRIOR ENERGY SUPPRESSED (PCG NU REPLAY) and
SHIPPED-PAIR FSC=0.143 with the other iteration stats, writes them to
simple_stats.txt (PCG_NU_SUPPRESSION_PCT, PCG_NU_SHIP0143,
PCG_NU_LAMBDA_REL + per-state keys), and advises:
- < 5%: prior inert — increase
pcg_nu_lambda_rel(~3x) - 5-60%: nominal — keep the strength
- > 60%: over-regularization risk — decrease (~3x)
Thresholds are provisional constants in simple_convergence
(PCG_NU_SUPP_INERT/OVER_PCT), inherited from the solvent readout's bounds;
recalibrate with a pcg_nu_lambda_rel strength ladder now that every run
prints the readout, reading the suppression % against the shipped-pair
crossing and the evidence trajectory.
DECISION (2026-08-28): NU replay supports trailing reconstruction (evidence pair = FSC pair)¶
The hard block on Q_NU + trail_rec is lifted; real-data refinement needs
both. The governing contract is the evidence pair is always the FSC pair of
the volassemble:
- Trailing, chain present: the base solves are the full-mass blended
chain solutions — the very statistics the ML replay re-reads
(
add_raw_accum_weightedof the same chain) — so current-iteration evidence from that pair (source=base_unfil) satisfies the "evidence from the pair the replay reuses" requirement. The original block's concern (the base pair is not the plain current-cohort pair) is resolved by matching the evidence to the replayed statistics, not to the cohort. - Trailing bootstrap (no chain yet): the FSC already comes lag-one from
the previous iteration's shipped half pair; the evidence comes from that
same pair (
source=previous_shipped, a new allowed evidence source next tobase_unfil). Lag-one evidence is precedented by the retired solvent envelope's one-iteration lag and is exactly as trustworthy as the bootstrap FSC thatP_tauwould be built from. The current-cohort fractional pair is NOT used: at small update fractions it can legitimately fail null calibration and would hard-stop real runs.
The accumulator arithmetic (chain blend weights, scale_raw_accum,
fixed-order reduction) is the test=pcg_frac_update-gated path
(validate_rec3D_pcg_fractional_updates) and is untouched: Q_NU attaches
after accumulation and before end_accum, exactly where set_ml_prior
does, so the prior is orthogonal to the tested trailing contracts. The
firing readout's reference remains the base solution of the same half (in
trailing, the blended base solve the replay warm-starts from), and the
LP-set handoff derives from whichever evidence pair was used. Shared-memory
trailing remains unsupported for PCG generally (pre-existing
accumulator-domain restriction, unrelated to the prior).
DECISION (2026-08-29): automsk=yes regenerates the evidence envelope under the Q_NU replay¶
Amends item 2 of the default-on decision: with automsk enabled, the NU
evidence envelope IS produced on the replay path -- regenerated inside
build_nu_replay_evidence, between optimize_nu_cutoff_finds and
cleanup_nu_filter, while the raw per-voxel evidence margins are live. The
matching-reference envelope therefore derives from the same frozen evidence
as the Q_NU precision: one analysis, two consumers, and still no second NU
pass. Cadence and artifact naming follow the same plan_state_postprocess
contract as the post-hoc paths (missing/incompatible mask, startit,
AMSK_FREQ). All three regeneration sites (gridding volassemble, PCG
post-hoc, PCG replay) now share the single producer
write_nu_evidence_envmask in simple_nu_filter; the replay envelope is
built from the static candidate bank (the replay analysis runs no
high-resolution shell extension), a deliberate, slightly shallower evidence
basis than a gridding-path envelope. This also arms the default-on
decision's item-3 fallback: if reference registration regresses without
envelope masking, explicit automsk=yes is now a functional knob on the
replay path.
DECISION (2026-08-29): refine3D_auto joins the pcg bypass; NU shell extension declared obsolete under Q_NU¶
refine3D_auto was the last workflow defaulting into the competition NU
machinery on the pcg backend: its unconditional nu_refine=yes default
hard-errored at the first volassemble, and its init-volume bootstrap ran the
competition prefilter (including the shell challenger) on the startup
halves. Both are fixed, mirroring the abinitio3D stage policy: nu_refine
defaults to no when rec_backend=pcg (explicit nu_refine=yes + pcg
remains a hard error), and the startup NU prefilter is bypassed under pcg --
raw-pair validation and its reconstruct-startup fallback stay
backend-neutral, but no _nu_filt startup references are produced and the
first matcher pass uses the raw native E/O references, exactly as every
later Q_NU iteration feeds the matcher. Doctrine going forward:
rec_backend=pcg with the Q_NU prior makes NU high-resolution shell
extension obsolete; nu_refine survives only on the gridding/competition
path pending its retirement (next section).
Gate C/D comparability note (2026-08-28 mask unification)¶
The gridding backend's spherical FSC mask was unified to params%msk_crop
(previously the broad rim radius), so gridding and PCG FSC/cFAR are now
directly comparable -- but gridding FSC-derived numbers recorded BEFORE the
unification (including baselines quoted in the Stage 5/6 records above) are
not comparable to post-unification gridding runs. Remaining Gate C/D
ablations must use same-policy baselines: rerun the gridding arm at the
current mask policy, or compare against the PCG backend directly.
CLOSED (2026-08-29): solvent-era suppression-threshold OPEN DECISION superseded¶
The open decision under the Stage 5 streptavidin record (lower
PCG_SOLVENT_SUPP_OVER_PCT from 60 to ~45-50; move the solvent default
0.1 -> 3e-2) is superseded: the solvent prior is scientifically retired and
Q_NU is the default regularized estimator, so the solvent suppression
guidance thresholds will leave the tree with the solvent code. No further
calibration of the retired prior is planned.
Stage 6.5 — competition-path retirement experiment (planned)¶
The experiment that would conclusively retire the gridding + post-hoc NU
competition path (including nu_refine shell extension) in favor of
rec_backend=pcg with the Q_NU prior. Code preconditions are complete as
of 2026-08-29: every workflow (abinitio3D, refine3D, refine3D_auto) runs the
pcg arm with zero prior flags, no competition machinery executing anywhere
(startup, per-iteration, or postprocess), envelope masking available under
automsk=yes, and cross-backend FSC/cFAR comparability restored by the mask
unification. Experiment design and acceptance criteria: TO BE DEFINED by the
user before execution -- candidate axes are the Gate C fixture set plus
real-data refine3D_auto pairs (gridding+NU competition vs pcg+Q_NU) judged
at matched convergence on resolution, cFAR, map quality, over-regularization
diagnostics, and wall-clock/cost. On acceptance: remove the retired solvent
prior code, the post-hoc NU competition path for pcg, and revisit
nu_refine's existence; record removals here.
Stage 6.6 — always-on adaptive band granularity (IMPLEMENTED 2026-08-29; build/runtime gates user-side pending)¶
The replay-setting successor to nu_refine-style resolution extension,
designed and implemented 2026-08-29. The problem: the
solver's finest band spans 8 A to Nyquist as ONE weight block (the
evidence probes stop at ~5-4 A), so high-resolution refinement pays a
granularity tax -- one confidence governs the whole fine tail, and the
matching-lp handoff is quantized to the candidate ladder near the
frontier. The competition path solved this with the shell challenger
(nu_refine=yes), which had to be OPT-IN because filter-side acceptance
was irreversible within the iteration: a wrongly accepted shell passed
its noise directly into the references.
Design: make the band structure a deterministic function of the current
evidence state, ALWAYS ON -- no flag, no successor to nu_refine=yes|no.
Keep the static 20/12/8/5 A ladder and extend it geometrically (the
existing ~0.6-0.67 per-step spacing) until the next boundary would pass
max(current base FSC=0.143 crossing x margin, crop Nyquist), with
matching evidence candidates per new boundary. Band count then grows with
resolution the way the abinitio crops do; the structure is derived from
the pair, shared by both halves, and frozen before either replay, per the
existing evidence contract.
Why always-on is safe where the shell challenger was not (risk inversion): subdivision has NO acceptance step. A new finest band earning low confidence adds penalty to the fine tail (the conservative direction -- today's one-block band 4 under-penalizes fine noise inside a partially-evidenced region); earning high confidence, the penalty retreats exactly where evidence supports it. Nothing is passed, nothing is cut, and the structure is refrozen from fresh evidence every iteration, so a poorly placed boundary self-corrects.
Implementation surface is evidence-side only: the solver
(set_nu_prior, band index, operator, preconditioner approximation) is
already nbands-generic. Changes: derive the ladder/limits per iteration
(replacing the NU_EVIDENCE_BAND_LIMITS constant as the sole source),
pack them in the compact state, generalize
expand_nu_evidence_band_weights past the fixed four, and thread the
state-carried limits to set_nu_prior (the provenance string already
records bands_A/candidates_A, so the evidence identity hash tracks
the structure for free). Side benefit: the matching-lp handoff steps
refine with the ladder (the 4.01/4.12/4.42 A staircase in the msp1 log
smooths out).
Gates: the Gate A partition test at each band-count transition (the bank is not a tight frame; refining the partition changes the operator -- measured, not assumed); the suppression readout watched across transitions (a mid-refinement subdivision redistributes where suppression concentrates); one lambda-ladder recheck at high band count. Cost note: two padded FFT pairs per extra band per application (~75% more Q_NU application cost at ~7 bands), fixed per solve and independent of particle count; a few extra candidates in the once-per-iteration evidence analysis.
IMPLEMENTATION RECORD (2026-08-29). As designed, evidence-side only:
- Constants
NU_EVIDENCE_MAX_NBANDS=8,NU_EVIDENCE_BAND_RATIO=0.64,NU_EVIDENCE_FRONTIER_MARGIN=0.8and the module-state active laddernu_evidence_band_limits_activeinsimple_nu_filter;nu_evidence_summary%band_limits/supported_fractionare now allocatable (band count is part of the frozen state). derive_adaptive_evidence_bands(bank submodule): always sets the active ladder -- static without a frontier (exact pre-6.6 behavior, which is what every non-replay caller andsimple_test_nu_envmaskget), extended geometrically toward max(0.8 x frontier, 2 x smpd) otherwise, two candidate probes per appended band (geometric midpoint + boundary).setup_nu_dmatsgains optionalevidence_frontier_lp(requiresevidence_source);init_nu_filtergains optionalextra_candidate_lps, appended strictly monotone past the static ladder, grid-snapped duplicates dropped, Nyquist- and cap-clamped.build_nu_evidence_statesizes band support, summary, provenance (bands_Anow dynamic), and the identity checksum from the active ladder;nu_evidence_state_is_validchecks band count in [4, NU_EVIDENCE_MAX_NBANDS], strictly decreasing positive limits, and that the first four entries ARE the static ladder;expand_nu_evidence_band_weightsand the solver (set_nu_prior/band index/operator/preconditioner) were already nbands-generic and are untouched.build_nu_replay_evidencetakes the evidence pair's FSC=0.143 crossing as the frontier and returns the state-carriedband_limitsalongsideband_w; both PCG execution paths threadres0143s(state)in and pass the returned ladder toset_nu_prior, so operator and evidence can never disagree (NU_EVIDENCE_BAND_LIMITSno longer reaches the strategy). The log line>>> NU ADAPTIVE EVIDENCE BANDS: <n> bands, finest boundary ...fires whenever the ladder extends.- No new parameter anywhere;
nu_refinehas no successor knob.
Runtime observables for the user-side gates: the adaptive-bands log line
appearing once resolution passes ~6 A (finest static boundary 5 A,
margin 0.8: first extension at frontier ~4 A / floor permitting);
pcg_nu_supported_fraction_band05+ lines and a longer bands_A list in
the evidence block; smoother matching-lp handoff steps; suppression
percentage continuity across a band-count transition.
Stage 6.6 first run: 1WCM truth-judged (2026-08-29) — unguarded subdivision DEGRADES; evidence-gated retention added¶
First adaptive-bands run, same fixture and budget as the Stage 6 Gate C
runs. The mechanism fired exactly per the rule: frontier 2.943 A ->
5 bands, finest boundary 3.2 A (next step 2.05 A below the 2.35 A floor,
correctly stopped); 10 candidates (the appended midpoint probe snapped
to the existing 4 A find and was correctly deduplicated, only 3.2 A
appended); dynamic bands_A/candidates_A in the provenance; all
harness gates PASSED. But the truth verdict is a clean DEGRADATION
against the 4-band control across the entire fine band, dropping below
even the gridding reference from ~3.7 A (truth-FSC at k=79/89/99/109:
0.739/0.440/0.219/0.098 vs the 4-band 0.813/0.550/0.308/0.148 and
gridding 0.766/0.472/0.227/0.117).
Diagnosis, visible in one line: band05 support = 0.000000. In the
solver partition the new band 5 spans EVERYTHING finer than 5 A, so the
zero-support subdivision replaced the fine tail's partially evidenced
weight ((1-0.19)^2 ~ 0.65) with the full penalty 1.0 -- over-suppressing
genuine 3-5 A signal band-wide. The design argument that low confidence
in a new band is "the conservative direction" was right about direction
and wrong about magnitude. The zero reading also exposes a detector
limitation: band confidence is softmax WINNER MASS on candidates at
least that fine, and at fine candidate spacing (3.2 vs 4.0 A) the
whitened objective barely separates neighbors, so the mass dilutes
toward zero at the frontier even where signal demonstrably extends (the
pair's crossing is 2.94 A). Suppression readout moved 35.9 -> 31.3%
across the partition change -- the predicted discontinuity, now
measured.
Fix (implemented): EVIDENCE-GATED RETENTION, the always-on form of
"subdivide when evidence supports it". After band support is computed,
appended bands are pruned finest-first unless their mean support reaches
NU_EVIDENCE_MIN_BAND_SUPPORT = 0.01; the static four are never pruned,
so pre-6.6 behavior is the guaranteed floor and a zero-support
subdivision self-neutralizes to the 4-band operator (logged as
'pruned to N band(s)'). Still no flag, no acceptance step on KEPT bands,
per-iteration refreeze unchanged.
VERIFIED (2026-08-29, same-day rerun): the guard behaves exactly as
designed. Extension proposed (5 bands, 3.2 A), then
'pruned to 4 band(s); appended band(s) earned no support'; bands_A
back to the static four while candidates_A retains the 3.2 A probe
(which keeps informing selected cutoffs and the LP handoff without
penalizing); suppression restored to 36.0/36.9% (4-band control:
35.9/36.8 -- continuity recovered); shipped-pair crossings identical to
the 4-band control (3.241/2.844); truth-FSC matches the 4-band control
to the fourth decimal across the band; all gates PASS. The retained
extra candidate perturbs the softmax slightly (band01 support 0.9227 vs
0.9279, temperature 4.18e-3 vs 6.04e-3) with no measurable truth effect
-- the acknowledged and acceptable footprint of a pruned proposal.
OPEN (Stage 6.6b, if adaptive granularity is to ever engage usefully): the fine-band confidence criterion needs a spacing-aware form -- winner mass systematically underestimates support as candidate spacing shrinks; a margin-based criterion (as the envelope uses) or cumulative-mass calibration against the null are the candidates. Until then, adaptive banding is expected to read as a no-op on most data (extension proposed, then pruned), which is safe and honest.
Stage 6.6 FINAL FORM (2026-08-29, user-directed): nu_refine mirror of the gridding challenger¶
The 6.6b spacing-aware criterion (a pairwise-margin sigmoid) was briefly implemented and WITHDRAWN unmeasured on user direction: too complicated, and unnecessary -- the codebase already contains the proven answer. The final design mirrors the gridding path exactly:
- The a-priori frontier-tracked candidate proposal (the
evidence_frontier_lpplumbing) is REMOVED. Candidate-bank extension in the evidence analysis is instead the existing nu_refine high-resolution shell walk (extend_nu_filter_highres_shells): one Fourier shell at a time from the populated frontier, accepted only on strict unary WIN-FRACTION at the tested frontier (>= 5% plus an absolute seed floor) -- the spacing-robust criterion that has gated gridding resolution extension all along, and precisely why winner-mass confidence is sound here: a band is only ever created over candidates that WIN somewhere, and dilution only zeroes dominated candidates. - Gated by the EXISTING
nu_refineflag, with its established meaning and workflow split: abinitio3D keeps the discrete static ladder (its stage policy pins nu_refine=no; the validated default-on Q_NU configuration is untouched); refine3D_auto restores its unconditional nu_refine=yes default on BOTH backends -- the gridding filter challenger there, the Q_NU evidence-bank extension here. No new parameter; the earlier backend-conditional nu_refine default is reverted. - The walk runs inside
build_nu_replay_evidencebetweenoptimize_nu_cutoff_findsandbuild_nu_evidence_state(the exact sequence of refine3D_auto's gridding bootstrap), logged as 'EVIDENCE BANK EXTENDED BY n ACCEPTED SHELL STEP(S) TO x A'. - The band ladder is derived from the ACTUAL bank at evidence-build time: the static four bands, extended geometrically (x0.64, cap 8) only while an accepted candidate is at least as fine as the next boundary. Static bank => exactly the four-band behavior; no band can exist without a challenger-validated probe. The evidence-gated retention guard stays as belt-and-braces behind the challenger gate.
- Guards updated:
nu_refine=yeson the pcg backend now requires the Q_NU replay (clear hard error otherwise, replacing the blanket 'does not yet support' errors); the post-hoc P_tau+NU pcg path keeps no extension support.
VALIDATED on the 1WCM harness (2026-08-29, both runs, all gates PASS):
- nu_refine=no control: EXACT parity with the verified four-band Q_NU Gate C result -- support fractions, suppression (35.9/36.8%), shipped crossings (3.241/2.844), and truth-FSC identical to all printed digits. With no walk there is no candidate footprint at all, so abinitio3D's discrete-ladder mode is the validated configuration verbatim.
- nu_refine=yes: the walk accepted 30 shell steps to 2.72 A (bank at the 24-candidate cap), band 5 (3.2 A boundary) earned REAL support (0.064, vs 0.000 under the a-priori proposal) because it sits over challenger-validated candidates that win at frontier voxels; ladder correctly stopped at 5 bands (next boundary 2.05 A below the finest accepted candidate). Truth verdict: IMPROVEMENT over the already dominant Q_NU control at every shell finer than ~3.3 A (2.91 A: 0.587 vs 0.578; 2.61 A: 0.344 vs 0.335; 2.49 A: 0.237 vs 0.227; gains persist to the band edge) against a ~0.003 truth-FSC dip in the 3.8-4.4 A region -- the intended granularity trade, favorable for a refinement tool. Suppression continuous (34.6/35.6%); shipped-pair inflation 4.6% vs the control's 3.4% (diagnostic only, inside guidance).
- One defect found and fixed by the first nu_refine=yes attempt: the provenance and identity-seed buffers (LONGSTRLEN=1024) overflowed with the 24-candidate provenance ('End of record' at the identity internal write); both are now XLONGSTRLEN.
Run record (2026-08-29): PfCRT abinitio3D -- Q_NU INERT at lambda_rel=0.1 on real data¶
First real-data abinitio3D with the Q_NU replay on PfCRT (EMPIAR-10330
final refined particle stack, mskdiam 160 A;
Processing/pcg_integration/PfCRT/ABINITIO3D_OUTPUT_RESTART1, code at
the pre-6.6 state -- equivalent here since abinitio3D pins
nu_refine=no). Outcome: map quality NOT competitive with gridding + NU
filter. Diagnosis from the output doc:
- Stages 1-5 (LP 20 -> 11): Q_NU off by design (filter mode none), mirroring the gridding path's NU-off early stages. Correct behavior.
- Stages 6-8 (LP 9.7 -> 6.0): Q_NU on at the dynamic default
lambda_rel=0.1, evidence rebuilt per iteration with sensible band
trajectories (band 1-3 support growing 0.76->0.83 / 0.69->0.78 /
0.15->0.67; band 4 low, 0.001->0.05), BUT prior-energy suppression
never exceeded 4.8% (0.3% at NU onset, 1.6% at the final rec) vs ~35%
on the 1WCM harness and the bgal run at the same lambda_rel. The
PCG NU PRIOR INERT (< 5%)diagnostic fired at essentially every NU-era iteration, recommending ~3x lambda. - Failure mechanism: on the pcg path Q_NU is the ONLY spatial regularization (post-hoc NU filtering skipped by design), so an inert prior means refinement aligns against effectively unshaped ML maps while the gridding path aligns against NU-filtered references. The data-scale anchoring of lambda_eff evidently does not normalize regularization strength across datasets.
- Secondary observations: null calibration much noisier than 1WCM (null_bias_median 0.46 vs 0.06), only 9 candidates survived, and the matching low-pass sat pinned at the static bank's 3.98 A floor through stage 8.
Next step (user-directed): rerun PfCRT at higher lambda_rel (0.3, then 1.0 if suppression still lands under ~15-20%; suppression is sublinear in lambda so the 3x hint may be conservative) to establish empirically what a suitable suppression target is. A suppression-targeted closed-loop auto-lambda (the replay already measures suppression every reconstruction) is the candidate long-term fix, to be designed only after the scan fixes the target band.
Q_NU operator performance pass (2026-08-30, operator-preserving)¶
apply_nu_precision cost 3*nb+1 padded FFTs per matvec (16 at nb=5)
against the kernelized data operator's 2 -- the prior was ~8x the data
term per PCG iteration. Two operator-preserving fixes landed (identical
operator up to floating-point reordering):
- Fourier-side adjoint accumulation: the bands are disjoint radial partitions, so the per-band masked spectra are assigned into one accumulator and synthesized with a SINGLE final inverse transform (IFFT is linear). Saves nb-1 inverse FFTs.
- Shared forward FFT (kernel path): the kernel matvec stashes its
pristine FFT(pad(p)) via
stash_nu_forwardbefore the Khat multiply (deapodization on, so the spectrum is of the iterate itself); the prior completes the mean-centering in Fourier space with the precomputed box-window spectrumnu_winhat = FFT(pad(1)), since FFT(pad(p - m)) = FFT(pad(p)) - m*FFT(pad(1)). Saves the prior's own forward FFT. The matrix-free path cannot share (its spectrum is of the deapodized iterate) and keeps its own forward.
Net: 3nb+1 -> 2nb+1 padded transforms on the kernel path (16 -> 11 at
nb=5), 2nb+2 on matrix-free. Allocator churn also removed: persistent
workspaces (nu_cmat0/nu_cacc/nu_cband/nu_xb, ensure_nu_workspaces)
replace the per-matvec allocations and all hot-path pad_vol/crop_vol
temporaries (direct get_rmat_ptr interior writes on a zeroed wimg).
Back-pocket option NOT taken (operator-changing, needs its own 1WCM
validation): running the radial band frame at the native box instead of
the padded lattice (~8x cheaper FFTs at padf=2). Memory flag stands:
nu_band_w is box^3 x nb (~2 GB at box 400, nb 8).
Verification requirement (R5-adjacent): rerun the 1WCM Q_NU control -- suppression, support fractions, and truth-FSC must reproduce the verified values to printed digits (FP reordering may wiggle the last digit); any larger deviation means the rewrite changed the operator.
PARITY VERIFIED (2026-08-30, 1WCM control, all gates PASS): suppression 35.89/36.82 and shipped crossings 3.241/2.844 identical to the verified control to all printed digits; all 129 truth-FSC shells bit-identical as printed; support fractions differ only in the sixth decimal (0.927938 vs 0.927939 etc.) -- exactly the FP-reordering signature expected from Fourier-side accumulation. Wall-clock not comparable on this run (host under load ~11 from concurrent jobs); judge the speedup from NU-era iteration timings in the next quiet PfCRT/refine3D run.
Active dev list (2026-08-29, user-prioritized)¶
Follow-on note: some PfCRT maps look acceptable, but with the prior inert that is the unregularized ML solve doing the work, not Q_NU -- the acceptable-looking outputs do not validate the prior.
- PRESSING -- NU-evidence locscale-style nonuniform postprocessing
(
nu_evidence_local_sharpening.md). A single isotropic B-factor does not work for most maps other than bgal and streptavidin; graded local amplitude scaling from the frozen NU evidence is the model-free fix. The proposal's Gate C/D precondition is now met (Q_NU cleared its gate program and the Stage 6.6 walk validated), so implementation is unblocked and elevated to the top of the queue. IMPLEMENTED (2026-08-29) as the isolatedpostprocess_nucommander +simple_nu_filter_sharpensubmodule (user-directed isolation from the standard postprocess path). v1 (confidence-gain band stack) over-sharpened the PfCRT core and is RETIRED as a recorded failure; v2 is the classical shrink-then-sharpen localized by the evidence (Guinier B inside the evidenced local Butterworth passband, null-flattened solvent). Design and records innu_evidence_local_sharpening.md§3b/§3c; validation per its §4 plan pending. - Auto-lambda design after regularization parameter testing. The PfCRT lambda_rel scan (0.3, then 1.0; in flight) establishes the target suppression band empirically; only then design the suppression-targeted closed-loop lambda_eff controller (the replay already measures suppression every reconstruction). R9: record the target band here before implementing the controller.
Scan record and root-cause analysis (2026-08-30). PfCRT suppression:
lambda_rel=0.1 -> 3.25%, lambda_rel=0.3 -> 8.6% (shipped-pair 3.93 A,
diagnostic banner NOMINAL). The x2.65 response to x3 lambda is almost
perfectly LINEAR -- the weak-prior perturbative regime, so 1WCM-like
shaping (~35%) needs an order of magnitude, not small multiples;
next points lambda_rel=1.0 and 3.0 to bracket, and where the
response bends sublinear is the saturation information the
controller needs. Root cause of the cross-dataset failure:
update_lambda_from_density anchors data_scale to the LOW-BAND
mean of the raw data diagonal D, so lambda_eff = lambda_rel *
mean(D_lowband) -- but the prior competes against D in the FINE
shells where the unsupported-band content lives. The effective
strength is lambda_rel * D_low/D_fine, and that spectral falloff
ratio is a dataset property (CTF envelopes, particle count, ice,
crop grid): gentle on the 1WCM phantom (0.1 -> 35%), ~10x steeper
on PfCRT (0.1 -> 3%). Candidate fixes, to be decided AFTER the
scan: (a) anchor data_scale to the prior-active shells (mean D over
shells finer than the coarsest band boundary, or D weighted by the
prior's spectral footprint), making lambda_rel dataset-invariant by
construction; (b) close the loop on measured suppression.
Discriminating test once a ~30-40% PfCRT point exists: does
fine-shell anchoring alone collapse the PfCRT and 1WCM operating
points onto one lambda_rel?
TARGET RECORDED (2026-08-30, user, R9): aim for ~60% prior energy suppressed. Empirical basis: at that operating point the NU effects are clearly visible on real data (msp1). This sits at the top of the diagnostic banner's NOMINAL band (5-60%), so the banner thresholds should be re-centered around the 60% target when the auto-lambda controller is designed; the controller's setpoint is 60%, not the 1WCM harness's ~35%.
AUTO-LAMBDA IMPLEMENTED (2026-08-30). Design recorded before
first run (R9):
- Plant model: one-pole amplitude-suppression law
s = glambda/(1+glambda), validated on the scan (PfCRT g=0.336
from the 0.1 point predicts s(0.3)=9.2% vs 8.6% measured; 1WCM
g=5.4 -- a 16x cross-dataset gain spread, which is why no fixed
lambda_rel can work).
- Controller: memoryless one-step secant per refinement iteration,
resolve_nu_autolambda in simple_rec3D_pcg_strategy (both exec
paths, before attachment): reads the previous iteration's
PCG_NU_STATS_FILE (lambda used + state-mean suppression, the
file the replay already persists), identifies g, solves for the
60% target. Deadband hold at 60 +/- 5%; multiplicative step clamp
x5; lambda_rel bounds [0.01, 30]; suppression floored at 0.1% and
capped at 99% for finite identification. Both halves and all
states share one lambda, same freezing discipline as the evidence.
- Activation: ONLY when pcg_nu_lambda_rel was left to its dynamic
default (l_pcg_nu_autolambda, set in the parameters dynamic
default block). An explicit strength pins lambda -- every recorded
control, harness run, and ladder point stays reproducible, and
explicit 0 = P_tau control is untouched (R5/R10 preserved). Cold
start (no stats file) keeps the 0.1 default, so single-shot
reconstructions in fresh directories are deterministic.
- Diagnostics: stats file gains a PCG_NU_AUTOLAMBDA key; the
convergence banner is re-centered on the target (INERT <5%, BELOW
TARGET <45%, ON TARGET 45-75% aim 60%, OVER >75%) and, in auto
mode, reports the lambda in use instead of advising manual
changes; the controller logs MEASURED/HOLDING/adapted-lambda lines
per reconstruction.
- Deliberately NOT done: fine-shell re-anchoring of data_scale
(operator-meaning change, would force re-baselining per R2; the
closed loop absorbs the anchoring error in one update). Recorded
as the cold-start improvement to revisit only if convergence to
target proves too slow on some dataset class.
- Validation plan: (i) 1WCM harness with explicit lambda -- must be
bit-identical to the verified control (controller inert when
pinned); (ii) PfCRT/msp1 refine3D or abinitio3D WITHOUT a lambda
flag -- expect convergence to 55-65% suppression within ~3
iterations from cold start, then station-keeping; watch the
shipped-pair inflation diagnostic while the controller holds.
FIXED 60% SETPOINT FALSIFIED -- PfCRT REGRESSION (2026-08-31). First production abinitio3D runs of the auto-lambda controller (c2b419ad, no lambda flag, nu_refine=yes): PfCRT regressed from 3.93/3.98 A shipped pairs (b082ef4a, pinned lambda_rel=0.3, supp 8.6%) to 5.86 A with the transmembrane helices resolved as sausages -- the controller drove lambda toward the 60% setpoint (g=0.336 implies lambda ~4.5, ~15x the proven strength). streptavidin, msp1 and FlhB held or improved at the same commits. Conclusion: the plant GAIN spread (16x) that motivated auto-lambda extends to the TARGET -- the good operating point is a dataset property (PfCRT ~9%, 1WCM ~35%, msp1 ~60%), so no fixed suppression setpoint transfers.
AUTO-TARGET IMPLEMENTED (2026-08-31, user-directed). The
setpoint itself is now controlled; design recorded before first run
(R9):
- Structure: cascade. Inner loop unchanged (one-step secant
auto-lambda tracking the setpoint). New outer loop
(resolve_nu_supp_target logic inside resolve_nu_autolambda)
owns the setpoint, driven by the shipped-pair FSC=0.143
trajectory -- the persisted over-regularization diagnostic, never
a resolution claim.
- Law: AIMD (additive-increase / multiplicative-decrease). Per
reconstruction, comparing the previous iteration's state-mean
shipped-pair crossing to the one before it (2% relative
deadband): improved -> setpoint +5 percentage points; degraded ->
setpoint x0.6; stalled -> hold. lp-limited stage plateaus stall
and therefore hold; crop/box changes that perturb the crossing at
stage transitions can only trigger a conservative back-off.
- Bounds [5, 75]% (banner inert floor to banner over ceiling); cold
start 15% -- gentle by construction (above PfCRT's proven 8.6%,
ramping toward msp1-class operating points only while the shipped
pair keeps improving: 60% is reachable in 9 improving
iterations).
- State: memoryless via PCG_NU_STATS_FILE, which now also carries
PCG_NU_SUPP_TARGET, PCG_NU_AUTOTARGET, PCG_NU_SHIP0143_AVG and
the one-step history PCG_NU_SHIP0143_PREV (lifted before each
rewrite).
- Activation: ONLY when pcg_nu_supp_target is left to its dynamic
default (l_pcg_nu_autotarget); an explicit value in [5,75] pins
the setpoint (reproducible controls, R2), and requires the
auto-lambda controller (an explicit lambda pins the strength
outright, hard error otherwise). Registered on refine3D and
abinitio3D (forwarded to the refine3D stages; stripped with the
other PCG-only keys off non-PCG child command lines).
- Diagnostics: the convergence banner reports AUTO-TARGET/PINNED
TARGET with the live setpoint; the manual-lambda advisory bands
are re-centered on the reported setpoint (default 60 when
absent); the controller logs IMPROVED/DEGRADED/STALLED setpoint
lines per reconstruction.
- Validation plan: (i) pinned lambda -- both controllers inert,
bit-identical to the verified control; (ii) PfCRT abinitio3D with
no flags -- expect the setpoint to hold near the cold start while
shipped pairs march, back off on any inflation, and the run to
recover the ~3.9 A pinned-lambda result; (iii) msp1 -- expect the
setpoint to ratchet toward its known-good ~60% while shipped
pairs improve, with no regression vs the fixed-60% runs.
AUTO-TARGET COLD START FALSIFIED -- msp1 abinitio3D REGRESSION
(2026-09-06). Ten repeated msp1 abinitio3D runs at dc2eb9a7
(rec_backend=pcg, no prior flags): _lp stage snapshots normal
through stage 6, overfitted from stage 7, final postprocessed maps
wrecked. Mechanism, established from the controller law and the
recorded dataset operating points (no new instrumentation needed):
- msp1 sits at ~60% suppression AT THE DEFAULT lambda_rel=0.1 (the
2026-08-28/29 zero-flag verification runs at pinned 0.1 were
healthy, 3.86 A Nyquist-limited; the fixed-60% controller held
there too). The AIMD outer loop cold-starts the setpoint at 15%,
so the very first inner-loop step sees supp ~60% vs target 15%
and steps lambda DOWN at the x5 clamp: 0.1 -> 0.02 -> 0.01
(floor) within three iterations of stage 6. Under R10 Q_NU is the
only replay precision, so lambda at the floor is an effectively
UNREGULARIZED ML solve (PRIOR INERT banner): the merged matching
reference carries un-shrunk in-band noise.
- The outer loop cannot recover: it ratchets +5 points only while
the shipped-pair FSC=0.143 crossing improves by >= 2 shells, and
lp-limited ab initio stages stall (hold) almost immediately, so
the setpoint stays near 15% for the rest of the run. Worse, the
signal is the FSC of NON-independent halves (merged-reference
matching): overfitting inflates it (read as "improved"), and
regularization that removes correlated noise deflates it (read as
"degraded" -> x0.6 back-off). The loop therefore settles on the
weakest prior consistent with the highest correlated-noise FSC.
- Compounding factor: with nu_refine=no the matching-band handoff is
the RAW finest selected label (min_pct=0, "numerically unchanged
vs gridding"), i.e. the static bank's 3.98 A floor whenever a
single voxel selects it. On the gridding path that was harmless
(the references are NU-filtered); against soft Q_NU references it
matches noise. b46f5c257 caps the handoff at the stage ladder
(lpstop ceiling), which restores the pre-NU matching band.
- Stage 6 survives because its first iterations still run at the
default strength and the schedule lp; stage 7 (nspace 5000,
greedy frac_best 0.85, finer crop, 12 iterations) starts with the
prior already at the floor and the project lp already promoted.
The final bootstrap_rec3D resumes the floored lambda (its
calibration keeps the ~15% target), then auto-B sharpens an
unregularized native-grid map.
Log evidence (ABINITIO3D_OUTPUT_RESTART1, one of the ten runs),
read 2026-09-06, confirming the mechanism with two additions:
- stage 6 (LP 7.5): iteration 1 at the default gives supp 42.4%;
the inner loop steps 0.1 -> 0.024 at once; the outer loop then
reads a one-shell shipped-pair change (4.41 -> 4.54 A) as
DEGRADED and backs the setpoint off 15 -> 9; lambda holds at
0.024 while the readout decays 11.5 -> 3.8% over ten iterations
AT FIXED LAMBDA (the readout is not a steady-state response, see
below); one late step to 0.060.
- stage 7 (LP 6.7): the crop change re-scales the plant (22.6% at
0.060), the outer loop backs off again to 5.4 and the inner loop
steps to 0.012. From then on the readout is NEGATIVE (-3 .. -2%,
floored to 0.1%) and, because the absolute +/-5 deadband contains
the 5% setpoint floor, every iteration logs ON TARGET and holds
0.012 for the whole stage: a deadband deadlock at an inert prior.
Shipped-pair crossing walks 4.5 -> 6.3 A across the stage.
- stage 8 (LP 4.5): two IMPROVED steps raise the setpoint to 15 and
the inner loop ramps lambda 0.012 -> 0.06 -> 0.30 -> 1.4 -> 2.3
-> 3.7 -> 7.5 -> 12.1 -> 20.0 while the readout never exceeds
14%: the plant gain has collapsed, the prior is stiff where it
acts and absent elsewhere.
- final bootstrap_rec3D: calibration at 0.1 reads 39% on the native
grid, adapts to 0.027 for the pinned 15% target, ships at 17.9%.
- matching handoff: constant per stage at the finest bank label
(4.415 / 4.120 / 4.013 A), i.e. the crop's static-bank floor every
iteration of every NU stage, unbounded at this commit.
- Readout validity: the ML replay warm-starts from the PREVIOUS
iteration's shipped ML half and runs maxits_pcg=2 iterations, so
the measured suppression is a lagged property of the iterate
(previous orientations, previous lambda), not the response to the
current lambda. The one-pole identification is therefore built
on a quantity that decays at fixed lambda (stage 6) and cannot
follow a 200x lambda ramp (stage 8). Negative readouts are the
signature: the warm-started replay carries more evidenced energy
than the fresh base.
USER VERDICT (2026-09-06): the fine NU handoff must not set the
abinitio3D matching band at all; the conservative lpstages ladder
value is the matching low-pass in the NU stages. b46f5c257 enforces
it (the ladder rides as the lpstop ceiling on every staged child
and the project handoff is clipped to it at every promotion).
LOG EVIDENCE AGAINST THE CEILING ON THE PCG PATH (2026-09-06, full log sets: msp1 10 runs at dc2eb9a7 vs 10 runs at b46f5c25 (ceiling); streptavidin 10 runs at 9201e797 vs 2 complete runs at a053ee07 (ceiling + legacy sigma)). Streptavidin is the clean comparison (identical workflow, full sampling, crop box 88 throughout, sigma init path identical at every stage): - through stage 5 the ceiling runs sit inside the healthy envelope (stage-5 FSC=0.5/0.143 4.29/3.71 in run 2 = the healthy median); - every healthy run improves at stage 6 from 4.29/3.7-3.8 to 3.6-3.9/3.2-3.6 A while matching at the bank's finest label 4.04 A (the raw handoff, every iteration of stages 6-8); - both ceiling runs match at the ladder 6.24 A in stages 6 and 7 and do NOT improve at all (run 2: 4.29/3.52 -> 4.29/3.61; run 1: 4.57/3.92 -> 4.57/4.04), then stage 8 (ladder 4.5 A) recovers part of the gap (3.92/3.27 vs healthy 3.7-3.8/3.1-3.2) after its full 25 iterations -- the observed "rescued in stage 8" and the ~500 s longer wall time. msp1 shows the same shape with a confound: the healthy set ran the cavg_ini route (nested cavgs stages, per-particle sigma init every stage), the ceiling set a checkpoint start at stage 4 with the group-only sigma init ("reusing existing grouped sigmas") at every stage. Even so: healthy stage 6 matches at 4.98 A and FSC=0.5 goes 6.7 -> 4.9 A within the stage; ceiling stage 6 matches at 7.54 A and FSC=0.5 stays at 7.0-7.2 A; ceiling stage 7 (6.72 A, nspace 5000, frac_best 0.85) randomizes the search in all 10 runs (projection distance 34 deg, in-plane 84 deg, shift 4.2 px, SCORE sdev 0.002 vs 0.007-0.013 healthy) and ends at FSC 6.575/6.575 A, one shell inside the band; stage 8 recovers to 4.35-4.48 A with the auto-lambda driven to 15-29. All ten healthy runs end at 3.86-3.96 A. Conclusion from the logs: on the pcg + Q_NU path the raw finest-label handoff was the mechanism that carried the NU stages past the ladder; the ladder ceiling removes it and the NU stages stall at the band. The earlier "LP overfitted from stage 7" judgement on the dc2eb9a7 msp1 set is not supported by its FSC trajectories (ten consistent runs, 3.9 A finals, FSC=0.5 4.0 A); the finest-label matching ran between FSC=0.5 and FSC=0.143 of the current map, not beyond it. Recommendation (not applied, user decision): revert the lpstop ceiling for the pcg path, or replace it with a data-driven cap (e.g. the base-pair FSC=0.5 crossing plus a shell or two) instead of the class-FRC ladder, which is not informative about the particle map once the NU stages begin.
SOLVE-SUPPORT POLICY CORRECTION (2026-09-06, user-directed). Dev item 5 (ca450fe54, 2026-09-01) and the 2026-09-02 review item made the conservative density envelope the replay solve support unconditionally in ML mode, independent of automsk; every msp1 and streptavidin run since 2026-09-01 (including the ten healthy streptavidin runs at 9201e797) solved the shipped pair on that envelope while the base pair stayed spherical, so their two FSCs were never comparable. The healthy msp1 set (dc2eb9a7, 2026-08-31) predates it. Rule now: no envelope masking anywhere unless automsk=yes; only then may envfsc=yes extend the envelope to the base solve. Implemented as pcg_density_support_enabled(params) gating build_pcg_state_support and both current-base-pair fallbacks (shared + distributed). The envfsc FSC evaluation itself (phase-randomized masked FSC) is unchanged and still envfsc-only.
SOLVE-SUPPORT POLICY, ENVFSC COUPLING (2026-09-09, user-directed).
automsk=yes now implies envfsc=yes on BOTH backends; envfsc is derived
in validate_parameter_consistency (logged when it overrides an
explicit or defaulted envfsc=no) and mirrored by
set_refine3D_envfsc_policy in the abinitio3D stage config so the
refine3D command lines read truthfully. Per backend:
- PCG: under automsk=yes the density envelope constrains BOTH the
base/unfil solve and the ML-regularized replay once a prior
reconstruction exists (the first base pair bootstraps on the sphere
and seeds the replay support). The strategy no longer consults
l_envfsc for the base support (l_base_support_constrained =
l_state_support; build_pcg_state_support throws if the coupling
is bypassed). The FSC pair is reported support-constrained, so
evaluate_halfmap_pair skips the phase-randomized re-masking; the
automask artifact is still written for its other consumers. The
envfsc=no + automsk=yes split of the 2026-09-01 three-rule policy
(spherical base, envelope replay) is retired.
- Gridding: the same envelope generator (automask3D of the base-pair
average at envmsklp, same binwidth) is applied post hoc to the FSC
pair with the phase-randomized correction, and the corrected FSC
feeds add_invtausq2rho exactly as the constrained-pair FSC feeds the
PCG ML prior. This is the closest post-hoc counterpart of the
constrained estimate a gridding assembly can offer: the mask
cannot enter the estimator, so the correction is the honest
substitute, and it is what makes the two backends' FSC-0.143/0.5
numbers and their regularization strength comparable under
automsk=yes.
Residual, deliberate asymmetries between the backends: (i) the PCG
solve support is built from the lag-one reference while the post-hoc
gridding envelope (and the automask artifact on both) is built from
the current base-pair average -- one iteration of envelope lag,
generous mask, negligible; (ii) the shipped PCG halves carry the
envelope, the shipped gridding halves carry only the soft spherical
support at msk_crop (the support-provenance sidecar records which),
so downstream masking (postprocess envfsc, NU evidence background) is
post hoc on gridding and already-in-the-map on PCG. automsk=no is
unchanged on both: sphere throughout, envfsc as requested. The final
classical reconstruct3D that refine3D/refine3D_auto/refine3D_states
issue no longer forces automsk=no (2026-09-09, after a refine3D_auto
run showed the shipped PCG map solved on the sphere while every
iteration had used the envelope): automsk rides along, filt_mode=none
and nu_refine=no still make the shipped map classical.
NU EVIDENCE ENVELOPE UNDER THE DOUBLE SUPPORT (2026-09-09, user-directed). The evidence envelope stays the mask controlling the NU filtering of the volumes: it is the only envelope that excludes detergent (micelle is best explained by the coarsest candidate, so its margin is null), which the density envelope cannot do. What changes is how its null is obtained, in two regimes keyed on how the base pair was solved: - SPHERICAL base pair (gridding; PCG bootstrap without a lag-one reference): the robust median + nu_msk_sigMAD of the margin over the observed support, unchanged in substance -- the generous sphere makes solvent the majority population -- with the pre-existing >50%-signal warning promoted to the validity verdict (l_null_majority). - ENVELOPE-CONSTRAINED base pair (PCG under automsk=yes): the estimator has removed the far solvent, so its margin is an exact zero spike (with the envelope near half the sphere the median lands on it and the MAD collapses, inflating the envelope), and the remaining support is not a solvent-majority mixture either (with a tight envelope the median lands in the signal and the envelope collapses to the best ordered core while the >50% check stays quiet). The null is therefore DESIGNATED by Euclidean geometry rather than estimated from a mixture: automask3D now exposes its binary core and dilated intermediates, and set_nu_evidence_null_shell takes the median/MAD on the dilation ring (dilated minus core), restricted to voxels the base support carries at full weight (its cosine skirt attenuates the noise). Labels are free on the observed density envelope and fixed solvent outside it (NU_ENVMASK_EXCLUDED_SCORE), nesting the evidence envelope inside the density envelope. Validity is shell sufficiency (NU_ENVMASK_MIN_NULL_VOX, NU_ENVMASK_MIN_NULL_FRAC of the domain); the core-to-shell median margin is logged as the separation diagnostic. The shell width IS binwidth, which used to be 1 layer in refine3D/refine3D_auto (parameter default) and 7 layers in abinitio3D (ENVMSKWIDTH_DEFAULT), i.e. a one-voxel shell and a far tighter PCG solve support in refine3D. RESOLVED (2026-09-09, user-directed): the density envelope dilation has a shared physical minimum, ENVMSKWIDTH_A_MIN = 7.5 A (the former abinitio3D default of 7 layers at 1.075 A/pixel), applied in validate_parameter_consistency whenever the envelope is in use (l_envfsc) as binwidth = max(binwidth, ceiling(7.5/smpd_crop)) at the sampling the envelope is built at (a lower bound, review 2026-09-09), so the same physical envelope comes out at every crop level and in every program; abinitio3D no longer injects a layer count. The minimum replaces the default only: an explicit binwidth wins in either direction. Whether 7.5 A is more than a dataset needs is measured, not argued: the skirt attenuates rather than includes (inside the PCG estimator a weight-0.4 skirt voxel is density at 40%), so the full-weight ring is the only clipping margin, and the NU evidence reports how much of that ring it labels signal (>>> NU DILATION RING OCCUPANCY): signal in the ring means the dilation is capturing density, a null ring means it is pure margin and binwidth can be tightened for that specimen. In both regimes the density-term median/MAD follow the null set, and if the null is invalid or the envelope is empty nonuniform_filter_state arms the density envelope itself as the background (source='density_envelope', logged as EVIDENCE FALLBACK). The PCG callers hand the state support and l_base_support_constrained (in the distributed bootstrap blend: constrained only if both contributions were) to nonuniform_filter_state; gridding passes nothing and stays in the spherical regime. The standalone nu_filt3D route sets no shell. Also removed: the vestigial regeneration planner (plan_state_postprocess / AMSK_FREQ / NU_ENVMASK_ACTION_) -- the envelope has always been regenerated every cycle from the live evidence and the artifact has no in-workflow reader. Side-by-side lines to compare: NU ENVELOPE OCCUPANCY, "Null model", "Null shell voxels", "Core median margin" against "Null median margin", and whether the background was armed from the evidence envelope or the fallback.
CODE REVIEW RESPONSE (2026-09-09, automsk_yes_code_review.md). The masked-FSC bias of a support-constrained PCG pair (P1) is acknowledged and deliberately NOT corrected: a common window on both halves can contribute correlated power, but the constraint is the point of the estimator, and the policy is to REPORT what was done rather than to hide it -- evaluate_halfmap_pair now names the mode on every evaluation (>>> FSC MODE in the log and the resolution text: spherical support / envelope post hoc with phase-randomized correction / estimator-constrained without correction). pcg_mskfile (P1) is now returned by build_pcg_state_support as the state support regardless of automsk, so FSC mode, provenance sidecar and the NU null regime see a constrained pair. set_nu_evidence_null_shell (P1) reads the four volumes through get_rmat_ptr and allocates only the two packed masks, and logs the requested geometry diagnostics per state (envelope/support Dice, fraction of the current dilation ring retained at full weight of the base support, shell voxels of ring). core/dilated (P2) are requested from automask3D only on the constrained branch. The dilation minimum (P2) uses ceiling. The public policies (P2: automasking, abinitio3D, refine3D, refine3D_auto, reconstruct3D_pcg) are updated in the same change, including the stale refine3D_policy claim that references are multiplied by the NU envelope. Not done: reusing the gridding FSC envelope in the NU consumer (ownership boundary; one automask3D per state per iteration is the accepted cost). The review's validation matrix (nine cases, peak RSS on a production box) is the user's to run.
PfCRT REGRESSION gridding_vs_pcg (2026-09-10, ~/for_claude/PfCRT_regression): abinitio3D, 10 gridding vs 9 PCG restarts, same data. Gridding 8/10 good (3.98-4.3 A, side chains), 2 fail at 8.2 A. PCG 2/8 "almost side chains" at 4.9-5.0 A, 6 fail at 7.2-8.4 A, 57% slower. Log analysis: every refinement solve was ITS=2 fixed (maxits_pcg=2, rtol=0). The base solve exited at relative residual 0.15-0.2; the ML replay NEVER got below 1 in any PCG run at any refinement iteration, and in the three hardest failures (R4, R5, R8) started at 20-100 at the first ML stage and grew monotonically to 10^3-10^4 -- growth across iterations is only possible through the warm start from the previous iteration's ML half, an unconverged transient of a different system (new P_tau, new poses) that two iterations cannot pull back; R8's base warm start joined in (resid 1.0-1.5) and the reconstruction collapsed to 103 A. The shipped ML halves (NU auxiliary member, matching references) were CG transients. The systematic 5-vs-4 A gap of the surviving runs has the same origin one level down: at the first NU iteration (stage 6) the gridding base pair supports 6.4% of the mask at 7.96 A and 1.25% at 5.97 A, so the NU handoff promotes the matching low-pass to 5.97 A at once and the map goes 8.9 -> 4.5 A within the stage; the PCG base pair, two warm iterations from a stale start, puts 0% at 7.96 A, the matching low-pass stays at 10.02 A and the take-off never happens. DECISION (user): retire the cross-iteration warm starts. The base solve starts from zero and the ML replay from the shell-shrunk CURRENT base solution (the P_tau optimum in closed form), every iteration; maxits_pcg stays 2 (the simulated-data calibration: two iterations beat gridding, beyond five the residual moves but nothing interpretable in the map does). override_base/ml_warm_start_from_previous removed; the solve summary line reports INIT= (start residual) next to RESID= so a start worse than nothing can never again hide in the sidecar files. The solve-kind field of the support-provenance sidecar is now provenance only. RESULT (2026-09-10, ~/for_claude/PfCRT_regression/cold-start, first run at c9808eff): 3.98 A, side chains, identical to the good gridding runs. The stage-6 take-off is back: matching low-pass promoted to 5.97 A at the first NU iteration (it 86), 4.44 A by it 90, FSC-0.143 7.96 -> 4.44 A within the stage (gridding: 8.9 -> 4.5). The cold base solve reaches RESID 0.06-0.09 in two iterations, BETTER than the warm-started 0.15-0.2 -- the warm start was harming the base too. The ML replay from the shell-shrunk base: INIT 3-8 -> RESID 1-5 in stages 3-5 (the shrinkage start is worse than zero for the ML system while the FSC is low across most shells; harmless here, the base pair drives FSC and NU), INIT ~0.2 -> RESID ~0.04 from stage 6 on, a brief INIT 1.4-2 at the stage-8 crop transition. Cost unchanged by the change: PCG master phase 15.9 s/it vs gridding 4.2, refine3D parts 12.3 vs 7.6 s/it (the workers write the raw accumulators), 7463 s vs 4817 s total.
msp1 STAGE-7 COLLAPSE ROOT CAUSE (2026-09-06, from the full log sets):
the external-init repeats (5_abinitio3D) ran next to the completed
healthy set (4_abinitio3D) of the same project. The legacy sigma
loader's sibling-directory carry-over (../*/
DECISION (2026-09-06, user): Q_NU REMOVED FROM THE PCG BACKEND IN ITS ENTIRETY, together with the auto-lambda and auto-target controllers, the stats file, the bootstrap calibration pass, the parameters (pcg_nu_lambda_rel, pcg_nu_supp_target), the UI entries, the convergence readout, the evidence replay in the strategy, the operator (set_nu_prior/apply_nu_precision and workspaces) and the test=pcg_priors gate. Rationale: an elegant estimator that ends in a parameter optimization the competition never needed; the belt on PfCRT and the msp1 trajectories are the record. The PCG path now follows the gridding path exactly: the unregularized (_unfil) pair seeds the discrete filter bank, the P_tau-regularized pair joins the competition through the auxiliary route (ml_reg=yes, nu_refine=no), nu_refine=yes runs the same shell walk (refine3D_auto), and the finest selected label is the matching handoff. Implemented by extracting the gridding volassemble NU section into simple_nu_state_filter (src/main/volume) and calling it from both the gridding assembly and the PCG master (shared + distributed). What stays from the PCG work: the projected solve support under automsk=yes, trailing accumulator chains, warm starts and crop-change embedding. The compact evidence-state module remains for the envelope, postprocess_nu and the envmask test.
PfCRT REGRESSION 2026-09-07 (abinitio3D, 7/7 runs at 6.0-6.5 A vs the July reference 8/10 at 4.1-4.3 A; both full sampling, identical commands apart from rec_backend). Map-level audit: every stage snapshot of every restart docked (both hands) against the July restart-1 snapshots and scored by cross-FSC. Through stage 5 the September maps are as close to the July reference as July's own restarts (median cross-FSC 20-10 A: 0.41 vs 0.41 at stage 5), so the global-search stages are NOT the problem despite their low orientation overlap (0.12; the likelihood sampler draws near-uniformly over the top-K, dist 13-22 deg on PfCRT vs 3-5 deg on msp1/streptavidin, but the maps are equivalent). The two sets part in stage 6, the first NU stage (stage-6 cross-FSC 10-7 A: July 0.50, September 0.27) and never re-converge. Two causes, both in the NU stages: (1) the matching cap implemented on 2026-09-06 was the ladder's FINAL value lpfinal, 6.0 A for PfCRT (2D classes stop at 6 A), not the 4.5 A hard bound the user asked for; the raw NU handoffs of the September runs (4.44/4.14/3.98 A) equal the bands the July run actually matched at, and the cap pinned them at 5.97 A. (2) Early stopping: the annealing sampler reaches overlap 0.9/0.95 within 3-9 iterations of every NU stage; July successes ran 12/12/25 iterations, every early-stopped run (July 4/5/3 iterations -> 5.9/4.9/8.0 A; September 3 iterations in stage 8 everywhere) failed with the FSC still improving. Fixes: cap = LPSTOP_BOUNDS(1) (coarser explicit lpstop retained); minits = maxits in NU stages. Not implicated by the data: rec_backend (stage 2-5 maps match July's), the 20 A stage-1 floor and ladder shift, the FSC=0.5 promotion, inpl_cont, the sigma bootstrap. Remaining second-order candidate for PfCRT-class specimens: the sampler temperature (audit pfcrt_audit_a6ae317ae_vs_head.md item 1).
The controller observations above stand as a record; the guard rails
below were drafted and then WITHDRAWN (not applied), so the validated
adaptive configuration (embb, exp_gate, PfCRT) is unchanged.
Withdrawn guard rails (not applied):
- Deadband half-width capped at half the setpoint
(min(5, 0.5*target)), so a readout below half the target always
steps the strength up; the 5%-floor deadlock cannot recur.
- NU_AUTOLAMBDA_LAMBDA_MIN 0.01 -> 0.1 (= the dynamic default).
The controller may only STRENGTHEN the prior relative to the
validated default; no recorded dataset (PfCRT, 1WCM, bgal,
streptavidin, msp1, embb) has a good operating point below it.
- Setpoint seeding: at the first controller step the AIMD setpoint
is seeded from the suppression the strength in use actually
delivers, max(15%, measured) (stats-file key
PCG_NU_AUTOTARGET_SEEDED, cleared with the stats file). PfCRT-class
datasets (3% at the default) still ramp from 15%; msp1-class ones
hold at their default operating point and ratchet from there.
NU_AUTOTARGET_COLD_START replaces the literal 15 in the
controller; the parameters default block keeps 15 as provisional.
- Open: whether staged abinitio3D should run the controllers at all. The 2026-08-28/29 healthy msp1
runs were pinned at 0.1; the readout is warm-start-lagged at
maxits_pcg=2 and the reward signal is a merged-reference FSC, so
the recommended interim policy is to pin lambda_rel at the default
for the staged workflow (or freeze both loops after seeding) and
let the controllers act only in gold-standard refine3D_auto.
- Open (not changed here): the AIMD reward signal. A shipped-pair
FSC from merged-reference (non-gold-standard) matching is not a
valid over-regularization diagnostic in abinitio3D; consider
freezing the setpoint at the seeded value in staged abinitio3D
(adapt only in gold-standard refine3D_auto), or gating the ratchet
on the base-pair FSC at the stage ladder limit. Also open: the
nu_refine=no raw-finest handoff should use the same 5%-supported
percentile as nu_refine=yes on the pcg path, now that the
reference is not NU-filtered.
FINAL-RECONSTRUCTION Q_NU POLICY (2026-08-31, user-directed).
The original-sampling final reconstructions of abinitio3D and
refine3D_auto previously dropped the PCG backend and its prior
entirely (abinitio3D: prep_final_rec_cline rebuilt the child line
without rec_backend, shipping a gridding P_tau final map;
refine3D_auto: a plain unregularized objfun=cc rec3D). The
objfun=cc passes exist for sigma availability -- registration-box
sigmas are crop-incompatible at the final box, which is exactly
what bootstrap_rec3D solves (cc pass -> half-map sigma
derivation -> euclid ML pass). Policy now: the final map is
reconstructed on the refinement's backend, and on pcg the Q_NU
replay regularizes it in-solve. Implementation:
- bootstrap_rec3D strips the Q_NU keys off its unregularized
sigma pass (no ml_reg, activation contract) and lets them and
filt_mode flow into the regularized pass.
- abinitio3D prep_final_rec_cline forwards
rec_backend/maxits_pcg/rtol and, when the final stage uses
ml_reg, sets filt_mode=nonuniform and forwards the pinning keys.
- refine3D_auto's final rec now runs bootstrap_rec3D (previously
unregularized cc), with bootstrap sigmas written at endit+2 so no
crop-box sigma star is overwritten, and the same pcg/Q_NU
forwarding.
- Grid-transition calibration (corrected 2026-09-02): lambda_rel
does not transfer unchanged across the staged-crop -> native-grid
transition. The change in downscaling changes the effective Q_NU
plant gain even though the suppression target remains meaningful.
With auto-lambda active, bootstrap_rec3D therefore retains the
learned suppression target, lets the unregularized sigma pass clear
the old-grid response, runs one current-grid Q_NU calibration solve
without postprocessing, and then reruns the regularized reconstruction
after resolve_nu_autolambda has adapted from that measurement. Only
the calibrated replay is shipped. A missing target starts at 15%.
Explicit lambda controls skip calibration and remain pinned as usual.
- Final cold-solve convergence and sharpening (corrected 2026-09-03):
both calc_final_rec (abinitio3D) and the refine3D_auto final bootstrap
use the shared five-iteration PCG budget floor, while larger explicit
budgets survive and an explicit residual tolerance may still stop early.
The final global B-factor is still estimated from the prior-free
_even_unfil/_odd_unfil average, but its temporary Guinier-fit copy is
windowed by the conservative density envelope. This removes the broad
spherical bootstrap background from the slope without applying a mask to
any PCG output. Iteration postprocessing and non-PCG paths are unchanged.
PfCRT REGRESSION ROOT CAUSE -- HANDOFF GATE + FROZEN-EVIDENCE
DEADLOCK (2026-08-31). The post-auto-target PfCRT abinitio3D run
(9b2424b5) still stalled at 5.86 A with the matching low-pass pinned
at 5.97 A, "RIDING FROZEN EVIDENCE (no resolution advance)" and
"SHIPPED PAIR STALLED" every iteration -- the controllers were
holding on a stall they could not break, so the prior strength was
never the driver. Git bisection of the record: both offending
changes landed together in a6a5cc1e/9f3c3c9e ("PCG NU prior
optimizations for speed"), the first commit after the good pinned
b082ef4a runs (3.93/3.98 A):
- The LP-set matching handoff switched from the raw finest selected
per-voxel cutoff (minval(cutoffs)) to the 5% assignment-support
percentile (nu_evidence_finest_supported_lp at
NU_ALIGN_LP_MIN_ASSIGNED_PCT). On a small membrane protein only
a small core carries fine evidence while the micelle belt
dominates the assigned support, so the gate collapsed the matching
bandwidth to ~the FSC crossing. streptavidin/msp1/FlhB resolve
compactly and uniformly, so the percentile ~= the minimum and they
were unaffected.
- The frozen evidence cache rebuilds on FSC=0.143 ADVANCE -- but the
alignment search is capped at the evidence-derived low-pass, so no
advance can occur: a self-sealing loop (the age-5 forced rebuild
regenerates from maps aligned under the cap and the gate
re-collapses the handoff). The plateau then reads as convergence.
Fixes (both, user-directed):
- Handoff decoupled from the support gate: raw finest selected
cutoff (min_pct=0), restoring the proven b082ef4a behavior.
Sparse-but-real fine evidence must be allowed to pull the search
band forward; over-extension widens the search, over-restriction
deadlocks it. The gridding-path gate
(get_nu_filtmap_finest_selected_lp) is untouched.
- Binding-band rebuild condition in nu_evidence_needs_rebuild: the
cache entry records the Fourier index of the matching low-pass it
handed off (handoff_find); once the FSC crossing comes within
one shell of that cap, the alignment band is the binding
constraint and the evidence rebuilds from the live pair (with the
nu_refine shell walk re-attempted). Principle: the cache may trade
staleness for speed in the Q_NU band weights, but the SEARCH
BANDWIDTH must never be governed by a frozen statistic.
- Validation plan: PfCRT abinitio3D, no prior flags -- expect the
matching low-pass to extend ahead of the FSC again and the run to
recover the ~3.9 A result; streptavidin/msp1/FlhB reruns -- expect
no change (gate inert for compact particles, cache rides only
while non-binding, so the speed win survives where it was ever
legitimate).
MATCHING-REFERENCE REGRESSION -- PLAIN NONUNIFORM TOPOLOGY
(2026-08-31). First refine3D_auto+pcg PfCRT run (auto-lambda and
auto-target behaving: supp 13.3%, lambda 0.177, target held at the
15% cold start): shipped pair stuck at the gridding run's
ITERATION-1 state (FSC=0.500 at 7.96 A / 0.143 at 4.14 vs gridding
3.98/3.61 by iteration 2), postprocessed maps massively
over-sharpened. Root cause chain:
- The matcher's plain-nonuniform contract consumes INDEPENDENT
even/odd _nu_filt references with NO further filtering
("filtering done when volumes are assembled",
simple_matcher_refvol_utils). The redundancy policy
(2026-08-28: Q_NU in-solve => no post-hoc NU filter, no _nu_filt
products) silently turned the matcher's first-iteration raw-refs
fallback into the PERMANENT state: every iteration matched
independent raw Q_NU half maps, unfiltered, at the evidenced
matching lp (~4 A) from iteration 1.
- abinitio3D never sees this because GOLD_STD_STAGE=TURNED_OFF
rewrites every nonuniform stage to nonuniform_lpset -- MERGED
single-reference matching, no independent per-half registration.
The redundancy policy was only ever validated in that topology.
- Independent per-half matching against each half's own
unsuppressed in-band noise (mild auto-targeted replay: 13%
suppression barely regularizes the refs; the old fixed-60%
setpoint was inadvertently cleaning them) overfits each half to
its own noise -> mid-resolution half-map divergence with a
matched-noise 0.143 tail. Textbook gold-standard overfitting.
- Over-sharpening: abinitio3D pins bfac=0; refine3D_auto lets
postprocess auto-estimate B by Guinier on the SHIPPED map. Q_NU
(and ML) amplitude suppression steepens the Guinier slope ->
auto-B strongly negative -> massive sharpening.
Fixes (2026-08-31, both implemented):
- filter_pcg_nonuniform_maps: the Q_NU skip-everything branch is
now LP-SET-ONLY (merged-reference topology keeps handoff-only
behavior). Plain nonuniform falls through and generates the
derived _nu_filt matching references (base = _unfil pair under
ml_reg; the ML/Q_NU pair enters as the finest-bank aux
replacement only when nu_refine=no, per the aux-channel
contract); the matching-lp handoff is then the filter-bank
finest selected lp, as in gridding. Shipped primary outputs
remain the raw Q_NU maps.
- postprocess_volume_from_files: automatic B-factor is now
always estimated from the _even_unfil/_odd_unfil pair
average when present (prior-free amplitudes; the estimate
targets the underlying signal decay), falling back to the
shipped map only when no unfil pair exists.
3. Test nu_refine=yes with rec_backend=pcg in abinitio3D. The
nu_refine=no rationale was gridding-specific (the ML-regularized aux
competitor supplied beyond-bank resolution implicitly,
l_use_aux = l_ml_reg .and. .not. l_nu_refine); the pcg path has no
aux channel, so nu_refine=no is a hard 4 A evidence/matching-lp
ceiling. The walk is gold-standard-gated so early noisy stages
should accept nothing; verify on the bgal replay first, watching
per-stage extension lines against the stage lp schedule.
4. Solvent constraint via the lowest-bin convention (IMPLEMENTED
2026-09-01, user-directed; design recorded before first run).
Context: the
2026-08-05 fundamentals changes removed the envelope from the NU
filter (spherical-only setup_nu_dmats, a31c7e8ad) and flipped
refine3D_auto to spherical FSC (envfsc=no, c7dfd5055); the retired
solvent prior left the PCG solve with no solvent constraint at all.
envfsc=yes is restored as the refine3D_auto hard default
(2026-09-01): the density-envelope automask + phase-randomization
corrected FSC (evaluate_halfmap_pair) again steers matching lp,
NU gating, and convergence. This item drafts the remaining half:
reinstating the solvent constraint in the NU filter and the Q_NU
prior WITHOUT reopening the reason the envelope was removed.
Design principles:
- ONE mask, one convention, both consumers. The conservative
density automask (image_msk%automask3D: lp-smooth -> otsu
binarize -> connected components -> grow -> cos_edge) at the
conservative envmsklp, computed ON THE FLY from the current
base average at assembly time, never written to file for this
purpose. NOT the aggressive NU evidence envelope (amsklp-scale),
which keeps its narrower jobs (detergent-omitting
matching-reference masking, diagnostics) and is never used for
the FSC or the solvent constraint.
- POST-ASSIGNMENT CLAMP, not objective support. The spherical
support of setup_nu_dmats is an invariant (envelope-restricted
objectives broke the null statistics -- the reason for
a31c7e8ad). The objective, whitening, and null estimation stay
spherical and untouched. The envelope enters only AFTER
optimization:
(a) NU filter: voxels outside the envelope have their selected
label overridden to the COARSEST candidate (lowest-resolution
bin) before ordered-label Potts smoothing, which then owns
the boundary. New optional envelope argument on the
label-selection/optimize_nu_cutoff_finds handoff; absent
argument = current behavior, bit-identical.
(b) Q_NU prior: expand_nu_evidence_band_weights assigns voxels
outside the same envelope to the coarsest band at the
maximum lack-of-evidence weight, so the replay applies its
strongest fine-shell suppression in solvent -- the in-solve
solvent-flattening constraint the retired Q_s solvent prior
was reaching for, expressed through the existing band
machinery (no new prior term, R10 mode-exclusivity
untouched).
- Implementation (as built): module-level clamp state
(nu_solvent_lmask/nu_l_solvent_clamp) with public
set_nu_solvent_envelope/clear_nu_solvent_envelope; armed by the
caller after setup_nu_dmats, cleared by cleanup_nu_filter, so
absent = bit-identical current behavior for every other caller
(nu_filt3D, bootstrap refs, tests). Filter clamp in
optimize_nu_cutoff_finds (label -> coarsest candidate before
Potts, logged as ">>> NU SOLVENT CLAMP: N support voxels...").
Evidence clamp in build_nu_evidence_state (solvent -> null
assignment, zero band support, zero uncertainty; skipped voxels
bypass the softmax entirely, so the summary statistics --
null_fraction, supported_fraction, uncertain_fraction -- reflect
the constraint; provenance gains solvent_clamp=density_envelope,
which flows into the identity hash, so a frozen state built with
the clamp can never be mistaken for one without). Armed at three
sites: filter_pcg_nonuniform_maps (plain-nonuniform pcg),
gridding volassemble setup_nonuniform_filter, and
build_nu_replay_evidence (Q_NU evidence, frozen with the state)
-- each computing the density automask on the fly from the base or
evidence pair average at envmsklp. Mask cost: one automask3D per
state per assembly (same order as the envfsc path already pays).
- Freezing discipline: the Q_NU clamp mask is part of the frozen
evidence state (rebuilt with it), so operator and evidence cannot
disagree across the cache lifetime; the search-bandwidth binding
rule from the frozen-evidence deadlock fix is unaffected (the
clamp only ever COARSENS solvent voxels, never the molecular
region the handoff reads).
- Validation plan: (i) A/B against the envfsc-restored build alone
(A = envfsc=yes only, B = A + solvent clamp) on PfCRT
refine3D_auto, both backends -- the clamp's contribution must be
measured, not confounded with the FSC restoration; (ii) 1WCM
harness with the clamp active: truth-FSC must not degrade
(constraint is solvent-only by construction); (iii) suppression
readout shift: expect the measured %-suppression to RISE at
unchanged lambda (solvent now contributes), so watch the
auto-lambda/auto-target interplay -- the setpoint controller must
not compensate the clamp away by lowering lambda; if it does,
restrict the suppression readout to the in-envelope region.
FIRST-RUN FINDINGS AND FIXES (2026-09-01, streptavidin
refine3D_auto+pcg). Shipped pair 3.05-3.12 A, controllers and
clamp active; two defects surfaced and were fixed:
- Matching low-pass pinned at the static bank finest (4.036 A)
while the evidence extended to 3.1 A: the plain-nonuniform
matching-reference pass built only the static ladder. Fixed:
filter_pcg_nonuniform_maps now runs the proven shell walk
(extend_nu_filter_highres_shells) after optimization when
nu_refine=yes, so the matching handoff advances with the
evidence (">>> NU MATCHING BANK EXTENDED..." line).
- The predicted controller interaction FIRED: total-energy
suppression read 40% at lambda 0.1 (solvent term), auto-lambda
drove lambda to the 0.01 floor, molecular regularization
collapsed to ~6%. Fixed as designed: the suppression readout is
now restricted to the evidenced region -- reconstructor_pcg
accumulates exact per-band energies y_b^T W_b y_b total and over
l_nu_evidenced (any input band weight < 1) during
apply_nu_precision; get_nu_prior_stats returns the evidenced
penalty and report_nu_solve_stats feeds the controllers the
evidenced-region suppression.
- Output tidied: routine NU diagnostics (Potts sweeps, whitening
profile, envelope ICM sweeps, envmask stats block, evidence
calibration/provenance details, solvent-automask banner lines,
ML warm-start notices, prior-energy detail) demoted behind
NU_DEV_OUTPUT. Kept by default: the low-pass assignment table
(the per-iteration NU fingerprint, restored on review), the
solvent clamp count, a one-line evidence summary (id + null +
band supports), one line per half with lambda_eff + evidenced
suppression, and the controller lines.
- Distributed-exec ergonomics (2026-09-01, user-directed, applies
to ALL distributed SIMPLE applications): partition scripts no
longer tee their output into the master terminal -- parts append
to SIMPLE_SUBPROC_OUTPUT only, and the scheduler prints one
summary line per completed phase (">>> solve_accum calls enter concurrent
sections; validation, NU firing statistics, diagnostics/logging,
image construction, and teardown are serial again. Each operator
now owns an explicit half-team FFTW plan size, and nested OpenMP
levels are enabled only around the solve pair then restored. PCG
profiling uses stateless local timestamps rather than the shared
legacy timer. This removes the deterministic code path behind the
prior box-300 failure: spherical set_mask -> mask3D_soft no longer
runs in an OpenMP region. The shared/direct route remains serial because its
builder image batch is not half-owned. Remaining gate: measure
peak resident memory and wall time at boxpd=600, and retain the
implementation only if the two prepared workspaces fit with useful
speedup in the final abinitio3D reconstruction.
- NU work deduplication (2026-09-01, user-directed): the Q_NU
evidence phase and the matching-reference generation ran the
full setup + solvent clamp + optimization + shell walk twice per
iteration on the same base pair. With nu_refine=yes (identical
setups: same _unfil pair, no aux channel), single state, and
source=base_unfil, the evidence phase now RETAINS its optimized
setup (retain_nu_filter_setup / nu_filter_setup_is_retained in
simple_nu_filter; cleanup_nu_filter always clears retention) and
filter_pcg_nonuniform_maps consumes it directly -- skipping
setup, clamp, extension and envmask regeneration -- then tears
it down. Retention is keyed on the finest-lp output being
requested (the signal that a matching volassemble follows), so
standalone reconstructions never leak a retained setup.
Multi-state and nu_refine=no keep the two-pass behavior (the
module holds one setup; the aux channel makes the setups differ).
- Second-run confirmation (streptavidin): matching bank extends
(10/9 accepted steps to 3.1-3.2 A), matching lp advances via the
support-gated promotion (3.43 A while the raw finest sits at 3.1
A with ~1% assignment -- the gate working as designed), evidenced
suppression keeps lambda off the floor spiral (settles ~0.01
with supp ~10-13% against targets 9-14%), B-factor sane (-62 to
-68). One controller wart fixed: the shipped-pair crossing is
Fourier-shell-quantized and the 3.05<->3.12 A adjacent-shell
flip re-triggered DEGRADED/IMPROVED steps; the AIMD comparison
now stalls on any change of fewer than two shells.
- Direct PCG support constraint (EXPERIMENTAL, 2026-09-01,
user-directed). Distinct from every prior and from the solvent
clamp: a real-space support constraint on the solve itself. The
machinery pre-existed --
set_maskalready solves the projected system (P H P) u = P b with the soft spherical support -- so the mode is: install an ARBITRARY [0,1] envelope as P. reconstructor_pcg%set_mask_volume(mskvol): caller-supplied volume as the support (validated, clipped to [0,1]); both solve entries now also project the INITIAL GUESS onto the support (a warm start's out-of-support content was previously carried into the output untouched -- latent even for the spherical case).- CLI:
pcg_mskfile=<vol.mrc>on reconstruct3D and refine3D (pcg-gated, hard-errors without the pcg backend or a missing file). All PCG solve sites route throughset_pcg_solve_support: pcg_mskfile when given, spherical mskdiam support otherwise -- workflow policies are untouched, so this is a pure play/experiment surface. - Composition: NOT a replay prior (R10 untouched); composes with P_tau/Q_NU (set_nu_prior already folds self%mask into the band weights, so the support automatically shapes the prior too), and with the trailing chain (constraint is solve-side only; the accumulators stay constraint-free, so masks may change across iterations and the mode switches off cleanly).
- Caveats recorded: the naive half-map FSC is optimistic inside the support by construction -- judge with the phase-randomized envfsc; the mask is shared between halves (the classic solvent- flattening compromise); the Fourier-shell preconditioner does not commute with a real-space P (still valid, mildly suboptimal).
- Deferred (design in the 2026-09-01 discussion): the soft-penalty variant (+ lambda_m diag((1-M)^2)) and the fixed-background focused mode (x = x_fix + P*delta, RHS = P(b - A x_fix)) -- both drop out of the same set_mask_volume machinery when wanted. Mode semantics for the record: HARD is a change of variables (x = P u), so out-of-support unknowns are REMOVED -- no strength parameter exists, conditioning improves by the support fraction, and a wrong mask deletes real density. SOFT is a Gaussian prior with precision lambda_m(1-M)^2, so out-of-support density is SHRUNK, not deleted, and survives where the data insists -- graceful under mask error but reintroduces a dataset-dependent strength (the auto-lambda failure mode). The Q_NU solvent clamp is the band-limited middle: soft, fine-frequencies-only.
SOLVE SUPPORT POLICY (2026-09-01, user-directed; supersedes the
first per-solve draft). Principle: with PCG a mask belongs in
the ESTIMATOR, not in post-processing -- post-hoc masking cannot
improve a map, a support constraint can. Three rules (user,
2026-09-01), over the two PCG passes (1) unfil and (2)
regularized:
- automsk=yes enables the conservative density envelope (Cyril's
automask3D at envmsklp) as solve support at all.
- envfsc=yes + automsk=yes: BOTH passes take the envelope.
- envfsc=no + automsk=yes: pass (1) keeps the SPHERICAL support
(the FSC pair stays unconstrained), pass (2) takes the envelope.
- Consequence, and the reason envfsc is the right switch: when
pass (1) is envelope-constrained, the FSC pair is ALREADY
masked by the estimator, so the envfsc masking +
phase-randomization preprocessing in evaluate_halfmap_pair
must not run again -- it is now gated on rec_backend (skipped
for pcg, kept for gridding, where it remains the only way to get
the envelope into the estimate). The envelope is still derived
and returned there, because the automask artifact has other
consumers (postprocess envfsc, matcher fallback, final rec).
- ONE support envelope per state feeds both passes, so when both
take it the FSC pair and the shipped pair stay on the same
footing.
- Derived from the reference volume the iteration matched against
(lag-one, the same lag the matching references carry), which
also resolves the ordering problem: the support exists before
the base solves. Missing/startvol reference => spherical
fallback.
- The envelope is the CONSERVATIVE density automask at envmsklp
(Cyril's 20 A mask), used as-is. An earlier draft coarsened it
(max(30 A, 2*envmsklp)) out of concern for the NU evidence's
null calibration and whitening; that was over-engineering
(user, 2026-09-01): the 20 A envelope is already generous --
protein plus micelle, dilated, soft-edged, and about half the
spherical support on both datasets measured (PfCRT: 941k of
1.93M support voxels outside it; streptavidin: 102k of 217k) --
and NU refinement predates the shell whitening and works on
plain Euclidean unaries, so the whitening MAD is not a design
constraint. The evidence readiness contract (null_fraction
outside [0.01, 0.90] hard-errors) is the guard if a specimen's
envelope ever proves too tight for the null calibration; watch
pcg_nu_null_fraction in the evidence banner on first runs.
- Precedence in set_pcg_solve_support: explicit pcg_mskfile > the
per-state density envelope > spherical mskdiam.
- Backend split, for the record: gridding can only put the
envelope into the MEASUREMENT (post-hoc mask + phase
randomization); PCG puts it into the ESTIMATION (support
constraint). Same envelope, same intent, different mechanism --
and never both at once.
- refine3D_auto startup: reconstruct -> mask -> re-reconstruct -> refine (IMPLEMENTED 2026-09-01, user-directed). Root cause of the PfCRT collapse, established by elimination (the envelope-masking and solver-convergence hypotheses were both falsified by controls -- automsk=no reproduced the collapse exactly, and reconstruct3D at maxits_pcg=2 from the same orientations gives 5.97/3.93 A with cFAR 0.78, better than refine3D_auto's first iteration): the first matching had no proper references to match against.
- The matcher's nonuniform contract is that filtering is
assembly-owned (
if( l_ml_reg .or. l_nonuniform_mode ) then ! filtering done when volumes are assembled). At iteration 1 the_nu_filtreferences do not exist yet, so it fell back to the RAW input half maps -- and then applied NO filter to them, matching unfiltered noise out to the FSC=0.143 band (~3.9 A on PfCRT, where that curve has already fallen to 0.16 and is about to cliff). abinitio3D's final stage, by contrast, matched a MERGED, NU-filtered reference at an explicit ~6 A lp. - The gridding path was immune because
prepare_nu_bootstrap_refs_from_raw_halvesgenerated NU-filtered startup references; it returned immediately for pcg ("the Q_NU replay regularizes in-solve"), which is true from iteration 2 onward and false for iteration 1. -
High-contrast specimens (bgal, streptavidin) survive matching unfiltered half maps; a low-contrast detergent-solubilized membrane protein does not. Hence "works on big things or things with no detergent". Implementation, three parts: (a)
reconstruct3Don the pcg backend now produces the SAME reference products a refinement iteration does:distr_executecallsfilter_pcg_nonuniform_mapsafter the master when filt_mode is nonuniform, generating the_nu_filtmatching references and writing the evidence-derived matching-lp handoff into the project. Previously it called the master and returned, so no workflow could reconstruct-then-refine correctly. (b) refine3D_auto runs a STARTUP sequence before any matching:calc_pspec(particle-spectrum sigmas) then ONE regularizedreconstruct3Dcarrying the refinement's own filt_mode/automsk/ envfsc/nu_refine. It leaves the automask, the NU evidence envelope, the_nu_filtreferences and the lp handoff in place, and its output becomes vol1.prepare_nu_bootstrap_refs_from_raw_halvesis deleted as superseded. SIGMA BASIS (2026-09-01, second iteration of this design): the first version usedbootstrap_rec3D, which derives sigmas from the startup half maps. That exists for the case where no box-compatible sigmas CAN exist (the final rec, crop box -> native box); at the refinement box calc_pspec can estimate them directly from the particles, and its estimate conditions the euclid system far better (bgal startup base residual 0.23 with the half-map sigmas, 0.08 with calc_pspec's). Worse, refine3D then re-derived its own regardless, because the former sigma bootstrap predicate was positional (startit <= 1) and never checked whether usable sigmas existed -- so the startup was regularized against sigmas the refinement discarded. That predicate now also requires the absence of a sigma star for the starting iteration, so one basis serves the whole workflow. Fresh runs are unaffected (no star exists, so the bootstrap still runs). The startup'sPCG_NU_STATS_FILEis deleted afterwards: the NU replay controllers compare consecutive REFINEMENT iterations, and leaving the startup readout behind made the first auto-target comparison startup-vs-iteration-1 -- not like for like, and it took a real setpoint step off it (bgal: 15 -> 9). (c) The matcher can no longer match unfiltered references silently: when the nonuniform references are missing it flags the fallback, applies the FSC-optimal filter to the raw half maps instead of nothing, and hard-errors if there is no FSC either. This is the structural guard -- it protects every workflow, not just this one, and would have made the original defect loud. -
PfCRT matching-reference collapse -- SUPERSEDED HYPOTHESIS (2026-09-01, retained as a record of what was falsified). Reproduced twice in refine3D_auto+pcg: iteration 1 matches with the SPHERICAL reference mask (the envelope does not exist yet) and is healthy (orientation overlap 0.875); iteration 2 is the first to consume the NU-evidence-envelope-masked references and collapses (overlap 0.236, in-plane distance 17 deg, shift increment 1.4 px avg / 11.3 max, FSC 0.500 6.47 -> 7.39 A, B-factor -79 -> -142). Mechanism: the NU evidence envelope OMITS DETERGENT BY DESIGN (PfCRT occupancy 8.9% of the spherical support), so the matching reference is protein-only while the particle images contain the micelle. Under the Euclidean objective the unexplained micelle dominates the cost, the assignment randomizes, and the re-centering shows up directly as the shift-increment jump. This is a MODEL-DATA MISMATCH, not overfitting and not a prior miscalibration -- soluble specimens (streptavidin, bgal) are immune because their envelope covers the whole particle. FALSIFIED by the automsk=no control, which reproduced the collapse identically (6.75/3.98 -> 7.76/4.93 with no envelope masking anywhere, versus 6.47/3.98 -> 7.39/4.93 with it). The reference mask was never the cause; see item 7 for the actual one. The matcher mask change this motivated was reverted. Retained because the reasoning about reference/image content consistency is still the right frame -- it simply pointed at the wrong mask.
- (reverted)
prepare_matching_reference_maskusing the density envelope for matching references. -
refine3D_auto gains a STARTUP BOOTSTRAP (user-directed): bootstrap_rec3D runs before any matching (unregularized pass -> sigmas from the half maps -> regularized pass with the refinement's own filtering settings), so the masks, the NU evidence products and the _nu_filt matching references all exist at iteration 1. Iteration 1 then matches the SAME kind of reference as iteration 10; previously the reference convention changed underneath an already converged alignment, which is what made the mismatch above fire as a cliff rather than a gradient. abinitio3D never needed this because its stage ladder introduces ml_reg / NU filtering / envfsc / automsk gradually, while the alignment is still loose.
-
PfCRT refine3D_auto ROOT CAUSE -- COARSEST-BANK FLATTENING IN THE MATCHING REFERENCES (2026-09-02, confirmed by elimination; fix implemented). With the startup bootstrap in place (1d0071e77) the collapse moved from iteration 2 to iteration 1 and tracked the first consumption of NU-filtered matching references in every topology. Control chain that isolated it:
- automsk=no reproduced the collapse (mask exonerated -- and the earlier automsk=no falsification of the envelope hypothesis was confounded by the then-unfixed unfiltered-refs defect of item 7; both hypotheses pointed at reference/image content mismatch, both via the wrong mechanism);
- rec_backend=gridding reproduced it (Q_NU replay, auto-lambda, evidence reuse, PCG solve support all exonerated);
filt_mode=none automsk=no autoscale=nowas HEALTHY at iteration 1 (overlap 0.981, angular moves 0.98 deg avg, shifts 0.17 px, FSC=0.143 holding at 3.98 A, cFAR 0.73, B-factor -70) -- exonerating the euclid sigma cold start, the native box-300 geometry, nspace 20000, and prob_neigh, since only the reference recipe differed. Mechanism: on a detergent-solubilized specimen the NU bank puts 72-73% of the spherical support in the coarsest 19.4 A bank (solvent clamp ~49% plus in-envelope null ~24%, i.e. ~45% of the density envelope interior -- the micelle). The matching reference then omits density every particle image contains; under the euclid objective the unexplained micelle dominates the cost, pose discrimination is lost, and the alignment scatters (3.93 -> 4.93 -> 5.86-5.97 A plateau read as convergence). abinitio3D is immune (nonuniform_lpset merged refs at box 160, where null ~= clamp only); compact soluble specimens are immune (nearly nothing inside their envelope goes null). Score-spread flatness (0.380 +/- 0.002) appears in healthy runs too and was never evidence. First fix attempt -- FSC-OPTIMAL FALLBACK IN THE MATCHING PRODUCT (6e5135689) -- TESTED AND RETIRED (2026-09-02). Seeding the_nu_filtmatching references with the FSC-optimal filtered base pair wherever no finer bank label was positively selected did NOT rescue the alignment. The test ran at automsk=yes defaults, where the matcher still multiplied the references with the NU evidence envelope before reprojection -- the restored fallback content outside the ~10-19% envelope was erased again by the multiplication, which stood in every collapsing configuration. The multiplication, not the filter flattening, is the standing suspect; the fallback machinery was reverted. Superseding policy (2026-09-02, user-directed). Principle: a reference must never HARD-REMOVE density that is present in the particle images; density excluded by an envelope is DOWN-WEIGHTED through heavy low-pass filtering of the background (cisTEM precedent: envelope + heavy background low-pass does not break alignment). Three parts, both backends:- Reference-envelope multiplication REMOVED. The matcher's
mask_matching_referenceapplies the spherical soft mask ONLY;prepare_matching_reference_maskand the evidence/density envelope multiplication (zero_env_background+mul) are deleted. No reference is ever multiplied with an envelope before reprojection. - automsk=yes now means: the filter-field BACKGROUND is the
complement of the NU evidence envelope, derived from the SAME
evidence pass (the setup unaries; no automask3D double-compute).
write_nu_evidence_envmaskgainsl_arm_background: one call derives the envelope, writes the artifact, and arms the background clamp (set_nu_solvent_envelopewith source='nu_evidence_envelope'; provenance -- and thus the frozen evidence identity -- records the clamp source). Background voxels take the coarsest bank candidate in the filter field, which both generates the matching references (heavy background low-pass, detergent down-weighted but present) and derives the Q_NU precisions (maximum lack-of-evidence outside the envelope) -- one field, both consumers. Armed beforeoptimize_nu_cutoff_findsat all three sites (build_nu_replay_evidence,filter_pcg_nonuniform_maps, volassemble'ssetup_nonuniform_filter), so the existing pre/post-Potts clamp mechanics apply unchanged; the later plan-gated envmask writes are skipped in this mode (already written in-pass). - automsk=no keeps the conservative density-envelope solvent constraint of dev item 4 (on-the-fly automask3D), bit-identical.
- The PCG SOLVE support is deliberately NOT the evidence mask: it
stays the conservative density envelope from the lag-one
reference (
build_pcg_state_support), per the solve-support policy -- the evidence mask is too tight to constrain the solve. -
Validation plan: PfCRT refine3D_auto at full defaults -- expect iteration 1 to hold ~3.9 A with high overlap now that the references explain the micelle at 19.4 A instead of omitting it; streptavidin rerun -- expect no regression (its evidence and density envelopes nearly coincide, and its background is essentially solvent).
-
PCG+NU code-review response (2026-09-02,
pcg_nonuniform_code_review.md, corrected implementation). - P1 post-hoc filter:
filter_pcg_nonuniform_mapsno longer runsnu_filter_volsfor ANY mode -- it is the evidence-derived scalar matching-lp handoff for bothnonuniformandnonuniform_lpset, nothing else. No_nu_filt/_nu_locresproducts exist on the pcg backend; plain nonuniform matches the PRIMARY per-half Q_NU maps (matcherl_pcg_qnupath: primary maps authoritative, no fallback flag, no FSC-optimal or other extra filter). The evidence-phase setup-retention machinery in the strategy was removed as its consumer no longer exists (the nu_filter module API remains). NOTE the standing tension with the 2026-08-31 "matching-reference regression -- plain nonuniform topology" record, which retired exactly this raw-Q_NU-refs matching for gold-standard noise overfitting: circumstances have changed (evidence-envelope background suppression in-solve, auto-lambda/auto-target live), but the overfitting risk of independent per-half matching against each half's own in-band noise is not structurally removed -- watch mid-resolution half-map divergence with a matched-noise 0.143 tail on validation runs. - P1 evidence background: the solvent/background clamp in
build_nu_evidence_statenow encodes the COARSEST bank candidate (supports [1,0,0,...] -> band weights [0,1,1,...]) instead of the post-Potts null label whose zero supports suppressed all non-DC background detail. The constraint is installed before the Potts sweeps, fixed voxels participate as boundary conditions, and the readinessnull_fractionis retained from the unconstrained broad spherical evidence field, counted only over that packed support, so neither the full-box exterior nor envelope size can invalidate its calibration diagnostic. The coarsest band's support is categorical and is not inferred by comparing a grid-quantized cutoff with its nominal Angstrom boundary. - P1 solve support:
build_pcg_state_supportis independent ofautomsk; the shared route now builds and installs the per-state density support exactly like the distributed route (base solves under envfsc=yes, regularized replay always). A valid start volume is a density source. If no reconstruction exists yet, the base solve necessarily bootstraps on the sphere for eitherenvfscvalue; its current merged pair then seeds the density-supported replay. - P1 FSC preproc:
evaluate_halfmap_pairtakesl_pair_support_constrainedand skips the envelope+ phase-randomization preprocessing only when the pair was actually density-constrained in the estimator -- never inferred from the backend name. PCG spherical-fallback pairs under envfsc=yes now get the ordinary envelope-corrected FSC. - P2 Q_NU mandatory:
validate_nu_replay_requesthard-errors any pcg+NU configuration without the active euclid ML replay and pcg_nu_lambda_rel > 0; strength-zero/P_tau remain valid only with filt_mode=none (bootstrap_rec3D pass 1 forces filt_mode=none and is unaffected).pcg_mskfileis likewise rejected on NU routes (development escape hatch isolated to filt_mode=none). [SUPERSEDED 2026-09-09: validate_nu_replay_request and the NU-route rejection went with the dead-code removal; pcg_mskfile is accepted on every PCG route and, since the 2026-09-09 review, reported as the state support: build_pcg_state_support returns it with l_have=.true. regardless of automsk, so the FSC mode, the support-provenance sidecar and the NU evidence null regime all see a constrained pair.] - P2 refine3D_auto envfsc: now a guarded (genuinely overridable) default.
- P2 _pproc: PCG skips all post-hoc mask multiplication in
postprocess_volume_from_files, including derived_pproc/_mirrproducts. Non-PCG behavior is unchanged. - P3:
nu_evidence_envelope_masking.mdcarries a SUPERSEDED banner; automasking/nonuniform policies updated 2026-09-02. - Runtime follow-up: a density-supported iteration-1 base pair exposed a
numerical mismatch between the narrower solve support and broader NU
evidence sphere. Exact zero/zero boundary voxels had entered the radial
MAD whitening fit as zero-noise samples, and the explicit zero predictor
could then overflow even though ordinary cross-half candidate unaries
remained finite.
nu_objective_noise_profilenow excludes only those unobserved exact-zero pairs from scale estimation while retaining the spherical evidence domain;nu_objectivecomputes standardized Huber losses in double precision with a high finite cap. The focused envelope test includes this hard-supported-pair geometry. Rebuild and runtime rerun are pending. nu_refinefollow-up:refine3D_autokeeps its defaultyes, while staged abinitio3D's heavily usednu_refine=noPCG state is preserved exactly (eight signal candidates, integer coordinates, unit candidate masses, four bands, raw-finest matching handoff). Adaptive PCG discovery alone is capped at FSC=0.143 plus two Fourier shells and uses the final ordered-label tie tolerance. Once extension candidates exist, their coordinates continue from the unchanged static ladder in Fourier-shell distance normalized by the finest static interval; normalized Voronoi widths remove candidate-count bias from soft band support. Adaptive matching uses the 5% supported cutoff plus the same two-shell FSC headroom. Theautomsk=yesevidence envelope is explicitly the preliminary static-bank boundary, fixed before adaptive challenges.- Observed-domain calibration (review follow-up, 2026-09-02 pm): the
whitening fix above left every OTHER evidence statistic on the full
sphere. With the base pair density-constrained (refine3D defaults,
iteration 2 onward) about half of the spherical support is an exact
zero/zero cluster at gap ~0, which pinned the lower-quartile null
center, diluted the spatial beta, could collapse the temperature
median into its fallback, and inflated the readiness
null_fractionby support geometry (a small particle in a generous sphere could cross the 0.90 ceiling and hard-error).setup_nu_dmatsnow records a packed observation mask (nu_observed_mask, same exact-zero test as the profile);build_nu_evidence_stateevaluates the null-bias center, beta, temperature, null/uncertain/band-support fractions on observed voxels only, freezes unobserved voxels at the explicit null in both Potts sweeps, and ships them with zero band support and unit uncertainty. The summary carriesobserved_fractionand the provenance string recordsstatistics_domain=observed_support. The spherical NU support itself is unchanged (skill invariant). - Solve/support provenance: every shipped state volume now has a
<vol>_pcg_support.txtsidecar (solve_support=density|sphere,solve_kind=base|regularized|mixed). The trailing bootstrap reads it for the lag-one FSC/evidence pair, soevaluate_halfmap_pairskips the envelope+phase-randomization preprocessing exactly when that pair was density-constrained in the estimator; a pair without a sidecar (imported) stays unconstrained. A bootstrap blend is constrained only if both contributions were. Base solves also use the kind field to reuse only a compatible own-half base solution:_unfilfor a regularized predecessor, primary for a base-only predecessor, and no primary warm start for a mixed predecessor. There is no legacy_unfilfallback for volumes without a sidecar (review 2026-09-02 pm: a stale_unfilbeside an imported map would survive two CG steps; such jobs are rerun from scratch). - PfCRT refine3D_auto log audit (2026-09-02 pm, Jul02 vs Sep02 runs):
the July refine3D_auto took the abinitio3D map from 4.14 to 3.53 A in
four iterations. At HEAD, abinitio3D orientations bootstrap to 3.88 A
(pcg, sphere) / 4.14 A (gridding), i.e. at least as good as July, and
refine3D_auto then DEGRADES them on both backends. Gridding: the 5%
support gate added to
get_nu_filtmap_finest_selected_lpon 2026-08-30 capped the matching band at 5.0 then 5.97 A against a 4.1 A map (July: raw finest label, 3.98 A); overlap plateaued at 0.85, cFAR 0.75 -> 0.34, helices unresolved. Fix: the volassemble handoff passesmin_assigned_pct=0. PCG:pcg_nu_matching_lowpassput the band at FSC+2 shells (3.79 A vs 3.88 A) every iteration against unfiltered Q_NU references; orientations "converged" (overlap 0.998, dist 0.14 deg) while the FSC=0.5 crossing went 4.09 -> 7.06 A and cFAR 0.75 -> 0.54: per-half self-consistency, not signal. The headroom is nowNU_ALIGN_LP_FSC_HEADROOM_SHELLS = 0(experiment; the binding-band rebuild test prevents the frozen-evidence deadlock at 0). Also observed: SCORE 0.38 +/- 0.002 across all particles on both backends (relative whitened residual d ~ 0.97), consistent with the likelihood softmax over the top-K being effectively uniform; not the primary cause here since abinitio3D at HEAD uses the same sampler and is fine. - Cold restart (review 2026-09-02 pm):
solve_coreno longer hard-errors on non-positivedot(p,Hp); it returns the pre-step iterate withstop_reason=PCG_STOP_INDEFINITE.solve_with_cold_restartwraps every production solve: a warm-started solve (previous base/ML half, or the replay's base-solution seed) is retried once from zero, a cold solve fails immediately, and a second indefinite outcome is fatal. Rationale: a warm start converging toward the approximate kernel's fixed point concentrates the residual in the operator-error modes, the same regime as over-iteration; a mid-refinement crash there would discard a long run for a recoverable condition. Follow-up review keeps the retry itself compute-only: each concurrent distributed half returns its final outcome with any restart trigger recorded, while logging and fatal handling occur in serial finalization. Public solver calls without an outcome retain their previous immediate failure. - Shell-walk flag renamed
l_require_margin(wasl_tie_tolerant): it demands anu_label_smooth_is_bettermargin win, i.e. it is STRICTER than the raw<test, which the old name inverted. - Diagnostics
half_pair_parallel=now reports whether the pair actually ran as concurrent sections (one empty half runs serially). - Shared route: a missing evidenced matching low-pass is logged and the
previous
lprides, matching the distributed route. -
bootstrap_rec3DimportsNU_AUTOTARGET_MIN/MAXinstead of duplicating the bounds. -
Final-reconstruction stage made standalone (2026-09-07).
- msp1 residual-pass crash reported after the 09:25 fix carried exactly
the same code addresses as the 09:18 log (0x12ac0b2
choose_and_run_strategy, 0x12b2812 refine3d_exec._omp_fn.0, 0x166a8d3
alloc_hash): same binary, i.e. the pre-fix shared-orientation race, not
a second race. Rebuild
simple_private_execon that machine; addr2line on 0x12ac0b2 must no longer name the old choose_and_run_strategy line. - Export gate: one event, the master's own THROW at the cold base solve of the final reconstruction on the residual sigmas (dot(p,Hp)=NaN at CG iteration 1). Its backtrace appears before the "10 PART(S) COMPLETED" line only because stderr is unbuffered; no worker died. D is validated finite by update_lambda_from_density, so the NaN enters through the RHS or the preconditioner floor. The bootstrap reconstruction on the image-power STAR (iteration N) solved; the final one on the residual STAR (N+1) did not, so the residual groups are the suspect (scale, tiny shells). Diagnosis needs sigma2_it_N.star vs sigma2_it_N+1.star from the run directory.
-
bootstrap_rec3Dmoved to simple_commanders_refine3D and now owns the complete sequence: image-power seed (N), euclid ML bootstrap map, refine=sigma residual pass against it, consolidation (N+1), final euclid ML map on the residual sigmas; it returns vol1..N and which_iter=N+1 on the command line. abinitio3D's calc_final_rec (takes the commander as class(commander_base) from the abinitio commander) and refine3D_auto call it and carry no copy. Standalone reproduction of the failing stage on any project with 3D orientations:simple_exec prg=bootstrap_rec3D projfile=... pgrp=... mskdiam=... nparts=... nthr=... rec_backend=pcg(UI now exposes rec_backend, maxits_pcg, rtol). -
Canonical sigma2 store review, streptavidin 2/10 failures (2026-09-07).
- Proven non-equivalences with the legacy STAR store at review time: (a) freshness, canonical committed residuals(N) right after matcher pass N and its stage-boundary and final reconstructions consume them, legacy's STAR N holds residuals(N-1); (b) the final reconstruction: legacy refreshes sigmas at native sampling whenever the registration box differs from the native box, canonical reused any structurally consumable state and skipped the refresh. Precision, averaging and grouping are the same.
- Failure surfaces only canonical had: hard abort on any active record with a non-finite or non-positive shell; generation/digest/checksum/ coverage aborts on merge; silent re-seed from image power on any validation failure. Design hazard: the state was registered as an absolute path that survives every project copy, so a copied execution project consumed and mutated the originating run's sigma2_state.bin; candidate and range names carried no transaction identity.
- Fixes: registration by name resolved against the project file's
directory (absolute only for explicit cross-directory states);
abinitio3D drops an inherited registration at start; candidate and
range names carry the generation they commit
(
<stem>.g<N>.next,<stem>.g<N>.part<NN>.range) and a worker checks the candidate's generation; invalid records are skipped with a warning in reduction and commit validation; the canonical final reconstruction applies the legacy registration-box rule. Testsimple_test_exec prg=sigma2_stateextended (invalid-record skip, scoped names). Uncompiled. -
Follow-up ordering correction (2026-09-08): canonical consolidation now occurs after shared/distributed assembly, so every reconstruction owned by iteration N consumes the sigma model that scored iteration N. The final symmetry-search candidate remains pending through symmetric reconstruction and is committed by the abinitio3D stage immediately afterward. This restores the legacy lag-one visibility boundary without persistent iteration history.
-
Final-reconstruction sequence validated on msp1 (2026-09-07). The standalone
bootstrap_rec3D(image-power seed, gridding ML bootstrap map carrying the workflow's filt_mode/nu_refine/automsk,refine=sigmaresidual pass against it, consolidation, PCG final map with the cold-solve floor) produced the best msp1 map ever obtained (user report). The earlier standalone run had a PCG bootstrap map and a 2-iteration cold final solve (3.912 A on both maps, 812 s); the difference is the regularization-consistent residual reference and the 5-iteration cold solve. This is the reference configuration for final reconstructions. Numbers of the validated run (commit 95d4adbf, 131223 particles, nparts=10, nthr=8, rec_backend=pcg): NU matching band 3.863 A at the residual pass; final base solves 5 its, residual 2.55e-2/2.58e-2 (was 3.7e-2 at 2 its); FSC=0.5 4.479 A, FSC=0.143 3.962 A, cFAR 0.4280, B-factor -130.3; ML replays 5 its, residual 3.89e-2/3.87e-2 (was 6.2e-2); 586.5 s total (was 812.5 s: the gridding bootstrap pass costs ~115 s against ~300 s for the PCG one, more than paying for the longer final solves). The FSC crossing moved by one shell; the visible gain sits in the far better converged solves and the consistent weighting, not in the nominal resolution.
Run record (2026-09-08b): PfCRT -- the NU competition on truncated-CG pairs; nu_input=gridding¶
Superseded record 13 (reverted commit fef7bd9eb, the "gridding-equivalent"
M^-1 b input: histograms even further from July's, stage 7 collapsed to
random orientations, cFAR 0.80 -> 0.09; that input carried the 1% shell-mean
density floor and a per-half least-squares scale fit, neither of which a
gridding pair has).
Facts established from the logs and the maps on disk (all runs 16.8k particles, mskdiam 160, NU stages 6-8):
- Search side unchanged since the 2026-08-10 PfCRT-validated state
(33c9d3663):
simple_strategy3D_prob, the eul_prob sampler, the euclid objective insimple_polarft_corrand the matcher band code differ only by PCG dispatch, diagnostics and cosmetics. - The NU competition code differs from July (a6ae317ae) only by the radial whitening (53f173787). The even/odd noise profile measured on the final native-box pairs is flat for PCG pairs (sigma(r) 4.2-4.5 across the sphere, halved only in the soft-support edge shell) and 1.5e-2 -> 0.94e-2 for the July gridding pair (the fixed under-deapodization fade); the whitening is not the cause.
- Label histograms at matched FSC. July gridding pair, stage 7, FSC 6.3 A:
84% of the sphere at >= 10.0 A, 4.7% at 5.97, 1.8% at 5.0, 0.6% at 4.14.
Sep 7 PCG base pair, stage 7, FSC 6.1-6.3 A: 43% at >= 10.0 A, 32-35% at
7.96, 15-18% at 5.97, 0.1% at 5.0, 2.6-3.6% at 4.14 (the 5.0 label skipped,
the finest label populated). Sep 8
M^-1 binput, FSC 6.5 A: 44% at 5.97. - Same binary (124b277f
nu_filt3D, static bank, mskdiam 160) on the final native-box_unfilpairs: July gridding pair (FSC0143 3.98 A) 25% of the sphere at <= 5.97 A; current_code_fails PCG pair (FSC0143 5.18 A) 33%; Sep 7 PCG pair (FSC0143 6.09 A) 20.5%. A 5.2 A PCG pair receives more fine labels than a 4.0 A gridding pair.
Mechanism: the cross-half unary penalizes a finer candidate by the noise it admits from the other half. A truncated-CG solution (two warm-started iterations, or the five-iteration cold final) is spectrally regularized: its poorly determined high-frequency modes stay damped in both halves, so the finer candidates are nearly free and the competition selects them wherever the halves share any content -- including the previous iteration's map carried by the warm start (the base FSC stayed at 6.3 A for four iterations of essentially random orientations in the Sep 8 run: memory, not data). On high-SNR specimens (streptavidin, msp1, embb) those labels are also supported by the data, hence no failure there; on PfCRT they are not, the NU references carry unsupported 5-8 A content, the search fits it and stalls (4.5 A cap) or collapses. The PCG maps themselves are not at fault; the competition was designed for a pair with the full-band independent noise of the data, which the gridding backend supplies and a regularized solve does not.
Implemented (opt-in, default unchanged so the four working datasets are not
touched): nu_input=gridding on abinitio3D, refine3D, refine3D_auto
and reconstruct3D. reconstructor_pcg%set_keep_gridding_half makes
end_accum keep E T^-1 b -- the exact sampling-density division of the
folded RHS with the sampl_dens_correct rule (divide where rho > 1e-6, zero
elsewhere), no shell floor, no prior, no support mask, the solver's own
scale and convention -- and both PCG master paths seed the NU competition
from that pair (bootstrap: blended with the previous pair like the base
pair). The base pair remains the FSC oracle, the _unfil product, the ML
warm start and the auxiliary resolution. Test matrix requested: PfCRT x
{rec_backend=pcg default, rec_backend=pcg nu_input=gridding,
rec_backend=gridding}, master logs (ABINITIO3D_OUTPUT_RESTART*) kept.
Addendum (2026-09-08c, user-directed): FSC-anchored candidate cap, both
backends, default on. A pre-processing low-pass of the input pair was
considered and rejected: candidates beyond the cutoff would have identical
raw unaries but are smoothed at candidate-specific radii, so the finer label
wins by footprint in about half the voxels and Potts turns that into patches.
Capping the bank instead: init_nu_filter(fsc_res=) keeps the static labels
coarser than fsc/NU_BANK_FSC_HEADROOM (1.5, about two labels), never fewer
than two, nu_bank_cap_find bounds the shell walk (max_find), the cap is
logged once per state (NU BANK CAP). The ML pair as competition input was
also considered and rejected: P_tau shrinks the other-half noise identically
in both halves, so within the band the fine candidates get cheaper still.
Measured on the final native-box pairs (E-O slice spectra, noise power
relative to the 8 A shell): July gridding 1.11/1.28/1.35 at 6/5/4.4 A (rising
toward Nyquist), PCG 1.01/0.96/0.90 (falling); the July real-space fade
(sigma 1.5e-2 centre, 0.94e-2 edge) is not a band limit and the gridding
insertion ran to the crop Nyquist in July as now.
Addendum (2026-09-08d): the tie mechanism and the like-for-like selection.
Candidates are smoothed at radii of 1.5 x LP (capped at 30 A), so
near-identical raw unaries are separated by the smoothing footprint: the
smaller radius wins at local minima of the unary field, the larger at
maxima, an intermediate radius almost never. The Sep 7 PCG base-pair
histograms follow the radius table (radii for 5.97/5.0/finest): box 140,
4/3/3 px: 2.8% / 7.2% / 0.08% (5.0 and 4.44 tie exactly, 5.0 wins); box 150,
4/4/3 px: 15% / 0.1% / 3% (5.97 and 5.0 tie, 4.14 wins by footprint); box
160, 5/4/3 px: 7.5% / 2.7% / 2.4% (all differ, both populated). Offline with
the standalone competition (Mac binary e45b237a, mskdiam 160, final
native-box pairs): the Sep 7 ML pair (FSC0143 5.86 A) put 18.3% of the
sphere at 3.98 A against 2.3% for its base pair -- the ML pair as sole
input needs the tie handling fixed first. A uniform 10% relative margin
(finer label must beat the incumbent by 10%) was tried and rejected the
same day: honest base pairs collapsed to 79-82% at 19.4 A (the natural
differences between coarse labels are below 1%, at the fine end about 12%)
and the ML skip pattern persisted (3.5/7.8/4.1% at 5.97/5.0/3.98). The
implemented rule is scale-consistent instead: raw unaries are kept
(raw_dmats_mask); the selection walks coarse to fine and at each level
smooths the candidate and the incumbent at the candidate's radius, so
identical unaries tie exactly and the coarser label keeps; the Potts sweeps
never promote a voxel beyond its entry label. Cost: n(n+1)/2-1 smoothing
passes instead of n. Offline check pending (rebuild, then
scratchpad/run_nu_all.sh): pass = July base/ML pairs close to their
uncapped-binary histograms (4/12/20/19/21/10/9/6% coarse to fine for the
base pair), Sep 7 ML pair finest label from 18% to low single digits, skip
pattern gone.
Addendum (2026-09-08e): nu_input=gridding defect. The first runs with the
gridding half produced a hollow sphere at stage 6. Cause: the RHS scatter
carries one padsc (padf**3 = 8) and the calibrated operator padsc**2
(Khat = padsc**2 x folded density), so the exact quotient b_hat/rho
sits padsc**2 = 64 above the solution's scale; CG absorbs that in the
preconditioner, the reverted M^-1 b attempt hid it with its least-squares
fit, and the NU references synthesized from a 64x reference destroyed the
matching. build_gridding_half now divides by padsc**2, and
test=pcg_recon gates the kept half against the converged streamed solve
(LS scale within 25% of 1, correlation above 0.9). Offline check of the
like-for-like selection (2026-09-08d) with the rebuilt Mac binary passed:
Sep 7 ML pair finest label 18.3% -> 0.09%, skip pattern gone on all PCG
pairs, July pairs' finest labels unchanged (5.3% at 3.98 A) with the mid
band one label coarser (7.96 A: 20% -> 10%), 10.7 s at box 300.
test=pcg_recon with the corrected gridding half: LS scale 1.248,
correlation 0.945 against the converged streamed solve (5 iterations to
rtol 1e-3); the residual 25% is the operator's CTF deconvolution beyond the
plain density quotient, a constant of the kernel, not a layout error.
Addendum (2026-09-08f): nu_input=ml, user design. The ML-regularized pair
is the sole competition input, no auxiliary member, on both backends
(nu_static_aux_replacement is false under it; the gridding assembly reads
the regularized pair whenever it is needed; both PCG paths pass ml_even/odd
as the base input and the base pair through the unused auxiliary slot). The
base pair keeps every other role. Validation: nu_input=ml with NU active
requires ml_reg=yes under euclid. The FSC cap still applies to all inputs
as a bound on the finest-label handoff; it can be retired once the ML input
is validated.
Run record (2026-09-08g): PfCRT abinitio3D on e45b237a (FSC cap, old argmin, rec_backend=pcg defaults), one restart. Structure correct: docked against the July final, cross-FSC 0.143 at 4.09 A, band means 0.81 (20-10 A) and 0.63 (10-7 A), the level of July's own restart-to-restart agreement (0.54-0.61); the stage 6/7/8 snapshots dock at 7.2/4.5/4.5 A. Resolution pinned: final FSC0143 4.50 A, FSC05 5.97 A, B -132; the handoff asked for 4.14 A from stage 7 iteration 3 and 3.98 A from stage 8 iteration 1 and was clamped to the 4.5 A NU-stage ceiling every iteration; the FSC sat at exactly 4.500 A for 30 iterations (this dataset's FSC tracks the matching band; July matched at 4.14/3.98 A and reached 4.37/4.3 A). Stage 6 stalled at 7.2-7.4 A with orientation overlap 0.11-0.14 and 11-13 degree jumps (July: 0.6 and 3.5 degrees at the end of stage 6) with the band at 5.0 A; the old argmin still over-populated the retained bank (stage 7 iteration 3, FSC 5.75: 17.7% at 5.97 A, 1.0% at 5.0, 2.3% at 4.14 -- the radius-4/4/3 skip pattern; stage 8: 30% at 7.96, 11% at 5.97, 10% at 5.0, 2-3% at 3.98 for a 4.5 A map). The like-for-like selection on this run's final base pair gives 19/7.4/8.1/0.85% instead of 32/12/11/3.2%. The FSC-anchored bank cap behaved as designed (handoff never beyond FSC/1.5). Controller changes: NU-stage lpstop ceiling removed (explicit lpstop only), minits=maxits retired (stage 8 overlap 0.96-0.997 for its last ten iterations with the FSC flat; streptavidin 2000 s vs 1300 s). Next run: like-for-like + uncapped handoff + early stopping, default input and nu_input=ml arms.
Run record (2026-09-09): f4dfb1bd (like-for-like selection, uncapped NU-stage handoff, early stopping), PfCRT. Base input, three restarts: 4.20 / 4.03 / 4.63 A; cross-FSC 0.143 against the July final 3.93 / 3.93 / 4.09 A, band means (10-7 A) 0.68 / 0.62 / 0.58, better than July's own restart agreement (0.54-0.61); within-run FSC never finer than the cross-run crossing (no overfitting signature); handoff walk 7.96 -> 5.97 (stage 6), 5.0 -> 4.14 (stage 7), 3.98 (stage 8); stage-6 histogram 72% at 11.9 A at FSC 8.6 A (July 82%). R3 never populated 4.14 A in stage 7, settled at the 5.0 A band (overlap 0.96) and early-stopped stage 8 after 7 iterations. nu_input=ml, three restarts: 8.4 / 9.1 / 8.4 A, cross-FSC against July 7.4 A; every NU stage early-stopped (9/7/3, 12/6/3, 4/5/3 iterations); labels drifted coarser each iteration (7.96 A from 35% to 1.7%, 11.9 A up to 65%) because P_tau is a global per-shell shrinkage driven by the global FSC and the under-converged replay re-applies it to its own warm start. Both nu_input options, the kept gridding half (set_keep_gridding_half/get_gridding_half), the auxiliary-slot bypass and the pcg_recon gate were removed the same day; the base pair is the competition input, as in July.
Record (2026-09-09b): one support, applied once -- gridding aligned to the deapodized, support-masked PCG products¶
User request: align the gridding FSC to the deapodized halves, mask the
gridding products so they are equivalent to the PCG reconstructions, and
stop applying the mask twice anywhere on the PCG path. Audit of every
mask3D_soft consumer of a reconstruction product:
| site | before | after |
|---|---|---|
gridding restore_gridding_pair |
FSC on undeapodized base halves (legacy parity); deapodized, unmasked halves written; merged map fills the box |
halves and _unfil halves deapodized then mask3D_soft(msk_crop, backgr=0.) (the PCG set_mask profile: 12 px cosine band centred on msk_crop); FSC on the masked finals; merged map masked in restore_merged_volume; sidecar solve_kind=gridding written |
evaluate_halfmap_pair |
mask3D_soft(msk_crop) on copies of both backends' halves (second application on PCG halves) |
no mask; spherical_mask_radius argument removed |
PCG warm starts (override_*_warm_start_from_previous) |
mask3D_soft on the previous half (already P u) |
none |
solver entry (solve, solve_accum) |
mask_mul(x): u0 = P x_prev = P^2 u_prev, exit x = P u -> P^3 per warm-started iteration in the band, compounding over a stage (with the strategy re-mask; P^2 without) |
mask_div(x): u0 = x/P where P >= PCG_SUPPORT_DIV_MIN = 0.1, zero below; exact for support-masked input, amplification capped at 10x on the outermost ~2.5 px for content not proportional to P (resampling ringing, foreign maps), which is regenerated from the data each iteration instead of compounding |
postprocess_volume_from_files |
no mask for rec_backend=pcg (by name), spherical/envelope mask for gridding (now a second application) |
no post-hoc mask for any volume with the support-provenance sidecar, or rec_backend=pcg; classical mask otherwise (imported maps) |
matcher mask_matching_reference |
mask3D_soft after Fourier filtering |
unchanged: reference preparation (compact support after ringing, user start volumes); P^2 in the band, deliberate |
Why the solver entry matters with two CG iterations: in the band the
constrained system P H P u = P b has the unconstrained solution x = H^-1 b
wherever P > 0, but the Krylov directions z = P M^-1 r vanish like P
there, so two iterations barely move the band and the output inherits the
start. With a shrinking start (P^2-P^3 per iteration) the cosine edge
hardened toward its inner end over a 20-30 iteration stage; with the exact
conversion the band keeps the accumulated corrections of all previous
iterations, which is what a warm start is for.
Consequences to expect in the backend comparison: the gridding FSC loses the
apodization taper's rim down-weighting (it was slightly optimistic at high
resolution), the gridding _unfil NU inputs now match the PCG support, the
gridding Guinier B-factor is estimated on masked _unfil halves like PCG,
and gridding _pproc maps are no longer masked a second time. The sidecar
moved to simple_halfmap_diagnostics (write/read_support_provenance,
file name unchanged). Uncompiled; user compiles and runs the comparison.
Record (2026-09-09c): backend comparison review -- shared observation, transactional sidecar, hard domain + window, cost records¶
Findings of doc/refactoring_notes/abinitio3d_reconstruction_backend_comparison_review.md
verified against the code and acted on:
| finding | verified | change |
|---|---|---|
| 3.1 cropped observations prepared differently | yes: gridding norm -> Fourier crop -> taper at crop box (prep_imgs4rec), PCG norm -> taper at native box -> native plane of the crop disk (three accumulation sites); no-crop order also differed (taper/norm) |
prep_rec_observation in simple_matcher_ptcl_io, called by prep_imgs4rec (crop step) and all three PCG accumulation loops; test=pcg_recon stage 13 parity gate |
| 3.2 sidecar lifecycle | yes: stage-boundary simple_rename of the state volume left <vol>_pcg_support.txt behind (base warm start refused at the next stage's first iteration), final/symmetric copies unpaired, sidecar written before the map |
copy/rename/remove_support_provenance in simple_halfmap_diagnostics, applied at the abinitio3D stage rename, final and symmetric copies, refine3D start-volume rename, noise starts and imports; PCG masters write the map before the sidecar |
| 4.1 wall time only | yes | per-part bench files with context and thread-seconds; master phase line with threads, thread-seconds and peak RSS |
| 4.2 support formulations not equal | yes: where 0 < P < 1 the converged constrained solution is the unconstrained one (P cancels), so the band depended on the CG state |
hard solve domain window > 0 + one soft output window (install_support, window_mul, window_div; PCG_HARD_SOLVE_SUPPORT toggle); stage 14 regressions |
The window band regression (stage 14a) asserts, over shells with window in [0.15, 0.85], that the constrained 40-iteration solve equals the windowed unconstrained solve within 15% RMS; 14b that six repeated two-iteration output-space warm starts keep the band RMS within 10%. Uncompiled; the user compiles, runs the test gate and the paired comparison.
11. The NU machinery as the prior infrastructure¶
The nonuniform-regularization machinery
(doc/policies/nonuniform_filtering_policy.md, src/main/nu_filt/) is the
mature evidence engine the prior now builds on directly. LocScale-2.0 derives
local confidence from a pseudoatomic reference; NU derives it from
cross-validated half-map prediction with an explicit noise model. The
2026-08-27 design joins the formerly separate envelope, local-resolution, and
graded-confidence ideas into one solver input.
11.1 Reuse without binary collapse¶
Reuse:
- the radially whitened symmetric cross-half Huber unary;
- the static candidate bank and accepted
nu_refineextensions; - spherical objective support and null-estimation discipline;
- ordered-label spatial regularization;
- selected-cutoff/local-resolution diagnostics and LP-set handoff.
Replace:
- coarsest-versus-best envelope margin as the sole prior statistic;
- binary MRF segmentation into molecular/solvent;
- component filtering, growth, cosine edge, and persistent envelope artifact;
- envelope complement and weighted-solvent-mean
Q_s; - a second post-replay NU analysis that can disagree with the prior state.
11.2 Required new evidence state¶
Add the null candidate and compact the full unary curve into ordered band
support/confidence before dmats_mask is released. The state is constructed
from current base halves between solve phases, not lagged from a previous
replay. It is shared read-only by the two half replays and by downstream NU
diagnostics. The base maps remain observable over the full spherical support,
so a weak domain excluded from the replay's useful bandwidth can still recover
evidence in a later refinement iteration.
Do not include the ML auxiliary replacement pair when constructing this state.
That pair is a post-hoc NU candidate under the current workflow and would let a
regularized result participate in estimating the precision that regularizes
it. nu_refine frontier evidence may participate only when the challenged
shell was evaluated from the unregularized base pair and accepted under the
recorded frontier contract.
11.3 Solver representation¶
The first operator is the compact multiscale frame of §5.3. Prefer broad bands with an explicit normalization and a cheap positive preconditioner approximation over an 8--16-label FFT stack. The selected cutoff is a diagnostic/hand-off value; the solver consumes graded band confidence, not a hard label mask. Measure the incremental value over post-hoc NU filtering: the in-solve prior participates in deconvolution and Krylov conditioning, whereas post-hoc filtering can only modify the finished estimate.
11.4 Feedback discipline¶
The envelope-specific lag-one shrinkage loop disappears, but resolution-field feedback remains because later particle alignment consumes the NU replay map. Track confidence/cutoff overlap and entropy across iterations, and retain the omitted/weak/coarse-domain recovery tests. The ordinary base-pair FSC remains the only resolution claim; shipped-pair correlation is diagnostic. Never feed the NU evidence field into FSC solvent correction or replace the spherical NU support with it.
12. Recommended reading order¶
Wilson 1942 [3]; Wilson 1949 [4]; Singer 2021 [5]; Gilles and Singer 2022 [6]; Wang 1985 [7]; Terwilliger 1999 [9]; Terwilliger et al. 2020 [10]; LocScale-2.0 [1].
References¶
-
A. Bharadwaj, R. de Bruin, and A. J. Jakobi, "Confidence-guided cryo-EM map optimisation with LocScale-2.0," Nature Communications, vol. 17, article 8778, 2026. doi:10.1038/s41467-026-75327-8
-
A. J. Jakobi, M. Wilmanns, and C. Sachse, "Model-based local density sharpening of cryo-EM maps," eLife, vol. 6, e27131, 2017. doi:10.7554/eLife.27131
-
A. J. C. Wilson, "Determination of absolute from relative X-ray intensity data," Nature, vol. 150, p. 152, 1942. doi:10.1038/150152a0
-
A. J. C. Wilson, "The probability distribution of X-ray intensities," Acta Crystallographica, vol. 2, pp. 318-321, 1949. doi:10.1107/S0365110X49000813
-
A. Singer, "Wilson statistics: derivation, generalization and applications to electron cryomicroscopy," Acta Crystallographica Section A, vol. 77, pp. 472-479, 2021. doi:10.1107/S205327332100752X
-
M. A. Gilles and A. Singer, "A molecular prior distribution for Bayesian inference based on Wilson statistics," Computer Methods and Programs in Biomedicine, vol. 221, article 106830, 2022. doi:10.1016/j.cmpb.2022.106830
-
B.-C. Wang, "Resolution of phase ambiguity in macromolecular crystallography," Methods in Enzymology, vol. 115, pp. 90-112, 1985. doi:10.1016/0076-6879(85)15009-3
-
K. D. Cowtan and P. Main, "Improvement of macromolecular electron-density maps by the simultaneous application of real and reciprocal space constraints," Acta Crystallographica Section D, vol. 49, pp. 148-157,
-
T. C. Terwilliger, "Reciprocal-space solvent flattening," Acta Crystallographica Section D, vol. 55, pp. 1863-1871, 1999. doi:10.1107/S0907444999010033
-
T. C. Terwilliger, O. V. Sobolev, P. V. Afonine, P. D. Adams, and R. J. Read, "Density modification of cryo-EM maps," Acta Crystallographica Section D, vol. 76, pp. 912-925, 2020. doi:10.1107/S205979832001061X
-
D. Kimanius, G. Zickert, T. Nakane, J. Adler, S. Lunz, C.-B. Schönlieb, O. Öktem, and S. H. W. Scheres, "Exploiting prior knowledge about biological macromolecules in cryo-EM structure determination," IUCrJ, vol. 8, pp. 60-75, 2021. doi:10.1107/S2052252520014384