Canonical Sigma2 State Refactoring¶

Date: 2026-09-04

Status: cut over on 2026-09-10. Canonical persistence is the only runtime sigma2 path across 2D, 3D, streaming, reconstruction, project merge, and Flex. Legacy files remain supported only at the explicit sigma2_convert boundary.

Purpose: single living design, implementation plan, and validation record.

1. Decision Summary¶

Replace partition-local sigma2_noise_partN.dat files and iteration-numbered sigma2_it_N.star files with one current sigma2_state.bin per particle-project lineage.

The canonical store contains the latest native-grid residual spectrum for each global particle row and the latest grouped model derived from those records. Its identity does not include nparts, worker number, execution mode, or iteration. Only the last committed generation is retained.

Runtime workflows neither read nor write STAR sigma files. A dedicated sigma2_convert command provides explicit backward-compatible import and export at the format boundary. It is not part of normal execution.

All new or invalid sigma state is initialized from particle power spectra. Existing particle records survive append operations; only new rows are initialized. Logical removal uses the project state flag. Physical reorder or compaction requires an explicit row map. A native box or smpd change invalidates the store; crop-only stage changes do not.

Distributed updates use a transactional candidate. Worker-exclusive files are the safe default. Direct writes to disjoint candidate ranges are enabled only on a filesystem/site configuration that has passed the production concurrency diagnostic.

Both current scientific policies remain supported: sigma_est=global and sigma_est=group.

2. Why This Model¶

The current in-memory distinction is valid:

per-particle residual spectra are the authoritative update state;
grouped even/odd spectra are derived models used by Euclidean matching and ML reconstruction/restoration.

The current file identities are not scientific requirements. Part files encode scheduler ranges, and iteration STAR files encode file-selection history. This makes restart depend on the old nparts and can corrupt the meaning of fractional updates if individual residuals are reconstructed from a group mean.

For an updated subset U, the correct group sum is

sum(i not in U) r_i + sum(i in U) r'_i

Replacing unchanged records by the old group mean is generally not equivalent. The canonical store therefore preserves every r_i; regrouping and repartitioning become independent operations.

3. Canonical Store Contract¶

Ownership and identity¶

builder%esig remains the runtime owner. A focused sigma-store domain API owns validation and state transitions; byte-level operations belong in src/fileio. Commanders and strategies orchestrate the lifecycle but never calculate offsets or parse the format.

The project registers the canonical path explicitly. Workflows must not scan directories for the highest-numbered sigma file.

The versioned header records:

magic, format version, scalar kind, native shell bounds;
particle count, native box, native smpd;
grouping policy, group count, and section offsets;
committed generation, provenance, state, and integrity metadata;
an order-sensitive particle-layout identity.

The layout identity is a digest of the ordered stable particle keys owned by the project (lineage plus normalized stack reference and stack index). The particle state flag is excluded so activation changes do not invalidate the layout. On append, the old rows are accepted only if their prefix digest matches the committed layout. A same-size reorder therefore cannot be mistaken for the old layout.

Stored data and invariants¶

The file contains:

a fixed header;
the current native-grid grouped even/odd model;
one fixed-size native-grid spectrum per physical particle row;
integrity information for the committed generation.

Global particle row is the only record key. nparts, worker ranges, and iteration are absent from the committed format. Inactive rows may remain stored but are excluded from consolidation and ordinary consumers.

Every committed store must satisfy:

header and file-size validation;
layout and native-grid compatibility;
complete, finite, positive active-particle records;
grouped curves equal to reduction of the committed particle records under the recorded grouping policy;
integrity validation for all committed sections.

The grouped section is always native-grid. Cropped consumers select the needed shell interval through the existing grid-adaptation path.

4. Lifecycle Policy¶

Initialization¶

A missing or invalid store is rebuilt with calc_pspec for every active particle, followed by grouped reduction. This applies to a native-grid mismatch and to a reorder for which no trusted map exists.

Canonical workflow bootstrap checks validate the registered header, grid, layout, grouping, and committed state. Existence of an iteration STAR is not a runtime criterion.

bootstrap_rec3D requires an associated particle project. It initializes the canonical state through calc_pspec, reconstructs a bootstrap map on that seed, runs a residual-only refinement pass, and commits the resulting per-particle records before producing the shipped reconstruction. A Euclidean ML-regularized reconstruction with no populated particle table has no canonical sigma basis and is unsupported; there is no transient half-map sigma fallback.

Updates and grouping¶

Workers read and write global particle ranges. A full update replaces every active record; a fractional update replaces only selected records and copies all others unchanged into the candidate. The previously committed generation remains visible through current-iteration assembly. After those consumers, the master reduces active particle records into global or group curves and commits the complete generation for the next iteration. At the abinitio3D symmetry boundary the final commit is deferred through symmetric reconstruction.

Particle-set changes¶

Append: verify the old-layout prefix, copy old records, run calc_pspec only for the new suffix, regroup, and commit.
Logical removal: keep the record but exclude state=0 rows from grouping and consumption.
Physical compaction/reorder: rewrite through the explicit old-to-new map produced by the owning project operation. Rebuild if no trusted map exists.
Native-grid change: rebuild all records; persistence migration never interpolates shells.

5. Distributed Write and Recovery¶

The committed sigma2_state.bin is immutable. Every update targets the generation-scoped candidate sigma2_state.g<N>.next (N = the generation the update commits; worker ranges are sigma2_state.g<N>.part<NN>.range, both since 2026-09-07); only the master may create, size, validate, publish, or discard it.

Protocol:

The master removes a stale candidate, creates a new exclusive candidate, copies the committed generation when unchanged rows must survive, records the scheduled global ranges, flushes, closes, and launches workers.
Each worker writes exactly its assigned records, flushes and syncs its output, closes it, and only then emits the normal completion sentinel.
After the distributed barrier, assembly and any other reconstruction owned by the current iteration consume the previous committed generation.
The master then opens fresh handles, verifies exact non-overlapping range coverage and checksums, derives the grouped section, and validates the complete candidate.
The master flushes and syncs the candidate, atomically renames it over the committed file, and syncs the containing directory. A failed validation leaves the previous committed file untouched. The symmetry-search stage delays steps 4-5 until its symmetric reconstruction has completed.

The implemented range-I/O mechanism is:

local (safe default): each worker writes one exclusive temporary range file; the master validates and merges them. This avoids concurrent multi-client writes and cache interactions. These files are candidate material, not persistent state, so their names may contain worker/range information without making the committed state depend on nparts.
direct remains a possible later optimization: workers would use GFortran unformatted stream write(pos=...) on disjoint candidate ranges. It has no activation path in this refactor. Adding one requires a production concurrency diagnostic for the exact filesystem and mount configuration.

All current workflows therefore use local; there is no workflow parameter that can opt into unvalidated shared-file writes.

The implementation provides explicit low-level support for flush -> file fsync -> close, atomic rename without the delete-first behavior of simple_rename, and directory fsync. Filesystem probing and mount reporting remain optional conveniences, not substitutes for a future direct-I/O diagnostic.

6. STAR Compatibility Converter¶

sigma2_convert is a dedicated command and module with no dependency from the runtime sigma-store API. It supports:

Exact legacy import: assemble a canonical store from a complete set of legacy part files, using their recorded global ranges, and validate exact coverage. An accompanying grouped STAR may be checked against the derived model but is not authoritative.
STAR-only import: expand the STAR grouped curve according to the current project group membership to seed per-particle records, then derive and validate the canonical grouped section. The command labels this conversion as lossy and records provenance=star_group_seed.
STAR export: write the current grouped section to a user-selected STAR path for older tools. Export does not create iteration history or change canonical state.

Import requires an explicit project and input path; export requires an explicit output path. The converter never runs automatically, searches for iterations, or silently replaces a valid store. If conversion is not requested, an old project follows the normal power-spectrum initialization path.

7. Rollout and Implementation Plan¶

The canonical path was introduced as a typed opt-in for validation. After the maintainer gate, the temporary selector and legacy runtime implementation were removed. Canonical state is now unconditional; the converter remains.

Store and safe transactions — complete: API, header/layout digest, local range merge, integrity and recovery, and explicit conversion boundary. Direct shared-file writes are deferred as an optional optimization.
Initialization and reduction — complete: per-particle power spectra, prefix-preserving append, blockwise reduction, and bootstrap identity checks.
3D migration — complete: matcher full/fractional updates, refine3D variants, external-reference emission, and gridding/PCG reconstruction.
2D/restoration migration — complete: cluster2D, abinitio2D checkpoint paths, probabilistic assignment, and class-average restoration.
Streaming/secondary migration — complete: isolated chunk/pool lineages, safe dynamic-pool rebuild, project concatenation, Flex, cleanup, retention, and project-output behavior. Nano remains correlation-only and therefore has no sigma state to migrate.
Cutover — complete: canonical state is the only runtime path. Legacy runtime I/O and its CLI selector are removed; sigma2_convert remains the compatibility boundary.

Implemented scope¶

The canonical implementation provides:

a versioned binary store with fixed offsets, record and section integrity, order-sensitive particle-layout identity, deep validation, candidate copying, exact local-range coverage, blockwise even/odd group reduction, file and directory sync, and atomic publication;
explicit project registration with unconditional canonical persistence;
shared-memory and distributed transactions for 2D and 3D matchers, including fractional updates, update_missing, probabilistic modes, and the optional CC residual-emission path;
power-spectrum initialization and committed-state recovery for abinitio2D checkpoint continuation, particle abinitio3D, abinitio3D_cavgs, refine3D variants, direct reconstruct3D, bootstrap_rec3D, and Flex PCA;
canonical loading for class-average restoration plus gridding and PCG reconstruction at native and cropped working grids;
isolated state ownership for stream chunks and changing pools. Normal append preserves an identity-verified prefix; an arbitrary dynamically selected pool without an explicit old-to-new map is rebuilt from particle power;
exact row-wise canonical concatenation and regrouping in chunk aggregation and merge_projects. Mixed canonical/legacy general project merges discard the inherited registration so a later workflow rebuilds it rather than consuming a stale first-project path;
an explicit sigma2_convert developer command for exact legacy part import, lossy grouped-STAR import, and grouped-STAR export, with project identity, native-grid, grouping, and exact-range checks;
canonical persistence on the owning 2D, 3D, stream, reconstruction, and Flex programs. Normal workflows do not create legacy part files or iteration-numbered sigma STAR files, and expose no persistence selector.

simple_test_sigma2_state covers both grouping policies, prefix identity, candidate preparation, exact range merge, commit publication, corruption and coverage rejection, and preservation of the prior committed generation.

The safe local-range protocol is the completed production implementation. Direct multi-client candidate writes are deliberately deferred as an optional performance project; they are not required for canonical semantics or cutover.

8. Acceptance Criteria¶

Storage and recovery:

changing nparts, numlen, or shared/distributed mode reuses the same committed state;
patterned disjoint writes pass repeated multi-process checks on every direct-enabled filesystem, including boundaries inside pages/stripes;
missing, overlapping, truncated, or interrupted ranges reject the candidate and preserve the prior file byte-for-byte;
unknown or failed filesystem validation selects the local merge path.

Scientific equivalence against the pre-refactor baseline:

full and fractional updates preserve updated and untouched particle rows;
sigma_est=global and sigma_est=group reproduce grouped even/odd curves;
Euclidean scores, assignments, and sigma-weighted reconstruction/restoration inputs agree within the existing numerical tolerance;
probabilistic modes retain matcher-owned residual updates; objfun=cc and ml_reg retain their present consumption gates.

Lifecycle and compatibility:

append preserves the old prefix and initializes only new rows;
inactive rows, explicit compaction maps, untrusted reorder, native-grid changes, and crop-only stages follow Section 4 exactly;
an equal-size reordered project is rejected by the layout digest;
exact legacy import, lossy STAR-only import, and STAR export are tested;
normal workflows create no part or iteration STAR sigma files;
every listed 2D, 3D, streaming, and secondary consumer uses the canonical API.

9. Validation Record¶

On 2026-09-04, source-only checks for the first implementation slice passed:

git diff --check;
repository Fortran source-index generation, including both new modules;
command/UI registration audit, with only the three pre-existing unrelated name mismatches.

The maintainer subsequently reported that simple_test_sigma2_state passes. The first simple_test_units run exposed that simple_abspath(..., check_exists=.false.) does not reliably resolve a future file on this platform. Registration was changed to construct an absolute path from the current directory when the target does not yet exist, and the maintainer then reported that simple_test_units passes. Record tested operating systems, filesystems and mount options, scheduler/backend, direct/local mode, numerical baseline revision, and results here before cutover.

After adding the base-refine3D opt-in, the source-only checks were repeated:

git diff --check passed;
repository Fortran source-index generation completed and the generated module graph remained acyclic;
the command/UI registration audit again reported only its three pre-existing unrelated name mismatches.

On 2026-09-05, after the base-refine3D plumbing and the added update-preparation coverage, the maintainer reran simple_test_sigma2_state. It completed with SIMPLE_TEST_SIGMA2_STATE NORMAL STOP.

Before attempting a production 3D run, the validation order was changed to start with a laptop-scale abinitio2D comparison. The refine3D UI opt-in was removed, and the canonical path was connected to shared-memory abinitio2D, its in-memory cluster2D iterations, and final class-average regularization. Source-only checks for this pivot passed: focused git diff --check, Fortran index generation, and an acyclic generated module graph; the UI audit retained only its three pre-existing unrelated mismatches. A concurrent user edit in doc/algorithms/abinitio2d.md has separate trailing whitespace and was left untouched. The maintainer subsequently reported that the canonical shared-memory abinitio2D run completed gracefully and that its class averages look excellent, providing the scientific gate for the distributed slice.

The distributed cluster2D path now uses the same canonical transaction as the shared-memory path: master candidate preparation before worker dispatch, partition-local range production, and exact-coverage master consolidation and commit. At that intermediate stage checkpoint resume was still explicitly gated. This distributed slice passed focused git diff --check, Fortran source-index generation with an acyclic module graph, and the UI audit with only its three pre-existing unrelated mismatches. The maintainer subsequently reported that the distributed test passed.

The first abinitio3D_cavgs exposure then passed focused git diff --check, Fortran source-index generation with an acyclic module graph, and the UI audit with the same three pre-existing unrelated mismatches. Compilation and runtime validation were left to the maintainer, and docked multi-state canonical mode was still explicitly gated for that first 3D slice.

The maintainer's first distributed abinitio3D_cavgs run reached normal refine3D and map-symmetrization completion, then exposed a legacy-only sigma file check in the standalone gridding reconstruction child. The shared load_sigma2_groups boundary now accepts the project explicitly: canonical mode resolves the registered committed state, validates native grid, ordered layout and grouping identity, and loads its grouped curves; legacy discovery and STAR loading remain unchanged. All gridding and PCG reconstruction callers were updated through that common boundary. Focused git diff --check, Fortran index generation with an acyclic module graph, and the unchanged UI audit passed. The maintainer subsequently validated both shared-memory and distributed abinitio3D_cavgs execution with the canonical store, including the standalone reconstruction consumer that exposed the original failure.

Post-run artifact inspection found one remaining sigma2_it_98.star. The final original-sampling reconstruction was rebuilding its child command line from selected parameters and omitted the then-required store selector; its missing-STAR check therefore entered the legacy half-map bootstrap and wrote an iteration STAR before the regularized reconstruction. The final reconstruction was changed to propagate the selector and sigma_est. The current canonical-only path validates the registered store against the original native grid, project layout, grouping, and committed state. A valid store is consumed directly; an invalid native identity is rebuilt from the associated particle project with calc_pspec, as required by Section 4. That legacy bootstrap was subsequently removed at cutover.

Focused git diff --check, Fortran source-index generation, and an acyclic generated module graph passed after this correction. The maintainer subsequently rebuilt and reran the canonical workflow and confirmed the clean-artifact result with no sigma2_it_*.star created.

The maintainer then validated an independent multi-state canonical abinitio3D_cavgs run. The remaining class-average 3D path was the docked split. Its old pre-reconstruction calc_group_sigmas call exists only to materialize the legacy iteration STAR from already current particle sigmas. Canonical state needs no equivalent mutation because split relabelling changes neither ordered particle identity nor global/stack membership. Canonical docked mode now skips that legacy barrier and reuses the committed generation for the split reconstruction; the following matcher creates and commits its normal candidate transaction. Source-only validation passed, but shared-memory and distributed docked runtime tests remain outstanding.

The integration pass then removed the remaining artificial workflow gates and completed canonical support across the current runtime surface. This added abinitio2D checkpoint recovery, refine3D sparse/update-missing and CC emission transactions, direct reconstruct3D and bootstrap initialization, stream chunk/pool ownership, prefix-preserving append, exact canonical state concatenation in both chunk aggregation and merge_projects, the explicit converter, and Flex initialization. At that validation stage, canonical selectors were exposed on the owning workflows pending the consolidated matrix.

After the integration pass, source-only validation completed successfully:

focused git diff --check for src/main, src/fileio, and the sigma test;
repository Fortran source-index generation;
an acyclic generated module graph;
command/UI registration audit with only the same three pre-existing unrelated name/instance mismatches.

Compilation and runtime execution were not performed by the agent, in accordance with repository policy. The maintainer will run the consolidated matrix in Section 10.

On 2026-09-10 the cutover removed the CLI selector, its typed parameter and derived logical, every runtime legacy branch, iteration-STAR discovery and partition-file propagation, and the stream sigma-directory handoff. The explicit STAR/legacy-part import and STAR export converter remains intact.

A post-cutover source review found no correctness regression and removed what the cutover had stranded: the legacy grouped-STAR helpers and their unit test in simple_euclid_sigma2 (the converter keeps write_groups_starfile and read_sigma2_groups_file), the caller-less imgkind=sigma2 os_out accessors on sp_project, the sigma2_noise_part / sigma2_it_ name constants, and a dead iteration-STAR symlink in the rec3D-backends test runner. which_iter on bootstrap_rec3D and the endit + 2 offsets in refine3D_auto and abinitio3D only number the residual sigma pass and its iteration files; the committed state carries no iteration number, and the comments now say so.

10. Outstanding Maintainer Test Matrix¶

The earlier validation record captures incremental pre-cutover gates. The following post-cutover matrix remains open until results are recorded here.

Build the changed executables and run simple_test_sigma2_state plus the normal unit suite.
Run canonical abinitio2D shared and distributed, then resume a checkpoint at a later stage.
Run canonical particle abinitio3D and the already established abinitio3D_cavgs cases, including docked multi-state if available. For the Streptavidin regression, verify the final symmetry-stage commit is deferred, the symmetric reconstruction completes, and the commit follows immediately; repeat the canonical run at least ten times against the established baseline.
Run canonical refine3D shared and distributed with a fractional update and update_missing=yes; cover an external-reference CC initialization path.
Run direct canonical reconstruct3D and bootstrap_rec3D from a project with no registered state and verify that power-spectrum initialization occurs.
Run a small canonical stream through chunk and pool updates, including one pool membership change and one append.
Exercise sigma2_convert exact part import, grouped STAR export/import, and refusal to overwrite an existing canonical output.
Merge two canonical chunk projects and two canonical general projects; verify the merged project owns a new state and runs with a different nparts. Also merge a mixed canonical/legacy pair and confirm that no stale state path is retained.
Run canonical Flex PCA once and confirm it either reuses a matching state or initializes one from particle power.
For every workflow, confirm that no sigma2_noise_part*.dat or sigma2_it_*.star runtime artifacts are produced and that changing nparts does not change the registered committed state identity.