The store#

Point a policy at a path and the same sweep gains a cache, bounded memory and resumability, with no change to the function or the contract:

result = layer(space, policy=SweepPolicy(store="runs/layer.zarr"))

That is one mechanism, not three. Results stream into a zarr store as they land, one region per point, and a status variable rides along saying what happened to each.

status#

code

label

meaning

0

pending

never attempted

1

ok

computed and stored

2

failed

the call raised; the value is nan

3

skipped

excluded by skip_where

It comes back with the result and is written to the store, so “what did that run actually do” is answerable without re-running anything, including while the run is still going: read the status variable from the store from another process.

Resuming#

Only points without an ok status are computed. Interrupt a long sweep and call it again with the same store; a fresh run against a finished store makes zero calls and costs a read.

layer(space, policy=policy)  # 60 calls
layer(space, policy=policy)  # 0 calls

Because the points that succeeded are read back rather than recomputed, they are bit-identical rather than merely close. With a stochastic engine that is the difference between a cache and a coincidence.

What a store is tied to#

A store carries a fingerprint of the contract, the version string and the statics, and refuses to open under a different one. This is deliberate: it is what stops results computed before a bug fix from mixing silently with results computed after it. Bump version when the physics changes, and keep one store per version; idioms shows the comparison that makes possible.

Extending an axis and resuming is refused rather than silently misaligned. There is no store reindexing.

One writer at a time#

A lock enforces one writing run per store. Readers are never blocked. Since a dead process on another host cannot be probed reliably, force_unlock=True is the honest override rather than a staleness heuristic.

Multi-process cooperation, where cluster tasks claim points atomically, is out of scope. The documented answer is to partition the space at submission, one store per task, and merge afterwards.

The chunk grid#

The store’s loop-dim chunk grid is sized from a memory budget (64 MB by default), not pinned to one point per chunk, and writes are buffered into it. A size-1 grid would turn every point’s write into a read-modify-write of that whole chunk, which measures about eighteen times slower on a 40000-point sweep. Override it per dim with store_chunks={"y": 256} if you need to.

The trade is that a wider chunk keeps more computed-but-unflushed points in memory. Nothing un-flushed was ever persisted, so a hard kill loses only what the open chunk was holding, and it is recomputed rather than lost. A PointFailed raise flushes every open chunk before propagating.


Worked in the store and failures.

Deduplication#

A satellite scene rarely has as many distinct parameter rows as it has pixels. Retrieval products come off a classifier or a coarse grid, so neighbouring pixels repeat.

result = layer(space, policy=SweepPolicy(dedup=True))

Identical input rows are collapsed before anything is called, each distinct one is computed once, and the results are expanded back onto every position that shares them.

What it costs and what it buys#

The plan reports the distinct-row count before running:

SPACE
y      tau, ssa      300
x      tau, ssa      300
points              90000
dedup  60 unique (89940 duplicates)

With an expensive engine, removing 89940 calls is the whole story. The duplicate positions still have to be filled, which is a gather in memory and a batched write pass into a store, and that cost does not go away. So:

  • expensive callee, repeating scene: near-total saving;

  • free callee: deduplication is pure overhead, and the plan tells you so before you pay for it.

The distinct-row count is a property of the scene rather than of its size, so the saving grows as the image does.

It never changes a value#

Deduplication is a cost decision. It moves no value, no dim and no shape, and a parametrised suite enforces bit-identical results across policy combinations as a release gate. Duplicated positions are copied, not recomputed, so even a stochastic engine gives exactly equal results with and without it.

That property is also a trap worth knowing about: repetitions that are meant to differ have to differ in their inputs, or deduplication will correctly collapse them into one. That is what the seed carrier variable in idioms is for.

What it does not do#

Unique rows are extracted in memory. There is no streaming deduplication, so a space too large to hold its own loop variables is out of scope.


Worked in dedup on a scene.