Caching and Provenance

BioImageFlow caches the output of every node. When you re-run a workflow, only nodes whose inputs or parameters have changed are recomputed.

How caching works

Each node has a result key under storage_path/cache/v1/. The result key identifies the reusable cache selection for the node’s logical inputs, parameters, environment, and upstream cache state. Each successful execution publishes an immutable record, and current.json selects the record that cache hits must use.

The implementation also computes a diagnostic logical signature from:

• Tool class name and version

• Environment hash (dependencies)

• Resolved parameter values

• Selected upstream record references when available, with diagnostic fallback material while a plan is pending

• Source-code hash (dev_mode only)

This signature is exposed as NodePlan.logical_signature for diagnostics, but it is not the public cache identity. Use NodePlan.final_result_key and NodePlan.selected_record_id for cache/provenance state. If current.json selects a valid record for a node’s final result key, the cached DataFrame is loaded instead of re-executing the tool.

Note

dev_mode=True adds the tool’s source code hash to the signature, so editing process_row invalidates the cache. Leave it off in production — only the package version should matter then. Pass it as wf.compute(target, dev_mode=True) or wf.plan(dev_mode=True).

Cache location

Results are stored under the storage_path you pass to Workflow:

bif_data/
└── cache/
    └── v1/
        └── results/
            └── <result-shard>/
                └── <result-key>/
                    ├── current.json
                    └── records/
                        └── <record-id>/
                            ├── dataframe.parquet
                            ├── manifest.json
                            └── assets/

Immutable records are not deleted by normal invalidation. Invalidation removes cache selection state such as current.json.

What invalidates the cache

Any of these changes produce a different result key:

• Parameter change: e.g., sigma=1.0 to sigma=2.0

• Upstream change: if a parent node’s hash changes, all descendants recompute

• Tool version change: updating the package version of a tool

• Source code change (dev_mode only): modifying the tool’s Python source

External file references are path-based in the current cache. If an input file is modified in place without changing its path, the cache may still select the existing record. Change the path, change an explicit parameter, or call Workflow.invalidate(...) when file contents change at a stable path.

Inspecting cache state without executing

plan() returns a per-node plan without running any tool code — useful for GUIs that want to render cache state, or for scripts that need to see what compute() would do:

from bioimageflow import NodePlanStatus

plan = wf.plan()                          # dict[str, NodePlan]
for name, entry in plan.items():
    print(name, entry.status, entry.final_result_key, entry.selected_record_id)

Each NodePlan carries:

• node_name — scoped name ("outer/inner_1" for nested workflow tools)

• final_result_key — result key when all required upstream selected records are known

• selected_record_id — selected immutable record ID when the node is cached

• status — one of the five NodePlanStatus values below

• upstream — scoped names of this node’s direct upstreams

• pending_upstreams — upstream nodes whose selected records must be produced before this node’s final key is known

• logical_signature — diagnostic logical signature, not a cache key

Status	Meaning	Suggested UI affordance
CACHED	current.json selects a valid reusable record for the final result key; compute() would short-circuit if it consumes the same upstream records.	Green / “up to date”
PRIOR_SELECTION_MISS	The planned final result key has no selected record, but the same node has another selected current record.	Yellow / “needs rebuild”
UNEXECUTED	No reusable record exists yet for this node/result lineage.	Grey / “not yet run”
PENDING_UPSTREAM	At least one upstream selected record is not known yet, so the final result key cannot be reported.	Grey / “waiting on upstream”
SKIPPED	The node is disabled, or has a disabled upstream that prevents execution.	Struck-through / muted

A workflow node aggregates its internals: it reports CACHED only when every internal entry is cached.

plan() does not start Wetlands worker pools. It uses the direct planning path and does not run tool code.

Invalidating cache when a parameter changes

When a workflow lives in a long-running process (a GUI, a server) and an upstream parameter changes, you may want to clear cached results explicitly rather than rely on hash drift to trigger re-execution. invalidate() clears the cache selection for the named nodes — and, by default, the selection of every node transitively downstream:

cleared = wf.invalidate(["segment"])      # cascade=True is the default
cleared_nodes = {selection.node_name for selection in cleared}

wf.invalidate(["segment"], cascade=False) # just "segment"

The return value is a set of InvalidatedSelection entries for current.json pointers that were actually removed. Each entry reports the node name, result key, selected record ID when it was readable, and whether the pointer was removed normally or as corrupt metadata. Passing an unknown name raises KeyError.

Warning

invalidate() is not safe to call concurrently with compute() on the same workflow. The library does not currently expose a public lock primitive; callers that need to invalidate while a compute is in flight must coordinate externally (cancel + join + invalidate).

Forcing re-execution

The recommended recipe is invalidate():

wf.invalidate(["segment"])  # invalidates segment + everything downstream
result = wf.compute(target)

Changing a parameter value works as a fallback — even a trivial change produces a new cache key.

Cache cleanup

Automatic deletion of published cache records is not part of runtime execution. Published-record pruning is an explicit storage maintenance operation.