Build Cache and Store

Sigil keeps content-addressed build inputs and outputs in a store. Reusing a verified object avoids recompiling it; changing an input or toolchain creates a different object instead of overwriting the old one.

Default location

The default store is:

1. $SIGIL_STORE, when set
2. $XDG_CACHE_HOME/sigil/store
3. $HOME/.cache/sigil/store

SIGIL_STORE is an explicit override. Sigil expects that location to be writable and reports an error immediately when it is not.

Read-only stores in sandboxes

Some sandboxes, including managed Codex workers, expose the normal user cache as readable but not writable. Sigil detects this by attempting a small write; it does not rely on Unix permission bits, which may not reflect sandbox policy.

When the default store is read-only, sigil build automatically uses two layers:

• the existing default store as a read-only fallback
• a writable store under /tmp as the upper layer

Reads search the upper layer first and then the fallback. New content, build records, locks, and garbage-collection roots are written only to the upper layer. No Codex configuration or environment variable is required, and the build prints both paths when it activates this mode.

The upper path is stable for a given default-store path, so workers that share the same /tmp can also share the overlay. It is named like:

/tmp/sigil-store-overlay-v1-<store-path-hash>

/tmp is normally ephemeral. Losing the overlay is safe: Sigil will recreate it and can still reuse verified objects from the fallback. It may, however, need to rebuild objects that existed only in the overlay.

Integrity and concurrency

Every object is verified against its content hash before Sigil publishes it from either layer. A corrupt object in a read-only fallback is treated as a cache miss and is not modified. Sigil can then rebuild the object into the writable layer.

Writers coordinate through per-content locks in the writable store. A verified cache hit does not acquire a lock, which also permits reuse from an immutable store. Permission failures are reported as errors rather than being mistaken for lock contention.

Package reuse and bytecode outputs

An unchanged package can skip its build task using a manifest under the configuration's .sigil-package-cache directory. For each registered bytecode artifact, schema 2 records the role, path, size, and SHA-256 of the required outputs: .sgb bytecode, .json documentation, and .sgb.sgci compiler-input metadata, plus the paired .sge effect proof for a module. Sigil hashes these output bytes on every package-reuse check; an unchanged size or modification time alone does not establish integrity.

Package input files are also freshly content-hashed before reuse. A linked regular file includes both its link spelling and its target bytes. Broken links and links to directories decline package reuse, as do packages that depend on an input root whose identity cannot be established. Independent packages can still reuse their own manifests. This check assumes inputs remain stable during the build; it does not create a concurrent filesystem snapshot.

A missing or changed required output causes the package task to run. The ordinary bytecode action can still reuse the lower store: it restores verified bytecode, documentation, and module effect proofs, updates the documentation's source path for the current checkout, and regenerates compiler-input metadata. Restored effect proofs must still validate against their bytecode and dependency inputs. A malformed effect proof is an error; cache reuse does not silently replace it. Old or malformed package manifests miss safely and are replaced after a successful complete build. No store reset is needed.

Optimizer summaries (.sgs) are optional. Their absence does not prevent package reuse. The optimizer validates existing summaries before consuming their facts; a missing or invalid summary uses local analysis instead. A lower store restore removes the destination's previous summary because that restore does not publish a summary for the restored bytecode.

Custom or legacy actions that register bytecode without all required outputs can still build successfully, but do not save a reusable package manifest. Other artifact groups retain their existing validation rules. Restoration writes individual files; this contract does not provide atomic publication of an entire output set to concurrent readers.

Bytecode input identity

The supported cache path is a fresh, direct sigil build invocation, including standalone packages normalized to a workspace. Default source builds and known compile-sigil-module / compile-sigil-modules task steps describe their current source/output mappings. Effective target features are selected before extracting imports. The dependency graph and compiler use the same ordered library roots. Source and transitive macro/reader inputs contribute content identities, so a same-size edit with restored modification time still invalidates reuse.

The modeled declaration positions are the import clauses of a file's one define-library or library form, including those inside a library-level cond-expand, and, for a script-style file with no library form (a top-level (import ...) followed by expressions, loaded at runtime rather than imported), its top-level import forms and those inside a top-level cond-expand. Quoted data is not treated as executable code.

A source with a declaration form the extractor cannot model compiles fresh and declines reuse; it is not a build failure. (0.22.1 failed the build on these forms, which rejected script-style files under src/; 0.22.2 returned them to a fresh compile.) The build prints one UNCACHED <file>: unmodeled dependency declaration or feature condition: <form> line naming the innermost unmodeled form, and --verbose prints the same form on the package's NO-CACHE line. Unmodeled forms include library-availability conditions such as (library ...), include declarations, a second library form in one file, and explicit imports outside the modeled positions (procedure bodies, parameter default expressions, a top-level import next to a library form). The unmodeled form's own dependency is not a graph edge, so the compile order does not account for it; the file's modeled import clauses remain edges.

The package's bytecode plan cannot be established, so its manifest cannot authorize a skip until the unmodeled form is gone. The package's other files keep the per-file bytecode store: each key names its own inputs, including the freshly compiled .sgb of every dependency in the package, so an importer of the unmodeled module is restored only while that module compiles to the same bytes (until 0.22.10 the store was off for the whole package). The file itself never registers a producer proof, so a consumer in another package that imports its module also declines reuse. Ordinary unknown compiled-artifact provenance still causes a cache miss and fresh compile when the dependency declarations themselves are supported.

Alternate-reader (#lang) outputs execute normally but decline package and bytecode-store reuse, and do not establish producer authority. The reader implementation's imports do not prove imports in the syntax it returns. Useful reuse for these outputs is deferred until generated dependencies are provable; this does not establish ordering or VM isolation for arbitrary reader output.

Compiled dependency metadata is a hint, not evidence that an arbitrary external artifact has a complete dependency list. Successful known source actions and freshly verified package results establish invocation-local producer records. These bind source, bytecode, metadata, logical module name, and selected imports. Missing, malformed, stale, or unknown compiled-only inputs decline reuse. Conflicting module producers or replacement generations cannot renew authority in a VM that may retain an earlier module. Early standard-library actions may therefore compile before sufficient source-producer evidence exists; complete unchanged packages and qualified downstream modules can still reuse results.

The compiler identity includes the running executable's bytes and supported startup library contents, regardless of the reported version. A bundled CLI binds its archive through the executable. A development CLI binds the complete fixed sibling lib installation: sorted .sgb, .sgs, and .sgci membership and bytes, including hidden entries and regular-file symlink targets. Source shadows, native extension files, unlisted/unreadable directories, unsupported links, and arbitrary startup overrides decline bytecode reuse. This installation identity is computed once per build and assumes those inputs remain stable for the invocation; it is not a concurrent snapshot.

Arbitrary task steps (a lambda or other step the planner cannot describe) still run, and their package manifest does not authorize a package skip. That uncertainty also disables dependent package skips. Module caching stays on around them: each module's key names its own inputs, and a step that only writes files the key does not name cannot change what the module compiles to. The build does check what each such step leaves in the build VM. It snapshots every registered module (binding count, macro count, and a counter bumped when a macro is replaced) before and after the step. If the step registered a new module the compiler does not own, or added a binding to or replaced a macro in any existing module (the compiler's own included), the build prints NO-CACHE a task step registered or redefined <module> and nothing else it compiles is cached: no bytecode store lookups or stores, no native object reuse, no build-action cache and no package manifests, for the rest of that build. A later compile could otherwise expand the step's in-memory code while its key names the files on disk.

The top-level environment package manifests run in, (sigil user), is the one module exempt from that check (it is the list +step-environment-modules+ in (sigil build)). The exemption holds because no cached module can see it: a module that imports (sigil user) is declared uncacheable, and so is a module whose macros, while expanding, evaluate code in another module (eval-string-in-module), or look up by name, or read a binding of, a module outside the expanding module's import closure (find-module, load-module, an eval'd import, %hygiene-ref, module-ref). Lookups inside that closure, such as the hygienic references the expander itself emits into the macro's own module, are not counted: the module's key already names every module in it. eval and interaction-environment in a transformer see the module being compiled, not (sigil user). Package manifests are always evaluated in (sigil user), including when a task step loads one on a release-config (native-backend) compiler. Known gap: redefining the value of a binding a module already has (set!, or define of an existing name) stores into the binding's cell with no hook, so the check does not see it; the unit test test-task-step-guard.sgl and gate case TS carry it as an expected failure. Running arbitrary steps in a fresh VM or subprocess would close it.

Native action hooks keep their existing behavior; this contract does not establish complete native header, external tool, or arbitrary action identities.

Programmatic/repeated workspace builds, watch reuse of a compiler VM, and direct bytecode actions without qualified cache hooks compile afresh instead of using mtime-only skips. This can cost incremental performance for direct task users. The declaration checker does not expand macros to discover dynamically generated imports or arbitrary compile-time file I/O. The policy does not unload modules or isolate undeclared VM, filesystem, or environment mutations performed by arbitrary task code. Action isolation and atomic snapshot/publication are separate concerns.

Checking input invalidation

After building the development CLI from the current sources, run:

make test-cache-inputs

This explicit integration target runs the compiled dependency, source producer, and compiler identity regressions. Each harness copies the complete unbundled development tool and uses fresh fixtures, output directories, and stores. The compiler identity control requires an unbundled executable so it can change executable bytes while preserving the reported version. The target prints the parent evidence directory; each harness retains its own result and raw logs. CACHE_INPUT_TIMEOUT sets the per-process limit in seconds (default: 600).

The controls cover unknown or misleading compiled dependency metadata, same-size source edits with restored mtimes, feature-selected restoration, changed compiler bytes, and useful reuse after unrelated changes. They are a separate, slower gate from the Scheme tests in make test-build.

Choosing another store

Set SIGIL_STORE when you deliberately want one writable store:

SIGIL_STORE=/work/cache/sigil-store sigil build

Because this is an explicit choice, Sigil does not silently add a /tmp overlay if the selected path is unwritable. This makes configuration mistakes visible and keeps CI behavior predictable.