SQLite cache
patchula-cache uses rusqlite 0.41 with bundled SQLite, so the system
SQLite’s version and features never matter. HTTP bodies are stored with Zstd
level 3.
Layout
Section titled “Layout”A cache is a directory. Cache::open_dir creates it and opens
cache.sqlite (with SQLite’s -wal and -shm files) inside it. The partial
clones for lockfile tools go in worktrees/ and the tool store in tools/,
unless PATCHULA_TOOLS_DIR moves it. The package caches the tools share
between runs go in tool-caches/, partitioned by the credentials a run
holds: public for a run without any (safe to share because repositories
that commit credentials are refused), private-<hash of the entries' scopes> otherwise, with no read-only lower layer between them. Pruning is
throttled per cache root and takes tool-caches/prune.lock so concurrent
prunes never over-evict. The server also keeps server.lock
there. The CLI’s default directory is shared by every checkout on the machine,
so concurrent local runs use one database through WAL and the busy timeout.
Opening takes an exclusive lock on cache.lock while it checks and migrates
the database, and migrations read the schema version inside an immediate
transaction. In the tool store, each tool run holds patchula-store.lock
shared, and removing versions (clearing an unproven install, tools prune)
holds it exclusive, so no run loses a tool it is using.
Threads and transactions
Section titled “Threads and transactions”Open one Cache per batch and clone its handle. One OS thread owns writes,
migrations and corruption recovery, with a queue of 128 jobs. Four more each
own a read-only WAL connection and a 32-job queue; round-robin dispatch runs
SQLite reads and Zstd decoding in parallel without blocking async workers.
Readers keep statements short, so decoding never holds a WAL read transaction.
Reader corruption disables the pool for that handle and retries through the
recovery coordinator.
Read access updates are coalesced and flushed in one transaction before writer
jobs and on shutdown, so reads never take SQLite’s write lock. Body inserts and
release merges are transactional. The database uses WAL, a five-second busy
timeout and synchronous=NORMAL, since cache data is reproducible. Only one
batch should own a cache path during corruption recovery: quarantine is not
coordinated across processes.
HTTP entries
Section titled “HTTP entries”Keys normalise method case, the URL (authority, default port, path, fragment)
and header names. Query order is kept, because repeated parameters can be
order-sensitive. Callers choose the vary-relevant headers; for authenticated
data they include a digest of the credential namespace, never a raw token.
GitHub’s namespace is the credential scope, which names the credential’s
reach (see token scope); hints,
catalogs, listings and inventory-actor state are keyed by it too.
The conditional-GET protocol the registry cache and the GitHub client share —
keys, entries, validators, 304 merges, Retry-After and bounded decoding —
lives in patchula_http::conditional; each client keeps its own freshness
policy, sender and error mapping on top.
An entry stores the status, validators, selected response headers such as
Link, the compressed body, fetch time, expiry and Cache-Control hints.
Expiry keeps validators. An invalid compressed body is a miss, and its
validators are never sent.
Large registry bodies use a versioned local representation in the key. npm,
Cargo sparse indexes, PyPI and GitHub REST release heads store a derived
document with only the fields the Count reads, never the raw response.
Derivation runs on spawn_blocking; a successful derived response streams
straight into its derive worker, so the raw document is never held whole.
The request’s host permit is held until its body is read, so in-flight bodies
stay inside the host limit. Reads that arrive already buffered derive under a
four-job limit. A 304 updates only metadata and reuses the compact body. A
derived representation is name@v{VERSION}~{fingerprint}, where the version
belongs to the transform’s own code rather than the release and the
fingerprint to the locked versions of the crates it depends on: a new release
keeps the cached entries of every transform whose code and dependencies did
not change, and a change to one transform misses only that representation.
tests/derived-versions.tsv guards the contract — it hashes the source files
that produce each representation’s bytes, and the owning crate’s parity test
fails when the code moves without a version bump; the maintenance step is to
bump the Derived‘s version and update the row in the same commit. The row
also lists the Cargo.lock packages the bytes depend on. Those are not hashed:
the cache crate’s build script reads Cargo.lock and both tables and
fingerprints each row’s locked versions (the first 12 hex digits of a sha256)
into its key, so a dependency bump, such as the Count’s own update PRs on
this repository, invalidates exactly the representations that list the crate
and needs no version bump. A row listing - has no fingerprint and its key is
plain name@v{VERSION}. The fingerprint covers only the listed crates’ locked
versions. A git or [patch] source whose code changes without a version
change, and a transitive dependency that is not listed, change nothing in it:
list such code in the row’s sources, or bump the version by hand. A workspace
or path crate cannot be listed at all, since Cargo.lock holds it at the
workspace version; list its files as sources. The parity test checks only that every listed package
is in Cargo.lock, so a typo fails, and the build fails outside the workspace
rather than key without the lock. The fingerprint follows ~ because + is
the retired per-build marker migration 014 deletes on sight; names may
contain none of @, ~ or +. A derive may only read code in its row’s listed files, and
since whole files are hashed a formatting or comment edit in one forces a
bump too — batch cosmetic edits in those files. The parity test also scans
the crate’s sources for Derived::new("<name>" (and the pull-token
representation constant), so a representation cannot exist without a row,
and a listed file that is renamed fails naming the row and the missing
file. Entries written
under the retired name+build scheme are never read again and are deleted
when the cache opens; raw and derived namespaces never share validators.
Opening the cache also prunes superseded keys. The build script emits each
tabled name’s version (the row’s, which the parity test holds equal to the
code’s) beside its fingerprint, so the cache knows the one key this build
reads for every row of both tables. After migrations it deletes every state
row and derived HTTP body under a tabled name whose key is not that one: an
older version, an older fingerprint, or the plain key from before
fingerprints. It is one indexed join per table: a name’s keys are the range
name@v to name@w, found through the state table’s primary key and an
index on the representation an HTTP key names (migration 015). Names that
are no longer tabled are not pruned. Two builds that share a cache directory
and differ in a version or a locked dependency prune each other’s entries for
those names, so each misses once after the other has run.
The default 256 MiB budget counts compressed HTTP body bytes. An access clock approximates LRU and evicts whole HTTP rows, validators included, until the cache fits, also on open after the budget is lowered. Release lists and state values are outside the budget, so it does not cap the file size.
Releases and state
Section titled “Releases and state”Release order is the upstream’s newest-first order, never inferred from semver
or dates. merge_releases starts at the head and stops at the first item
already stored. Later pages of the same scan pass the previous page’s last
inserted version to merge_release_page as after, which tells a refresh
prefix apart from initial pagination. The newest item is also kept in
release_heads.
Typed values the Count keeps between runs (hints, extractions, catalog views,
receipts and the like) live in state; the server queue, installations and
repositories keep their own tables. A state row is one value, keyed by a
namespace and a canonical-JSON scope. A State implementation names its
NAMESPACE and VERSION, forming name@v{VERSION} (with a dependency
fingerprint for a code-versioned state, below), and declares the Scope struct
whose named fields say what the entry is about.
Values are JSON. A stored value that fails to decode is a miss, logged at
debug level; database errors remain errors. Each call site owns its value and
scope types, so a change to a stored shape bumps VERSION or NAMESPACE
rather than silently reusing entries written for the old one. The exception
is a shape that reads both ways: auto-merge history rows went from whole REST
pull requests to the five fields its readers use, and each shape decodes as
the other, so they stay at version 1.
A value derived from an immutable input by this build’s code — an evaluated
config, an extraction, a parsed changelog, a not-ours verdict — sets
CODE_VERSIONED instead, and VERSION names the deriving code: a new
release keeps the cached entries of every state whose derivation code did not
change, and a change to one state’s derivation misses only its namespace.
tests/state-versions.tsv guards the contract — it hashes the source files
and source trees (an entry ending / covers every file under it, so a file
added to a listed tree fails too) the derivation reads, and the owning crate’s
parity test fails when the code moves without a version bump; the maintenance
step is to bump the VERSION and update the row in the same commit. The row’s
listed Cargo.lock packages key the entry the way they key a derived
representation: the namespace is name@v{VERSION}~{fingerprint}, so a
dependency bump misses exactly the states that list the crate, with no
version bump. Exempt states carry no fingerprint. A derivation may only read code in its row’s listed
files and trees; over-include rather than miss, and batch cosmetic edits in
listed files. The parity test also scans every impl State in the
workspace’s non-test sources — #[cfg(test)] modules and tests/
directories do not count — and requires each to be either a row in
tests/state-versions.tsv or an entry with a reason in
tests/state-exempt.tsv; an unlisted state, a listing whose impl is gone
and a flag that disagrees with the table all fail, so states in crates
without a parity test of their own stay covered. The rule for which table
a state lands in: a value this build’s code derives from an input whose
key does not capture the decoded result, kept without a TTL — a permanent
memo — is code-versioned. A state refetched within a bounded freshness
window, or holding facts that are not code-derived — receipts, history,
permits, run timestamps, hints — is exempt; a stale hint only costs a
wasted read and never a wrong answer. Entries written under the
retired name@vN+build scheme are never read again and are deleted when the
cache opens, and so are a code-versioned state’s entries under any key but
its current one (see the prune above).
Schema and corruption
Section titled “Schema and corruption”Migrations are transactional and tracked with SQLite’s user_version. A
migration that retires a representation costs one unconditional fetch per
affected registry. A database from a future schema is rejected untouched.
patchula cache schema prints the newest schema the binary supports; the
GitHub Action keys its saved cache by it, so a run never restores a database
its binary would reject.
Opening runs quick_check. SQLITE_CORRUPT, SQLITE_NOTADB or a failed
check closes the connection, moves the file and its WAL and SHM sidecars to a
unique .corrupt-<timestamp> name, warns through tracing and rebuilds.
Corruption during an operation rebuilds and retries it once. If that fails, the
cache warns and falls back to memory so the batch keeps running. Permission,
busy and future-schema errors are not treated as corruption. The application
must initialise tracing for the warnings to show.
Host limits
Section titled “Host limits”concurrencyin the settings, or--concurrency, sets repository scan parallelism (default 8).- Registry requests use per-host semaphores,
http.defaultHostConcurrency(32) each, independent of repository work.http.hostConcurrencyoverrides single hosts, such as["api.github.com"] = 8. Limits must be 1–100. - GitHub REST and GraphQL share one governor, sized by the
api.github.comhost limit. Without one, a CLI run takes the default host limit capped at 16 and the server keeps 4. That is below GitHub’s 100 simultaneous requests, but does not by itself avoid its point and CPU secondary limits. - The shared reqwest pool keeps up to 32 idle connections per host.
- A registry 429 sets a cooldown for that host and is retried at most three
times, honouring
Retry-Afteror backing off 1, 2 and 4 seconds. Waits over 60 seconds and exhausted retries fail, and queued callers wait out the cooldown. Docker Hub pull-quota errors on manifest GETs are returned at once and hold back manifest GETs for the quota window (six hours when Docker Hub names none). HEAD and tag-list requests don’t use the pull quota, so their 429 cools the whole host only for a window the counter names; otherwise it is an ordinary throttle. - Overlapping identical registry and derived reads share one request. Each shared answer lives only while a caller waits on it, so memory follows the reads in flight, not every URL read since the server started.
- Only successful responses are cached, so a rate-limited pagination attempt never becomes a complete catalog.
Freshness floors
Section titled “Freshness floors”Three settings keep answers reusable past what upstream validators alone would allow. The user-facing summary is in sources.
http.registryFreshnessSecs(3600) applies to OCItags/listpages with neither ETag nor Last-Modified. After expiry every page is fetched again, so additions, deletions and retargeting anywhere are seen. Manifest and tag digests keep their upstream policy.http.goFreshnessSecs(3600) applies to publicproxy.golang.org@v/listand@latest, even when the upstream max-age is shorter..infodocuments keep the upstream policy.http.advisoryFreshnessSecs(3600) keeps the(id, modified)pairs OSV’squerybatchlists for each{ecosystem, package, version}asosv-listingstate. Inside the window the listing answers locally and only stale or missing queries are posted; a failed batch stores nothing, and a revalidation (an explicit recheck, dashboard tick or retry) always asks OSV.
Zero restores upstream freshness at once, also on an existing cache. Floors
count from the original fetch time and are never extended by reads. no-cache
and no-store still apply. No other source gets a floor. New versions,
deleted tags and new advisories can go unseen for up to the window.
Registry pull tokens
Section titled “Registry pull tokens”Anonymous registry pull tokens are keyed by registry and image, under the HTTP
body budget. Only the bearer is stored. Expiry follows issued_at,
expires_in and HTTP Age, less a ten-second margin and capped at one hour;
a missing expires_in means 60 seconds. A 401 evicts the token and allows one
shared refresh.
Registry credentials
Section titled “Registry credentials”A response read with a private registry
entry’s credentials is keyed by that entry’s prefix and a one-way digest of its
credentials as well as the request, so a read without the entry, with another
one, or after its secret or account changes never sees it. The key never holds
the secret. Such responses follow the registry’s no-store and Vary: *,
digest-addressed ones included. A response that varies on authorization is
kept only when the key names who authorized it: an operator entry’s scope
does, a token the registry issued does not. Tokens a registry issues to an account
stay in memory and are never written to the cache.
Manifest digests
Section titled “Manifest digests”Digest checks send HEAD with the same Accept negotiation as GET and their
own cache key. The top-level Docker-Content-Digest identifies the
multi-architecture index. A missing header or an unsupported HEAD (405 or 501)
falls back to GET, hashing the body if needed. Docker Hub does not count HEAD
checks against the pull quota.
Write receipts
Section titled “Write receipts”A successful branch and PR reconciliation stores a verified-write-receipts
state entry per repository and branch prefix: the commit and tree OIDs, the
base, the planned update and
the canonical PR state (a line-ending-normalised body hash, labels and the
auto-merge flag), never credentials. Warm runs read a fresh GraphQL inventory
of heads and open PRs, and an exact match skips history, tree and PR reads.
A stored receipt is proof every branch the Count wrote shared the prefix, so
the refs walk passes it as refs(query:): GitHub’s filter is a
case-insensitive substring match, and the client keeps only the names that
start with the prefix. The walk narrows only while a receipts-prefix state
entry, written beside every save, names the repository’s current
prs.branch: a prefix flip away and back leaves the older prefix’s receipts
in place but no longer proves the other branches, so the run walks every
branch once and the restore that verifies the receipts again marks them.
Every branch the receipts name must also appear in the answer — one that
does not, renamed outside the prefix while its pull request stayed open or
deleted, takes the full write path rather than settling around a branch it
cannot see. No receipt — a first run, a cold cache or a changed prefix —
walks every branch as before, and that run is a full write whose cleanup
lists every branch itself. One consequence remains: a copy of a Count branch
under a name outside the prefix has no receipt, so it is invisible to a warm
run and is deleted by the next full write instead.
Any mismatch, or unknown or incomplete inventory data, falls back to full
reconciliation. In behind mode a base move invalidates the receipt. At -vv,
diagnostics report save and restore counts and the first failed condition per
branch.
The inventory’s issue walk filters issues by filterBy.createdBy, and the
login it asks for is remembered per credential scope as inventory-actor
state: the first page’s viewer.login is stored as soon as it is read, even
when the walk goes on to fail. A read whose answer’s viewer.login does not
match the stored hint retries unfiltered once, so a wrong or stale entry
costs one request and never narrows the issue set wrongly.
Branch cleanup keeps a code-versioned not-ours entry per repository and commit
OID whose message records no base: such a commit can never be the Count’s,
so its branch is not read again while it points there. Nothing else about
ownership is cached; a compare, an author check or a signature check runs
on every full write.
Catalog views and release age
Section titled “Catalog views and release age”Successful catalog views are stored as catalog state, scoped to the
package, whether the versions host may stand in and the credential scope. A
dependency with a positive effective waitHours
reuses a catalog younger than clamp(waitHours - 1, 1, 24) hours, 23 hours
for the default 24-hour wait. Zero keeps the usual HTTP freshness. Dependencies
sharing a catalog use the shortest wait among them. A release that must wait a
day does not need a fresh catalog on every run.
Only the catalog is cached; policy always uses the current time, so a known
held-back release can become eligible during reuse. The report records
metadata.lookup.catalog.fetchedAt and reused next to evaluatedAt.
Repository rechecks, checked PR boxes and selected dashboard controls bypass
reuse for the dependencies they name. Failed lookups are never cached as
views. Mutable Git tags and image digests keep their own freshness: bounded
GraphQL reads resolve the refs currently needed without revalidating the lists.
mise listings
Section titled “mise listings”What mise ls-remote --json printed is stored as mise-ls-remote state, since
it has no validators. The scope holds the tool, its options (the bracketed
text the tool argument carries, so one set of options never shares another’s
listing), a version prefix such as lts, whom the listing serves, whether mise may read the mise-versions mirror
and the pinned mise version. A listing younger than an hour is reused without
starting mise. A manual retry replaces it, and only successful lookups are
stored.
Whom a listing serves follows the token mise was handed (see
scoped tokens). Sources::mise_access
decides it from the tool and the repository the lookup is made for (its name
and visibility), and it is the third element of the batch’s lookup key and
part of the catalog scope. Release notes look a repository’s mise tools up
through a client made for that repository, so they get the same access and
read the same listings.
public, with ametadata: readtoken scoped to a repository that asked for the tool. That token reads no repository’s private releases (not even the minted one’s, should the tool name it under an old name), so the listing is public, and one lookup in the batch serves every repository that asks for it. A tool is listed this way when every GitHub repository its backend names (github:,ubi:,aqua:,packslip:,cargo:with a GitHub URL, or each backend the registry snapshot lists for a shorthand) is another one, or its backend reads a package registry or is a core tool. When the Count cannot tell (go:,http:,spm:, an unlisted shorthand, any tool with options and the like), the listing is public only for a repository GitHub reports as public.- The run’s credential scope and the repository’s numeric id otherwise, with
a
contents: readtoken: the listing of a tool released from the repository, or of an unknown one in a private repository (or one whose visibility was not read), serves that repository alone. The id is the one the mint names, so a new repository under a renamed one’s name reads none of its listings or catalogs. - The run’s credential scope for credentials that cannot mint a scoped token (a personal token, an Action’s own token) and for a local directory: mise gets the run’s token, and the listing serves only that credential, never a scoped lookup.
A batch lookup checks that its key’s access is the one the repository it is made for gives, and fails otherwise: a public key never runs without a repository to mint for. Off github.com no token reaches mise, so every listing there is public.
Options are part of the tool’s package, so they are part of the lookup key,
and a key’s via is a repository whose own entry asked for exactly that tool
and those options. A tool’s options may send mise to another host
(api_url), and mise hands its GitHub token to any host that looks like a
GitHub API. That token is always minted for a repository that wrote the
options, so it reaches no further than its writer already can. Flake inputs still hand nix the run’s token.
© 2026 Sweepy
