#Upstream snapshots and publishing data

Upstream is provenance at a point in time. CI only verifies the pinned bytes and scores the rules on them. It never runs upstream, and never checks whether upstream would still answer the same. A snapshot goes stale only when a maintainer decides to take a new one.

Each gated tool-pack's gates.yaml is its upstream manifest. It holds the pins (upstream.version, upstream.commit, the runtime), the file types upstream lints (upstream.lints), and the per-rule facts declared at the pins (rules). A maintainer reviews these at each bump.

• A rule the labels do not name is not judged on the file. Labels mark intent by line; a column, if present, is provenance.
• Judging is in conformance/README.md, "How samples are judged". The archive, samples.lock and SAMPLES.md are in pack-format.md, "Pack data".
• node scripts/s3/pack-fixtures.mjs samples fetches samples alone. Samples never extract into the tool directory.
• Samples staged in publish/samples/ are laid over the locked set, so a pull request's CI judges them before publishing.
• writeSampleSet checks the samples as the publish does, and writes a REUSE.toml licensing the staged copy.
• A label naming no rule of the tool-pack fails the whole pack's conformance check, which catches typos.
• The builders are a library API with no command of their own.
• No rule says how far to trim a set; SAMPLES.md says how it was trimmed, if it was.

COPYLEFT_SOURCES lists repositories found copyleft, with the evidence.

The licence is the one in force at the sample's commit, for its file. That is the licence files of the nearest directory above it that has any (a monorepo package's own, else the root's). At a commit with none, it is the README's licence section or package.json's license.

A SOURCE_LICENCES entry records the repository's licence and files. Under at it records each pinned commit where they differ (jest was BSD-3-Clause with a patent grant until 2018). at also records licences no allowlist entry covers (NONE where nothing states one, CC-BY-4.0 for mocha's docs). The builder then drops the sample, and a publish staging it is refused, whatever the repository's licence is today.

publishedSamples refuses:

• a sample whose repository has no entry, or whose labels declare a source.license other than the entry's at the sample's commit;
• a sample from a copyleft source;
• a path that is neither LICENSES.md nor <sample-id>/<path>, or a sample not staged whole: its file beside its labels, the file's sha256 equal to the labels' source.sha256.

checkLicenceTexts reads, for each repository and commit the staged samples name, the licence files SOURCE_LICENCES records for that commit, from GitHub (gh api, cached in .rerail/pack-data/licences/). It refuses the commit when a file is not there, the files do not read as the recorded licence, they name a copyleft licence, or the commit has no licence (NONE). Both checks run when writeSampleSet stages a set and again in the publish workflow's prep, the only way a sample reaches S3.

The approver's summary lists every repository the archive takes files from, with its licence.

The archive's LICENSES.md lists each repository, its licence, its files, and the licence texts read at the samples' commits. A set converted from a test set without reading licence texts (the ESLint test set's) carries only the licence id the test set recorded, with no text. So an archive holds licence texts only for repositories whose licence files were read, and SAMPLES.md says which: all, none, or the ones it names.

A maintainer starts a bump by hand (Start a bump) after reading the nightly report (Upstream releases).

A re-snapshot follows a generator, recipe or runtime pin change, or makes data only Linux makes.

A local derive runs a pull request's own pins, generators and recipes.

Generators run upstream's code, so they run only in a maintainer's sandbox or the bump workflow, never pull request CI.

source.units gives each unit's origin. Its rules are the upstream rules whose tests the unit came from. That makes it one of that rule's own units for stays-silent and upstream-rule. A unit with no entry (Rerail's probes, or unknown origin) is scored but is no rule's own evidence.

ESLint-based generators share scripts/pack-data/eslint-expected.mjs.

A generator that lints each case under its own language (ecmaVersion, sourceType, parserOptions) records it as languageOptions in source.units. It does so where it differs from ESLint's defaults for the file's extension. The runner evaluates the unit under it (pack-format.md).

A variant lives at fixtures/renamed/<the original's path>, renamed by scripts/pack-data/rename/ecmascript.ts:

• Renamed: identifiers, keeping length, character shape and the order of every two names (one bijective map per file), so sorting rules judge the same order.
• Protected: keywords, standard library, common host, framework and test-runner API names, hook-shaped and …Sync names, and package exports (imported names, members of a package's binding, props of its components).
• Kept: literals, since rules read them (a number's digits, a URL, a font name).

Upstream lints the copy like any unit, so its expected findings are upstream's own. Its source.units entry carries renamedFrom, the original's key. The runner scores variants apart from the originals, as holds-under-rename.

A port fails if it matches the test cases' names instead of the construct. It also fails if renaming takes every finding and none survives (held vacuously, which blocks). A rename_exempt reason in the rule's gates.yaml entry exempts it (conformance/README.md). A family without variants yet takes its overfit evidence from samples alone.

Last, each generator calls writeProvenance (or write_provenance from Python). The two files go in packs/<id>/upstream/<tool>/ and are archived with the data (pack-format.md, "UPSTREAM.md"):

• provenance.json: each pack test data file with its source, upstream path and how it was made.
• LICENSES.md: each source's licence and licence text.

The generator names the files it copied from upstream. The writer classifies Rerail's own files (the tool's UPSTREAM.md listing, authored/, the tool's recorded output). It reads each source from its tracked packs/<id>/licenses/<project>/UPSTREAM.md.

The derive fails, writing neither file, on any mismatch with those records:

• a pack test data file with no entry;
• a file outside its source's "Taken from it" globs, or such a glob matching no file;
• an upstream path under a "Not copied" prefix;
• a checkout at a commit other than the Source row's.

A generator that writes neither file also fails the derive.

A runtime is pinned by its package.json, pyproject.toml or composer.json alone: every direct dependency at an exact version and in overrides, no committed lockfile.

• Python pins its interpreter in .python-version, which uv installs.
• PHP pins its interpreter in .php-version. Composer cannot install it, so derive refuses any other php.
• Every PHP process runs with no php.ini (php -n, PHP_INI in runtimes.ts).
• runtimes/php/extensions.txt lists the extensions runtimes/php/build.sh compiles in; a php loading any other set is refused.

Transitive packages resolve at install. If one changes what upstream answers, pin it too and take a re-snapshot.

fixtures:derive runs the pull request's generators and recipes and the upstream code they fetch. Review that code first, then run it in a sandbox:

• its own clone, in a container or VM;
• no reachable AWS or GitHub credentials (no AWS_* variables, ~/.aws or gh login);
• Linux for rust/clippy, whose answers are Linux's (Clippy lints only code rustc compiles for the host).

For a change already on main, the bump workflow's resnapshot input does this in CI. The sandbox stages the whole set into publish/, and the git bundle it hands back is data.

fixtures:derive needs the network (pinned upstreams and their packages) and uv. The PHP and Blade packs need the pinned PHP (scripts/pack-data/runtimes/php/build.sh) and Composer; the Rust pack needs rustup. Everything else installs under .rerail/pack-data/.

It fails if a derivation changes a tracked or authored file. --update (a pin change) instead keeps the tracked files it changes (a licence record) for the commit.

When checking a pull request, step 2's fixtures:pull lays publish/ over the locked archive. git status then lists every file where the derivation differs from what the pull request publishes.

Anything else under publish/ fails the publish.

fixtures:stage copies every file the locked archive lacks or holds differently. An overlay cannot delete a file, hence --all for a replace publish.

Review sees the bytes in the diff, with *.expected.json collapsed. CI lays publish/ over the pulled data and judges it. Structure metric #3 blocks the merge while any publish/ is tracked.

The request comment runs on pull_request_target with the base branch's workflow, so on forks too. It checks out nothing; its token can only write pull request comments.

• While anything is left to publish, it keeps one comment. Its rows: one per tool-pack with its publish/ files and removed UPSTREAM.md authored rows, plus publish steps by hand and with gh.
• On a fork's pull request, a proposal (Forks), the comment is "Pack data in a proposal from a fork": the same rows, no publish steps, and a note that the pull request porting the change publishes it.
• It deletes a waiting comment once nothing is left to publish. The comment blocks nothing; structure metric #3 does.

The dispatch. The run is titled Publish pack data for #<n>.

• replace off (default): the locked archive, less files of removed authored rows, with publish/ laid over it.
• replace on: exactly what publish/ holds. Use it with whole staged sets (fixtures:stage --all) to remove files or regenerate a set. Bump and re-snapshot pull requests always use it (Bump pull requests).

prep runs only on main, with main's code and the read role.

• It refuses a pull request that is not open, a head that is no commit on a valid branch, and, for a branch of this repository, a head on the default or a protected branch.
• It fetches refs/pull/<n>/head as objects (it must be the head GitHub names). It packs each publish/ from those git blobs (scripts/s3/pack-fixtures.mjs pack --from), never checking the pull request out.
• Staged samples go through publishedSamples and checkLicenceTexts (Sample licences), the second reading their licence files from GitHub with the workflow's token. Every file bound for S3 must pass the licence rules (pack-format.md, "Licensing"), samples by their recorded licence.
• A refusal fails prep without asking for approval. The log says why: fix the pull request and dispatch again.
• The approver's summary, per archive: kind, tool-pack, sha256, files, bytes, licences with file counts, and Sources (a samples archive's repositories and licences; for test data, the licenses/*/UPSTREAM.md records).
• Then each archive's added, changed and dropped paths (up to 50 per list; a replace publish's dropped files appear nowhere else), and how many publish/ files the commit will delete.

upload waits for the test-data-upload deployment, whose required reviewer is a code owner who has read the summary (REVIEW.md).

• With no approval on record it fails and puts nothing, since an environment without a required reviewer would gate nothing. Only then does it assume the write role.
• It verifies each archive against prep's job outputs, not the artifact's own manifest.
• It puts each create-only (S3 refuses an existing key), tagged rerail:commit (the pull request's head), rerail:pr, rerail:run-id and rerail:uploader (the approver).
• It confirms every object, even after a failed put: S3's head, plus a fresh download verified against its lock.

finish runs only once every object is confirmed.

• It builds one commit on the pull request's head with git plumbing alone (no checkout, no hooks): the new test-data.lock and samples.lock, with every published tool-pack's publish/ deleted.
• It pushes as the neralabs-ci app, whose token makes CI run again. The plain push only succeeds as a fast-forward of the head prep packed; it never forces.

• "action needed": someone pushed to the branch since prep, or the branch is on a fork, which is never published from (Forks). The run is red; the comment gives each lock's text and the reason.
• failed: an object was not put and confirmed (a failed put or confirm, a rejected approval, or a cancelled job). The failed job names each object's failure in the comment. No lock changes.

Re-running is safe because puts are create-only and content-addressed. Confirm accepts an object S3 already holds once it verifies. Once the locks are committed, Pack test data is in S3 passes and conformance runs on the published data (CI).

A fork's pull request runs no CI: CI is costly, so maintainers don't approve fork runs. When a maintainer ports the change to a branch here (CONTRIBUTING.md), publish/ comes along. CI judges it there, and the publish runs for the ported pull request. The workflow can read a fork's head via refs/pull/<n>/head and posts its locks in the comment, since finish cannot push to a fork's branch.

An archived file is not in the pull request's diff, so an overlay cannot delete it. Removal changes the archive's file set, which provenance.json must match or prep refuses it. So derive the tool-pack again after removing (fixtures:derive on the pinned toolchain; Linux for rust/clippy) and stage the changes.

A removed row alone makes a publish request, with nothing in publish/. The comment counts it, and prep drops each authored file the record listed at the merge base with main and no longer lists.

The old archive stays in S3 for the commits that name it.

It archives the data as on disk, uploads it tagged with the caller's gh login, and rewrites the lock (--dry-run shows what it would archive). node scripts/s3/pack-fixtures.mjs samples push does the same for a samples archive in the local cache.

Neither runs prep's licence check or asks for approval, so pull request data goes through the workflow. They exist for a maintainer with write credentials who must put an archive in S3 outside a pull request. One case is an archive a lock names but S3 lacks; fixtures:pull and Pack test data is in S3 point to them for it.

S3 writes are create-only, so nothing published is overwritten.

From a CI run. Each .github/actions/pack-test-data step sets an archives output, one <pack> <tool> <kind> <sha256> line per archive (kind test-data or samples). Its pull adds them as a table to the job summary. A fetching job that hits the cache runs no pull, so its output lists the locks its cache key names. To replay, check out the run's commit and run npm run fixtures:pull; its locks name those sha256s.

One archive. Fetch rerail/archives/<pack>/<tool>/<sha256>.tar.zst (samples: rerail/archives/samples/<pack>/<tool>/<sha256>.tar.zst) from s3://nera-test-fixtures/. The sha256 is the tar's, so check it after zstd -d.

A pin is the release the rules were ported against. For every gated tool-pack the report finds upstream's latest stable release (the release check below) and the tag and commit GitHub names for it. It compares that commit with the pin statically through GitHub's API. It compares both trees, by blob, over the paths in gates.yaml's watch: block (pack-format.md). That catches adds, changes, deletes and renames. It downloads only changed test files, from both commits.

It reads GH_TOKEN or GITHUB_TOKEN; the workflow passes its read-only GITHUB_TOKEN. It does not check that a tool-pack's own pins agree. CI does, on every pull request and push (test/scripts/upstream/releases.test.ts, "this checkout's pins agree").

The job summary (also --summary, or --json as data) has one row per tool-pack:

• Status: up to date; behind with nothing relevant changed (how many releases; shared code alone is nothing relevant); changed; or error (a tool-pack it could not read; the others still report).
• Gap: pin to latest release, a breaking one flagged.
• Tests: changed test files the generator reads, including a Clippy test's expected .stderr. For the ESLint family, RuleTester cases added, removed and changed; for other tools, changed files count as changed fixtures. For phpstan, exactly the files its tests name, read at both commits.
• Rule code: ported rules whose code changed. For psalm, each issue's class and the analyzers that raise it; for phpstan, each identifier's rule class. bladestan's rules are phpstan's, in another repository, so it reports shared code only.
• Shared: changed shared code, a weaker signal. It is listed, but alone it leaves the status at "behind, nothing relevant changed".
• New tests not read: tests upstream added for a ported rule that the generator does not read (a ruff #[test_case(Rule::…)] in a ported plugin, a Clippy test whose //~ annotations name a ported lint, another test file in an ESLint rule's directory). Porting them is a generator change.

It fails only on its own errors, never on what upstream published.

For every gated tool-pack it reads the release gates.yaml pins (upstream.version) and the one its runtime installs (scripts/pack-data/runtimes/<upstream.runtime>/package.json, pyproject.toml or composer.json). The package is the name upstream is installed as, which may differ from its repository:

• ESLint family: from eslint.release.installed.
• Other adapters: listed in ADAPTER_PACKAGES (PHPCSStandards/PHP_CodeSniffer is squizlabs/php_codesniffer, tighten/tlint is tightenco/tlint).
• Clippy ships with Rust, so its release is a toolchain's. gates.yaml pins its version and commit and names the clippy-<release> runtime, whose rust-toolchain.toml pins the channel. packs/rust/upstream/common.py refuses any toolchain but TOOLCHAIN; conformance.json names Clippy 0.1.<minor>. None of these is the repository's rust-toolchain.toml, which pins the toolchain Rerail is built with.

Latest stable release per registry:

Versions compare by numeric components, ignoring a leading v. The first differing component is the gap: patch, minor, major, or breaking-minor for a minor release ahead of a 0.x pin. Rust and Clippy compare on major.minor only, so a Rust patch release is not behind.

It prints one table row per tool-pack (Tool-pack, Ecosystem, Package, Pinned, Latest stable, Gap, Status), also to the job summary under GitHub Actions. It warns for each tool-pack behind, pins that disagree, or an unreadable registry, and always exits 0. --json <file> writes every pinned and latest release ([{pack, tool, toolPack, pinned, latest, gap, breaking}], sorted by tool-pack; latest and gap null when unknown) for the bump start.

It reads npm run upstream:releases -- --json on main's head. A bump targets each tool-pack's latest stable release; a re-snapshot, its pin. A named tool-pack that is not gated, already latest, or has an unreadable registry fails the run before anything starts. all with nothing behind starts nothing.

Then, with only its GITHUB_TOKEN (contents: write, actions: write), it recreates the work branch auto/bump-run/upstream at main's commit. It dispatches upstream-bump.yml there once with the list ([{pack, tool, version}], sorted by pack then tool). Its job summary lists what it started.

The bump opens one pull request on auto/bump-upstream (auto/resnapshot-upstream for a re-snapshot). The next start replaces its commits, published ones included. To keep a tool-pack the open pull request carries, name it again.

The bump reads each file back to check only pins moved.

• gates.yaml: upstream.version, upstream.commit (the commit of the release's GitHub tag; an annotated tag is followed to its commit), and an ESLint upstream's eslint.release.tag. A release without a tag of the pinned form stops the bump.
• The runtime's pin file: upstream's package, and on npm every package of its scope at the same release. A runtime directory named <name>-<version> is renamed for the new release, and every gates.yaml naming it follows.
• conformance.json's tools[].version.
• Generator pins (packs/<pack>/upstream/<tool>/* and the pack's shared upstream/* scripts): the commit, npm's gitHead where a generator pins that instead, ...VERSION = "<release>" and ...TAG = "<tag>" constants, and an entry keyed by upstream's package.
• Reference tools a pack pins by hand (fixtures/vue/tools.mjs, fixtures/sql/common.py): upstream's package entry and the commit and tag beside it.
• packs/<pack>/licenses/<project>/UPSTREAM.md: the Source row's tag and commit, and the release in its heading. A row with no tag (—) naming npm's gitHead, like react-hooks', moves to the new gitHead and keeps —.
• Clippy: TOOLCHAIN, RUSTC_VERSION, CLIPPY_VERSION and CLIPPY_COMMIT in packs/rust/upstream/common.py, and licenses/clippy/UPSTREAM.md's Source row and the Rust release in its heading.

Anything still naming the old release (README prose, a comment, the rest of UPSTREAM.md, a recipe under scripts/pack-data/recipes/, a renamed runtime's old name) is listed in the summary for the reviewer.

The bump leaves gates.yaml's rules alone. The reviewer checks them against the new release: a rule deprecated, added to a recommended configuration, or gone.

Deriving runs upstream's code at a release nobody has reviewed, with unlocked transitive packages and Composer. It never runs in main's cache scope or next to a credential, so the work is split:

1. upstream-bump-start.yml runs no tree code beyond the release check, and dispatches the bump on the work branch.
2. upstream-bump.yml, the producer, runs only on the work branch and refuses any other ref.
   • scope empties the branch's cache scope.
   • test-data pulls every archive under the read role, with main's code.
   • bump legs, one per tool-pack, each failing alone, hold no OIDC token. Each moves its pins (upstream:bump), derives on Linux every tool-pack the bump made stale (fixtures:derive -- --update), and stages each whole set into publish/ (fixtures:stage --all). It uploads the patch, summary and bump.json as the bump-<pack>--<tool> artifact.
   • With resnapshot, each listed version must be the pinned release, which is derived again.
3. upstream-bump-publish.yml runs on main when a bump run ends, with main's code, treating artifacts as data.
   • run checks the bump run's branch, its age, and that main's code is unchanged since its commit.
   • verify, one leg per artifact, redoes the pin edits with main's code. It admits a patch only if it makes exactly those edits plus the files deriving and staging write, and changes no lock.
   • compose makes one commit per tool-pack, Bump <pack>/<tool> to <version>, on main's head, and the pull request body: pin diffs, upstream facts, what still names the old release, and each leg's own report marked unverified.
   • pull-request pushes to auto/bump-upstream (auto/resnapshot-upstream for a re-snapshot) with the app's token and opens or updates the pull request. It publishes nothing; the app needs no Actions permission.
   • work-branch deletes the work branch.

A bump that cannot be made (pins already disagree, a release without a tag) fails its leg with the reason. The other tool-packs are still proposed.

The pull request body, the bump publish run's job summary and the sticky comment (pack-data-publish-request.yml, which recommends replace on a bump pull request) say the same. Its prep summarises the data, a code owner approves the upload, and its finish commits the locks and removes publish/. Until then the pull request cannot merge.

Dropping one commit leaves the other tool-packs' bumps as they are. A branch built from the bump's commits before its publish still carries publish/, so the new pull request must be published. To keep the dropped release, restart its bump once the cause is fixed. Or bump it by hand (npm run upstream:bump, then derive, stage and publish).

Pack test data works through .github/actions/pack-test-data with fetch: "true". It looks up the cache key (from the locked sha256s alone) without restoring it. On a miss it downloads from S3 under the read role's OIDC token. It runs only this commit's scripts/s3/pack-fixtures.mjs.

Jobs that read the data hold no token. They restore with fetch: "false", verify each archive against its lock, extract it, and fail on a miss. So a job that can mint a token never extracts a cache entry. A job running upstream code (the bump) has no token to mint.

Pack test data is in S3 runs node scripts/s3/pack-fixtures.mjs check under the read role. It does a head-object per archive in every test-data.lock and samples.lock, downloading nothing. It fails with each missing key, its lock and its remedy, so a pull request cannot merge with data S3 lacks.

Conformance is a matrix of shards; both it and tools run over every pack with publish/ laid over the pulled data. Each shard scores a share of every pack (conformance:languages:check -- --shard <k>/<n>). tools builds and judges the reports from their results (-- --results <dir>). It uploads the summary and comment as the conformance-summary artifact, which the conformance-report job posts on the pull request (conformance/README.md). On a pull request they run only when it changes what the check reads, including anything under packs/. Other changes (docs, the dashboard, the hosts) skip the shards, the comment and tools' conformance steps.

Lint and type-check: structure metric #3 also fails when a tool directory lacks its test-data.lock, a lock is malformed, or pack test data is tracked in the pulled layout (repository-structure.md).

CI result needs every other job. The ruleset requires it, since GitHub counts a job skipped after a failed dependency as passing. A job skipped by the changes job's path filter, or for a fork or Dependabot, passes it.

Pack test data is a point-in-time artifact: no CI job checks it reproduces from the pins. The approving code owner checks that what a pull request publishes follows from it, and the conformance check judges the rules on it.

A fork's pull request runs no CI: maintainers don't approve fork runs, and what goes in is ported to a branch here (Forks). Dependabot gets no OIDC token. So test-data, test-data-in-s3, conformance, tools (and with it the conformance comment) and the other jobs that read pack data run only for pushes and same-repository pull requests.