#Language packs

"Something its language lacks" is a key, option, format style, suppression comment or another family's grammar. The host's schema checks a rule's shape only. Every write, and validateCodeRule from @neralabs/rerail/rules, compiles the rule in the engine. The engine refuses it as invalid, or as language-not-supported when the registry plans the language or capability.

• Adapter files: Python (adapter/scope.rs, value.rs, ...), Rust (adapter/rust.rs), Vue (adapter/vue.rs) and Sql (adapter/sql.rs).
• with_adapter! is the one exhaustive match on a family. It, Family, GrammarDialect, Family::of and honours are generated from the registry; no enum is edited by hand.
• Every other call site asks the adapter. One that needs a concern's associated type calls a function generic over A: Adapter from inside the macro.
• Call names carry their family (CallRules::NAMES). Value semantics (truthy, type_name, equals, order, relation, numeric, integer, to_str) belong to ValueRules.
• Refusals. A concern the registry does not mark supported for a family is Refused. One gate (gate) reports a refused feature as invalid and a planned one as language-not-supported.
• Embedding families take the embedded language's semantics, over the embedded program: a document of its own, read with that family. Blade delegates to Php over the compiled view; Vue to Ecma over the script-block program. Their own concerns name the embedded language's for the compile's checks. A semantic key over the file's own tree is refused (compile::documents::check_sides).
• The adapter kit (adapter/kit.rs) runs one generic suite over every adapter, on the document each family's semantics read. It checks each adapter reads exactly the capabilities its registry entries support.

The program comes from vue::view, via script-block (embed/script_block.rs): TypeScript for lang="ts", TSX for lang="tsx", else JavaScript. Another language (coffee) embeds none. The program has its own grammar and text, with segments mapping that text to the file's bytes.

The adapter does not read names inside macro token trees and macro_rules! bodies. Each concern is proven against a recording of the Rust 1.99.0 toolchain under fixtures/rust/ (generate.py beside each; run by hand, never in CI).

Rust refuses languageOptions, resolves.imported, and other families' grammars and format styles.

Suppression is clippy (suppress/clippy.rs), checked against clippy-driver over UI fixtures and the ripgrep and tokio corpora (fixtures/rust/suppress).

• Reads rustc's and Clippy's lint-level attributes (allow, expect, warn, deny, forbid), outer and inner, including those a cfg_attr applies when its predicate holds. With suppress.clippy.msrv, it also reads the #[clippy::msrv] in force.
• Per file only. Levels set in another file (a crate root's #![allow(..)] over its modules) or on the command line are not seen.
• cfg predicates other than all, any and not over clippy are false.

That dialect is GrammarDialect::Sql, the vendored tree-sitter-sql with Rerail's patches (native/grammars/tree-sitter-sql/PATCHES.md). Where dialects disagree it reads PostgreSQL's; the pack declares the rest as divergences.

A query block binds:

• its FROM/JOIN relations (Form::Relation);
• its UPDATE/DELETE/INSERT target (Form::Target);
• around CREATE POLICY or ALTER POLICY, the policy's table (Form::Target), seen by the queries in its USING and WITH CHECK, as SQLFluff's RF01 reads it on PostgreSQL;
• around a WITH statement, its CTE names (Form::Cte).

A qualified column's table part resolves to a relation or target of its block or an enclosing one; a CTE only through a relation over it. The pack's ports read syntax, value, compare, scope and suppression. None reads arguments; that capability's own oracle proves it.

calls and formatArgs are refused in languages/sql.json: SQL gives the engine nothing to read for them. A rule reading either is invalid for sql, never language-not-supported. The module doc of adapter/sql.rs has the evidence:

• calls: no SQLFluff rule follows a call into a routine. A routine body is a string literal of another language ($$ ... $$, PL/pgSQL), not parsed SQL. A .sql file has no workspace-resolvable call graph: no import or module links one file's routines to another's.
• formatArgs: no SQLFluff rule reads a format string, and ISO SQL has no format function. PostgreSQL's format() and SQLite's printf() are dialect extensions.

Each dialect has:

• label: the file language hosts and the store use.
• suffixes: lowercase, with the dot. Compound ones such as .blade.php win over shorter ones.
• variant, supported only: the engine's GrammarDialect::<variant>.
• parser, supported only: tree-sitter, or forte, Rerail's port of Forte (docs/blade-parser.md). Validated, but drives no behaviour yet.
• grammar, supported only: vendored:<name> (built by build.rs), native:<name> (a parser Rerail implements) or ast-grep:<Language>.
• tapeTag: required when supported; a planned dialect may declare one to reserve it. Persisted, unique across entries, never reused. Taken: Rust 9, SQL 10, Vue 12 (11 is free for a second SQL dialect).

Embeds. A supported entry may embed only supported languages. An Embedder builds at most one program per file and via: a document derived from the file's (tape::DerivedImage), whose own tape the store keeps as a row beside the file's (file_documents).

Versioning. A new optional field or value is a rerail.language/v1 change. Renaming or removing a field, or changing a value's meaning, needs rerail.language/v2.

Validation. The build and test/engine/code-rule/language-registry.test.ts fail when:

• ids, dialect labels, suffixes or tape tags repeat;
• a supported dialect lacks a variant, parser, grammar or tape tag;
• a planned language supports a capability or names runsOn;
• entries name different capabilities;
• runsOn names another family's dialect;
• embeds names an unknown language.

A rule of a planned language, or one reading a not-supported-yet capability, is refused as language-not-supported and kept inactive:

• Compiling the rule fails with the stable code language-not-supported (the message starts with it and names the language), never a schema error.
• The catalog loads the rule but keeps it inactive. Audits skip it, never fail on it, and do not skip a pack that holds it.
• The rule APIs (GET /api/repository/rules, GET /api/rerail-rules, GET /api/library-rules) and activity API (GET /api/rules/activity, measured: false) return it with inactive: { code, reason }.
• Audits parse none of a planned language's files: its suffixes select no dialect, so regex clauses read them as text.
• Pack tests report its rules as not supported yet without failing (validation contract).

A test that needs "a planned language" names it. It has family exemplar, one dialect exemplar (suffix .exemplar), every capability not-supported-yet, no suppressions and no tape tag. It is in no package and never in the product:

• Not in the registry. languages/ and the generated host module omit it; npm run generate:languages reads languages/ alone.
• Test builds only. native/build.rs validates it like any entry, and against the registry for repeated ids, labels, suffixes and tape tags. It emits it only as #[cfg(test)] items: PLANNED_LANGUAGES lists it only in test builds (TEST_PLANNED_LANGUAGES names languages only tests plan). It is no part of the engine's identity.
• Mocked in host tests. The product-build native binding does not know it. Host tests mock the generated module with exemplarRegistryModule() and the native module with nativeRefusingExemplar (test/engine/support/exemplar-registry.ts, exemplar-engine.ts), which refuses only exemplar rules and passes the rest to the real binding.
• Real refusal. A native test checks the engine's language-not-supported refusal. So does test/engine/code-rule/language-not-supported-engine.test.ts, unmocked, by compiling a rule of a product language the registry still plans.

The test-only HTML grammar embeds TypeScript, TSX and JavaScript through Via::TestScript (language::test_embeds, native/src/code_rule_dsl/embed/test_script.rs). Its program is an HTML file's scripts, then its on* handlers. Its tests (embed/tests.rs) cover the embedder loop, derived documents and their dialects stored apart from the file's tape, and scope and usage over a program mapped to the file out of order. The generated Via and with_embedder! carry it only under #[cfg(test)].

For step 2, the build generates GrammarDialect::<Variant>, Family::<Family>, Family::of, honours and the with_adapter! arm; no enum is edited by hand. For step 3, a new family implements:

• ScopeRules;
• ValueRules, with const LANG: Family = Family::<Family> and the hooks truthy, type_name, equals, order, relation, numeric, integer and to_str;
• ArgumentRules;
• CallRules, with const NAMES: Option<Family>, the family whose namespace its call names live in;
• FormatArgRules;
• SyntaxRules.

A new dialect in an existing family needs no per-dialect table: ECMAScript's scope reads each dialect's grammar (scope::ecma::script).

Each missing suppression kind needs:

• one row in suppression_kinds! in native/src/code_rule_dsl/suppress.rs (Variant key: Spec = linter::spec, "detail";), with its parser in suppress/<linter>.rs;
• one entry in codeRuleSuppressSchema in src/engine/code-rule/code-rule-schema-semantic.ts;
• its name in the registry entry's suppressions;
• its use in the adapter's suppresses.

build returns one embed::Program or none: its dialect, its tree over its own text, the SegmentMap mapping that text to the host's bytes (copied or written over them, in any order), and the host syntax errors it reads through. The engine handles the rest:

• Storage. The program is a derived document (tape::DerivedImage): text, source map, host errors and its own tape, in its grammar's codes, checked like every tape. The store keeps that tape as its own row, related to the file's by file_documents (store).
• Access. A clause reads it with ast.embedded: <via>; a host rule reaches it with embedded; a program node reaches the host with host (matcher::embedded).
• Positions. Witnesses, captures and fixes land at the file bytes they stand for. Each document is an origin::Origin into its source, and origin::InFile places nodes and ranges in the file (fixes only over copied bytes). Nothing else maps a position.
• Ranges. A node covers its pieces' file ranges (embed::segments::FileRanges), except where the program is out of file order and joined only over white space. A single span reports the range it starts in. Positions are half-open: a boundary belongs to the range starting there.
• Syntax errors. Each document has its own, as file ranges (soundness::SyntaxErrors::of_doc). A program with empty host_errors is complete whatever the host's errors.
• Calls read the program whose dialect's calls join the graph (calls::calls_document).

The embedding family's adapter (step 3) names the embedded family's concerns, as Blade's does PHP's.

conformance.json, per ported rule: its upstream tool and rule, id (the rule's own), policy (its document), how findings compare (line or line+column) and the capabilities it exercises.

• A port's clauses report the upstream rule's code.
• Every declared capability needs a ported rule or an exemptCapabilities entry with evidence.
• New ports start as candidates under candidates/rules/ (candidates and promotion).

upstream/<tool>/: each recipe is one module per pack, listed in scripts/pack-data/recipes/index.ts.

• Generators. packs/python/upstream/ruff/generate.py is one. ESLint-based ones share scripts/pack-data/eslint-expected.mjs, which also writes rename variants under fixtures/renamed/.
• Expected findings. npm run fixtures:derive -- --pack <id> copies the tool's tests into fixtures/ and writes fixtures.expected.json: the pinned tool's findings on them (expectedFindingsSchema in conformance/languages/pack.ts, each unit's upstream origin in source.units).
• Publishing. That derived test data is never in git. It is staged into upstream/<tool>/publish/ (npm run fixtures:stage -- --pack <id> --tool <tool>) and published by the publish workflow, which commits the tool-pack's test-data.lock (docs/upstream.md).
• Provenance. The generator writes the tool's provenance.json and LICENSES.md last, from the pack's licenses/<project>/UPSTREAM.md records (docs/upstream.md).
• Authored cases. List each in the tool's own UPSTREAM.md, or keep a corpus with its ground truth under upstream/<tool>/authored/.
• Get other packs' data with npm run fixtures:pull.

licenses/<project>/UPSTREAM.md has:

• a License: <SPDX expression> line;
• a Source table: repository, tag, 40-hex commit, licence;
• a Taken from it table mapping each pack-relative glob of copied files to its upstream path and copy method;
• a Not copied line per upstream path under a licence the allowlist refuses.

Add the tool's licence to package.json's license (Apache-2.0 AND MIT for an MIT tool), and annotate the rule documents in REUSE.toml with that expression.