RFC: Layered policies with extends

[jzonthemtn]

Summary

Let a policy declare a base policy it extends, so a family of related policies can share their common rules and state only how they differ. Adds an optional top-level extends (and an abstract flag) to the redaction policy schema, EXTENDS / ABSTRACT to the POLICY declaration in PhiSQL, and a defined merge contract that every Phileas runtime implements.

Motivation

Many policies are variations of one another. The federal court privacy rules are a concrete case: FRCP 5.2 (civil), FRCrP 49.1 (criminal), and FRBP 9037 (bankruptcy) apply the same partial redactions to SSNs, taxpayer IDs, birth dates, minors' names, and financial accounts. 49.1 adds home addresses, 9037 excludes the debtor from the minor-name rule, and each has its own exemptions. Today each must be a complete, separate policy, so a fix to the shared rules has to be made three times and the copies drift.

The same pattern appears wherever a standard policy needs local changes: a court order that requires more or less redaction in one case, a customer's adjustments to a shipped policy, or a department variant of an organisation-wide policy. Layering lets the standard policy keep evolving underneath those variants.

What does this change touch?

• Redaction policy schema (new extends and abstract fields)
• Grammar (EXTENDS, ABSTRACT on the policy declaration)
• Catalog (merge contract; no new reserved keywords)
• Compile contract (EXTENDS compiles to extends)
• Phileas runtime (base resolution and merging, in Java, Python, and .NET)

Proposed change (sketch)

Schema (additive; the current schema/<version>/schema.json edited in place). Two optional top-level properties:

"extends":  { "type": "string",  "description": "Name of the base policy, resolved the way Phileas resolves policies by name." },
"abstract": { "type": "boolean", "default": false, "description": "An abstract policy can be extended but not used to filter." }

Resolution. extends names a policy using the existing naming rule (catalog/policy.yaml: the file basename, with hyphens and underscores equivalent). Phileas resolves it from the same policy store it already loads named policies from.

Merge contract.

• Single inheritance. Chains are allowed; a cycle is a load error.
• For a filter present in both, the derived policy's strategies are evaluated before the base policy's. Existing first-match semantics then make derived strategies take precedence.
• List-valued identifiers (pheyes, identifiers, dictionaries) merge by id; entries without an id are appended.
• A derived filter with "enabled": false disables the inherited filter.
• ignored and ignoredPatterns: union.
• config, crypto, fpe, generators: derived values win per key.
• metadata is not inherited.
• Filtering with an abstract policy is an error.

PhiSQL.

policyDecl
    : (abstractKw=ID)? POLICY policyName=ID (extendsKw=ID baseName=ID)? (DESCRIPTION description=STRING_LITERAL)?
    ;

ABSTRACT and EXTENDS are matched as identifiers in these positions and checked by the compiler, not added to the lexer as reserved keywords. Reserving them would be a major version bump under CONTRIBUTING, because either word could already be used as an identifier, such as a policy named extends.

Worked example. A base policy and a derived one, using only existing strategies. (Restricting date truncation to birth dates is a separate change and not part of this RFC.)

personal-identifiers.phisql:

POLICY personal_identifiers
  DESCRIPTION 'Partial redaction of personal identifiers.';

DEIDENTIFY
  SSN         AS LAST_4,
  CREDIT_CARD AS LAST_4,
  DATE        AS TRUNCATE_TO_YEAR;

REDACT IDENTIFIER('account_number') WITH LAST_4;

frcrp-49-1.phisql:

POLICY frcrp_49_1 EXTENDS personal_identifiers
  DESCRIPTION 'FRCrP 49.1: personal identifiers plus home addresses.';

REDACT STREET_ADDRESS, ZIP_CODE WITH REDACT;

Compiles to frcrp-49-1.json:

{
  "metadata": { "description": "FRCrP 49.1: personal identifiers plus home addresses." },
  "extends": "personal-identifiers",
  "identifiers": {
    "streetAddress": { "streetAddressFilterStrategies": [ { "strategy": "REDACT" } ] },
    "zipCode": { "zipCodeFilterStrategies": [ { "strategy": "REDACT" } ] }
  }
}

At load time Phileas resolves this to the base's ssn, creditCard, date, and identifiers filters plus the two address filters.

Expected versioning impact

PhiSQL spec minor (new grammar and catalog entries; ABSTRACT and EXTENDS are contextual, not reserved). The schema change is additive, so per CONTRIBUTING the current schema/<version>/schema.json is edited in place rather than minting a new schema version.

Backward compatibility

• Every existing policy and .phisql file is unaffected: without extends, no merging happens and filtering is unchanged.
• A policy that uses extends is rejected by a runtime that embeds the schema from before this change, because the schema sets additionalProperties: false and the runtimes validate against it. That is the intended failure: silently ignoring extends would filter without the base's rules.
• Every runtime that loads a layered policy must therefore pick up the updated schema and implement layering first. Per CONTRIBUTING, the schema addition is complete only once Phileas implements it.

Alternatives considered

• Copy policies (status quo). Simple, but shared fixes must be repeated in every copy and the copies drift.
• Compose at authoring time instead. An IMPORT in PhiSQL could merge policies at compile time and emit complete, flattened JSON, with no schema or runtime change, and each deployed policy would be self-contained for audit. This RFC proposes runtime extends because base changes then reach derived policies without recompiling them, and because applications can create small per-case overrides at runtime against a standard policy. This is the main trade-off to settle before accepting.
• Per-strategy overrides by id. Letting a derived policy patch individual fields of a base strategy by id is more precise, but makes the merge contract considerably more complex. Not proposed now.

Open questions

• Should extends be able to pin a base version, so a change to the base cannot silently change derived policies?
• PhiSQL syntax for disabling an inherited filter (the JSON form is "enabled": false).
• Should a concrete policy be validated for completeness at load (for example, every referenced name resolves), with that check skipped for abstract policies?

Acceptance Criteria

• The current schema/<version>/schema.json is edited in place to add optional top-level extends (string) and abstract (boolean, default false), as CONTRIBUTING requires for an additive change.
• The merge contract above is documented in the catalog, including resolution, ordering, list merging by id, enabled: false, per-section rules, metadata not being inherited, cycles, and abstract policies.
• The ANTLR grammar and the EBNF both accept ABSTRACT and EXTENDS on the policy declaration as contextual words, without reserving them, and an existing identifier named abstract or extends still parses.
• The Java and Python reference compilers emit extends and abstract, and the .NET package exposes the updated schema.
• A base and derived example pair under spec/ compiles, and both validate against the updated schema in CI.
• All existing examples still compile to unchanged JSON and validate against the updated schema.
• A policy that extends itself is a compile error.
• Issues exist in philterd/phileas, philterd/phileas-python, philterd/phileas-dotnet, and philterd/phileas-conformance for runtime support and conformance cases, and are linked here.
• The open questions are each answered in this thread before the RFC is accepted.