Skip to content

Parity: SSN detection misses Unicode hyphens and line-wrapped identifiers #82

Description

@jzonthemtn

Description

The formatted SSN patterns in ssn_filter.py handle ASCII hyphens, unformatted numbers, and ASCII spaces. They do not cover nonbreaking hyphens (U+2011), or an ASCII hyphen followed by a newline between groups.

Fixture difference: the Java audit used 078-05-1120, but Python explicitly excludes that identifier. Use 123-45-6789 for this parity regression so the formatting fix does not require weakening the existing exclusion rules. The exclusion itself is not the defect reported here.

Expected behavior and parity

Track parity with philterd/phileas#385, opened for finding R4 from the Philter 4.0 release audit. Recognize supported Unicode separators and line wrapping within an otherwise valid SSN. Redaction must cover the complete original identifier and preserve surrounding text.

Use these fabricated test inputs (decode the escapes before filtering):

Case Input
ASCII control SSN: 123-45-6789
Nonbreaking hyphens SSN: 123\u201145\u20116789
Line wrapping SSN: 123-45-\n6789

The two variant gaps were identified by source inspection in this implementation; an end-to-end runtime reproduction has not yet been performed. Confirm the behavior with regression tests before implementing the correction.

Acceptance criteria

  • Confirm the ASCII control and reproduce both variant failures through the detector and redaction pipeline.
  • Align supported Unicode separators and line-wrap patterns with the behavior agreed in SSN detection misses Unicode hyphens and line-wrapped identifiers phileas#385.
  • Preserve accurate offsets into the original text if normalization is introduced.
  • Add negative cases to prevent matching unrelated numbers across lines and preserve existing SSN validity/exclusion rules.
  • Verify detection spans and final redacted output, including repeated identifiers and surrounding Unicode text.
  • Document intentional exclusions.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions