Separate wildcard (*) single word lexicon entries into a new autotag data type #36

Summary

Some single word lexicon entries are patterns, not literal words. Two kinds of pattern syntax are used:

• * as a wildcard, e.g. *-year-old matches 10-year-old, 25-year-old, etc.
• [...] character classes, e.g. [0-9] matches any single digit, and [0-9]* matches zero or more digits, so [0-9]*ish matches 9ish and 10ish. These appear in the upstream USAS lexicons but are not in this repository yet.

Pattern entries are currently mixed with ordinary entries in which the same characters are literal, e.g. f***. Tools such as PyMUSAS therefore can't tell which entries to read as patterns and which to match exactly.

This issue proposes moving every single word lexicon entry whose lemma or pos is a pattern into its own file, with a new data type value of autotag.

Proposal

• New data type autotag in language_resources.json. The file format is exactly the same as the single data type (same headers and rules, see Single word lexicon file format). The only difference is how the lemma and pos fields are read. In an autotag file they are patterns:

  • [...] is a regex character class matching exactly one character, e.g. [0-9] matches one digit and [a-z] one lowercase ASCII letter.
  • * straight after a character class means "zero or more characters from that class" (as in regex), e.g. [0-9]*ish matches 9ish and 10ish (see the warning below).
  • * anywhere else means "zero or more of any character" (regex .*), e.g. *-year-old matches 10-year-old.
  • Every other character is literal, e.g. ., +, ?, (. When an entry is converted to a regex, these characters must be escaped.
  • • The pos field cannot be just *. Every autotag entry must give a POS pattern, so that patterns don't match tokens of every POS. A pos pattern can still contain * alongside other characters, e.g. N*.

  PyMUSAS support is tracked in UCREL/pymusas#28, which needs to cover [...] and [...]* as well as *.

• File naming convention: semantic_lexicon_LANG_autotag.tsv, saved in the language's folder. LANG is the same language suffix used by that language's single word lexicon file, e.g. English/semantic_lexicon_en.tsv → English/semantic_lexicon_en_autotag.tsv.

• In single files, *, [ and ] in the lemma field are always literal. The existing Implicit POS rule, where a pos value of exactly * means any POS, stays as it is.

Affected lexicons

At the moment, Danish and English are the only single word lexicons that contain pattern entries. All of these use *, and no entries in the repository use [...] yet. Both lexicons also contain at least one entry where * is literal.

Entries from the upstream USAS lexicons that use [...] (e.g. [0-9]*ish) should go into the matching autotag file when they are added to this repository.

Out of scope: Portuguese/semantic_lexicon_pt.tsv contains * in many lemmas, but how to handle them is undecided (see #5). It is unchanged until #5 is resolved.

Tasks

• Create Danish/semantic_lexicon_dk_autotag.tsv and English/semantic_lexicon_en_autotag.tsv, following the naming convention above, with the same headers as their single counterparts (lemma pos semantic_tags), and remove the moved entries from the original files.
• no autotag entry has a pos value of exactly *
• Add both files to language_resources.json with "data type": "autotag".
• Document the autotag data type in the README (USAS Lexicon Meta Data and Lexicon File Format sections). Cover the pattern syntax ([...] character classes, * after a class meaning zero or more of that class, * elsewhere meaning zero or more of any character; every other character is literal), including that [...]* can match zero characters and how to require at least one. State that *, [ and ] in single files are literal, and give the autotag file naming convention.
• Update CONTRIBUTING.md so that contributors:
  • know autotag lexicon files exist, follow the single word lexicon file format, and support *, [...] and [...]* patterns,
  • know that [...]* also matches zero characters (e.g. [0-9]*ish matches ish), so they may need to add a non-optional class, e.g. [0-9][0-9]*ish,
  • put pattern entries in a semantic_lexicon_LANG_autotag.tsv file rather than in the single word lexicon file,
  • add the file to the meta data file with "data type": "autotag".
  • know that every autotag entry needs a POS pattern, and that a pos value of just * is not allowed,
• Accept autotag everywhere single is accepted and validate it the same way, plus a check that every [ in an autotag lemma/pos starts a well-formed, closed character class (e.g. by compiling the converted regex):
  • txt_to_tsv.py
  • test_collection.py
  • test_all_collections.py
  • lexicon_statistics.py (report autotag as its own type)
  • create_pos_tagsets.py
• Regenerate lexicon_statistics.md and the generated_pos_tagset.tsv files (the generated POS tagset files should be identical, but would be good as a check to ensure that create_pos_tagsets.py is refactored correctly).