Check generated POS tagset against the expected POS tagset in CI #35

Summary

In the future it would be good to have a step in the CI pipeline that checks the POS tags within a language's generated POS tagset (generated_pos_tagset.tsv) against the expected POS tags given within that language's pos resource, and fails the CI if any generated tag is not in the pos resource.

Background

• The pos resource type in language_resources.json points to a user created POS tagset file, which states the POS tagset that the language's lexicon files are expected to use. See the POS tagset file format section (Tag and Definition columns).
• create_pos_tagsets.py creates generated_pos_tagset.tsv per language folder from the POS tags actually used within the language's single and MWE lexicon files (POS and Count columns). See the Generated POS Tagsets section.
• At the moment the pos resource files are not used to check the lexicon files, nor the generated POS tagsets, and create_pos_tagsets.py is not run in CI. Therefore a lexicon file can use a POS tag that is not in the stated tagset and nothing will flag it.

Proposal

Add a script and CI step that, for each language that has a pos resource in language_resources.json:

1. Reads the expected POS tags from the Tag column of the pos resource file.
2. Reads the POS tags from the POS column of the language's generated_pos_tagset.tsv.
3. Fails the CI (non-zero exit code) if any generated POS tag is not within the expected POS tags, printing the offending tags.

Matching rules

• One direction only: the CI only fails when a tag in the generated POS tagset is missing from the pos resource. Tags that are in the pos resource but not in the generated POS tagset (not used by any lexicon file) do not fail the CI.
• Wildcard tag *: this is a valid implicit POS value (see the Implicit POS section), it is allowed to be in the generated POS tagset without being in the pos resource.
• Case sensitive: tags are compared exactly. As create_pos_tagsets.py lower cases all POS tags before saving them, this script would have to be adjusted so that it can compare case sensitive tags while also maintaining it's current function.

Prerequisites

This issue is blocked by issue #34 , which has to be resolved first.

Points to decide

• Regenerating in CI: should CI run create_pos_tagsets.py to get fresh generated files before comparing, rather than trusting the committed generated_pos_tagset.tsv files?

Acceptance criteria

• Issue Add a pos resource to every language and add tagset name / tagset url meta data #34 is resolved.
• A script exists that performs the comparison for all languages.
• The CI fails when a generated POS tag is not in the pos resource, with a clear message of the offending tags.
• The CI passes when all generated POS tags (other than *) are in the pos resource, even if the pos resource has additional tags.
• The README documents the check.