Added course pipeline: crawl, cache, classify, extract, and CTDL map

[nsoto-tech]

No description provided.

[rohit-joy]

@nsoto-tech Create a top level folder named xtra-cli. We'll turn these into readily usable commands from local machine or an automated workflow.

Under that, have a folder for each stage of the ETL pipeline.

1. crawling
2. downloading
3. extraction
4. transformation
5. scoring

and a lib folder for shared/reusable code between these scripts.

Under these folders, you can move the rest of the scripts. There will be multiple such scripts under these folders.

You can see existing patterns for command structures here: https://docs.google.com/document/d/1WD0AQAWuY0klJYznBST7-O93mHXSlR-k0LRQQPAqCKU/edit?tab=t.0#heading=h.xr3byxpiy9rz and the code that follows that structure here in ceops CLI: https://github.com/CredentialEngine/ce-registry/tree/main/ceops . Try to follow the same structure and add test coverage for each command you are wanting to check in. Your tests can of course include sample html or pdfs which you may use for unit or integration testing your scripted commands.

Note that the same scripts will be used for both benchmarking as well as actual extractions. So there is no difference between "golden" or non-golden from the script point of view. Creating golden examples is a business scenario that is enabled through the scripts and promoting/copy/pasting files from one folder to another.

[nsoto-tech]

@rohit-joy xtra-cli/ is set up as you described: one folder per ETL stage (crawling, downloading, extraction, transformation, scoring) plus lib for shared code.

Commands follow the ceops noun-verb form. Same scripts for a live extract and for a pack that later gets copied into a reviewed folder. Golden is that destination after review (pack promote), not a second pipeline.
python xtra-cli/xtra_cli.py catalog crawl --pack my_pack --url https://catalog.brookdalecc.edu --limit 5
python xtra-cli/xtra_cli.py page download --pack my_pack --normalize
python xtra-cli/xtra_cli.py course extract --pack my_pack
python xtra-cli/xtra_cli.py course transform --pack my_pack
python xtra-cli/xtra_cli.py course score --reference my_pack --candidate dumps
Same scripts for extract and for packs that later get copied. Tests live in xtra-cli/tests/ with sample HTML. Engine still sits in lib/ for this PR (cli.py still there) next is folding that into the stage folders. Schema golden_value left until we bump the contract.

[rohit-joy]

@nsoto-tech Thanks for the updates. Some questions:

1. What is a pack? This might constrain us at this point, whereas we need to be able to run the scripts on any catalog of any size. ZIPping and folders by packs will likely slow us down. Just ISO8601 Timestamp yyyy-MM-ddThh:mm:ssZ for UTC time.
2. I like the crawl verb. But there may be multiple strategies to crawl. So organize the scripts such that you can specify which strategy to use to crawl. e.g. Crawling with playwright, crawling with AI agent, crawling with third party crawling service, etc. We need to be able to readily add such strategies in due time, if not today. So the command could potentially look like xtra catalog crawl --with-playwright --url ... as a way to specify playwright strategy.
3. We will do the same thing for extract and transform commands also. So those need to follow the same pattern as crawl.
4. Limit 5 seems to be limiting to 5 pages. This is fine for testing.
5. What I'm more interested in is seeing a concurrency limit so we are not inadvertently putting pressure on the 3rd party sites or getting blocked by their reverse proxy or bot detectors. So we should also have a parameter to download slowly as in download one page with exponential back offs with minimum 3 minutes or so, so that we are not triggering getting blocked and also not hurting the sites. Note that crawling is going to run nearly all the time to keep our cache up to date, and there are thousands of catalogs with thousands of pages to crawl.
6. Regarding the file structure and command invocation, please try to follow the same structure as in the ceops CLI commands. There is a neat folder structure that is in the form of src/noun/noun/verb.py and corresponding tests in tests/noun/noun/test-verb.py.
7. Reuse, reuse, reuse - Take what you need from ceops CLI to read and write from Azure Blob storage. You can use the Azurite emulator locally to write tests and test locally. When it comes to production, we need to be reading/writing straight with Azure Blob storage account containers.
8. When in doubt, try to align with what ceops CLI is doing. If there is still a doubt, then ask me proactively. :)

[rohit-joy]

Name the test files with prefix test_.

[rohit-joy]

Every source file in the src folder must have a corresponding test_ file under tests with the same folder structure.