read the readme

[floswald]

I'm working on a readme parser. I want to flag interesting terms for the replicator. I'm looking for help with search terms, in particular complicated datasets. Can you give me a hand with some recurring keywords? what other complex datasets/ organizations come to your mind? thanks

    searches = ["confidential", "proprietary", "not available", "HPC", "intensive", "IPUMS", "LEHD", "Statistics Norway", "FDZ", "IAB", "RDC", "Statistics Sweden", "CASD", "VisitINPS"]

[larsvilhuber]

Can't really help, except suggest we parse the READMEs crossed with my internal databases' flag for access difficulty. If you have somebody to work on it, I can gladly give access.

[floswald]

yeah well...that would be myself :-) if you can share that I could have a look, not sure how useful this is anyway. Parsing PDFs is working surprisingly bad. I am going to suggest people submit an .md preferrably.

[larsvilhuber]

Asking people to submit MD files is a recipe for failure and backlash, most economists don't know what that is.

Use a PDF-to-MD (markitdown, https://github.com/microsoft/markitdown) or PDF-to-ASCII (ps2ascii) converter, there's even a Linux command PDFGREP which works very well.

[larsvilhuber]

@floswald have you made any progress?