are Signalcommands the only commands where package name != command name?

[floswald]

Hi lars,

I wanted to implement your stata scanner without having to deploy stata, i.e. just read the source code and go through a list of keywords. is this a futile approach?

trying to understand whether this list here

replication-template-development/tools/Stata_scan_code/ado/auxiliary/signalcommands.txt

Line 1 in 7332c52

Signals Package

is what one would use to pick up stata commands which do not bear the name of their package.

suppose I get

ssc hot, n(1000)

Could I reasonably assume (yes there will some errors, but fine) that for the 1000-20 (files in that list) I can confront a user's code with my list of 980 patterns, and say if I match binscatter, I could claim that you are using the binscatter package (see net describe binscatter, from(http://fmwww.bc.edu/RePEc/bocode/b))

thanks!

[larsvilhuber]

Background on the Stata scanner:

• the most current version is at https://github.com/labordynamicsinstitute/Statapackagesearch . The version in the template repo is a bit older, and not fully dynamic (on my todo list, forgot to add it to the issues, now done: Update Stata scanner #62 )
• The better way to parse the keywords is embedded in that version. It pulls keywords from https://github.com/labordynamicsinstitute/Statapackagesearch/tree/main/auxiliary but that process, too, is not (yet) dynamic.
• The current set of keyword-to-package mapping comes from Sergio Correia, who in turn transformed the output from the SSC mirror at https://github.com/labordynamicsinstitute/ssc-mirror/
• I have injected some commands into the SSC mirror scripts that actually generate the raw data for this, but have not finished parsing it see https://github.com/labordynamicsinstitute/ssc-mirror/blob/main/.github/workflows/mirror.yml#L55 An example is https://github.com/labordynamicsinstitute/ssc-mirror/actions/runs/15204148070/artifacts/3183115554. This needs to be used, within a GHA workflow, to generate the right mapping file.
• This in turn needs to be used by https://github.com/labordynamicsinstitute/stata-keyword-crosswalk/

So the right place to fix this is in https://github.com/labordynamicsinstitute/ssc-mirror/, to there generate a list of keyword-package crosswalks as an output, and store it in the releases branch.

Some of this has been addressed (as a dead-end branch) in the stats branch of that repo https://github.com/labordynamicsinstitute/ssc-mirror/tree/stats

[larsvilhuber]

A further note on the Stata scanner:

• the reason the Stata scanner is programmed in Stata was so that it would be easy to use for the target audience (classic Stata users)
• It absolutely makes sense to use Python to do the same kind of parsing
• I would use a "bag of words" approach, which are pre-cooked tools to parse all the words by their frequency, and possibly context.
• You should be able to implement the logic behind the Stata Package search into Python easily.

[floswald]

ok i see - that's what I thought/hoped. I'm looking into this right now. Not sure I understand why one needs the bag of word approach here, can you explain? Is it because some package/command names are plain english words which you would also find in a sentence? My idea was that I only look at the code files, so I don't have to worry about english prose and can forget about the context of the word. So then, if I find binscatter I can be sure that this a stata command and not an english word (well let's assume that word existed).
I'm implemeting this in julia, there's plenty of tools around as well.

[larsvilhuber]

There are signal words that are plain english word (table, etc.) because people have chosen that for ado files.

IMHO, Julia is not the right tool for something that is pure text processing, but knock yourself out.

[larsvilhuber]

@floswald Did you do this in a way that I could pull into this workflow?

[floswald]

nope sorry...