Add "pup" (like "jq", but for HTML) to the docker image

[marco-m]

Hello,

thanks for dynamic-metalink-resource, we are happy users of it!

We are tracking a product whose release is announced only via HTML, no JSON :-(
As such, to use dynamic-metalink, we do a series of ugly sed on the HTML page.

Looking around, I found pup (https://github.com/ericchiang/pup) which can really be described as "jq, but for HTML".

Would you consider adding it to the resource?

thanks!

[marco-m]

Forgot to mention I will be happy to provide a PR.

[dpb587]

Hi - happy to hear this resource has been helpful for you! I know what you mean about parsing HTML pages – it's no fun. The pup option sounds vaguely familiar; I think I checked it out a few years ago. I like the concept. Having checked it out again this afternoon, it doesn't seem like it is still maintained (e.g. many issues pending feedback, the latest release segfaulted for me, no recent commits).

Before trying to support it in this resource, do you have thoughts about:

• What are some other alternatives to pup? Are they more maintained?
• Have you tested and verified using pup for the use cases you'll want it for? Is it more practical than grep/sed?
• If you're asking about public HTML/server/assets and willing to share concrete case(s), I'd be happy to help brainstorm if there might be alternative, creative ways to scrape and automate.

Thanks!

[marco-m]

Hello, you make very fair points and I understand your hesitation. I too noticed that unfortunately pup is unmaintained and I just assumed it would work for my case. I will come back with an answer to your questions.

[marco-m]

Ok, I did an investigation. Here is what I found:

• no reasonable alternatives to pup
• maybe the segfault you mentioned is because you are on macOS and followed the brew instructions in the README? They are known to be buggy, a simpler brew install pup gives a perfectly working executable, I tested it. See Update brew install instructions ericchiang/pup#116
• I tested and verified for my use case. The URL is public so you can have a look if you want. To get a semver string, I can simplify from

      curl -s https://veertu.com/download-anka-build |
      grep -E -o 'download-link-cell">[^>]*</div>' |
      grep -E 'Controller.*Linux' -A 1 |
      grep -Eoi ">Ver \d+\.\d+\.\d+<" |
      grep -Eoi "\d+\.\d+\.\d+"
1.1.0

to

      curl -s https://veertu.com/download-anka-build |
      pup ':contains("Anka Controller") + .download-link-cell text{}' |
      grep -Eoi "\d+\.\d+\.\d+"
1.1.0

So from my point of view, although not maintained any more, the latest release v0.4.0 of pup from github is quite stable. Also, adding pup to the image doesn't make the resource less robust, it depends whether pup is used or not, and even if it is used my (limited) tests show that it is working fine. What do you think?

[marco-m]

Hello @dpb587, did you have any chance to look at my reply above?