options: expose SOCKS5, cookies-file, pause, strip-query, keep-* and a modern User-Agent#36
Merged
Merged
Conversation
…a modern UA Surface the crawl options the engine gained between 3.49-3 and 3.49-12 in the Android options UI, keeping parity with the WinHTTrack set (xroche/httrack-windows#31): - Proxy: an HTTP/SOCKS5 selector that prepends socks5:// to the single -P value and defaults to the SOCKS port 1080 instead of 8080. - Spider: --cookies-file (-%K), a Netscape cookies.txt to preload. - Flow control: --pause (-%G), a random MIN:MAX-second delay between downloads. - Expert: --strip-query (-%g). - Links: the --keep-www-prefix / --keep-double-slashes / --keep-query-order toggles that opt out of parts of the -%u URL hack. - Browser ID: drop the hardcoded "Windows 98" User-Agent default. The field now starts empty, so no -F is emitted unless the user sets one and the engine applies its own current versioned token. Adds a JUnit test over the new argv emission: the SOCKS5 scheme and port default, the value tokens, the %-flag bundling, and the empty-skip that makes the User-Agent suppression work. Closes #6 Signed-off-by: Xavier Roche <roche@httrack.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Surfaces the crawl options the HTTrack engine picked up between 3.49-3 and 3.49-12, which the Android GUI predates, so it lines up again with WinHTTrack (xroche/httrack-windows#31). The proxy panel gets an HTTP/SOCKS5 selector that prepends
socks5://to the single-Pvalue and defaults to the SOCKS port 1080. The spider panel gets a cookies-file field, flow control a random--pauseinMIN:MAXseconds, the expert panel--strip-query, and the links panel the threekeep-*toggles that opt out of parts of the-%uURL hack.It also drops the hardcoded
Mozilla/4.5 ... Windows 98User-Agent default. The browser-identity field now starts empty, so no-Fis passed and the engine fills in its own current versioned token unless the user sets one.A JUnit test covers the new argv emission: the SOCKS5 scheme and port default, the value tokens, the
%-flag bundling, and the empty-skip that the User-Agent change relies on. There's no instrumentation suite in this repo, so the widgets themselves are compile-verified only, not device-tested.Closes #6