Add --user-agent and -user-agent-suffix CLI options

[liquidsec]

#1453

[codecov]

Codecov Report

Attention: Patch coverage is 45.45455% with 6 lines in your changes missing coverage. Please review.

Project coverage is 93%. Comparing base (f36633f) to head (13cdc27).
Report is 37 commits behind head on dev.

Files with missing lines	Patch %	Lines
bbot/scanner/preset/args.py	46%	6 Missing ⚠️

Additional details and impacted files

@@          Coverage Diff          @@
##             dev   #2190   +/-   ##
=====================================
- Coverage     93%     93%   -0%     
=====================================
  Files        376     376           
  Lines      29225   29236   +11     
=====================================
+ Hits       27006   27013    +7     
- Misses      2219    2223    +4     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[TheTechromancer]

Thanks for doing this one.

In the interest of keeping --help as lean as possible, what do you think of having only --user-agent as a cli flag?

--user-agent-suffix can still be a config option. I think we can have a threshold where, if it's a popular option and people are asking for it to be a cli flag, we can "graduate" it kind of like we did for --exclude-cdn.

Mainly I want to avoid a situation like nuclei where we have so many cli flags, that it makes it hard to find the one you're looking for. Keeping the help output small will be a real challenge considering all of BBOT's functionality and the functionality we're adding, like the upcoming server / agent stuff etc.

[liquidsec]

I think some feedback from the community might be good on that. Personally, I would probably only ever use the suffix one. Obviously I know how to make a preset / add the option and its not too big of a deal, but I think for the bug bounty crowd, there's a lot of situations where they will say: you must insert your hackerone username in user-agent field, etc...

I guess I don't feel strongly either way, I think it's a useful flag that a select user base will make of, but I also hate cluttered CLI help pages.

[liquidsec]

I think -c user_agent_suffix=whatever is still miles more convenient than googling common user-agents, picking one, pasting it in, putting quotes around it, and inserting a suffix manually, so if you want to start with it out of args i'm ok with that.

[liquidsec]

@TheTechromancer I left it in there but commented it, to make it easy for people who really want it to enable it themselves

[TheTechromancer]

We should probably add user_agent_suffix: "" in defaults.yml as well

[liquidsec]

Removing it made the combining logic not work, so i added it back but suppressed it from help. Effectively an undocumented feature.

[liquidsec]

We should probably add user_agent_suffix: "" in defaults.yml as well

the way it works, user_agent_suffix never actually exists as a config option. It just lives for within the args long enough to change user_agent. If we want to do that we might just need to revisit the whole implementation