Implement HTTP 1.0 & fix TCP_client

[gpotter2]

This PR implements:

• HTTP 1.0 layer. Based on scapy_http but:
  • added sessions support: follow chunks/streams of TCP HTTP (Content_Length and Chunked)
  • added decompression & re-compression support (3 default methods implemented: gzip, lzip, zlib)
  • fixes build: quite a few things were broken
  • add support for all HTTP fields
  • fixed for PEP8
  • improved Python 3 support, general bytes handling (especially with regexes/split)
  • Removed deprecated fields (Let's not import already-deprecated stuff)
  • added tests: test chunk reassembly, content_length threshold, TCPSession, gzip decompression (most common)
• TCPsessions. Used to sort TCP packets via their seq/ack. Used to decode HTTP flows. Could be extended to other TCP protocols
• TCP_client & Automaton: fixes for Automaton: support _ATMT_Supersocket on Windows. Also update it to h'share the StringBuffer of the TCPSession, so that it can also handle TCP fragmentation.
• documentation about HTTP/TCP_client
• add other specific tests (gzip compact utiles...)

Note: because of Python 2-3 compatibility, several utils were needed:

• gzip_decompress/gzip_compress: gzip.decompress/gzip.compress isn't available on Python 2.7.

Important note: sr() is NOT supported YET, because it would require to change sndrcv() to handle Sessions. This is done in #1999. (The 2 PRs must be merged before it may be implemented) (done)

This code was able to decode a JPEG image sent through several HTTP chunks, despite TCP re-transmission. You can check it out by yourself by un-commenting a line in the chunked-image test

Rationale

• Why scapy_http ?

It has been around for quite some time. It allows users/projects that were using it to migrate seamlessly, to use the new features.
I do not want to change the API, and it allowed to start on an existing basis.

• What's next ?

This PR should be able to fully replace scapy_http which development has been paused (stopped?). That would allow to deprecate scapy_http and continue any further development here.
I've asked the maintainers if they want to merge the projects, therefore release a last scapy_http version with a warning deprecating it. I doubt they want to invest more time into maintaining it 😄

See invernizzi/scapy-http#46

[codecov]

Codecov Report

Merging #1925 into master will increase coverage by 0.18%.
The diff coverage is 87.08%.

@@            Coverage Diff            @@
##           master   #1925      +/-   ##
=========================================
+ Coverage   87.12%   87.3%   +0.18%     
=========================================
  Files         198     199       +1     
  Lines       44679   45078     +399     
=========================================
+ Hits        38926   39356     +430     
+ Misses       5753    5722      -31

Impacted Files	Coverage Δ
scapy/contrib/isotp.py	89.44% <ø> (+5.43%)	⬆️
scapy/plist.py	86.22% <ø> (ø)	⬆️
scapy/pipetool.py	89.97% <100%> (+0.06%)	⬆️
scapy/compat.py	100% <100%> (ø)	⬆️
scapy/config.py	84.16% <100%> (+0.03%)	⬆️
scapy/sendrecv.py	84.29% <100%> (+0.02%)	⬆️
scapy/arch/pcapdnet.py	70.4% <100%> (+0.75%)	⬆️
scapy/error.py	100% <100%> (ø)	⬆️
scapy/layers/inet.py	72.84% <100%> (+0.02%)	⬆️
scapy/automaton.py	87.2% <70.17%> (+1.76%)	⬆️
... and 9 more

[gpotter2]

@guedou @p-l- I've finally finished this PR. Feel free to review it 😄

I would have liked to know your thoughts before actually taking any action such as contacting scapy_http maintainer.

On the licensing question: That's GPLv2-GPLv2 so there's no problem.

[gpotter2]

@guedou This is ready for review. p-l- has already reviewed a version (not the latest, but there have only been slight bug fixes since). It appears that tox is failing to install on Travis with 3.4 - unrelated.

[guedou]

Nice PR. See my comments.

[guedou]

Do we need to add the pcap files? They are quite big.

[gpotter2]

I can probably reduce the size of the pcaps.
They're used to test each dissection method

[gpotter2]

@guedou I've compressed the bigger pcap (39.8kb -> 24.8kb). Compressing the other one (8k) didn't result in any gain.

I'm not sure if I can spare more space :/ I would have liked to keep real examples for our HTTP tests.

• The first one contain the smallest chunked image I could find. HTTP chunks only happen on files that are big enough :/
• The second one is a compressed payload. We could try to reduce it even more by setting something custom, but I'd rather have a test that uses real payloads.
• The "unecessary" packets contained in the pcaps are packets use in TCP ACK, which are very small. Deleting them doesn't reduce the size, but makes the pcap less "real"

[gpotter2]

This test: https://github.com/secdev/scapy/blob/master/test/linux.uts#L11
can now be rewritten using Automaton ATMT-SuperSocket conversion mechanisms. Below is a recompiled HTTP payload (2 packets)

>>> load_layer("http")
>>> a = TCP_client.tcplink(HTTP, "www.google.com", 80)
>>> a.send(b"HEAD / HTTP/1.0\r\n\r\n")
>>> a.sniff(session=TCPSession)
<Sniffed: TCP:1 UDP:0 ICMP:0 Other:0>
>>> _[0]
<Ether  dst=[...] src=[...] type=IPv4 |<IP  version=4 ihl=5 tos=0x0 len=None id=32140 flags= frag=0 ttl=121 proto=tcp chksum=None src=[...] dst=[...] |<TCP  sport=http dport=54773 seq=1135001053 ack=1005612689 dataofs=5 reserved=0 flags=A window=60720 chksum=None urgptr=0 |<HTTP  |<HTTPResponse  Http_Version='HTTP/1.0' Status_Code='200' Reason_Phrase='OK' Accept_Ranges='none' Cache_Control='private, max-age=0' Content_Type='text/html; charset=ISO-8859-1' Date='Sun, 09 Jun 2019 19:22:39 GMT' Expires='-1' P3P='CP="This is not a P3P policy! See g.co/p3phelp for more info."' Server='gws' Set_Cookie='NID=185=kq_kBczt9GkdFomd4pTsi8yue5UIAoODedJ8g7DHFYS1a5M2gGWzGRjXJ6uZgKs4Yim-KtnKEhqzg3pFLVngRtcGj0NQaHzEvkbzMyMDnrmUMaEVYSfVLUZPrX0PgNUOxg2n7_HPmULQ27vBSREWyPTu46QToUwZ8OGGRlBX5bM; expires=Mon, 09-Dec-2019 19:22:39 GMT; path=/; domain=.google.com; HttpOnly' Vary='Accept-Encoding' X_Frame_Options='SAMEORIGIN' X_XSS_Protection='0' |>>>>>
>>>

I also updated the dissection method (removed a ton of junk). I've moved more stuff into TCPSession rather than the HTTP part, which makes it way less complicated

[p-l-]

After a first quick read...

[p-l-]

That's great! I have a couple of minor suggestions.

[p-l-]

@guedou your call.

[gpotter2]

A last small fix & one improvement. This now works properly (it will also follow chunks...):

load_layer("http")
req = HTTP()/HTTPRequest(Accept_Encoding=b'gzip, deflate', Cache_Control=b'no-cache', Connection=b'keep-alive', Host=b'www.secdev.org', Pragma=b'no-cache')
a = TCP_client.tcplink(HTTP, "www.secdev.org", 80)
payload = a.sr1(req)
open("www.secdev.org.html", "wb").write(payload.load)

feel free to try it out on your platform 👍 I only tested the automatons on Windows, but they should work as well elsewhere.

This should be ready to merge.

[gpotter2]

Another pretty good demo is available:

load_layer("http")
http_request("www.google.com", display=True)

whereas in wirehark:

[gpotter2]

@guedou Tests now pass. this is mergeable

[guedou]

@gpotter2 do you want me to squash the commits or keep them?

[gpotter2]

Squashing will be better. There has been some stuff that was reverted

[p-l-]

@guedou this is OK for me.

[guedou]

Cool PR. I made some simple comments.

[gpotter2]

I had actually been so focused on HTTP dissection that I overlooked if the build was always correct.
Turns out there were a few issues (corrected in the last commit) with Unknown_Headers. This is now fixed.
As both building and dissection are complete, this is (actually) now ready.

Sorry for the last-minute commits

[gpotter2]

@guedou @p-l- This should be mergeable.
The Scapy-http maintainer said we would work deprecationnout once this has been merged, to make sure it's ready for 2.4.3 :-)

[guedou]

I will try to release v2.4.3rc3 by Wednesday.