A Bite of HTTP/2.0: Underlying exploration of HTTP2 and new attack frontiers

Background:

After accumulating some scenarios, the testing requirements for Web Fuzzer HTTP2 should gradually be put on the agenda; many times, when encountering H2 websites during testing, users still have to switch to Burp to continue testing, and are forced to endure the interaction and Repeater/Intruder has quite a big problem with scalability. But in fact, after some simple exploration, HTTP2 support is not particularly complicated, and can be supported with some “innocent” underlying infrastructure settings.

HTTP2 barriers to protocol testing

There is a huge difference between HTTP/2 and HTTP/1.1. The most intuitive feeling is that HTTP2 enforces TLS by default, and H2C (HTTP 2 Cleartext) becomes an “unrecommended protocol”; this is indeed the case. This brings us great inconvenience in studying “HTTP2”: we can’t analyze H2 packets through wireshark by default?

Of course, with some other means, we can see the specific data packets of H2. This is not the focus of this article, so we will not introduce this part. I will write another content later to explain how to use H2C.

Introduction of RFC7541 HPACK

hpack is mainly a compression algorithm for http headers. After compression by this algorithm, the size of the header will be clearly reduced. At the same time, it is also very intuitive that “we have no way to directly observe the header described by the protocol.”

Let’s take a simple example of what the data looks like before and after HPACK compression:

//The simplest format
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.116 Safari/537.36

// Original data header HEXDUMP format
([]uint8) (len=129 cap=243) {
 00000000 55 73 65 72 2d 41 67 65 6e 74 3a 20 4d 6f 7a 69 |User-Agent: Mozi|
 00000010 6c 6c 61 2f 35 2e 30 20 28 57 69 6e 64 6f 77 73 |lla/5.0 (Windows|
 00000020 20 4e 54 20 31 30 2e 30 3b 20 57 69 6e 36 34 3b | NT 10.0; Win64;|
 00000030 20 78 36 34 29 20 41 70 70 6c 65 57 65 62 4b 69 | x64) AppleWebKi |
 00000040 74 2f 35 33 37 2e 33 36 20 28 4b 48 54 4d 4c 2c |t/537.36 (KHTML,|
 00000050 20 6c 69 6b 65 20 47 65 63 6b 6f 29 20 43 68 72 | like Gecko) Chr|
 00000060 6f 6d 65 2f 38 33 2e 30 2e 34 31 30 33 2e 31 31 |ome/83.0.4103.11|
 00000070 36 20 53 61 66 61 72 69 2f 35 33 37 2e 33 36 0d |6 Safari/537.36.|
 00000080 0a |.|
}

// Original data header HPACK format
([]uint8) (len=99 cap=99) {
 00000000 40 88 e0 82 d8 b4 33 16 a4 ff d8 d0 7f 66 a2 81 |@.....3...f..|
 00000010 b0 da e0 53 fa e4 6a a4 3f 84 29 a7 7a 81 02 e0 |...S..j.?.).z...|
 00000020 fb 53 91 aa 71 af b5 3c b8 d7 f6 a4 35 d7 41 79 |.S..q..<....5.Ay|
 00000030 16 3c c6 4b 0d b2 ea ec b8 a7 f5 9b 1e fd 19 fe |.<.K............|
 00000040 94 a0 dd 4a a6 22 93 a9 ff b5 2f 4f 61 e9 2b 0f |...J."..../Oa. + .|
 00000050 32 b8 17 68 20 65 70 85 c5 37 0e 51 d8 66 1b 65 |2..h ep..7.Q.f.e|
 00000060 d5 d9 73 |..s|
}

H2 Frame: Various frames

In addition to the most basic changes in TLS requirements, H2’s “frame”-based communication has also caused a series of difficulties for research and some content.

This frame is the same, we describe the changes in the way that everyone can best understand:

  1. The SETTING frame sets various parameters in the negotiated H2 connection.
  2. UpdateWindows frame: The window in which communication frames can be set
  3. Header frames, header data compressed by hpack, can not only transmit part of the data, but also mark the end of the stream.
  4. Data frame, similar to chunk data, can also be divided into multiple frames, and END_STREAM can mark the end of the stream.
  5. Control frames: PING / SETTING_ACK / GOAWAY / CONTINUATION and other various frames can greatly enhance the expressiveness of H2

You can have an impression here. In fact, we will not focus on exploring the contents of various data frames in this article.

H2 full-duplex communication

After we understand the concept of data frames, we will quickly realize that during the process of data packet transmission, we can transmit data and control frames at the same time. Yes, this is the simplest expression of full-duplex.

In fact, in the Golang standard library implementation process, this implementation is more clear (pseudo code representation):

go readLoop();

...
...

fn readLoop() {
     for {
         frame, err = framer.ReadFrame();
         if err != nil { break }
         
         switch frame.type {
         case http2.MetaFrameHeader:
             ...
         case http2.DataFrame:
             ...
         case http2.SettingFrame:
             ...
         case http2.UpdateWindow:
             ...
         case http2......
         }
     }
}

How do I build an HTTP2 connection?

Generally speaking, to build an HTTP2 connection, we need to build TLS first. In order for everyone to quickly understand this process, I summarized the steps to implement HTTP2 connection in Yaklang into an easy-to-understand sequence diagram:

After we have a rough idea, we can basically understand that HTTP2 will only perform the familiar HTTP communication after the communication is established.

Of course, because I mainly discuss the process of a communication, I intentionally made some shielding in the above content, such as STREAM_ID, and common specific configuration content in SETTING.

So when we actually look at the previous handshake process as a fixed “ceremony”, we may pay more attention to how the “HTTP/1.1” data packets are converted into “HTTP/2.0”, so we will I will explain the conversion process of this data package.

How to convert HTTP/1.1 packets to HTTP/2.0?

This process is actually not sophisticated at all.

HTTP1.1 Header to HTTP2 HeaderFrame

Note: The source of the following content is the Golang HTTP2 standard library

GET /path HTTP/1.1
Connection: close
User-Agent: Demo-User-Agent
Host: Example Domain

The above packet is a very simple case, we can tell at a glance

  1. Method is GET
  2. RequestURI is /path
  3. There are three headers

Then after thinking about it, the HTTP2 protocol does not seem to stipulate that fields such as Method RequestURI have positions similar to HTTP1.1, so there must be other mechanisms to additionally store these important contents.

HTTP1 key fields to HTTP2 table

Type HTTP/1.1 field HTTP/2.0 field Remarks
HTTP request Method :method
ReqestURI :path CONNECT method does not exist
Scheme :scheme Does not exist under CONNECT method
Host :authority
HTTP response StatusCode :status

We found that when the fields in HTTP1 are converted to HTTP2 through the contents of the above table, “:” will be added to the key fields.

This is very critical: in the hpack implementation of the Golang standard library, we distinguish this situation through IsPseudo. We can understand that Pseudo is the description of the content of these keyword headers, which is the key to being compatible with HTTP/1.1.

Header that should not appear again

Of course, as we all know, there are many HEADERs in HTTP/1.1 that are not suitable for H2, such as Connection, Proxy-Connection, etc. These headers will actually be removed by default in the Golang http2 standard library. Which ones will be removed specifically? What about the content?

We make the following summary:

  1. Connection
  2. Proxy-Connection
  3. Transfer-Encoding
  4. Upgrade
  5. Keep-Alive

Cookie Compatibility and Recommendations

In the standard library, cookie handling follows performance optimizations

Per 8.1.2.5 To allow for better compression efficiency, the
Cookie header field MAY be split into separate header fields,
each with one or more cookie-pairs.

In fact, according to the content of the description, this is just a suggestion. If you want to improve performance, you can do this. During the actual test, there was no penalty for not doing this. In order to retain more complex and diverse cookies, we For content, you can choose cookies to be exactly the same as the original content.

HTTP1.1 to HTTP2.0 data frame

In fact, compared with the header, this content actually has no other processing obstacles. It can be implemented directly using http2.Framer.WriteData(streamId, endStream, bodyBytes).

Thinking about attack boundaries

After we understand the basic issues of HTTP2, we will need to think about what potentially testable aspects the introduction of HTTP2 will bring?

HTTP/2.0 Downgrade HTTP/1.1

As we all know, if HTTP2 fails, it will automatically downgrade to HTTP1. Generally speaking, this process is insensitive to most clients, especially browsers; for the sake of compatibility, the server rarely displays “Only allowed” HTTP2″ situation occurs. We don’t need to describe this process too much; but one point that is easily overlooked is “How to upgrade HTTP/1.1 to HTTP2”

HTTP/1.1 upgraded to HTTP/2.0

In fact, most people don’t know about this upgrade process. We can briefly describe the upgrade method. It is actually very similar to websocket. The specific method is to add three extra headers to the original 1.1 data packet to upgrade Connection.

Connection: Upgrade, HTTP2-Settings
Upgrade: h2
HTTP2-Settings: AAMAAABkAAQCAAAAAAIAAAAAA

Through this method, the problem that the Golang standard library does not support H2C can be bypassed.

Well, this method is also easy and cost-effective to discover the basis for whether a web server supports HTTP2 or not.

HTTP/2.0 data frame fragmentation

As we all know, in HTTP/1.1, chunk can directly fragment the body, but under the HTTP/2.0 protocol, framer.WriteData can set END_STREAM, so as long as END_STREAM is not set, any “fragmentation” can be performed.

We use a simple piece of code to explain this process:

func TestFramer(t *testing.T) {
   var buf bytes.Buffer
   framer := http2.NewFramer( & amp;buf, & amp;buf)
   for _, block := range funk.Chunk([]byte(`hello world
hello world
hello worldhello worldhello worldhello worldasdfasdfh
`), 10).([][]byte) {
      framer.WriteData(1, false, block)
   }
   framer.WriteData(1, true, nil)
   spew.Dump(buf.Bytes())
}

After we execute the above code, the execution result is:

([]uint8) (len=159 cap=311) {
 00000000 00 00 0a 00 00 00 00 00 01 68 65 6c 6c 6f 20 77 |..........hello w|
 00000010 6f 72 6c 00 00 0a 00 00 00 00 00 01 64 0a 68 65 |orl.........d.he|
 00000020 6c 6c 6f 20 77 6f 00 00 0a 00 00 00 00 00 01 72 |llo wo.........r|
 00000030 6c 64 0a 68 65 6c 6c 6f 20 00 00 0a 00 00 00 00 |ld.hello .......|
 00000040 00 01 77 6f 72 6c 64 68 65 6c 6c 6f 00 00 0a 00 |..worldhello....|
 00000050 00 00 00 00 01 20 77 6f 72 6c 64 68 65 6c 6c 00 |..... worldhell.|
 00000060 00 0a 00 00 00 00 00 01 6f 20 77 6f 72 6c 64 68 |.....o worldh|
 00000070 65 6c 00 00 0a 00 00 00 00 00 01 6c 6f 20 77 6f |el..........lo wo|
 00000080 72 6c 64 61 73 00 00 08 00 00 00 00 00 01 64 66 |rldas.........df|
 00000090 61 73 64 66 68 0a 00 00 00 00 01 00 00 00 01 |asdfh.....|
}

Visible to the naked eye, we use HTTP2.0 Framer to fragment the transmitted data to achieve a chunk-like effect.

Note

In actual use, most HTTP2 clients will actively remove Transfer-Encoding and use the END_STREAM flag of HTTP2 DataFrame for fragmentation. However, this is a gentleman’s agreement, and whether it is implemented or not is actually implemented by the specific server and middleware.

Fragmented writing of HTTP/2.0 header frames

Of course, inspired by the above case, HTTP2 header frames can naturally be written in slices. In more detail, when writing HTTP2 header frames, we can control the content by controlling various parameters:

http2.HeadersFrameParam{
   //Which specific data stream is this frame data?
   StreamID: 1,
   
   //Specific transmitted data
   BlockFragment: hpackHeaderBytes,
   // Determine whether this data stream has ended. Generally speaking, if there is no Body to be transmitted, it can be set to True.
   EndStream: false,
   // Confirm that Headers transmission is completed
   EndHeaders: false,
   
   // Controlling Padding and priority will not be discussed yet.
   PadLength: 0,
   Priority: http2.PriorityParam{},
}

We won’t go into details about this, but it’s worth noting that the EndHeaders field can transmit multiple complete hpack-encoded headers, which is a very interesting setting.

HTTP2 Frame-Stream constructs “space-time interleaving”

Generally speaking, Framer is the transmission medium. A request or response transmitted in the medium is called a Stream;

Therefore, in the general client implementation of the http2 standard library, each time a new Stream is created, it is n + 2; if the Request is N, the Response will be N + 1, then the next Request will be N + 2;

Then open your mind! We can individually control which Stream a data frame serves through the STREAM ID. That is to say, I can transmit at the same time:

  1. Stream2-Header[1]
  2. Stream1-Data[1]
  3. Stream2-Data[1]
  4. Stream1-Data[end]

In this way, Stream2, as interfering data, can be interspersed in the Data of Stream1. After uninstalling HTTPS, it can quickly blind network devices that cannot restore the HTTP2 data flow, and can also interfere with various “static” rules of normal IDS/IPS.

Summary

Most of the content described above comes from the Golang http2 standard library. After we deeply understand the process and key points described above, we can implement the HTTP2 client ourselves.

During the actual implementation process, the HTTP part of the Yaklang engine removes various “restrictions” in the standard library and directly uses Framer and hpack of http2 to build the HTTP2 client, which can control any step in the communication to implement HTTP2 communication. .

HTTP2.0 support for Web Fuzzer

Through our above description, we have achieved the conversion of HTTP/1.1 style data packets to HTTP2.0 data frames, so

When the Proto part of the data packet is HTTP/2.0 and HTTPS is turned on, Web Fuzzer will automatically switch to HTTP2.

Happy Game!

Yak official resources

Yak language official tutorial:
https://yaklang.com/docs/intro/
Yakit video tutorial:
https://space.bilibili.com/437503777
Github download address:
https://github.com/yaklang/yakit
Yakit official website download address:
https://yaklang.com/
Yakit installation documentation:
https://yaklang.com/products/download_and_install
Yakit usage documentation:
https://yaklang.com/products/intro/
Quick FAQ:
https://yaklang.com/products/FAQ