Decode Chunked HTTP Bodies in C++
A server can send a response before it knows the body's final size. With HTTP/1.1 chunked transfer coding, your client reads a sequence of sized chunks instead of one Content-Length.
Before you start
Complete A Small HTTP Server. Run the commands in two Pi terminals in examples/network. The transport calls, receive deadline and header validation build on the preceding client lessons.
Outcome: decode a chunked body even when a size line, payload or CRLF spans TCP reads, without treating TCP EOF as successful completion.
Chunked coding is message framing, not encryption or compression. This lesson supports a deliberately restricted profile, not all HTTP chunk extensions or trailers.
A minimal response header selects:
| Transfer-Encoding: chunked
|
After the header terminator, the body hello can be encoded as follows. Each \r\n shown stands for actual CRLF bytes:
| Wire bytes |
Meaning |
2\r\n |
Next payload has hexadecimal size 2. |
he\r\n |
Two payload bytes, then a framing CRLF. |
3\r\n |
Next payload has hexadecimal size 3. |
llo\r\n |
Three payload bytes, then a framing CRLF. |
0\r\n |
Last-chunk size line. |
\r\n |
Empty trailer section ends the message in this profile. |
The decoded body contains only hello, not the size lines or framing CRLFs. A payload can itself contain newline bytes: its declared size, not a line search, determines where its data ends.
Sizes are hexadecimal: A is ten bytes and 1000 is 4096 bytes. A zero-size chunk is the last chunk; it is not an empty TCP read.
Keep three parser states
flowchart LR
accTitle: Decode chunk sizes, payloads and the final empty line
accDescr: Read a complete hexadecimal size line. A nonzero size leads to that many payload bytes and their CRLF, then back to size. A zero size leads to the empty trailer line. Only after that line is the response complete. Incomplete data remains buffered in every state.
S["Size line<br/>Wait for CRLF"] -->|"Nonzero N"| D["Data<br/>Wait for N bytes plus CRLF"]
D -->|"Append only N payload bytes"| S
S -->|"Zero"| T["Final empty line<br/>Wait for CRLF"]
T -->|"Empty trailer section"| E["Complete"]
Each state may need more input. The parser keeps parse_at and chunk_size between reads; it does not assume one chunk per recv().
After a read, the inner loop processes every complete unit already buffered. A single receive can contain the end of one chunk and the start of another. A 16-byte receive buffer also forces ordinary response headers to span calls.
Run a real chunked exchange
Build:
| cmake --preset native-debug
cmake --build --preset native-debug --parallel 2
|
Terminal A starts the included one-request fixture, with an outer process limit:
| timeout 35s python3 fixtures/http_chunked_server.py 8082
|
It binds only to loopback and prints listening 127.0.0.1:8082. Within 30 seconds, run in terminal B:
| ./build/native/debug/http_chunked_client 8082
|
Expected output:
| status=200 bytes=15
Hello from Pi.
|
The fixture sends Hello and from Pi.\n as two chunks, writes the zero chunk and empty trailer section, then exits after that request. Restart it before testing another path:
| ./build/native/debug/http_chunked_client 8082 /missing
|
That returns status 404 and the decoded Not found\n body. Generated dates and fixture log messages are not part of the decoded output.
The C++ server from the preceding lesson still sends Content-Length, not chunked coding. Do not pair this restricted decoder with it: the deliberate profile mismatch is an error, not a TCP fault.
Preserve framing checks and bounds
| Rule in this decoder |
Why it matters |
| HTTP/1.1, status 200 or 404 |
The selected response profile has a message body; other status/method rules are not guessed. |
| One Transfer-Encoding equal to chunked |
No duplicate fields or unsupported coding chains. |
| No Content-Length field |
Do not trust conflicting framing information. |
| Hex size line of 1–16 digits |
Reject signs, spaces, extensions and unbounded line growth. |
| At most 4096 decoded bytes |
Check each announced size against the remaining body allowance before copying. |
| At most 4096 header bytes and 16384 wire bytes |
Bound metadata and chunk-framing overhead separately from decoded payload. |
| Payload followed by CRLF |
Do not silently accept malformed chunk boundaries. |
| Empty trailer section only |
Wait for the final empty line after the zero chunk. |
| Fixed receive deadline |
Partial progress never renews the wait. |
The size check uses the remaining allowance, 4096 - body.size(), so an enormous announced size is rejected before adding it to another offset or allocating a payload buffer. Many tiny chunks can exceed the wire cap even while their decoded body remains below 4096.
Field names and coding-token comparisons are case-insensitive. Other header syntax follows the previous ASCII-only teaching profile. Content-Encoding and a Trailer header are not supported.
Reduced profile, not full chunked support
Chunk extensions and nonempty trailers can be valid HTTP. This example explicitly rejects them, along with coding chains, content decoding and unrelated statuses. Do not relabel those valid features as protocol errors or use this decoder as an internet-facing security boundary.
A complete implementation must apply the request/status body rules and the full transfer-coding precedence rules. This lesson selects one safe-to-observe local format instead of pretending that it handles every response.
Read the complete standalone decoder
Complete file: app/http-chunked/client.cpp. Socket setup and the GET string remain visible. Header parsing selects the restricted chunked profile, then the state machine reconstructs the body.
Full standalone decoder: app/http-chunked/client.cpp
| #include <algorithm>
#include <arpa/inet.h>
#include <array>
#include <cerrno>
#include <charconv>
#include <chrono>
#include <cstdint>
#include <iostream>
#include <poll.h>
#include <stdexcept>
#include <string>
#include <string_view>
#include <sys/socket.h>
#include <sys/time.h>
#include <system_error>
#include <unistd.h>
int main(int argc, char** argv) {
int fd = -1;
int status = 0;
try {
if (argc < 2 || argc > 4) {
throw std::invalid_argument(
"usage: http_chunked_client port [path [receive_ms]]");
}
unsigned int port = 0;
const std::string_view text = argv[1];
const auto [end, error] =
std::from_chars(text.data(), text.data() + text.size(), port);
if (error != std::errc{} || end != text.data() + text.size() ||
port == 0 || port > 65535) {
throw std::invalid_argument("port must be 1..65535");
}
const std::string_view path = argc >= 3 ? argv[2] : "/hello.txt";
if (path.empty() || path.front() != '/' || path.size() > 200 ||
!std::all_of(path.begin(), path.end(), [](unsigned char c) {
return c > 32 && c < 127 && c != '#';
})) {
throw std::invalid_argument("path must be a visible ASCII origin "
"path without spaces or fragments");
}
int receive_ms = 3000;
if (argc == 4) {
const std::string_view budget = argv[3];
const auto [last, invalid] = std::from_chars(
budget.data(), budget.data() + budget.size(), receive_ms);
if (invalid != std::errc{} ||
last != budget.data() + budget.size() || receive_ms < 1 ||
receive_ms > 10000) {
throw std::invalid_argument("receive_ms must be 1..10000");
}
}
fd = ::socket(AF_INET, SOCK_STREAM | SOCK_CLOEXEC, 0);
if (fd < 0) {
throw std::system_error(errno, std::generic_category(), "socket");
}
const timeval setup_timeout{3, 0};
if (::setsockopt(fd, SOL_SOCKET, SO_SNDTIMEO, &setup_timeout,
sizeof(setup_timeout)) < 0) {
throw std::system_error(errno, std::generic_category(),
"setsockopt");
}
sockaddr_in server{};
server.sin_family = AF_INET;
server.sin_port = htons(static_cast<std::uint16_t>(port));
server.sin_addr.s_addr = htonl(INADDR_LOOPBACK);
if (::connect(fd, reinterpret_cast<const sockaddr*>(&server),
sizeof(server)) < 0) {
throw std::system_error(errno, std::generic_category(), "connect");
}
const std::string request =
"GET " + std::string(path) +
" HTTP/1.1\r\nHost: 127.0.0.1:" + std::to_string(port) +
"\r\nConnection: close\r\n\r\n";
std::size_t offset = 0;
while (offset < request.size()) {
const auto sent = ::send(fd, request.data() + offset,
request.size() - offset, MSG_NOSIGNAL);
if (sent < 0) {
throw std::system_error(errno, std::generic_category(), "send");
}
if (sent == 0) {
throw std::runtime_error("send made no progress");
}
offset += static_cast<std::size_t>(sent);
}
// HTTP's empty header line ends this bodyless GET, not TCP EOF.
using Clock = std::chrono::steady_clock;
const auto deadline =
Clock::now() + std::chrono::milliseconds(receive_ms);
std::array<char, 16> buffer{};
std::string response;
bool headers_done = false;
std::size_t body_start = 0;
enum class Phase { Size, Data, FinalEmptyLine };
Phase phase = Phase::Size;
std::size_t parse_at = 0;
std::size_t chunk_size = 0;
std::string body;
bool complete = false;
int http_status = 0;
while (true) {
const auto remaining = std::chrono::ceil<std::chrono::milliseconds>(
deadline - Clock::now())
.count();
if (remaining <= 0) {
throw std::runtime_error("receive deadline");
}
pollfd ready{fd, POLLIN, 0};
const int result = ::poll(&ready, 1, static_cast<int>(remaining));
if (result < 0) {
if (errno == EINTR) {
continue;
}
throw std::system_error(errno, std::generic_category(), "poll");
}
if (result == 0 || Clock::now() >= deadline) {
continue;
}
const auto received =
::recv(fd, buffer.data(), buffer.size(), MSG_DONTWAIT);
if (received < 0) {
if (errno == EAGAIN || errno == EWOULDBLOCK || errno == EINTR) {
continue;
}
throw std::system_error(errno, std::generic_category(), "recv");
}
if (received == 0) {
throw std::runtime_error(headers_done ? "truncated chunked body"
: "truncated headers");
}
response.append(buffer.data(), static_cast<std::size_t>(received));
if (response.size() > 16384) {
throw std::runtime_error("wire response exceeds 16384 bytes");
}
if (!headers_done) {
const auto boundary = response.find("\r\n\r\n");
if (boundary == std::string::npos) {
if (response.size() > 4096) {
throw std::runtime_error("headers exceed 4096 bytes");
}
continue;
}
body_start = boundary + 4;
if (body_start > 4096) {
throw std::runtime_error("headers exceed 4096 bytes");
}
const auto first_end = response.find("\r\n");
const std::string_view status_line(response.data(), first_end);
if (status_line.size() < 13 ||
!status_line.starts_with("HTTP/1.1 ") ||
status_line[12] != ' ' ||
!std::all_of(status_line.begin(), status_line.end(),
[](unsigned char c) {
return c == '\t' || (c >= 32 && c < 127);
})) {
throw std::runtime_error("invalid status line");
}
const auto code = status_line.substr(9, 3);
if (code != "200" && code != "404") {
throw std::runtime_error(
"status outside this lesson's 200/404 profile");
}
http_status = code == "200" ? 200 : 404;
bool transfer_seen = false;
std::size_t cursor = first_end + 2;
while (cursor < boundary) {
const auto line_end = response.find("\r\n", cursor);
const std::string_view line(response.data() + cursor,
line_end - cursor);
const auto colon = line.find(':');
if (colon == std::string_view::npos || colon == 0) {
throw std::runtime_error("invalid header line");
}
std::string name(line.substr(0, colon));
const std::string_view punctuation = "!#$%&'*+-.^_\x60|~";
if (!std::all_of(name.begin(), name.end(),
[punctuation](unsigned char c) {
return (c >= 'a' && c <= 'z') ||
(c >= 'A' && c <= 'Z') ||
(c >= '0' && c <= '9') ||
punctuation.find(c) !=
std::string_view::npos;
})) {
throw std::runtime_error("invalid header name");
}
for (auto& c : name) {
if (c >= 'A' && c <= 'Z') {
c += 'a' - 'A';
}
}
auto value = line.substr(colon + 1);
if (!std::all_of(
value.begin(), value.end(), [](unsigned char c) {
return c == '\t' || (c >= 32 && c < 127);
})) {
throw std::runtime_error("invalid header value");
}
while (!value.empty() &&
(value.front() == ' ' || value.front() == '\t')) {
value.remove_prefix(1);
}
while (!value.empty() &&
(value.back() == ' ' || value.back() == '\t')) {
value.remove_suffix(1);
}
if (name == "content-length") {
throw std::runtime_error(
"Content-Length forbidden in this profile");
}
if (name == "content-encoding" || name == "trailer") {
throw std::runtime_error(
"response feature unsupported");
}
if (name == "transfer-encoding") {
std::string coding(value);
for (auto& c : coding) {
if (c >= 'A' && c <= 'Z') {
c += 'a' - 'A';
}
}
if (transfer_seen || coding != "chunked") {
throw std::runtime_error(
"expected one chunked Transfer-Encoding");
}
transfer_seen = true;
}
cursor = line_end + 2;
}
if (!transfer_seen) {
throw std::runtime_error(
"missing chunked Transfer-Encoding");
}
headers_done = true;
parse_at = body_start;
}
while (!complete) {
const auto available = response.size() - parse_at;
if (phase == Phase::Size) {
const auto line_end = response.find("\r\n", parse_at);
if (line_end == std::string::npos) {
if (available > 17 ||
(available == 17 && response.back() != '\r')) {
throw std::runtime_error(
"chunk size line too long");
}
break;
}
const std::string_view size_text(response.data() + parse_at,
line_end - parse_at);
if (size_text.empty() || size_text.size() > 16 ||
!std::all_of(size_text.begin(), size_text.end(),
[](unsigned char c) {
return (c >= '0' && c <= '9') ||
(c >= 'a' && c <= 'f') ||
(c >= 'A' && c <= 'F');
})) {
throw std::runtime_error(
"invalid chunk size or unsupported extension");
}
const auto [last, invalid] = std::from_chars(
size_text.data(), size_text.data() + size_text.size(),
chunk_size, 16);
if (invalid != std::errc{} ||
last != size_text.data() + size_text.size()) {
throw std::runtime_error("invalid chunk size");
}
if (chunk_size > 4096 - body.size()) {
throw std::runtime_error(
"decoded body exceeds 4096 bytes");
}
parse_at = line_end + 2;
phase =
chunk_size == 0 ? Phase::FinalEmptyLine : Phase::Data;
} else if (phase == Phase::Data) {
if (available < chunk_size + 2) {
break;
}
if (response.compare(parse_at + chunk_size, 2, "\r\n") !=
0) {
throw std::runtime_error("missing chunk data CRLF");
}
body.append(response.data() + parse_at, chunk_size);
parse_at += chunk_size + 2;
phase = Phase::Size;
} else {
if (available < 2) {
break;
}
if (response.compare(parse_at, 2, "\r\n") != 0) {
throw std::runtime_error(
"nonempty trailers unsupported");
}
parse_at += 2;
complete = true;
}
}
if (complete) {
break; // The zero chunk AND empty trailer section end the body.
}
}
std::cout << "status=" << http_status << " bytes=" << body.size()
<< '\n';
// Close this socket; surplus bytes are not a second response.
std::cout.write(body.data(), body.size());
} catch (const std::exception& error) {
std::cerr << error.what() << '\n';
status = 1;
}
if (fd >= 0) {
::close(fd);
}
return status;
}
|
Temporary header and size-line views are used only while the wire string is unchanged. The persistent progress variables are offsets and counts, not views into storage that a later append can invalidate.
The client returns as soon as the zero chunk and final empty line complete the message, even if the peer has not closed. It ignores any already-read surplus bytes and closes its own descriptor; it does not implement connection reuse or pipelining.
Direct compilation:
| mkdir -p build/direct
c++ -std=c++20 -Wall -Wextra -Wpedantic \
app/http-chunked/client.cpp -o build/direct/http_chunked_client
|
The receive phase defaults to 3000 ms, optionally set after port and path. Setup calls have separate per-call timeouts. The fixture's outer 35-second limit bounds its process lifetime; the protocol diagram does not imply any timing guarantee.
Verify the final delimiter, not just the visible text
| ctest --preset native-debug -R '^network[.]http_chunked[.]' --output-on-failure
|
The 33 cases include the real fixture, combined and split wire data, empty/binary payloads, uppercase hex, mixed-case headers, truncated size/data/final lines, malformed sizes and CRLFs, length conflicts, unsupported extensions/trailers, aggregate limits, wire overhead and deadlines. A maximum-length size line with its CRLF split across writes is checked separately.
Successful synthetic peers remain open until the decoder returns. The truncation cases close before a required unit is complete. This separates message completion from connection completion.
| After receiving |
Result |
| Complete last data chunk, but no zero chunk |
Not finished. |
| Zero chunk line, but no final empty line |
Not finished. |
| Zero chunk and empty trailer section |
Complete in this profile. |
| EOF before either required ending |
Truncated chunked body. |
Check your understanding
All the expected text is visible, but the peer closes before the zero chunk arrives. May the client report a complete response because it already has the useful text?
Compare your reasoning
No. Chunked framing supplies the message ending. Without the zero chunk and trailer-section terminator, EOF leaves an incomplete response even if its text looks plausible.
Continue the course
Previous: A Small HTTP Server · Next: HTTP Connection Reuse · Course overview
References: chunked transfer coding, chunk extensions, trailers.