Message Boundaries in a TCP Stream

TCP carries bytes in order, but it does not preserve your application's message boundaries. Send two length-prefixed messages over the same connection and reconstruct them even when a header spans several reads.

Course examples on GitHub

Open this lesson's sample file · CMake build instructions.

Before you start

Complete Your First TCP Connection. Use two terminals on the Pi in examples/network.

Outcome: decode two complete messages from a stream, retain incomplete data and reject a truncated frame.

The previous lesson's server echoes raw bytes without interpreting them. Reuse it here; the new client constructs frames and parses their echoed bytes.

Define a TCP frame

Our small application protocol is:

Field	Size	Meaning
Length	4 bytes	Unsigned payload length in network byte order.
Payload	0–64 bytes	Exactly that many bytes; no terminating zero is added.

A frame is 4 + N bytes. The limit of 64 is our protocol's rule, not a TCP limit. Length counts bytes, including for UTF-8 text.

flowchart LR accTitle: Two frames concatenated in a byte stream accDescr: The first four bytes encode length five, followed by hello. The next four bytes encode length two, followed by Pi. TCP does not insert a separator between these frames. A["00 00 00 05<br/>Length: 5"] --> B["hello<br/>5 bytes"] B --> C["00 00 00 02<br/>Length: 2"] C --> D["Pi<br/>2 bytes"]

The header bytes above are hexadecimal. htonl() converts a 32-bit length to network order, and ntohl() converts it back. We copy those four integer bytes, not an entire C++ struct with possible padding.

Run two messages on one connection

Configure and build:

cmake --preset native-debug
cmake --build --preset native-debug --parallel 2

In terminal A, start the existing byte-echo server:

./build/native/debug/tcp_echo_server 9001

Within 30 seconds, run in terminal B:

./build/native/debug/tcp_frames_client 9001

Client output:

frame=1 bytes=5 data=hello
frame=2 bytes=2 data=Pi
frames=2

The server reports echoed bytes=15: eight header bytes plus seven payload bytes. It knows nothing about the two messages.

Buffer first, parse complete frames

The client deliberately receives at most three bytes per call. Even a four-byte length field therefore needs more than one read. A successful recv() may also return fewer than three bytes.

The pending buffer keeps everything not yet consumed. After appending new bytes, the parser follows this loop:

flowchart TD accTitle: Parse complete frames and retain the rest accDescr: Append received bytes to the pending buffer. Wait for a complete four-byte header, validate the decoded length, then wait for the entire payload. Emit a complete frame, remove only its bytes, and look for another frame. R["Receive more bytes"] --> B["Append to pending"] B --> H{"At least 4 bytes?"} H -->|No| R H -->|Yes| L["Decode length N"] L --> V{"N at most 64?"} V -->|No| X["Reject frame"] V -->|Yes| P{"At least 4 + N bytes?"} P -->|No| R P -->|Yes| F["Emit frame<br/>Remove its 4 + N bytes"] F --> H

A read can finish one frame and begin the next. Removing only the completed frame leaves that next prefix in pending. The inner loop also handles several complete frames arriving together when a larger receive buffer is used.

Read the standalone client

Complete file: app/tcp-frames/client.cpp. Socket calls, serialization, parsing and explicit cleanup are all in this one file.

#include <arpa/inet.h>
#include <array>
#include <cerrno>
#include <charconv>
#include <cstdint>
#include <cstring>
#include <iostream>
#include <stdexcept>
#include <string_view>
#include <sys/socket.h>
#include <sys/time.h>
#include <system_error>
#include <unistd.h>
#include <vector>

int main(int argc, char** argv) {
    int fd = -1;
    int status = 0;
    try {
        if (argc != 2 && argc != 4) {
            throw std::invalid_argument(
                "usage: tcp_frames_client port [first second]");
        }
        unsigned int port = 0;
        const std::string_view text = argv[1];
        const auto [end, error] =
            std::from_chars(text.data(), text.data() + text.size(), port);
        if (error != std::errc{} || end != text.data() + text.size() ||
            port == 0 || port > 65535) {
            throw std::invalid_argument("port must be 1..65535");
        }
        const std::array<std::string_view, 2> messages{
            argc == 4 ? argv[2] : "hello", argc == 4 ? argv[3] : "Pi"};
        std::vector<char> wire;
        for (const auto message : messages) {
            if (message.size() > 64) {
                throw std::invalid_argument("message exceeds 64 bytes");
            }
            // Encode only the uint32_t bytes, not an in-memory C++ struct.
            const std::uint32_t length =
                htonl(static_cast<std::uint32_t>(message.size()));
            const auto* header = reinterpret_cast<const char*>(&length);
            wire.insert(wire.end(), header, header + sizeof(length));
            wire.insert(wire.end(), message.begin(), message.end());
        }

        fd = ::socket(AF_INET, SOCK_STREAM | SOCK_CLOEXEC, 0);
        if (fd < 0) {
            throw std::system_error(errno, std::generic_category(), "socket");
        }
        const timeval io_timeout{3, 0};
        if (::setsockopt(fd, SOL_SOCKET, SO_RCVTIMEO, &io_timeout,
                         sizeof(io_timeout)) < 0 ||
            ::setsockopt(fd, SOL_SOCKET, SO_SNDTIMEO, &io_timeout,
                         sizeof(io_timeout)) < 0) {
            throw std::system_error(errno, std::generic_category(),
                                    "setsockopt I/O timeout");
        }
        sockaddr_in server{};
        server.sin_family = AF_INET;
        server.sin_port = htons(static_cast<std::uint16_t>(port));
        server.sin_addr.s_addr = htonl(INADDR_LOOPBACK);
        if (::connect(fd, reinterpret_cast<const sockaddr*>(&server),
                      sizeof(server)) < 0) {
            throw std::system_error(errno, std::generic_category(), "connect");
        }
        std::size_t offset = 0;
        while (offset < wire.size()) {
            const auto sent = ::send(fd, wire.data() + offset,
                                     wire.size() - offset, MSG_NOSIGNAL);
            if (sent < 0) {
                throw std::system_error(errno, std::generic_category(), "send");
            }
            if (sent == 0) {
                throw std::runtime_error("send made no progress");
            }
            offset += static_cast<std::size_t>(sent);
        }
        // The byte-echo server uses EOF to finish this demonstration session.
        if (::shutdown(fd, SHUT_WR) < 0) {
            throw std::system_error(errno, std::generic_category(), "shutdown");
        }

        // Force even a four-byte header to span reads.
        std::array<char, 3> chunk{};
        std::vector<char> pending;
        std::size_t frames = 0;
        while (true) {
            const auto received = ::recv(fd, chunk.data(), chunk.size(), 0);
            if (received < 0) {
                throw std::system_error(errno, std::generic_category(), "recv");
            }
            if (received == 0) {
                if (!pending.empty()) {
                    throw std::runtime_error(pending.size() < 4
                                                 ? "truncated header"
                                                 : "truncated payload");
                }
                if (frames != messages.size()) {
                    throw std::runtime_error("missing frames");
                }
                break;
            }
            pending.insert(pending.end(), chunk.data(),
                           chunk.data() + received);
            // One read may finish a frame and start the next one.
            while (pending.size() >= 4) {
                std::uint32_t encoded_length = 0;
                std::memcpy(&encoded_length, pending.data(),
                            sizeof(encoded_length));
                const auto length = ntohl(encoded_length);
                if (length > 64) {
                    throw std::runtime_error("frame exceeds 64 bytes");
                }
                const std::size_t frame_size = 4 + length;
                if (pending.size() < frame_size) {
                    break; // Keep these bytes and collect more.
                }
                if (frames >= messages.size()) {
                    throw std::runtime_error("unexpected frame");
                }
                const std::string_view body(pending.data() + 4, length);
                if (body != messages[frames]) {
                    throw std::runtime_error("frame echo mismatch");
                }
                std::cout << "frame=" << frames + 1 << " bytes=" << length
                          << " data=" << body << std::endl;
                pending.erase(pending.begin(), pending.begin() + frame_size);
                ++frames;
            }
        }
        std::cout << "frames=" << frames << '\n';
    } catch (const std::exception& error) {
        std::cerr << error.what() << '\n';
        status = 1;
    }
    if (fd >= 0) {
        ::close(fd);
    }
    return status;
}

Read the code in three passes:

1. Build wire: validate each payload, append its four-byte length, then append its bytes.
2. Send the stream: advance by each send() result, then finish the outgoing direction.
3. Parse pending: inspect a complete header, wait for the payload and consume exactly one frame at a time.

memcpy() reads the length into a real uint32_t object, avoiding an unaligned integer dereference into the byte buffer. The length is checked before adding it to the header size or allocating storage for that payload.

The string_view into pending is used before erase() changes the buffer. Comparing it with the original message checks this echo exercise; the length prefix itself determines the frame boundary.

Message boundaries are independent of EOF

The client prints each complete frame immediately. It need not wait for the peer to close before recognizing a message.

shutdown(SHUT_WR) remains here so the earlier byte-echo server knows when this demonstration session has no more input. It does not separate the two frames.

At EOF	Result
Both frames decoded, no leftover bytes	Success.
Some header bytes remain	Truncated header.
Header present, payload incomplete	Truncated payload.
No leftover bytes, fewer than two frames	Missing frame.

A zero-length frame still contains its four-byte header. EOF is recv() == 0, not a frame whose declared length happens to be zero. This particular exercise expects two frames; an ongoing service would define its own session rules.

Change the input and the read size

Restart the server, then send an empty first message:

./build/native/debug/tcp_frames_client 9001 "" Pi

The client should print a first frame of zero bytes and a second frame containing Pi.

Next, change std::array<char, 3> to std::array<char, 64> and rebuild. The same output must result. Both default frames now fit in one receive buffer, but TCP still does not promise a particular number of reads. Restore the original buffer afterwards.

To compile the new file directly:

mkdir -p build/direct
c++ -std=c++20 -Wall -Wextra -Wpedantic \
    app/tcp-frames/client.cpp -o build/direct/tcp_frames_client

Run it against the server with the same port arguments. Linux per-call I/O timeouts and explicit descriptor cleanup follow the preceding lesson.

Check your understanding

A header announces five payload bytes, but the connection ends after only he. Should the receiver emit a two-byte message?

Compare your reasoning

No. The length is the application's contract. The five-byte frame is incomplete, so the receiver reports truncation instead of silently changing the message.

Continue the course

Previous: Your First TCP Connection · Next: Disconnection and Timeouts · Course overview

References: TCP stream semantics, receive results, network byte order.