What writing an HTTP server taught me about frameworks

I built an HTTP server in C++ from scratch — no HTTP libraries. Not "no fancy frameworks." No libraries. Just raw syscalls, a poll() loop, and a parser I had to write myself. The project is called serveme, and it lives at github.com/shinraxtensei/serveme. This post is everything I wish someone had told me before I started.

The central insight is unglamorous: a web server is a loop. One loop. It asks the kernel which file descriptors are ready, services each one, and goes back to waiting. That's it. Every framework — Express, Nginx, Kestrel — is this loop, dressed in a nicer coat.

Everything is a file descriptor

The Unix model is elegant to the point of violence: a socket is just a file descriptor. The same read() / write() calls that work on files work on sockets. The kernel does not care. A TCP connection is just bytes flowing through a numbered slot in the process's open-file table.

When you call socket(), you get back an integer. That integer IS the socket. Everything else — binding, listening, accepting — is just setting properties on that integer and telling the kernel what you want to do with it.

// The entire socket lifecycle, start to finish
int sockfd = socket(AF_INET, SOCK_STREAM, 0);  // get an fd
//  AF_INET  = IPv4
//  SOCK_STREAM = TCP (reliable, ordered bytes)
//  0 = default protocol (TCP for SOCK_STREAM)

int reuse = 1;
setsockopt(sockfd, SOL_SOCKET, SO_REUSEADDR, &reuse, sizeof(reuse));
// SO_REUSEADDR: don't make us wait 2 min after restart (TIME_WAIT state)

fcntl(sockfd, F_SETFL, O_NONBLOCK);
// Critical: never block waiting. More on this in a moment.

sockaddr_in addr;
addr.sin_family = AF_INET;
addr.sin_addr.s_addr = INADDR_ANY;  // listen on all interfaces
addr.sin_port = htons(8080);        // htons: host byte order -> network byte order
bind(sockfd, (struct sockaddr*)&addr, sizeof(addr));

listen(sockfd, 100);  // kernel queues up to 100 pending connections for us

// Now the server socket sits there. When a client connects:
sockaddr_in client_addr;
socklen_t addrlen = sizeof(client_addr);
int clientfd = accept(sockfd, (struct sockaddr*)&client_addr, &addrlen);
// clientfd is a brand new fd — one per connected client

The blocking problem — why one slow client freezes everything

Here is the naive approach: accept a client, read their request, send a response, repeat. This works for exactly one client at a time. As soon as a second browser tab connects, it waits — in the kernel's queue — until you finish with the first one. This is what blocking I/O means.

Calling recv(fd, buf, len, 0) on a socket that has no data yet will sleep your process until data arrives. The OS suspends you, runs something else, and wakes you when bytes show up. That's fine for one client. For a server handling thousands, it's death. You can't be asleep for one client while another is trying to connect.

The traditional fix was threads: one thread per connection. But threads are expensive — each needs its own stack (~8 MB), scheduling overhead, and locking hell for shared state. Nginx famously abandoned that model and went to a single-threaded event loop. So did Node.js. So did serveme.

Multiplexing: asking the kernel "who is ready?"

The solution to blocking is I/O multiplexing: instead of blocking on a single fd, you hand the kernel a list of fds and say "tell me which of these have data, without blocking on any of them." The kernel does the waiting. You do the work.

There are three main syscalls for this on Linux/Unix: select(), poll(), and epoll(). select() is ancient (limited to 1024 fds, bitmask API). epoll() is modern Linux-only (O(1) ready-set, ideal for thousands of connections). poll() is the middle ground — a cleaner API than select(), portable, and exactly what serveme uses.

// The poll() API
struct pollfd {
    int   fd;       // the file descriptor to watch
    short events;   // what events you care about (POLLIN, POLLOUT)
    short revents;  // what events actually happened (filled by kernel)
};

// You give poll() an array of these, and a timeout.
int ret = poll(fds, nfds, timeout_ms);
// ret = number of fds that have events ready
// ret = 0 on timeout, -1 on error

// Flags:
// POLLIN  = data is available to read (or new connection on a server socket)
// POLLOUT = socket can accept writes without blocking
// POLLHUP = peer closed the connection
// POLLERR = error on the fd

The actual loop — and it is exactly what you think

Here is the core of serveme, the real handleConnections() function stripped to its skeleton. This is the entire server:

void Core::handleConnections() {
    // Add all server (listening) sockets to the poll array first
    for (auto& sock : serverSockets) {
        pollfd pfd;
        pfd.fd = sock->get_sockfd();
        pfd.events = POLLIN;   // we want to know when a connection arrives
        pfd.revents = 0;
        pollFds.push_back(pfd);
    }

    while (true) {                                  // <-- the infinite loop
        int ret = poll(pollFds.data(), pollFds.size(), 60);
        if (ret < 0) throw std::runtime_error("poll() failed");

        for (size_t i = 0; i < pollFds.size(); i++) {

            if (pollFds[i].revents & POLLIN) {

                if (check_servers_socket(pollFds[i].fd) != -1) {
                    // A new TCP connection arrived on a listening socket
                    // accept() completes the handshake, gives us a new client fd
                    Client* client = new Client(serverSockets[...]);
                    fcntl(client->fd, F_SETFL, O_NONBLOCK);  // must be non-blocking
                    pollFds.push_back(client->pollfd_);       // add to the watch list
                    map_clients[client->fd] = client;

                } else {
                    // An existing client has data ready to read
                    map_clients[pollFds[i].fd]->handleRequest();
                }
            }

            if (pollFds[i].revents & POLLHUP) {
                // Client closed the connection
                removeClient(*map_clients[pollFds[i].fd]);
            }
        }
    }
}

Non-blocking I/O — why O_NONBLOCK is non-negotiable

Poll tells you a fd is ready. But "ready" means "at least 1 byte is available." If you call recv() and ask for 4096 bytes, only 200 may have arrived. You'll get 200 and recv() returns immediately. Good. But what if you then call recv() again to get the rest? Without O_NONBLOCK, you block — waiting for 3896 more bytes while every other client in your poll array starves.

With O_NONBLOCK set via fcntl(fd, F_SETFL, O_NONBLOCK), the behavior changes: if no data is available, recv() returns -1 with errno == EAGAIN (or EWOULDBLOCK — same thing). That is not an error. It means "nothing here yet, come back later." You add the fd to your poll list, return control to the loop, and wait for the next POLLIN event.

// In SocketWrapper constructor — always set non-blocking
SocketWrapper::SocketWrapper(int domain, int type, int protocol) {
    sockfd_ = socket(domain, type, protocol);
    int reuse = 1;
    setsockopt(sockfd_, SOL_SOCKET, SO_REUSEADDR, &reuse, sizeof(reuse));
    fcntl(sockfd_, F_SETFL, O_NONBLOCK);  // ← non-negotiable
}

// And again when we accept() a new client fd
Client::Client(SocketWrapper* sock) {
    fd = sock->accept(*addr);      // get the client fd
    fcntl(fd, F_SETFL, O_NONBLOCK); // immediately make it non-blocking
    pollfd_.fd = fd;
    pollfd_.events = POLLIN;
}

Parsing is a state machine, not a string split

HTTP/1.1 requests arrive as raw bytes. There's no framing — no length prefix, no magic separator. You receive whatever the kernel decided to buffer, which might be half a header, two full headers, or a header and the start of a body. You have to reassemble them yourself.

The naive approach — split("\r\n") — breaks immediately on partial reads. The correct approach is a state machine: each byte you receive advances a state, and you only act when a state completes.

// States in serveme's request parser
enum Stat {                       // the real enum name in inc/client.hpp
    START         = 1 << 0,       // skip leading CRLFs
    FIRSTLINE     = 1 << 1,       // "GET /path HTTP/1.1"
    HEADERS       = 1 << 2,       // "Key: Value" lines
    DONE          = 1 << 4,       // headers complete

    // chunked transfer sub-states
    CHUNKED_START = 1 << 5,
    CHUNKED_SIZE  = 1 << 6,       // hex chunk length
    CHUNKED_DATA  = 1 << 7,       // chunk bytes

    // multipart/form-data sub-states
    MULTI_PART_START    = 1 << 8,
    MULTI_PART_BOUNDARY = 1 << 9,
    MULTI_PART_HEADERS  = 1 << 10,
    MULTI_PART_DATA     = 1 << 11,
    END                 = 1 << 12,

    // BODY isn't a single state — it's a MASK: "are we anywhere in the body?"
    BODY = CHUNKED_START | CHUNKED_SIZE | CHUNKED_DATA
         | MULTI_PART_START | MULTI_PART_BOUNDARY
         | MULTI_PART_HEADERS | MULTI_PART_DATA | END,
};

// The parser reads one byte at a time from the socket,
// accumulates into a line buffer, and advances state on CRLF:
void Client::handleRequest() {
    char buf[1];
    int ret = recv(fd, buf, 1, 0);  // one byte at a time — intentional
    request->line += buf[0];

    if (request->line contains "\r\n") {
        if (state == FIRSTLINE)  { ParseFirstLine(line); state = HEADERS; }
        else if (state == HEADERS) { ParseHeaders(line);                  }
        line = "";
    }
    if (request->buffer contains "\r\n\r\n") {
        state = BODY;   // blank line = end of headers
    }
}

The first line gives you method, URL, and HTTP version. Each header line is a Key: Value pair. A blank line (\r\n\r\n) marks end of headers. Then — depending on Content-Length or Transfer-Encoding: chunked — you may need to read a body. Each variant has its own state machine nested inside the outer one.

The config file is a grammar

One of the less-obvious parts of the project: you have to write your own config parser. Serveme uses a Nginx-inspired format. The grammar has three nesting levels: http → server → location, each with its own set of directives.

http {
    client_max_body_size  200000000;  # bytes

    server {
        server_name  mysite;
        listen       8080;            # or ip:port
        root         /www;
        error_page   404 404.html;

        location / {
            allowed_methods  GET POST;
            autoindex        on;
        }

        location \.py$ {             # matched by suffix string, not real regex
            fastcgi_pass  /usr/bin/python3;  # CGI handler
        }
    }

    server {
        server_name  upload;
        listen       8989;
        root         /upload;
        location / {
            allowed_methods  GET DELETE;
        }
    }
}

Parsing this requires a proper lexer + parser pair. The lexer tokenises the stream (identifiers, braces, semicolons). The parser builds a tree of Http → Server[] → Location[] objects, each with a directive map. Serveme even has a -d flag that generates a Graphviz .dot file so you can visualise the config tree — useful for debugging nested location blocks.

Virtual hosting — one IP, many servers

A single serveme process can serve multiple domains on different ports, or even the same port with different server_name values. The decision of which server block handles a request happens on the Host: header, not on the IP.

void Client::selectServer() {
    // Step 1: find all server blocks listening on this port
    std::vector<Server> candidates;
    for (auto& s : Servme::getCore()->get_http()->servers) {
        if (s.ipPort.second == socket->get_listenPair().second)
            candidates.push_back(s);
    }

    // Step 2: match server_name to Host header
    for (auto& s : candidates) {
        if (s.server_name == request->host) {
            server = new Server(s);
            return;
        }
    }

    // Step 3: fall back to first matching server (Nginx behaviour)
    server = new Server(candidates[0]);
}

The Host header arrives during the HEADERS parsing phase. As soon as we see it, we call selectServer() — so every subsequent decision (root path, allowed methods, max body size, error pages) is scoped to the correct server block.

MIME types — the boring detail that breaks everything

When you serve a file, the browser needs to know what it is. The Content-Type response header tells it. Serving style.css without Content-Type: text/css means the browser will refuse to apply it. Serving script.js as text/plain means it won't execute.

Serveme loads a mime.types file at startup — a simple two-column table mapping extensions to MIME strings — into a std::map<string, string>. When building a response, we look up the file extension in that map and set the header. No library, no database. Just a flat file and a lookup.

Path traversal: a stack and a rule

The moment you map a URL onto a file on disk, you inherit a classic exploit: GET /../../etc/passwd — a request that tries to walk out of your web root. Serveme defuses it before any path matching with a small stack-based normaliser.

std::string normalizePath(std::string path) {
    std::stack<std::string> out;
    // split the path on '/', then fold each segment:
    //   "."   -> skip            (current dir, no-op)
    //   ".."  -> out.pop()       (go up) — but ONLY if out isn't empty
    //   else  -> out.push(seg)   (descend)
    // A ".." against an empty stack does nothing: you can never climb above root.
    ...
}

Every request URL runs through this before it touches the filesystem, and the invariant is one line: a ".." can never pop above the root. No regex, no blocklist of "bad" strings to outsmart — just a stack and a rule, which is far harder to trick than string-matching for ../.

CGI — how dynamic content works without a framework

The fastcgi_pass directive points at an interpreter (/usr/bin/python3, /usr/bin/ruby). When a matching URL is requested, the server fork()s a child process, exec()s the interpreter on the script, wires stdin/stdout via pipes, and reads the response back. The child outputs a valid HTTP response; the server forwards it to the client.

This is the original CGI model — no persistent workers, no socket pools. One request, one process. Expensive, but it's the simplest model there is, and it makes the mechanism completely transparent. You see exactly how $_POST data ends up in a Python script: as environment variables and stdin bytes.

The honest part: the writes are naive

I keep saying "never block" — and serveme's read path holds that line: non-blocking sockets, one poll(), a byte at a time. Its write path is where the toy shows. Responses go out in a loop that tracks how far it got (sendPos), but a send() returning -1 is treated as fatal:

int sent = send(client_fd, responseStr.c_str(), responseStr.length(), 0);
if (sent == -1 || sent == 0)
    throw std::runtime_error(E500);  // <-- treats EAGAIN as a 500
sendPos += sent;

On a non-blocking socket a full kernel send buffer makes send() return -1 / EAGAIN — which is not an error. It means "the pipe is full, come back when it drains." The correct move is to buffer the unsent tail, flip the fd to POLLOUT, and flush when poll() reports writability. Serveme doesn't — and that gap is the line between a from-scratch webserv and Nginx. Knowing exactly where that line sits is half the reason the project is worth doing.

What every framework is quietly doing

Before this project, when something went wrong in Express or NestJS I would scan Stack Overflow for the symptom. After it, I scan the syscall layer first. The symptom is almost always one of these:

• EAGAIN / EWOULDBLOCK — the socket had nothing to give. Not an error; schedule a retry.
• Partial reads — recv() returned fewer bytes than you asked for. Accumulate and retry.
• SIGPIPE — you wrote to a socket the client already closed. serveme does signal(SIGPIPE, SIG_IGN) for exactly this reason.
• TIME_WAIT — the OS is holding the port after you restart. SO_REUSEADDR lets you bind immediately instead of waiting 120 seconds.
• send buffer full — write() returns EAGAIN on the send side. You must buffer your response, register for POLLOUT, and flush when the kernel signals writability.
• select() fd limit — select() only handles fds below 1024. If you accept() fd number 1025, it silently corrupts memory. Use poll() or epoll().

It's layers, all the way up

Once you've seen the floor, the whole tower makes sense. serveme's loop — accept, parse, route, respond — is the bottom layer of every web framework you've ever used. They don't replace it; they stack conveniences on top of it:

• The event loop — poll() + non-blocking sockets. Node hides it behind libuv, the JVM behind an NIO Selector, Go behind the runtime netpoller. Same move: ask the kernel who is ready, never block.
• HTTP parsing — the byte-by-byte state machine becomes a ready-made request object. A Java Servlet container (Tomcat, Jetty) or Node's http.Server hands you method, headers, and body already framed.
• Routing — the handler map. Express's app.get(), Spring's @RequestMapping, a NestJS @Controller route — all just "match this method + path to this function," which serveme does by hand.
• Middleware / filters / interceptors — the chain that runs around your handler: Servlet filters, Express middleware, NestJS guards and interceptors. Functions wrapped before and after the byte-level work.
• Dependency injection — NestJS providers, Spring's IoC container: a graph that hands your handler its collaborators, so it declares what it needs instead of constructing it.

Every one of those is sugar over the same four moves: a descriptor became ready, read the bytes, decide, write the bytes back. A Servlet is handleRequest() with a lifecycle bolted on. DI is a tidy way to pass that handler its dependencies. Routing is a std::map from paths to functions. Strip every abstraction away and you land exactly where serveme starts — at poll() and recv().

I built it without HTTP libraries precisely because using them hides the machinery. You don't understand Transfer-Encoding: chunked until you've written a hex-to-size state machine at 2 a.m. You don't understand backpressure until send() returns EAGAIN and you have to figure out what to do with the bytes you couldn't flush. The constraint is the lesson.