engine.md

Sans-I/O engine ownership

httpkit-engine composes core values and strict HTTP/1 codecs. Client and server instances have one owner; calls perform no I/O, clock reads, scheduler effects, or application callbacks. The API is in lib/engine/httpkit_engine.mli.

Admission and events

Each connection admits one exchange. A server stops at the first head; later pipeline bytes remain in the caller's suffix until incoming/outgoing bodies finish and output is acknowledged. This supports ordered pipelined input without an unbounded request queue.

IDs contain a local sequence number and connection identity. Equal numbers from different engines cannot be interchanged. Retaining an ID does not retain the entire engine. IDs cannot be fabricated through the public API.

At most one incoming event is pending. Data events own copied strings bounded by the codec step limit. Not polling an event backpressures input. Polling can advance an already-buffered End but never reads a transport. input_state distinguishes idle/head/body input, blocking, and closure; adapters must not start reads while blocked. This prevents speculative reads from stealing post-handshake bytes.

Complete reports incoming body completion once, separately from outgoing finish. An exchange retires only after both directions finish, output drains, and pending completion events are delivered.

Commands and output

Backpressured commands have no effect and may be retried. Accepted commands must not be retried. Metadata is checked before admission. Body misuse after commitment aborts the exchange; no second error response is inserted into a body.

Output is a FIFO of immutable strings and a current offset. Polling returns a stable slice; acknowledgement retires only a prefix of that slice. Invalid counts leave it unchanged. The default queue limit is 64 KiB. Reservations include conservative framing overhead before the body encoder mutates; a command too large ever to fit fails rather than returning permanent backpressure. max_send_size lets adapters split user data into admissible chunks.

The input counter reports pending engine-owned Data payload. The output counter includes queued serialized headers, framing and payload. Codec metadata, queue nodes, bookkeeping and adapter staging add bounded overhead. Application-retained events are outside engine ownership. Independent instances may run on different domains; one instance must not be mutated concurrently.

Early responses and shutdown

A final server response before upload completion aborts the incoming body, emits Body_aborted, and forces close after output drains. Unread upload bytes never become a new request. Explicit discard instead consumes and checks framing to Complete before safe reuse.

A client receiving an early final response stops an unfinished upload, drops unsent queued upload bytes, and prevents reuse. It never transparently retries. Expect blocks body commands until a 100 response or explicit continue_request policy decision. Informational responses have a configurable finite count, default 16.

An accepted finish completes encoding, not transport delivery. A final response with Connection: close or close-delimited framing also drops any remaining queued upload after finish. A persistent final response preserves an already finalized upload; reuse still waits for its acknowledgements. This distinguishes the general continued-transmission guidance in RFC 9110 §7.5 from the rejection/closing case in RFC 9112 §9.5. Aborting a writer that the caller has not finished remains httpkit's conservative early-final policy. Already in-flight bytes cannot be recalled. Both adapters ignore a successful write's acknowledgement when its queue was cancelled during the write, while retaining write-count validation and transport-error handling.

Shutdown stops admission and closes after the active exchange finishes. An adapter owns its deadline and aborts on expiry. Abort is idempotent, preserves the first failure, drops pending work and exposes one Closed event. A peer half-close after a complete request still permits the server to finish its response.

Handoff

CONNECT and 101 responses require matching request context. Upgrade selection must be offered and both Connection headers must nominate upgrade. Handoff cannot cross an unfinished HTTP request body. Handoff is emitted only after required HTTP output is acknowledged. The caller transfers the transport and unconsumed suffix together; HTTP parsing cannot resume afterward.

Evidence

The suite tests both roles, pipeline suffixes, input/output backpressure, stale/foreign IDs, early responses, discard, informational/Expect behavior, abort/EOF/shutdown, invalid acknowledgement, body mismatch, readiness, and handoff. An independent model checks acknowledged bytes against a literal expected prefix under generated command schedules. A client model checks fragmented response identity/order. Independent connections are tested on 1, 2, and 4 domains.

test/engine/client_response_test.ml checks 72 client response configurations: fixed/chunked uploads, Expect and open writers, finalized output at partial/full acknowledgement boundaries, persistent/closing/EOF framing, and 200/413 finals after 103. Every case uses whole, bytewise, every-split and seeded segmentation with work budgets 1, 7 and 16384. It checks exact response suffix ownership, response-body delivery, output cancellation or preservation, event identities, closure and reuse. Eio/Lwt controls also hold a transport write across the final head, complete it partially or fully, and require the response body plus joined cleanup with no further upload writes.

Generated-input targets reuse these models; native campaigns preserve findings and explicit replay evidence. AFL remains deferred. The installed engine consumer runs in bytecode/native modes without runtime adapters. The native adapter suites establish transport, deadline, cancellation and cleanup behavior; pure engine tests do not establish those properties.

test/engine/client_sequence_test.ml exercises 124 authored informational, Upgrade and CONNECT scenarios at work budgets 1/7/16384 under whole, bytewise, every-split and eight seeded schedules, with four request-header acknowledgement states. It checks count boundaries, Expect transitions, EOF before final headers, strict framing failures, protocol selection, exact suffix ownership, once-only handoff/completion and counter/identity isolation on a second exchange. Protocol names compare without case; protocol versions retain case. Multiple selected protocol layers remain unsupported by the current single-selection policy. The existing native client fuzz target runs the same scenario model with generated case, segmentation, budget and acknowledgement choices, alongside its original chunked-body model. These are bounded functional controls, not a release-duration campaign or proof of sustained resource safety.

Expect gates both payload writes and final framing, including empty-body finalization. Backpressure leaves the writer and output queue unchanged.

Internal transition boundaries

Receive and send progress are separate private variants. Notification delivery and output draining are separate from both; this preserves duplex behavior.

Trigger	Receive state	Send state	Retirement condition
Client submits head	Awaiting head	Writing	Wait for response and output
Server receives head	Reading, or Received if empty	Awaiting response	Deliver request and incoming Complete
Incoming body End	Received	Unchanged	Deliver Complete once
Outgoing finish	Unchanged	Sent	Drain queued bytes
Server early final	Aborted input	Writing	Emit Body_aborted; force close after output
Client early final	Reading response	Sent	Drop unsent upload; force close
Valid handoff response	Terminal HTTP input	Sent	Drain HTTP output before Handoff

complete_input, abort_input and complete_output establish the related invariants together. Discard changes retention policy while preserving framing validation and the normal incoming completion transition.

Segmented reuse controls

test/engine/reuse_test.ml drives fixed, chunked and empty-body pipelines under whole/byte/every-split/random input and work budgets 1/7/16384. It checks normal and discard delivery, body/event identity, exact suffix ownership, admission blocked through partial output acknowledgements, and single idle closure. Early final responses before and during body parsing must abort the body and close after draining output; they cannot reinterpret the unread suffix as a request.