(sigil http client)

Construct a url struct.

Test if a value is a url struct.

Get the scheme field of a url struct.

Get the host field of a url struct.

Get the port field of a url struct.

Get the path field of a url struct.

Get the query field of a url struct.

Parse a URL string into a url record.

Supports http://host:port/path?query and https://.... Use url-scheme, url-host, url-port, url-path, url-query to access the components.

(let ((u (parse-url "https://example.com:8080/api?key=val")))
  (url-host u))  ; => "example.com"

Convert a timeout in seconds to whole milliseconds (at least 1).

Is this a usable positive timeout?

Resolve host to a list of IP address strings in the resolver's own preference order, or #f.

The list is REVERSED on the way out. resolve-hostname walks getaddrinfo's results forward and conses, so what it returns is the RFC 6724 preference order backwards. Connecting in that order would try the least-preferred address first and give it the first slice of the budget, silently inverting v4/v6 preference on a dual-stack host whenever a connect timeout is set. The unbounded tcp-connect path gets the order right, so this would also have made the two paths disagree.

RESIDUAL: the resolution itself is NOT bounded. See the "Blocking segments" section above.

Connect to the first address that answers, with the whole attempt bounded by total-ms.

Each address gets a fair slice of the REMAINING budget rather than the whole of it, so a blackholed first address cannot consume the entire deadline and leave a working address untried. Same shape the TLS connect path already uses.

Connect a plain TCP socket, bounded by connect-timeout when one is given. Without a timeout this is the original blocking tcp-connect, byte for byte.

Connect to a server, using TLS if scheme is https. Returns connection object or #f on failure.

connect-timeout (seconds, or #f) bounds getting to the point where the request can be sent. For HTTPS that is TWO segments, and both are bounded by this one value:

• the TCP connect, so a blackholed address can't hang on the OS SYN timeout; and
• the TLS handshake, which the connect timeout does NOT reach. Once a peer has ACCEPTED, the connect phase is over. A peer that then never sends a ServerHello left the handshake read blocking forever, which under a cooperative scheduler freezes the whole process. That is the failure that ran a production service for 55 days with every health surface reporting it healthy.

Each phase gets the full value rather than a shared split, so a slow but working connect does not eat the handshake's budget. The worst case before the request is sent is therefore about twice connect-timeout, not once.

Plain HTTP has no handshake, and its connect is bounded the same way.

Omitted or #f keeps the original blocking connect on both paths.

Write data to connection (socket or TLS)

Read data from connection (socket or TLS)

Read binary data from connection as bytevector (socket or TLS)

Close connection

Write a byte range of bv to the connection, returning the number of bytes accepted (which may be 0 on a non-blocking connection), or #f. Both layers take start/end offsets, so a retry after a partial write never copies the remaining tail.

Wait a short tick for a plain socket to accept more bytes.

socket-write returns #f for EAGAIN and for a genuine error alike, and NOTHING here can tell them apart. An earlier version tried to: if select said the socket was writable, the #f must have been a real error. That is wrong, and wrong in the destructive direction. Linux reports a socket writable once half the send buffer is free, so against a peer that IS draining the sequence EAGAIN-then-writable is ordinary backpressure, and treating it as an error failed a perfectly good 4 MiB upload.

So we wait and retry, and let the DEADLINE end it. The cost is that a genuine write error surfaces as a write timeout rather than an immediate failure. That is the safe direction: a slow failure for a dead socket beats a spurious failure for a live one. The real fix belongs in socket-write, which should distinguish would-block from error rather than making its callers guess.

Two things about socket-select that its type signature does not say, and that a plausible-looking call gets wrong in the reassuring direction:

• the timeout is an exact integer of MILLISECONDS, not seconds. A float raises rather than being rounded.
• it returns (readable writable), a two-element list of lists, NOT a flat list of ready sockets. So (pair? (socket-select ...)) is ALWAYS true, which is how the mistake above got written.

Put a connection (socket or TLS) into non-blocking mode. Used by the timeout path so reads return immediately when no data is available, letting the read loop enforce a deadline.

Compute an absolute deadline in jiffies from a timeout in seconds, or #f when no (positive) timeout was requested.

Has the (jiffy) deadline passed?

Whole milliseconds left before deadline, possibly negative.

Is exn an "ack unconfirmed" error (request sent, response read failed)? Distinguishes a delivered-but-unconfirmed send from a genuine never-sent failure (the latter does not carry this mark).

Raise an "ack unconfirmed" error: the request was sent but the response could not be read (read deadline, empty/closed read, or unparseable response). Carries ack-unconfirmed-irritant.

Raise a clean timeout error (read deadline). Marked ack-unconfirmed because the deadline only fires after the request was written.

Raise a clean timeout error for the request WRITE.

Deliberately NOT marked ack-unconfirmed. The write deadline can only fire with bytes still unsent, so the server holds a partial request it cannot act on. That is a never-delivered send and retrying it is safe, which is the opposite of what the ack-unconfirmed mark tells a caller to do.

Build HTTP request string

Build header lines from headers (dict or alist)

Parse HTTP status line Returns (version status-code reason) or #f

Parse response headers from data Returns dict with keyword keys

Find the end of the header block (rnrn) in a bytevector. Returns the index of the first r, or #f if not yet present.

Does this method/status combination forbid a response body? HEAD never has a body; 1xx/204/304 never have a body (RFC 9112).

Inspect the accumulated response bytes and determine how the body is framed. Returns #f while the header block is still incomplete, otherwise a descriptor: (no-body) — no body permitted; complete at headers (length <body-start> <n>) — fixed Content-Length body (chunked <body-start>) — chunked transfer-encoding (until-close) — unframed; read until the peer closes This lets the reader stop as soon as the full body has arrived instead of waiting for the connection to close (EOF), which a keep-alive/half-open peer may never do.

Is the chunked body starting at start fully present in bv? Walks chunk-size lines until the terminating 0-size chunk and its closing CRLF (no trailers). Returns #f if more data is needed.

Is the response complete per its framing descriptor? combined is the accumulated bytevector (only needed for chunked).

Write the whole request, bounded by deadline when one is set.

Returns #t when every byte was accepted, #f on a genuine write error, and raises a write-timeout error if the deadline passes with bytes still unsent.

With deadline #f this is the original single blocking write. A request usually fits the socket buffer, so that write usually returns immediately — but "usually" is not a bound. A peer that accepts the connection and never drains its receive queue fills the window, and a large body (an upload, a big POST) then blocks here indefinitely.

With a deadline the connection is already non-blocking, so a write that cannot proceed returns 0 (TLS) or #f-with-the-socket-unwritable (plain) instead of blocking, and the loop polls until the deadline.

Read HTTP response from connection Returns http-response or #f on error. method is the request method (HEAD responses carry no body). deadline is a jiffy deadline (or #f). When set, the read loop enforces a timeout instead of blocking on a stalled read.

Read a full HTTP response from the connection. Reads as bytevectors and converts to string once at the end to avoid splitting multi-byte UTF-8 characters across chunks.

Termination is HTTP-framing-aware: once the header block has arrived, the response is considered complete as soon as the body is fully received per its framing (Content-Length, the chunked 0-terminator, or a body-less status/method). The reader does NOT wait for the connection to close — a keep-alive or half-open peer may hold it open indefinitely even after sending a complete response, which previously hung the read (or, with a deadline, tripped a spurious timeout on an already-delivered request). Only an unframed response (no Content-Length, not chunked) falls back to reading until EOF.

When deadline is #f, reads block. When set, the connection is non-blocking: an empty read means "no data yet", so we sleep briefly and retry until the response completes, the peer closes, or the deadline passes (which raises a timeout error).

Parse HTTP response from string

Decode chunked transfer encoding using byte-level operations. Chunk sizes in HTTP are byte counts, so we must work with bytes to correctly handle multi-byte UTF-8 content. Format: <hex-size>rn<data>rn ... 0rnrn

Find position of rn in a bytevector starting at pos

Find position of rn in a string starting at pos

Convert hex string to number

Find the end of HTTP headers (position of rnrn)

Skip CRLF sequence(s) at position

Make an HTTP request.

Low-level function for making HTTP requests. Prefer the convenience functions (http-get, http-post, etc.) for common cases.

The optional timeout: keyword (seconds) bounds the request WRITE and the response READ, each against its own deadline. The connection is made non-blocking and a timeout error is raised if the request cannot be sent, or no complete response arrives, in time. Omitted, both block exactly as before.

A read timeout is marked ack-unconfirmed (http-ack-unconfirmed?): the request went out, so retrying may re-deliver it. A WRITE timeout is not, because it can only fire with bytes still unsent, leaving the server a partial request it cannot act on.

The optional connect-timeout: keyword (seconds) bounds getting to the point where the request can be sent: the TCP connect, so a blackholed address can't hang on the OS SYN timeout, AND the TLS handshake, which the connect phase does not cover. Each gets the full value, so the ceiling before the request is sent is about twice it. Omitted/#f keeps the blocking connect and handshake.

NOT bounded by either keyword: DNS resolution. See the "Blocking segments" section at the top of this file.

(http-request 'GET "https://api.example.com/users"
              headers: #{ authorization: "Bearer token" })

(http-request 'POST "https://api.example.com/users"
              headers: #{ content-type: "application/json" }
              body: "{\"name\": \"Alice\"}"
              timeout: 25)

HTTP GET request.

(http-get "https://example.com/")

(http-get "https://api.example.com/users"
          headers: #{ authorization: "Bearer token" })

HTTP POST request.

If no Content-Type header is provided, defaults to application/x-www-form-urlencoded.

(http-post "https://api.example.com/data" "key=value")

(http-post "https://api.example.com/data"
           "{\"key\": \"value\"}"
           headers: #{ content-type: "application/json" })

HTTP PUT request.

(http-put "https://api.example.com/users/123"
          "{\"name\": \"Alice\"}"
          headers: #{ content-type: "application/json" })

HTTP DELETE request.

(http-delete "https://api.example.com/users/123")

(http-delete "https://api.example.com/users/123"
             headers: #{ authorization: "Bearer token" })

HTTP HEAD request.

Like GET but only retrieves headers, not body. Useful for checking if a resource exists or getting metadata.

(let ((res (http-head "https://example.com/file.pdf")))
  (http-response-header res "Content-Length"))

HTTP OPTIONS request.

Query server for allowed methods on a resource.

(let ((res (http-options "https://api.example.com/users")))
  (http-response-header res "Allow"))
; => "GET, POST, OPTIONS"

HTTP PATCH request.

Partially update a resource. Unlike PUT which replaces the entire resource, PATCH applies partial modifications.

(http-patch "https://api.example.com/users/123"
            "{\"email\": \"new@example.com\"}"
            headers: #{ content-type: "application/json" })

Parse HTTP response body as JSON.

Returns the parsed JSON value, or #f if the response is #f or parsing fails. Requires sigil-json package.

(let ((res (http-get "https://api.example.com/data")))
  (http-response-json res))
; => #{ users: #[...] count: 42 }

HTTP GET request expecting JSON response.

Makes a GET request and parses the response body as JSON. Returns the parsed JSON value, or #f if request fails or status is not 2xx.

(http-get/json "https://api.example.com/users")
; => #{ users: #[...] }

(http-get/json "https://api.example.com/users"
               headers: #{ authorization: "Bearer token" })

HTTP POST request with JSON body, expecting JSON response.

Encodes the body as JSON, sets Content-Type to application/json, and parses the response as JSON. Returns parsed JSON on any status code, or #f if the request failed entirely.

(http-post/json "https://api.example.com/users"
                #{ name: "Alice" email: "alice@example.com" })
; => #{ id: 123 name: "Alice" }

(http-post/json "https://api.example.com/users"
                #{ name: "Alice" }
                headers: #{ authorization: "Bearer token" })

Read HTTP response headers only, without consuming the body.

Returns a list: (status-code headers leftover-string) where leftover-string is any body data already read past the header boundary. Returns #f on failure.

Parse headers from data at the given header-end position. Returns (status-code headers leftover-string).

Stream response body from connection to an output port.

Writes leftover bytes (from header read) first, then reads remaining data as bytevectors and writes them to the port.

Download a URL to a file, streaming data directly to disk.

Unlike http-get which loads the entire response into memory, http-download streams the response body to a file, making it suitable for large downloads.

The on-progress callback receives (bytes-received total-bytes) where total-bytes may be #f if the server didn't send Content-Length.

Returns a dict with download info on success, or #f on failure.

(http-download "https://example.com/large-file.bin"
               "/tmp/file.bin")
; => #{ status: 200 size: 12345 path: "/tmp/file.bin" }

(http-download "https://example.com/file.bin"
               "/tmp/file.bin"
               on-progress: (lambda (received total)
                              (display (str received "/" total "\r"))))

Fetch url (a method symbol, optional request headers dict and string or bytevector body) and return the response as raw bytes:

#{ status: <integer> headers: <ordered alist of (lowercased-name . value)> body: <bytevector> }

or #f if the upstream could not be reached / the response was unparseable. Distinct from http-request in three ways a byte-exact relay needs:

• the body is a BYTEVECTOR, never decoded to a string;
• headers is an ORDERED alist that preserves order AND duplicates (e.g. multiple Set-Cookie), which a dict would silently collapse;
• REDIRECTS ARE NOT FOLLOWED — a 3xx is returned untouched (status + Location intact) so the caller decides whether to chase it. A reverse proxy must relay redirects, not follow them; a redirect-following wrapper can layer on top.

timeout (seconds, or #f -> 30) bounds the TLS connect and is the IDLE read deadline (it resets whenever bytes arrive, so a large but steadily-flowing body never times out). Sends Connection: close, so a body with no Content-Length is read to EOF.

(let ((r (http-fetch-bytes 'GET "https://example.com/app.wasm")))
  (and r (bytevector-length (dict-ref r body: #f))))

Stream an HTTP response without buffering its body.

on-head is called once with #{ status: headers: }. on-chunk is then called with each available BYTEVECTOR of decoded response-body data. HTTP/1.1 chunk framing is removed incrementally; arbitrary transport boundaries are preserved safely. Returns the head dict on a complete response, or #f on connect, framing, callback, or idle-timeout failure. As with http-fetch-bytes, timeout is an idle deadline.

Build an API URL from a base URL and path segments.

Common pattern across API clients: concatenate a base URL with slash-separated path parts.

(build-api-url "https://api.example.com" "v1" "users" "123")
; => "https://api.example.com/v1/users/123"

Create a response checker function for an API client.

Takes a name for error messages and an optional list of (status-code . message) pairs for specific error handling. Returns a function that checks an HTTP response and either returns parsed JSON on success or raises an error.

The optional parse-error keyword accepts a function (lambda (status body) ...) for custom error body parsing (e.g., JSON:API error extraction). When provided, it is called instead of the default handler for status >= 400 that don't match a specific handler entry.

(define check-response
  (make-response-checker
    name: "YouTube API"
    handlers: (list
      (cons 401 "Access token may be expired.")
      (cons 403 "Possible quota exceeded."))))

(check-response (http-get url headers: auth))

Create authenticated JSON API method wrappers.

Takes a function that returns auth headers and a response checker, and returns a dict with get, post, put, patch, and delete functions that handle JSON encoding/decoding and authentication.

(define api (make-json-api
              auth-headers: (lambda () #{ authorization: "Bearer tok" })
              check-response: my-checker))

((dict-ref api get:) "https://api.example.com/users")
((dict-ref api post:) "https://api.example.com/users" #{ name: "Alice" })

(No description)

(No description)

(No description)

(No description)

(No description)

(No description)

(No description)

(No description)

(No description)

(No description)

(No description)