One of our billing workers kept dying.

It would sit flat for hours, then climb into its memory limit in about four minutes and get killed. Kubernetes restarted it, and later the same day it happened again. Six times a day, every day.

The obvious first move is to look at what PHP is holding on to, and that's where it got strange. When PHP runs out of memory it tells you: a fatal error, a stack trace, the allocation that tipped it over. We had none of that. memory_get_usage() reported a flat 6 MB the entire time, right up to the moment the process vanished.

Those two facts together are the whole puzzle. Something was eating memory, and it wasn't the PHP heap. It was cURL, two sockets at a time, and the cause turned out to be a missing keyword.

We've chased this same shape in more than one Appwrite service now, and the fix has been the same every time: stop building a new HTTP client for every call, and move the library onto utopia-php/client.

What was actually leaking

The worker idles at ~330 MB against a 512Mi limit, which is the flat stretch below. Once a day the newly-due invoice batch runs, about a thousand invoices with several Stripe calls each, and it goes from comfortable to dead in roughly four minutes.

Luke traced it, and the code turned out to be completely unremarkable. Pay\Adapter::call() built a new Utopia\Fetch\Client for every request. In a one-shot FPM request that's fine; the process exits and the OS reclaims everything. In a long-lived worker it's a slow bleed, and the reason is a reference cycle that has nothing to do with the PHP heap.

curl_setopt stores your write callback on the CurlHandle. A closure declared inside an instance method captures $this. So the handle holds the closure, the closure holds the adapter, and the adapter holds the handle. Nothing can be freed by refcounting, and __destruct, with its curl_close, only runs when PHP's cycle collector fires. By default that means after 10,000 cycle roots have piled up. Until then every request strands an open keep-alive connection.

Who is holding the cURL handle?

function () { … }static function () { … }

requests

0

open file descriptors

9

native memory

0.0 MB

PHP heap: 6.0 MB, flat, in both modes. memory_limit only governs that heap, never the native memory underneath it.

Reduced down, that's this:

The fix is one keyword:

A static closure gets no $this, so the last edge never forms and refcounting frees the handle as soon as the adapter goes out of scope. Neither callback used $this in the first place, which is what makes the change safe and also what makes it so easy to miss.

Two file descriptors and roughly a megabyte of native TLS buffers per request, none of it visible to memory_get_usage(). Descriptors are capped per process too, so whichever ceiling you reach first decides how the thing dies: the kernel OOMKills you, or you start refusing connections with Too many open files. I've written that exact closure before, probably more than once, and it would never have occurred to me that static was load-bearing. Adding it to both callbacks in utopia-php/fetch took 150 sequential requests from 309 fds and 183 MB RSS to 9 fds and 31 MB, flat.

We shipped that and moved on, which was the wrong instinct. It fixed the symptom in one library and left three others still hand-rolling their own transport, each free to reinvent the same bug.

What the library is

utopia-php/client is a PSR-18 HTTP client for PHP 8.5. It's about 2,400 lines including both transports, and it doesn't try to be clever:

Utopia\Client itself does almost nothing at request time. It resolves the URI against a base, fills in default headers, optionally stamps a traceparent, and hands the request to an adapter. Connection lifetime, TLS, timeouts and error classification all live in the adapter. Anything policy-shaped lives in a decorator.

Which specs it holds itself to

Writing your own HTTP transport in five places means reading the specs badly in five places. Here's what the shared one is on the hook for.

PSR-18 decides what counts as an error

The sentence that does the most work:

A Client MUST NOT treat a well-formed HTTP request or HTTP response as an error condition. For example, response status codes in the 400 and 500 range MUST NOT cause an exception and MUST be returned to the Calling Library as normal.

A 429 is not a failure, it's an answer. So sendRequest() returns it, and only genuine "there is no response" conditions throw. PSR-18 splits those into two branches: RequestExceptionInterface for a malformed request or response, NetworkExceptionInterface for a transport that failed. The type answers one question, which is the only one you have at the catch site. Would trying again help?

The library's own hierarchy keeps that property all the way down.

Each adapter maps its native error codes into that tree. The cURL adapter matches on CURLE_* constants, guarded by defined() so a libcurl build without HTTP/3 doesn't fatal at load.

Another line from the spec you can find in the code almost verbatim:

If a Client chooses to decompress the message body then it MUST also remove the Content-Encoding header and adjust the Content-Length header.

Both adapters negotiate compression for you: the request advertises whatever codecs the transport can decode, and the response arrives as plaintext. Which means the Content-Encoding: gzip and Content-Length the server sent are now lies about the body you're holding, so the adapter drops both. Set your own Accept-Encoding and it gets out of the way entirely.

Retrying, and why only some requests get to

Retry is a decorator, and its default Backoff strategy reads almost directly off RFC 9110. Only idempotent methods (§9.2.2) are retried, so a lost response can't turn into a double charge. Only transient outcomes are retried: a NetworkExceptionInterface, or a 429 / 502 / 503 / 504. A numeric Retry-After (§10.2.3) beats the computed delay, because the server knows things the client doesn't.

With no Retry-After, the wait is exponential with full jitter: a value drawn uniformly from [0, ceiling) rather than the ceiling itself.

14 workers take a 503 at the same instant

Replay

Exponential backoff, no jitterbusiest 100ms: 14 retries

Exponential backoff, full jitterbusiest 100ms: 8 retries

Our workers run as a fleet. A fleet that backs off deterministically comes back at a struggling upstream in lockstep and keeps it struggling.

Every one of those decisions lives behind a single method, so a library with different rules writes its own:

utopia-php/storage does exactly that. S3 signals throttling in an XML body as often as in a status code, so S3\RetryStrategy parses the body first and retries SlowDown, ServiceUnavailable, Throttling and RequestThrottled. It also refuses to retry a 503 whose body parses cleanly into some other error code, which is the case a status-code-only rule gets wrong.

withBaseUri() is not string concatenation

It looks like a convenience until you send ../v2/users and find out which one your client implements. This one does dot-segment removal, and only applies the base when the request URI is actually relative. An absolute URI passes through untouched.

Everything else is a header you'd otherwise hand-roll

withBasicAuth() is RFC 7617's base64(user:pass). withBearerAuth() is RFC 6750's Bearer <token>. Part::file() builds an RFC 7578 part with its RFC 2183 Content-Disposition. withMinTlsVersion(Tls::V1_2) is an enum each adapter maps to CURLOPT_SSLVERSION or Swoole's ssl_protocols. withTracePropagation() forwards the active utopia-php/span trace as a traceparent, and refuses to overwrite one that's already on the request.

None of these are hard. They're just wrong in slightly different ways in every library that rolls its own.

The four rules underneath

Reuse over recreate

withConnectionReuse() keeps one connection alive per client and reuses it for every request to the same origin. curl_reset() clears per-request options while preserving the handle's connection cache; the Swoole adapter keeps a kept-alive coroutine client keyed by origin. It's opt-in, on the theory that a client built for one call shouldn't sit on a socket. Any long-lived service wants it on.

Six calls to the same origin

Replay

new Client() per call6 handshakes

DNS

TCP

TLS

#1

DNS

TCP

TLS

#2

DNS

TCP

TLS

#3

DNS

TCP

TLS

#4

DNS

TCP

TLS

#5

DNS

TCP

TLS

#6

->withConnectionReuse()1 handshake

DNS

TCP

TLS

#1

#2

#3

#4

#5

#6

connection setupthe actual request69% less wall clock

For Pay, that one change was the entire fix. Four hundred requests against a local echo server, before and after:

When you need concurrency rather than sequence, Client\Pool borrows a client from a utopia-php/pools pool per request and reclaims it afterwards, so N coroutines share a bounded set of connections instead of opening N of their own.

Every with*() returns a clone

In Swoole that's not a style preference. A shared client you can mutate is a cross-request bleed waiting to happen, and cloning means there's no way to reconfigure someone else's client from inside a request handler.

A default is also only ever a default. withHeaders() fills in a header the request doesn't already carry and nothing more, so a per-request Content-Type beats the client-wide one.

Policy lives in decorators

Retry implements the same Adapter interface it wraps, forwards every configuration helper inward, and overrides only sendRequest() and stream(). So retries, pooling and whatever you add stack in any order, and none of them turn into constructor flags on the transport.

The stream() override is my favorite detail in the library. It counts bytes handed to the sink, and once a single byte has been delivered it stops retrying, because replaying would duplicate data the caller already processed.

Bounded memory by default

stream() hands each chunk to a sink as it arrives, so SSE and LLM token streams cost the same memory as a ping. Uploads go the same way: cURL pulls the body through a read callback, Part::file() reads lazily, and Swoole sends files with zero-copy sendfile(). A seekable body gets rewound before each attempt, which is what makes a streamed upload safe to retry at all.

Where it ended up

The Pay migration was net -55 lines. Adapter::call(), handleError(), and nine METHOD_* constants went with it — seven of the nine had no caller. A PSR-18 client and a PSR-17 factory already mean "build a request, send it"; the indirection was only ever a second vocabulary for HTTP.

Messaging is the one worth reading if you're doing this yourself. It kept every adapter's public API identical while swapping curl_multi for Swoole coroutines over a bounded pool, and added a Closure(): ClientInterface factory so a caller can inject retries, a proxy, or a stub. The one constraint it documents loudly: your factory must produce a client that can negotiate HTTP/2, because APNs rejects HTTP/1.1 outright.

What we grep for now

• Climbing RSS with a flat PHP heap. Count fds, not memory_get_usage(). ls /proc/<pid>/fd | wc -l in a loop finds this in five minutes.
• new Client() or curl_init inside a request path. In a long-lived process a client is a resource with a lifetime, not a local variable.
• Non-static closures handed to curl_setopt. If it doesn't touch $this, make it static. Costs nothing, and it's the whole bug.
• Any utopia library still carrying its own private HTTP stack. That's the rest of the work list.

Repo: utopia-php/client, though development happens in the monorepo. @lukebsilver wrote it and did the Pay migration. I did messaging, which is how I ended up with opinions about APNs.

Reference

The specs

• HTTP semantics in RFC 9110, message framing in RFC 9112, HTTP/2 in RFC 9113, and 429 in RFC 6585
• PHP-FIG: PSR-18 for clients, PSR-7 for messages, PSR-17 for factories
• URIs in RFC 3986; credentials in RFC 6750 and RFC 7617
• Multipart in RFC 7578, building on RFC 2046 and RFC 2183
• Content codings: deflate, gzip, br, zstd
• TLS 1.3 and W3C Trace Context

The code

• utopia-php/client, developed in the monorepo package, on top of utopia-php/psr7 and utopia-php/pools
• fetch#22 — the reference cycle, with the fd and RSS measurements
• pay#32 — the migration that fixed the OOMKills, and the before/after numbers
• messaging#137 — curl_multi swapped for Swoole coroutines over a pool

Background

• AWS, Exponential Backoff and Jitter, which is where the full-jitter formula comes from
• PHP manual, Collecting Cycles, on when the collector actually runs