A 403 from an API is not necessarily from the API

Alert emails stopped arriving. No exception, no failed job, no log line worth reading. They were being sent and not showing up.

The API was returning 403. So the first hypotheses were the obvious ones: an expired key, an unverified sending domain, a rate limit, a suspended account.

All wrong, and all of them cost time.

Read the body, not the status

The thing that broke it open was capturing the actual response body from inside the running container rather than the status code from a log.

The body carried error code 1010. That is not from the email provider. It is a bot-protection rejection from the CDN sitting in front of it, and 1010 specifically means the request’s browser signature was refused.

The API never saw the request. Nothing about my account, my key or my domain was wrong. Something in front of the service decided the client was not allowed to speak to it.

The trigger was a default I never chose

The sending code used the language’s standard library HTTP client. That client sends a default user agent identifying itself as the standard library, with a version number.

That signature is on a blocklist. Not because anyone objected to my traffic, but because it is one of the most common signatures used by scrapers, so it gets banned broadly.

Confirmed it in two requests. Identical POST, identical payload, identical credentials. Default user agent: 403. Custom user agent naming my application: 200, with an accepted message ID.

One header. The failure had nothing to do with my code being wrong and everything to do with my code being anonymous.

The blast radius was wider than the report

The reported symptom was alerts. Alerts are low stakes; you notice eventually.

All three senders shared one HTTP helper. So the same ban was silently killing device verification codes and setup notifications. Both of those are things a real person waits for and does not receive, and neither had raised a single complaint yet because the failure was invisible on both ends.

That is the reason to trace a shared helper the moment you find a bug in one of its callers. The reported case is rarely the worst one — it is just the one somebody happened to be watching.

The reason it lasted so long

There were tests. The tests passed.

They injected a fake transport. Every test asserted that given a fake HTTP layer returning success, the calling code did the right thing. All true, and all irrelevant to a failure that lived in the transport.

A test double at a boundary means that boundary is the one thing you never test. That is not an argument against doubles — mocking the network is correct for most tests. It is an argument for having at least one test that does not, positioned exactly where the doubles stop.

The fix shipped with a regression test that drives the real transport and asserts the outbound request carries a custom user agent. It is slower. It is the only test in that file that would have caught this.

What I took from it

Capture the body. A status code tells you a request failed. The body usually tells you who failed it. If your logging keeps the code and discards the body, you have kept the least informative half.

Ask who answered. Between your process and a vendor’s application sit a CDN, a WAF, a load balancer and a proxy, any of which can answer on the vendor’s behalf with the vendor’s status codes and none of their reasoning.

Identify yourself. Set a real user agent on every outbound client, naming the application and a contact. It costs one line and removes a whole class of being mistaken for something else.

Find the shared helper. One reported failure through a shared client means every caller is affected. Check them before closing the ticket.

Related

• When the agent gets it wrong - the same method applied to agent output.
• Designing an MCP server - why error bodies have to carry enough to recover from.