Sistem durumu

08:52 UTC

Webhook deliveries to a subset of endpoints are failing with a TLS verification error. Events are queued and will be retried, so nothing is being dropped. Investigating.

09:14 UTC

A base image update on the delivery workers shipped a trust store that no longer carries a cross-signed intermediate that several customer endpoints still serve. Around 4 % of registered endpoints are affected.

09:48 UTC

We have pinned the previous trust store on the delivery workers and the queued deliveries are going out. The oldest queued event is 56 minutes old.

11:06 UTC

Top-up credits are arriving late. Payments are being received and recorded, and no funds are at risk, but balances are updating with a delay. Investigating.

11:21 UTC

OxaPay moved their callback traffic to a new source range this morning. Our allowlist rejected it, so callbacks were retried instead of accepted. About 340 callbacks are pending.

11:44 UTC

The new range is allowlisted and the retries are being accepted. The backlog is draining oldest first at roughly 90 callbacks a minute.

18:00 UTC

Maintenance window open, as announced on 2026-06-05. The Singapore entry point is draining and requests are being served from Frankfurt for the duration.

18:12 UTC

The kernel upgrade is finished but one node is not rejoining the load balancer. Singapore stays drained while we look at it. Frankfurt is serving all traffic and latency from Asia is elevated to roughly 300 ms.

18:31 UTC

The node was presenting a certificate issued before the upgrade. It has been reissued and the node is back in rotation.

05:44 UTC

Activations on Indonesian numbers have been expiring without a code at a much higher rate than normal since about 05:10. Purchases and the rest of the catalogue are unaffected.

06:30 UTC

The operator changed the route for inbound international SMS overnight without notice. Messages reach their network and are not forwarded to us. The affected ranges are out of routing as of 06:24.

07:05 UTC

Indonesia is now served entirely by the two remaining operators. Success rate is 88 % against 93 % normally, and stock is about a third lower. Webhook volume for Indonesia is down accordingly.

09:40 UTC

The operator has acknowledged the route change and is reverting it. We will not return the ranges to routing until our own probes have delivered cleanly for two hours.

07:29 UTC

A number of API keys are receiving 429 responses while comfortably inside their per-minute budget. Investigating.

07:48 UTC

A counter shard lost its expiry during a cache node replacement overnight, so counts from the previous hour were never cleared for keys hashed onto that shard. Roughly 6 % of keys are affected.

08:15 UTC

The affected shard has been flushed and the keys on it are being served normally. We are checking the remaining shards for the same condition.

10:22 UTC

Creating a top-up invoice is failing with an upstream error for most currencies. Invoices already created are unaffected and payments already sent are being credited normally.

10:40 UTC

OxaPay has confirmed an incident on their invoice API. Purchases, activations, rentals and the rest of the API are not affected; only creating a new top-up is.

11:35 UTC

Partial recovery upstream. USDT-TRC20 and TON invoices are being created again. BTC and ETH still fail intermittently.

12:41 UTC

All currencies are creating invoices again. We are keeping this open while we watch the error rate.

15:38 UTC

The dashboard is failing to load for some visitors with a chunk loading error. The API is unaffected, so automations are still running normally.

15:47 UTC

A deploy at 15:31 invalidated the asset manifest while open sessions were still holding the previous one. Anyone who had the dashboard open before the deploy is affected; a hard reload works around it.

16:02 UTC

The previous asset bundle has been re-published alongside the new one so both manifests resolve. Error reports have stopped.

09:15 UTC

Success rates on Brazilian numbers have fallen from 91 % to about 44 % since 08:30. Purchases still succeed, but a large share of activations are expiring without a code.

09:52 UTC

One operator range is being rejected by several services at once, which normally means the range has been flagged rather than that delivery is broken. Its pool score is set to zero so routing avoids it.

11:10 UTC

Brazilian success rate is back to 86 % on the remaining operators. Stock for Brazil is roughly 40 % below normal while the flagged range is excluded.

14:30 UTC

The operator confirms the range was flagged by a downstream aggregator and is migrating the block. We are keeping it out of routing until our own probes show it recovering.

04:07 UTC

The API is returning 500 for most requests and the dashboard will not load. The primary database is not responding. Investigating at the highest priority.

04:14 UTC

The primary lost its storage volume. Automatic failover did not trigger because the primary was still answering health checks on its keepalive port. We are promoting the standby manually.

04:26 UTC

The standby is promoted and the API is answering again. Error rate is back to baseline. We are now verifying whether any committed purchase was lost in the promotion.

02:00 UTC

Maintenance window open, as announced on 2025-06-10. Writes return 503 with a Retry-After header; reads and long-polls continue to be served from the replica.

02:18 UTC

The upgrade is complete and the primary is accepting writes. We are comparing query plans against the pre-upgrade baseline before calling this done.

18:41 UTC

Top-ups paid in the last half hour are not appearing on balances. Payments are being received and recorded; it is the credit step that is backed up. No funds are at risk.

19:02 UTC

The consumer that applies OxaPay callbacks to balances stopped acknowledging messages after a deploy at 18:10, and the queue has grown to about 400 callbacks. We are rolling that deploy back.

19:35 UTC

The rollback is live and the backlog is draining at roughly 60 callbacks a minute. The oldest pending credit is 51 minutes old.

20:20 UTC

Backlog cleared. Every payment received during the incident has been credited, including four invoices that expired while their callback was waiting.

06:52 UTC

Delivery times for Indian numbers are running above 60 seconds against a normal median of 9. Numbers are still being issued and the refund window is unchanged. Investigating.

07:20 UTC

One of our two Indian upstreams is queueing inbound SMS at its own gateway. India is being routed to the second provider while we wait on their side.

08:05 UTC

The shift is complete and new Indian activations are delivering at a median of 14 seconds. Activations bought before 07:20 that are still open will either deliver or auto-refund at the end of their window.

09:40 UTC

The upstream has drained its queue and reports a full disk on a gateway node as the cause. We are keeping traffic on the second provider until their median has been stable for an hour.