Incident Summary
On Friday, March 27, 2026, Capsule experienced several periods of slow performance and some temporary outages.
These were caused by two separate and unrelated issues:
- A recent software update that unintentionally caused parts of the system to become stuck.
- An uncommon data pattern which led to degraded performance under specific conditions.
Both issues have now been resolved. The first was fixed by rolling back the recent update, and the second by mitigating the data pattern that caused the issue.
At no point was any customer data impacted.
Timeline of Events
(All times are UTC)
- 00:01 – A small number of users began experiencing errors when using the web app and API.
- 01:00 – Errors increased significantly, triggering an alert to our on-call engineers. Investigation began.
- 01:35 – The team observed locking behavior and replacement servers deployed to mitigate.
- 02:09 – Service appeared stable, with further investigation ongoing.
- 08:37 – Errors reappeared for a small number of users.
- 08:46 – Errors increased again, and engineers were alerted. Initially thought to be a repeat of the earlier issue.
- 08:54 – Initial fixes were applied, partially restoring service.
- 09:12 – The earlier issue was fully identified and fixed by reverting the problematic update.
- 09:34 – While monitoring the fix, a second issue was observed.
- 09:54 – The cause of the second issue was identified, linked to an uncommon data pattern. Fixes were applied and service was restored.
- 12:30 – Reprocessing this data caused another spike in errors.
- 13:00 – Mitigation put in place for the data pattern, and full service was restored.
Next Steps
We are carrying out a full internal review of both issues. Based on this, we will introduce additional safeguards to prevent similar problems from happening again.