System Degradation
Resolved
Oct 14 at 05:08pm UTC
We’ve taken steps to address the service degradation and are closely monitoring performance.
Post Mortem
What happened: After upgrading our columnar database, an incompatibility stopped automatic consolidation on a core table.
Impact: Partial data loss and content triggering issues.
Cause: The new version didn’t work correctly with tables that use pre-computed summaries, leading to too many small data fragments.
Fix: Rebuilt the summaries to force consolidation and rolled back to the prior database version. Performance recovered.
Prevention: Added an alert on fragment count, paused similar upgrades, and expanded compatibility/performance testing before future releases.
We’re sorry for the disruption and appreciate your patience.
Affected services
Created
Oct 14 at 03:27pm UTC
We are experiencing service degradation. Our team is actively monitoring and working to restore full performance.
Affected services