Summary
On May 5, 2026, customers in the EU region experienced issues with Custom Dashboards not loading as expected, and some filtering/grouping actions were slower than usual on the Violations page. The issue was caused by a configuration update that included an invalid setting, which prevented the EU analytics data service from operating normally. The service was restored the same evening, and we confirmed no data was lost.
Key Timeline (IDT)
May 5, 2026, 16:51 IDT – A configuration update was deployed to the EU production environment.
May 5, 2026, 19:06 IDT – Monitoring detected availability issues and alerts were triggered.
May 5, 2026, 19:23 IDT – Investigation began.
May 5, 2026, 19:36 IDT – The invalid configuration was identified and confirmed as the cause.
May 5, 2026, 19:40 IDT – A controlled recovery approach was planned to ensure data integrity throughout remediation.
May 5, 2026, 19:50 IDT – Additional safeguards and validation steps were performed prior to remediation, including recovery validation in a non-production environment.
May 5, 2026, 21:45 IDT – A safe recovery procedure was confirmed.
May 5, 2026, 22:05 IDT – Production service was recovered and data accessibility was confirmed.
Root Cause
A configuration update introduced an invalid resource value. While the change initially appeared to apply successfully, it caused repeated failures in the automated update process responsible for keeping the service healthy. After continuing to fail, the process removed the running service components associated with the EU analytics service, leading to service unavailability for customers until restoration was completed.
Actions Taken
Investigated alerts and isolated the problematic configuration change.
Performed additional safeguards and validation steps prior to remediation.
Validated recovery steps in a non-production environment before applying them in production.
Restored service by safely reattaching existing storage and returning processing to normal operation.
Confirmed service availability and verified that data remained intact.
Action Items
Add stronger pre-deployment validation to catch malformed configuration values before rollout.
Improve monitoring and alerting to detect repeated update failures and destructive actions earlier.
Implement safeguards to prevent automated update workflows from removing stateful resources for this analytics service during invalid update scenarios.
Improve operational logging and visibility to speed up diagnosis during similar events.
Expand and regularly validate backup/restore coverage for the affected analytics stateful components.
Resolved
Summary
On May 5, 2026, customers in the EU region experienced issues with Custom Dashboards not loading as expected, and some filtering/grouping actions were slower than usual on the Violations page. The issue was caused by a configuration update that included an invalid setting, which prevented the EU analytics data service from operating normally. The service was restored the same evening, and we confirmed no data was lost.
Key Timeline (IDT)
May 5, 2026, 16:51 IDT – A configuration update was deployed to the EU production environment.
May 5, 2026, 19:06 IDT – Monitoring detected availability issues and alerts were triggered.
May 5, 2026, 19:23 IDT – Investigation began.
May 5, 2026, 19:36 IDT – The invalid configuration was identified and confirmed as the cause.
May 5, 2026, 19:40 IDT – A controlled recovery approach was planned to ensure data integrity throughout remediation.
May 5, 2026, 19:50 IDT – Additional safeguards and validation steps were performed prior to remediation, including recovery validation in a non-production environment.
May 5, 2026, 21:45 IDT – A safe recovery procedure was confirmed.
May 5, 2026, 22:05 IDT – Production service was recovered and data accessibility was confirmed.
Root Cause
A configuration update introduced an invalid resource value. While the change initially appeared to apply successfully, it caused repeated failures in the automated update process responsible for keeping the service healthy. After continuing to fail, the process removed the running service components associated with the EU analytics service, leading to service unavailability for customers until restoration was completed.
Actions Taken
Investigated alerts and isolated the problematic configuration change.
Performed additional safeguards and validation steps prior to remediation.
Validated recovery steps in a non-production environment before applying them in production.
Restored service by safely reattaching existing storage and returning processing to normal operation.
Confirmed service availability and verified that data remained intact.
Action Items
Add stronger pre-deployment validation to catch malformed configuration values before rollout.
Improve monitoring and alerting to detect repeated update failures and destructive actions earlier.
Implement safeguards to prevent automated update workflows from removing stateful resources for this analytics service during invalid update scenarios.
Improve operational logging and visibility to speed up diagnosis during similar events.
Expand and regularly validate backup/restore coverage for the affected analytics stateful components.
Investigating
We are currently experiencing a service disruption affecting our EU environment.
What’s impacted:
Custom Dashboards are temporarily unavailable
Some filtering capabilities on the Violations page may be slower than usual
Current status:
Our team has identified the issue and is actively working to restore full functionality. All data remains safe.
Next update:
We will continue to share updates as we make progress.