UI Slowness

Summary
On December 12, 2025, some customers experienced significant slowness and errors in the user interface, including frequent timeout and error messages. The issue began in the evening and was primarily caused by an unusually high number of policies created through an external integration, which led to increased system load and memory usage. The situation was stabilized by increasing system resources and scaling down the problematic integration. Service performance returned to normal after these actions.

Key Timeline (IDT)
December 12, 2025, 21:13 IDT: Initial reports of slowness and errors in the user interface.
December 12, 2025, 21:24 IDT: System warnings about memory issues and timeouts observed.
December 12, 2025, 21:32 IDT: Identified a specific tenant with a much higher number of policies than usual.
December 12, 2025, 21:39 IDT: System resources increased to stabilize performance.
December 12, 2025, 21:49 IDT: The external integration responsible for the excessive policy creation was scaled down.
December 12, 2025, 21:53 IDT: System load and errors decreased, and service performance improved.

Root Cause
The incident was triggered by an external integration that created an excessive number of policies, far above the typical amount. This led to increased memory usage and processing delays, which became visible to customers as slowness and errors in the user interface.

Actions Taken

1. Increased system resources to handle the higher load.
2. Scaled down the external integration responsible for the excessive policy creation.
3. Monitored system performance to ensure stability.

Action Items

1. Improve monitoring to detect abnormal increases in policy counts.
2. Add safeguards to prevent integrations from creating excessive policies.
3. Review and enhance resource management for similar scenarios in the future.

This incident has been resolved and all systems are stable. We will share a detailed RCA in the coming days.

We identified the root cause of the violations/policies pages slowness. A recently updated connector began creating new dynamic policies, which caused increased load on the database and impacted the pages. As a mitigation, we scaled down the connector and are already seeing the related load and traffic spike decreasing. We will continue to monitor the system closely and will publish an RCA soon to share learnings and next steps. We apologize for the inconvenience.

We observed high load the “detection service” which resulted in slowness on the Violations and policies page. As an immediate mitigation, we increased resources for the affected services. We are continuing to investigate the underlying cause and will share further updates, including an RCA.

Summary
We are currently experiencing slowness in the Web UI, primarily affecting the Violations page. Page loads and interactions may take longer than expected.

Customer Impact
Customers may notice slow loading times or delayed responses when accessing the Violations page.

Other areas of the platform may be unaffected or experience minimal impact.

Investigation Status
Our team is actively investigating the root cause of the UI slowness and monitoring system performance. We will provide updates as we learn more.

Next Update
Further updates will be shared as the investigation progresses.