History
The incident has been resolved. We will provide the RCA shortly.
The incident has been resolved. We will provide the RCA shortly.
Summary
Between May 15 and May 28, 2026, a processing delay was identified affecting dependency update workflows for several large accounts. During this period, customers may have experienced significant lag in the time required to process and reflect dependency updates within the system. The issue was caused by an inefficiency in how the database retrieved specific project records, leading to temporary performance degradation. The issue has been fully resolved, and processing speeds have returned to expected levels.
Key Timeline (IDT)
May 17, 2026, 09:00 IDT: Initial reports of increased processing times for dependency updates were observed.
May 24, 2026, 14:30 IDT: System resources were increased to provide temporary relief while the investigation continued.
May 28, 2026, 11:15 IDT: Technical analysis identified a specific database query that was not utilizing the most efficient lookup path.
May 30, 2026, 16:45 IDT: A corrective update to the database indexing structure was deployed.
May 31, 2026, 10:00 IDT: Monitoring confirmed that processing latency returned to sub-millisecond levels and the backlog was cleared.
Root Cause
The root cause was a performance regression in the update workflow’s data retrieval process. The database’s automated query planner began selecting a less efficient path to locate project references. Because the system was searching through a very large volume of records, this inefficient path resulted in the system scanning thousands of unnecessary rows for each update. This led to high disk activity and significant delays, particularly for accounts with a high density of shared dependencies across many projects.
Actions Taken
Resource Scaling: Temporarily increased database memory and processing power to mitigate the immediate impact on customers.
Database Optimization: Performed a deep analysis of query execution plans to identify the specific bottleneck.
Index Refinement: Replaced an existing database index with an optimized version that ensures the query planner always selects the most direct path for data retrieval.
Validation: Verified the fix across all environments to ensure the new indexing structure supports both data integrity and high-speed lookups.
Action Items
Enhanced Monitoring: Implement more granular alerts for database query performance to detect similar planning inefficiencies before they impact customers.
Optimization Standards: Update internal database design guidelines to prevent overlapping index patterns that can confuse automated query planners.
Performance Testing: Incorporate “cold cache” scenarios into performance testing for large-scale data operations to better simulate real-world disk I/O constraints.
Summary
On May 5, 2026, customers in the EU region experienced issues with Custom Dashboards not loading as expected, and some filtering/grouping actions were slower than usual on the Violations page. The issue was caused by a configuration update that included an invalid setting, which prevented the EU analytics data service from operating normally. The service was restored the same evening, and we confirmed no data was lost.
Key Timeline (IDT)
May 5, 2026, 16:51 IDT – A configuration update was deployed to the EU production environment.
May 5, 2026, 19:06 IDT – Monitoring detected availability issues and alerts were triggered.
May 5, 2026, 19:23 IDT – Investigation began.
May 5, 2026, 19:36 IDT – The invalid configuration was identified and confirmed as the cause.
May 5, 2026, 19:40 IDT – A controlled recovery approach was planned to ensure data integrity throughout remediation.
May 5, 2026, 19:50 IDT – Additional safeguards and validation steps were performed prior to remediation, including recovery validation in a non-production environment.
May 5, 2026, 21:45 IDT – A safe recovery procedure was confirmed.
May 5, 2026, 22:05 IDT – Production service was recovered and data accessibility was confirmed.
Root Cause
A configuration update introduced an invalid resource value. While the change initially appeared to apply successfully, it caused repeated failures in the automated update process responsible for keeping the service healthy. After continuing to fail, the process removed the running service components associated with the EU analytics service, leading to service unavailability for customers until restoration was completed.
Actions Taken
Investigated alerts and isolated the problematic configuration change.
Performed additional safeguards and validation steps prior to remediation.
Validated recovery steps in a non-production environment before applying them in production.
Restored service by safely reattaching existing storage and returning processing to normal operation.
Confirmed service availability and verified that data remained intact.
Action Items
Add stronger pre-deployment validation to catch malformed configuration values before rollout.
Improve monitoring and alerting to detect repeated update failures and destructive actions earlier.
Implement safeguards to prevent automated update workflows from removing stateful resources for this analytics service during invalid update scenarios.
Improve operational logging and visibility to speed up diagnosis during similar events.
Expand and regularly validate backup/restore coverage for the affected analytics stateful components.
A fix supporting both the old and new formats was deployed at 13:00 IST.
The incident has been resolved.
Summary
Between April 10 and April 14, 2026, some customers saw SAST scans and findings generated for branches that were not included in their configured branch-scanning settings. This resulted in unexpected findings, including duplicates and false positives, appearing in the product. The issue was caused by a recent update to the branch-scanning control logic that did not correctly apply customers’ branch configuration during that period. The scanning behavior was corrected on April 14, 2026, and the incident was fully resolved on April 27, 2026 after the unintended findings were removed.
Key Timeline (IDT)
April 10, 2026, 09:33 IDT — A production update introduced incorrect handling of branch-scanning configuration for some SAST events.
April 14, 2026, 13:48 IDT — A subsequent update restored expected enforcement of configured branch-scanning rules.
April 21, 2026, 19:00 IDT — Customer reports alerted us to unexpected SAST findings on non-configured branches.
April 22, 2026, 13:52 IDT — Root cause identified; scope assessment initiated.
April 23, 2026, 18:13 IDT — Impact analysis tooling executed to identify affected data and tenants.
April 27, 2026, 14:30 IDT — Cleanup completed; unintended findings removed and incident fully remediated.
Root Cause
A product update introduced an issue in the branch-scanning configuration check, causing the system to create SAST scans for branches outside of customer-defined scanning rules for a limited period (April 10–14). These extra scans produced unintended findings, which became visible to customers as additional or duplicate items.
Actions Taken
Restored correct enforcement of customers’ configured branch-scanning rules.
Assessed the scope of the impact across affected environments.
Removed unintended SAST findings created for non-configured branches.
Confirmed expected behavior after cleanup.
Action Items
Improve validation and safeguards around branch-scanning configuration checks during updates.
Enhance monitoring to detect unexpected increases in scan volume or findings.
Strengthen release checks for feature-gated behavior to prevent similar regressions.