Delayed SAST PR

Summary
Between January 8 and January 9, 2026, customers in the EU region experienced delays in SAST (Static Application Security Testing) pull request scans and pre-receive hook scans.
These delays resulted in slower feedback for developers and, in some cases, led customers to temporarily disable automated scans.
The issue was caused by a significant increase in container scanning activity, which introduced processing slowdowns in the shared scanning workflow.
On January 8, the underlying infrastructure was strengthened to align with the US environment; however, this change alone was insufficient to handle the increased load.
On January 9, the issue was fully resolved by increasing the number of processing replicas, alongside improved monitoring and scaling adjustments. Service levels have since returned to normal, and ongoing monitoring confirms continued stability.

Key Timeline (IDT)
January 8, 2026, 10:00 IDT: Initial reports of slow SAST scan processing in the EU region
January 8, 2026, 11:00–14:00 IDT: Increased delays observed in pull request and pre-receive hook scans
January 8, 2026: Infrastructure resources strengthened to align with the US environment; improvement observed but insufficient under continued load
January 9, 2026, 10:00–13:00 IDT: Recurrence of scan delays; some customers temporarily disabled automated scans
January 9, 2026, 14:30 IDT: Additional processing replicas added; monitoring and scaling enhanced
January 9, 2026, 15:30 IDT: Scan processing times returned to normal
January 10, 2026, 11:00 IDT: Continued monitoring confirmed system stability

Root Cause
A sudden and significant increase in container scanning activity, primarily driven by new customer integrations, exceeded existing processing capacity.
Although infrastructure resources were increased on January 8, the approach focused on strengthening individual nodes and did not sufficiently address the concurrency requirements.
The shared processing path across multiple scan types caused the elevated load to impact SAST pull request and pre-receive hook scans.

Actions Taken

1. Strengthened processing infrastructure on January 8 to align with the US environment
2. Increased the number of processing replicas on January 9 to better handle concurrent scan volumes
3. Improved monitoring and alerting to better detect scan latency issues
4. Adjusted scaling strategies to respond more effectively to traffic spikes
5. Continued close monitoring to ensure service stability

Action Items

1. Improve separation between scan processing paths to prevent one scan type from impacting others
2. Enhance monitoring and alerting for early detection of scan delays
3. Review and optimize scaling strategies, with clearer guidelines on when to scale vertically versus horizontally

Delays in SAST pull request and pre-receive hook scans in the EU region due to insufficient processing capacity under increased scan load.