SAST Scans are Delayed

Summary
On July 8, 2025, we experienced delays in processing SAST scan reports, particularly for CLI scans that require a fast response. This issue stemmed from an unusually large volume of findings in a batch of released scan reports, which led to significant congestion in our processing queues. As a result, many scan reports became stuck, causing delays in findings appearing on the platform for full repository and push-event scans as well. The incident lasted several hours until the queues were fully processed, and service was stabilized.

Timeline
08.07.25 02:06 PM (GMT+3) – A large batch of initial SAST scans was released.
08.07.25 02:30 PM (GMT+3) – The SAST report processing queue began to grow significantly.
08.07.25 03:37 PM (GMT+3) – The issue was noticed, and an investigation began.
08.07.25 05:23 PM (GMT+3) – Impact on CLI scans was observed.
08.07.25 06:13 PM (GMT+3) – An incident alert was published on the platform.
08.07.25 07:34 PM (GMT+3) – The processing queue started to decrease.
08.07.25 08:00 PM (GMT+3) – The queue was fully processed, and service stabilized.

Root Cause
The primary cause of this incident was an exceptionally large volume of SAST findings within specific scan reports. This volume extended the processing time of these reports, leading to a significant message queue buildup in our Kafka topics. This buildup directly impacted CLI scans by blocking the queues and causing extended scan times.

Actions Taken

1. An alert was communicated to support teams and users regarding the ongoing incident.
2. Continuous monitoring of the relevant processing queues was performed.
3. An in-depth investigation was initiated to identify the underlying causes of the issue.

Action Items

1. Separate Report Queues: We will separate CLI scan reports from the general reporting topic to prevent future blockages and ensure faster responses.
2. Establish Lag Alerts: We will implement targeted alerts for Kafka lags in SAST to identify and address similar issues.

The issue is now resolved.

We’ve seen a larger than usual number of big scans started. It caused a queue in scan, which we are seeing slowly dropping. We are monitoring the situation.

You might experience delays with CLI scans, and detections will arrive on the platform with a delay. We’re working on a fix.