Stuck SAST Scans

Summary
On December 3, 2025 (IDT), some pull requests briefly displayed different statuses between the product UI and external CI systems. This behavior resulted from a delay in processing certain status update messages, which caused some updates to be applied out of order. Once the delay cleared, the statuses aligned normally. A small corrective update was deployed to prevent this from repeating.

Key Timeline (IDT)
Dec 3, 7:40 PM - Initial customer report; issue temporarily resolved after a rescan.
Dec 3, 8:30 PM - Additional report received, suggesting the behavior may occur in more cases.
Dec 3, 8:32 PM - Investigation begins.
Dec 3, 8:40 PM - A delayed “Running” status is observed arriving after a “Success” update.
Dec 3, 9:09 PM - Source of the delay identified as increased message load in one update channel.
Dec 3, 9:14 PM - Incident formally logged to coordinate the investigation.
Dec 3, 9:15 PM - Confirmation that out-of-order updates are caused by the temporary message delay.
Dec 3, 10:06 PM - A small corrective update is deployed to restore expected behavior.
Dec 3, 10:10 PM - Message delay clears; statuses align as expected and no further impact is observed.

Root Cause
A recent change temporarily increased the number of messages sent through one of the update channels. This led to short-term processing delays, which caused some updates to arrive out of order. Although the system handled the newer updates correctly, the delayed updates were still forwarded to external systems, resulting in a brief mismatch in displayed statuses.

Actions Taken

• Rolled back the recent change to reduce message volume and restore expected behavior.
• Verified that message flow returned to normal once the temporary delay cleared.
• Confirmed consistency across all status update paths.

Action Items

• Improve consistency in event metadata to support more predictable ordering.
• Add guardrails to avoid applying delayed updates to external systems.
• Review distribution of message load to reduce the likelihood of future delays.

The issue is now resolved. We will provide the RCA document in a few days.

We’ve identified the root cause of the incident as a bad code change combined with lag in the offline topic of the scan service. We are currently reverting the code change.

The lag has cleared, so the impact is now limited. Customers can trigger a new scan either through the UI (Rescan) or by commenting #cycode_sast_rescan on the PR.

We will continue to monitor the system as it stabilizes.

Description:
We’re currently experiencing an issue with SAST scanning on pull requests.

Current Status:
Our team is investigating the root cause and working to restore full scanning functionality.

Impact:
Pull requests SAST Scans may not display accurate SAST results until the issue is resolved.

Next Update:
We will provide an update as soon as we have more information.