Summary
During the incident, customers experienced significant delays and temporary disruptions across SAST, SCA, CCA, and Secret repository scans and push events. The issue was caused by a surge in reachability scanner jobs that overwhelmed the processing queue, compounded by scanner pods requesting excessive CPU and memory, infrastructure resource limits being reached, and inefficiencies in job prioritization and retry logic. As a result, processing capacity was improperly consumed and a large job backlog accumulated. A series of corrective updates were deployed to stabilize the environment, and the processing environment has since returned to expected performance levels.
Impact
Customers experienced delays for SAST, SCA, CCA, and Secret repository scans and push events, with some requests delayed by several hours and a peak queue size of over 64,000 jobs. Lower priority scans such as Trivy, Syft, and CCA were most affected, though high-priority jobs were eventually processed without further delay.
Key Timeline (IDT)
21.06.2026, 17:16 IDT: A surge in reachability scanner jobs caused the CycodeX queue to grow rapidly.
22.06.2026, 10:07 IDT: The issue was identified by an on-call engineer.
22.06.2026, 12:55 IDT: We increased the scanning platform resources to process more jobs.
22.06.2026, 14:10 IDT: A fix that lowered new reachability scanners was deployed to production.
22.06.2026, 18:43 IDT: Existing reachability scanners' priority was lowered.
23.06.2026, 09:12 IDT: Scans with higher priority were processed. Only lower priority scans remained, including CCA.
23.06.2026, 13:51 IDT: A fix that reduced communication overload to Kubernetes was deployed. The scanning platform started processing scan jobs much faster.
23.06.2026, 17:34 IDT: The queue was fully processed.
Root Cause
The issue was triggered by a combination of factors:
Reachability scanner job surge -- A surge in reachability scanner jobs caused the CycodeX queue to grow rapidly, which led to resource bottlenecks in the cluster and a peak queue size of over 64,000 jobs.
Excessive pod resource requests -- Due to configuration bugs, scanner pods requested excessive CPU and memory, which prevented efficient scheduling and amplified the resource bottlenecks in the cluster.
Infrastructure resource limits -- AWS VPC subnet IP and EKS API limits were reached, restricting the cluster's ability to scale and schedule new work.
Job prioritization and retry inefficiencies -- Inefficiencies in job prioritization and retry logic meant lower priority scans (Trivy, Syft, CCA) competed for capacity and were most affected, while the backlog continued to grow.
Actions Taken
Increased cluster and node pool capacity.
Fixed job prioritization to deprioritize reachability scans.
Capped resource requests for scanner pods to enable efficient scheduling.
Deployed additional fixes to the scanning platform.
Opened AWS support tickets to address resource limits.
Restored monitoring and logging.
Cleared the job backlog; the queue now processes new jobs as they arrive.
Action Items
Improve monitoring to better understand the behavior of the processing environment.
Improve scanning optimization and prioritization for all scan types.
Resolved
Summary
During the incident, customers experienced significant delays and temporary disruptions across SAST, SCA, CCA, and Secret repository scans and push events. The issue was caused by a surge in reachability scanner jobs that overwhelmed the processing queue, compounded by scanner pods requesting excessive CPU and memory, infrastructure resource limits being reached, and inefficiencies in job prioritization and retry logic. As a result, processing capacity was improperly consumed and a large job backlog accumulated. A series of corrective updates were deployed to stabilize the environment, and the processing environment has since returned to expected performance levels.
Impact
Customers experienced delays for SAST, SCA, CCA, and Secret repository scans and push events, with some requests delayed by several hours and a peak queue size of over 64,000 jobs. Lower priority scans such as Trivy, Syft, and CCA were most affected, though high-priority jobs were eventually processed without further delay.
Key Timeline (IDT)
21.06.2026, 17:16 IDT: A surge in reachability scanner jobs caused the CycodeX queue to grow rapidly.
22.06.2026, 10:07 IDT: The issue was identified by an on-call engineer.
22.06.2026, 12:55 IDT: We increased the scanning platform resources to process more jobs.
22.06.2026, 14:10 IDT: A fix that lowered new reachability scanners was deployed to production.
22.06.2026, 18:43 IDT: Existing reachability scanners' priority was lowered.
23.06.2026, 09:12 IDT: Scans with higher priority were processed. Only lower priority scans remained, including CCA.
23.06.2026, 13:51 IDT: A fix that reduced communication overload to Kubernetes was deployed. The scanning platform started processing scan jobs much faster.
23.06.2026, 17:34 IDT: The queue was fully processed.
Root Cause
The issue was triggered by a combination of factors:
Reachability scanner job surge -- A surge in reachability scanner jobs caused the CycodeX queue to grow rapidly, which led to resource bottlenecks in the cluster and a peak queue size of over 64,000 jobs.
Excessive pod resource requests -- Due to configuration bugs, scanner pods requested excessive CPU and memory, which prevented efficient scheduling and amplified the resource bottlenecks in the cluster.
Infrastructure resource limits -- AWS VPC subnet IP and EKS API limits were reached, restricting the cluster's ability to scale and schedule new work.
Job prioritization and retry inefficiencies -- Inefficiencies in job prioritization and retry logic meant lower priority scans (Trivy, Syft, CCA) competed for capacity and were most affected, while the backlog continued to grow.
Actions Taken
Increased cluster and node pool capacity.
Fixed job prioritization to deprioritize reachability scans.
Capped resource requests for scanner pods to enable efficient scheduling.
Deployed additional fixes to the scanning platform.
Opened AWS support tickets to address resource limits.
Restored monitoring and logging.
Cleared the job backlog; the queue now processes new jobs as they arrive.
Action Items
Improve monitoring to better understand the behavior of the processing environment.
Improve scanning optimization and prioritization for all scan types.
Monitoring
The system has processed all jobs with higher priorities. There is a queue of lower priority jobs that should not impact overall Cycode scanning performance
Monitoring
The fix was deployed. The queue is decreasing and we're monitoring it
Identified
We have identified the source of an issue and currently deploying the fix. At the same time we scaled our scanning platform up to accelerate scanning