History
GitLab has resolved the incident affecting PR scanning and integration connectivity
Resolved: Brief slowdown of the scans API (US region)
On October 13, 2026, between 07:58 and 08:28 UTC, some requests
to the scans API in the US region failed or were slow. Scan pages in the Cycode UI and the Cycode CLI may have shown temporary errors, and some pull request scans arrived a few minutes later than usual.
We increased the memory available to the service, which fully resolved the issue at 08:34 UTC. All queued scans completed shortly after, and no data was lost. We have identified the cause and a permanent improvement is in review.
(1/2)
1. Summary
On October 5, 2026, starting at 15:55 IDT, customers on our US platform saw slow responses, intermittent errors, and delays in pull request checks and CLI scans. Some customers whose repositories require Cycode pull request checks couldn't merge until scanning resumed.
The issue started when a background maintenance process in our primary database fell behind and stopped completing, and the database stopped saving new data. Restoring service took several hours because the standby database first had to process a large backlog of changes.
The database was back and the platform was available again by 19:43 IDT. Delayed processing then caught up, and the incident was resolved at 23:24 IDT. Pull requests that were missed during the incident were scanned.
2. Key Timeline (IDT)
• October 5, 2026, 12:55 IDT: A routine database maintenance process starts running much slower than normal. No customer impact yet.
• October 5, 2026, 14:53 IDT: The next maintenance cycle starts and doesn't complete.
• October 5, 2026, 15:55 IDT: The database stops saving new data, and platform performance degrades.
• October 5, 2026, 16:03 IDT: The database reaches its connection limit. Customers start seeing errors.
• October 5, 2026, 16:21 IDT: Incident declared and response team engaged.
• October 5, 2026, 16:24 IDT: First update posted to the status page.
• October 5, 2026, 16:49 IDT: Switch to the standby database started.
• October 5, 2026, 17:00 IDT: Support case opened with our cloud infrastructure provider.
• October 5, 2026, 18:24 IDT: Alternative recovery path started in parallel. Workaround published for customers blocked by required pull request checks.
• October 5, 2026, 19:43 IDT: Database available again, and the Cycode API and Cycode Platform are accessible.
• October 5, 2026, 19:54 IDT: Pull request scanning resumes.
• October 5, 2026, 20:20 IDT: Processing capacity increased to clear delayed events.
• October 5, 2026, 20:45 IDT: High-volume workloads paused to reduce database load.
• October 5, 2026, 21:56 IDT: Pull requests missed during the incident identified and scanned.
• October 5, 2026, 23:09 IDT: All services fully operational.
• October 5, 2026, 23:24 IDT: Incident resolved.
3. Root Cause
Our database runs a routine background process that regularly saves recent changes to permanent storage. Two conditions contributed to that process slowing down:
• High-volume short-lived data operations: some of our processing workloads were constantly creating and removing large numbers of short-lived data structures.
• A delayed background sync: a data-sync process had fallen far behind, which stopped the database from cleaning up after itself.
The maintenance process slowed down and eventually stopped completing. The database could then no longer save new data, and requests started to queue. Our services retried those requests and opened many new connections, which used up the database's connection capacity. Customers saw this as errors and slow responses.
We switched to our standby database. Before it could serve traffic, it had to process all changes made since the last completed maintenance cycle, about four hours of activity. That extended the outage.
4. Actions Taken
• Reduced load on the database by temporarily scaling down non-essential processing.
• Started a switch to the standby database.
• Opened a support case with our cloud infrastructure provider and worked with them on an escalated basis.
• Prepared an alternative recovery path in parallel.
• Shared a temporary workaround for customers blocked by required pull request checks.
• Once the database recovered, increased processing capacity to clear delayed events.
1. Summary
On October 5, 2026, starting at 15:55 IDT, customers on our US platform saw slow responses, intermittent errors, and delays in pull request checks and CLI scans. Some customers whose repositories require Cycode pull request checks couldn't merge until scanning resumed.
The issue started when a background maintenance process in our primary database fell behind and stopped completing, and the database stopped saving new data. Restoring service took several hours because the standby database first had to process a large backlog of changes.
The database was back and the platform was available again by 19:43 IDT. Delayed processing then caught up, and the incident was resolved at 23:24 IDT. Pull requests that were missed during the incident were scanned.
2. Key Timeline (IDT)
October 5, 2026, 12:55 IDT: A routine database maintenance process starts running much slower than normal. No customer impact yet.
October 5, 2026, 14:53 IDT: The next maintenance cycle starts and doesn't complete.
October 5, 2026, 15:55 IDT: The database stops saving new data, and platform performance degrades.
October 5, 2026, 16:03 IDT: The database reaches its connection limit. Customers start seeing errors.
October 5, 2026, 16:21 IDT: Incident declared and response team engaged.
October 5, 2026, 16:24 IDT: First update posted to the status page.
October 5, 2026, 16:49 IDT: Switch to the standby database started.
October 5, 2026, 17:00 IDT: Support case opened with our cloud infrastructure provider.
October 5, 2026, 18:24 IDT: Alternative recovery path started in parallel. Workaround published for customers blocked by required pull request checks.
October 5, 2026, 19:43 IDT: Database available again, and the Cycode API and Cycode Platform are accessible.
October 5, 2026, 19:54 IDT: Pull request scanning resumes.
October 5, 2026, 20:20 IDT: Processing capacity increased to clear delayed events.
October 5, 2026, 20:45 IDT: High-volume workloads paused to reduce database load.
October 5, 2026, 21:56 IDT: Pull requests missed during the incident identified and scanned.
October 5, 2026, 23:09 IDT: All services fully operational.
October 5, 2026, 23:24 IDT: Incident resolved.
3. Root Cause
Our database runs a routine background process that regularly saves recent changes to permanent storage. Two conditions contributed to that process slowing down:
High-volume short-lived data operations: some of our processing workloads were constantly creating and removing large numbers of short-lived data structures.
A delayed background sync: a data-sync process had fallen far behind, which stopped the database from cleaning up after itself.
The maintenance process slowed down and eventually stopped completing. The database could then no longer save new data, and requests started to queue. Our services retried those requests and opened many new connections, which used up the database's connection capacity. Customers saw this as errors and slow responses.
We switched to our standby database. Before it could serve traffic, it had to process all changes made since the last completed maintenance cycle, about four hours of activity. That extended the outage.
4. Actions Taken
Reduced load on the database by temporarily scaling down non-essential processing.
Started a switch to the standby database.
Opened a support case with our cloud infrastructure provider and worked with them on an escalated basis.
Prepared an alternative recovery path in parallel.
Shared a temporary workaround for customers blocked by required pull request checks.
Once the database recovered, increased processing capacity to clear delayed events.
Paused the workloads that were creating high volumes of short-lived data operations, to reduce database load while delayed events cleared.
Scanned the pull requests that were missed during the incident.
4b. Action Items
Change how large batches of data are written to reduce database load.
Move our highest-volume workload to its own dedicated database, so a problem there can't affect the rest of the platform.
Add alerts and deeper monitoring for database connection usage and database health.
Give each service its own dedicated database credentials.
Process incoming events more efficiently to speed up recovery after a disruption.
Improve monitoring of processing delays and of internal event alerts.
Review database capacity and performance settings to add more headroom.
MemoryDB metrics are stable, the processing lag has recovered, and the system is now fully operational.
MemoryDB metrics are stable, the processing lag has recovered, and the system is now fully operational.
The system is back to being fully operational.
The system is back to being fully operational.
1. Summary
On August 19, 2026, some customers experienced slower page loading and data display, as well as delays when starting or completing pull request scans.
The disruption was caused by a managed memory database entering a repeated restart cycle and not recovering automatically. This created a temporary slowdown in processing. Expected service behavior was restored after corrective actions, including replacement and increased capacity for the affected service. The provider is continuing to investigate why the service was unable to return to normal operation automatically.
2. Key Timeline (IDT)
• August 19, 2026, 16:10 IDT: We identified performance degradation affecting parts of the platform and pull request scanning.
• August 19, 2026, 16:30 IDT: An initial corrective change was applied and service behavior was monitored.
• August 19, 2026, 16:46 IDT: We confirmed that some delays were continuing and expanded the investigation.
• August 19, 2026, 17:14 IDT: We began working with our service provider to investigate instability in the managed memory database.
• August 19, 2026, 18:46 IDT: Recovery actions were in progress, with temporary mitigations in place to reduce customer impact.
• August 19, 2026, 19:06 IDT: The managed memory database capacity update completed, and platform responsiveness and pull request scanning returned to expected behavior.
3. Root Cause
The managed memory database entered a repeated restart cycle and was unable to recover automatically. This caused delays in the processing systems that support platform responsiveness and pull request scans.
The affected service underwent a routine replacement process, followed by a capacity increase during the recovery effort. While the capacity increase completed successfully and restored expected behavior, AWS is still researching why the service did not come back up normally after the restart cycle. Their detailed root-cause analysis is pending.
4. Actions Taken
• Applied corrective updates to reduce immediate processing demand.
• Temporarily adjusted the status update process to reduce reliance on the affected service.
• Restarted affected processing components to restore responsiveness.
• Worked with the service provider to replace affected capacity and complete a capacity increase.
• Monitored platform performance and scan processing until expected behavior was confirmed.
4b. Action Items
• Review the service provider’s detailed root-cause analysis when available.
• Improve monitoring to detect similar recovery delays earlier.
• Add safeguards to reduce the impact of temporary processing slowdowns.
• Introduce separate Redis capacity for each service to isolate workloads and reduce the impact of an issue in one service on others.
1. Summary
On August 19, 2026, some customers experienced slower page loading and data display, as well as delays when starting or completing pull request scans.
The disruption was caused by a managed memory database entering a repeated restart cycle and not recovering automatically. This created a temporary slowdown in processing. Expected service behavior was restored after corrective actions, including replacement and increased capacity for the affected service. The provider is continuing to investigate why the service was unable to return to normal operation automatically.
2. Key Timeline (IDT)
August 19, 2026, 16:10 IDT: We identified performance degradation affecting parts of the platform and pull request scanning.
August 19, 2026, 16:30 IDT: An initial corrective change was applied and service behavior was monitored.
August 19, 2026, 16:46 IDT: We confirmed that some delays were continuing and expanded the investigation.
August 19, 2026, 17:14 IDT: We began working with our service provider to investigate instability in the managed memory database.
August 19, 2026, 18:46 IDT: Recovery actions were in progress, with temporary mitigations in place to reduce customer impact.
August 19, 2026, 19:06 IDT: The managed memory database capacity update completed, and platform responsiveness and pull request scanning returned to expected behavior.
3. Root Cause
The managed memory database entered a repeated restart cycle and was unable to recover automatically. This caused delays in the processing systems that support platform responsiveness and pull request scans.
The affected service underwent a routine replacement process, followed by a capacity increase during the recovery effort. While the capacity increase completed successfully and restored expected behavior, AWS is still researching why the service did not come back up normally after the restart cycle. Their detailed root-cause analysis is pending.
4. Actions Taken
Applied corrective updates to reduce immediate processing demand.
Temporarily adjusted the status update process to reduce reliance on the affected service.
Restarted affected processing components to restore responsiveness.
Worked with the service provider to replace affected capacity and complete a capacity increase.
Monitored platform performance and scan processing until expected behavior was confirmed.
4b. Action Items
Review the service provider’s detailed root-cause analysis when available.
Improve monitoring to detect similar recovery delays earlier.
Add safeguards to reduce the impact of temporary processing slowdowns.
Introduce separate Redis capacity for each service to isolate workloads and reduce the impact of an issue in one service on others.
(1/2)
Summary
PR secret scans experienced processing lag, causing delays in processing pull requests.
The issue was caused by timeouts when calling the model validation service. The model validation service was experiencing elevated latency and errors because compute nodes were scheduled on a subnet that had exhausted its available IPs.
This slowed PR secret scan processing and caused consumer lag to build up.
To resolve the issue, we temporarily disabled the feature that produces new PR secret scans so the queue could drain. We also resolved the underlying infrastructure issue by moving nodes to subnets with available free IPs.
Customer Impact
Customers experienced delayed PR secret scans.
Other scan types remained operational.
Root Cause
The secret scanner calls the AI model validation service to classify detections during PR scans.
The model validation call had a 30-second timeout configured. When the model validation service became slow or unresponsive, each failed request could block a scan consumer for the full 30 seconds before timing out.
The underlying issue was in the model-serving infrastructure. One of the private subnets had run out of available IPs, and new nodes were created in that exhausted subnet, leaving model-serving pods in a pending state.
This resulted in elevated latency and errors in the model validation service and reduced the processing capacity of PR secret scans.
Contributing Factors
• Model validation timeouts. The model validation service timeout was 30 seconds. When the service was degraded, scan consumers waited for the full timeout before the request failed, which created a processing bottleneck.
• No fallback to bypass AI validation for PR scans. There was no mechanism to skip AI model validation specifically for PR secret scans while allowing the rest of the scan to continue. The available kill switch stopped producing PR secret scans entirely. This meant the mitigation required a tradeoff between allowing processing lag to continue or temporarily stopping PR secret scans.
• Customer communication gap. Internal alerts for PR scan lag did fire, but the status page was not updated promptly to reflect the degradation. As a result, there was a period in which we had internal visibility into the issue while customers did not have the same visibility through the status page.
Mitigation and Recovery
A pre-existing feature flag was used to stop new PR secret scans from being produced.
This stopped new messages from being added to the PR secret scan queue and allowed the existing backlog to drain.
The backlog was cleared shortly after the mitigation was enabled.
In parallel, the underlying model-serving infrastructure issue was resolved by moving nodes to subnets with available free IPs, restoring model-serving capacity.
Once the infrastructure issue was resolved and the backlog had drained, PR secret scanning was restored.
Learnings and Corrective Actions
• Review the AI model validation timeout. The 30-second model validation timeout meant that when the model-serving layer degraded, each scan consumer could remain blocked for the full timeout before failing. We are reviewing the timeout and retry behavior. This work must also account for the risk that shorter timeouts and faster retries could increase request volume against an already degraded model-serving service.
Summary
PR secret scans experienced processing lag, causing delays in processing pull requests.
The issue was caused by timeouts when calling the model validation service. The model validation service was experiencing elevated latency and errors because compute nodes were scheduled on a subnet that had exhausted its available IPs.
This slowed PR secret scan processing and caused consumer lag to build up.
To resolve the issue, we temporarily disabled the feature that produces new PR secret scans so the queue could drain. We also resolved the underlying infrastructure issue by moving nodes to subnets with available free IPs.
Customer Impact
Customers experienced delayed PR secret scans.
Other scan types remained operational.
Root Cause
The secret scanner calls the AI model validation service to classify detections during PR scans.
The model validation call had a 30-second timeout configured. When the model validation service became slow or unresponsive, each failed request could block a scan consumer for the full 30 seconds before timing out.
The underlying issue was in the model-serving infrastructure. One of the private subnets had run out of available IPs, and new nodes were created in that exhausted subnet, leaving model-serving pods in a pending state.
This resulted in elevated latency and errors in the model validation service and reduced the processing capacity of PR secret scans.
Contributing Factors
Model validation timeouts. The model validation service timeout was 30 seconds. When the service was degraded, scan consumers waited for the full timeout before the request failed, which created a processing bottleneck.
No fallback to bypass AI validation for PR scans. There was no mechanism to skip AI model validation specifically for PR secret scans while allowing the rest of the scan to continue. The available kill switch stopped producing PR secret scans entirely. This meant the mitigation required a tradeoff between allowing processing lag to continue or temporarily stopping PR secret scans.
Customer communication gap. Internal alerts for PR scan lag did fire, but the status page was not updated promptly to reflect the degradation. As a result, there was a period in which we had internal visibility into the issue while customers did not have the same visibility through the status page.
Mitigation and Recovery
A pre-existing feature flag was used to stop new PR secret scans from being produced.
This stopped new messages from being added to the PR secret scan queue and allowed the existing backlog to drain.
The backlog was cleared shortly after the mitigation was enabled.
In parallel, the underlying model-serving infrastructure issue was resolved by moving nodes to subnets with available free IPs, restoring model-serving capacity.
Once the infrastructure issue was resolved and the backlog had drained, PR secret scanning was restored.
Learnings and Corrective Actions
Review the AI model validation timeout. The 30-second model validation timeout meant that when the model-serving layer degraded, each scan consumer could remain blocked for the full timeout before failing. We are reviewing the timeout and retry behavior. This work must also account for the risk that shorter timeouts and faster retries could increase request volume against an already degraded model-serving service.
Add a graceful degradation path for AI validation. The system currently has a kill switch for PR secret scanning, but does not have a graceful degradation mechanism for the AI validation step. We are working on a mechanism that can disable AI model calls specifically for PR scans, allowing scans to continue while avoiding the dependency on a degraded model-serving layer. The goal is to allow scans to continue with reduced functionality rather than stopping PR secret scanning entirely.
Improve alerting and incident communication. The alerting infrastructure detected PR scan lag, but the status page was not updated promptly. We are improving the process for escalating service degradation and communicating customer-facing impact through the status page.
Improve model-serving infrastructure resilience. The underlying infrastructure issue occurred because nodes were scheduled on a subnet with no available IPs. The immediate issue was resolved by moving nodes to subnets with free IPs. We are continuing to improve the resilience of the model-serving infrastructure to reduce the likelihood that infrastructure capacity issues can affect PR scan processing.
Conclusion
The incident was caused by degradation in the AI model validation service due to an infrastructure issue where nodes were scheduled on a subnet with exhausted IP capacity.
The resulting model validation timeouts slowed PR secret scan processing and caused consumer lag to build.
We mitigated the incident by temporarily stopping new PR secret scans, allowing the backlog to drain, and restoring model-serving capacity by moving nodes to subnets with available free IPs.
We are following up with improvements to model-serving resilience, timeout and retry behavior, runtime configuration, graceful degradation, alerting, and customer communication.
GitHub component: API Requests
Original GitHub incident: https://stspg.io/3xn46bst0bjh
Root Cause
The incident was caused by a deployment of one service that ran a database index creation. The team has identified an issue with the way we perform index creations as database migrations.
During a deployment the pods with newest image of the service attempted to create an index on a big table. The team has identified that the index creation took 7 minutes. However, during index creation pods were not responsive, and as a result, Kubernetes deemed them as unhealthy pods and attempted to retry those pods after 5 minutes.
As a result, because the pod got killed before the index creation was fully completed, the database transaction was rolled back. Then, subsequent pods attempted to create the index again, dying after 5 minutes.
This lead to the database being in unhealthy state, and the service was down.
The team has rolled back the deployment, and killed all replicas that attempted to create the index. Thanks to that, the service and the database was in healthy state again.
Why safety measures did not help
Cycode provides a safety mechanism that unblocks all Pull Request scans after a specific period of time, giving each scan a maximum duration before the Pull Request is unblocked. However, because the service that is responsible for triggering and completing scans, as well as this safety net, was down, the process couldn't behave as expected. We acknowledge this gap and are working on strengthening this area of our system.
Action items
The team is actively investigating enhancements and new safety protocols that can be put in place in order to have another safety net preventing Pull Request scans being stuck in case of any incident.
The team is investigating changes to the index creation process.
Root Cause
The incident was caused by a deployment of one service that ran a database index creation. The team has identified an issue with the way we perform index creations as database migrations.
During a deployment the pods with newest image of the service attempted to create an index on a big table. The team has identified that the index creation took 7 minutes. However, during index creation pods were not responsive, and as a result, Kubernetes deemed them as unhealthy pods and attempted to retry those pods after 5 minutes.
As a result, because the pod got killed before the index creation was fully completed, the database transaction was rolled back. Then, subsequent pods attempted to create the index again, dying after 5 minutes.
This lead to the database being in unhealthy state, and the service was down.
The team has rolled back the deployment, and killed all replicas that attempted to create the index. Thanks to that, the service and the database was in healthy state again.
Why safety measures did not help
Cycode provides a safety mechanism that unblocks all Pull Request scans after a specific period of time, giving each scan a maximum duration before the Pull Request is unblocked. However, because the service that is responsible for triggering and completing scans, as well as this safety net, was down, the process couldn't behave as expected. We acknowledge this gap and are working on strengthening this area of our system.
Action items
• The team is actively investigating enhancements and new safety protocols that can be put in place in order to have another safety net preventing Pull Request scans being stuck in case of any incident.
• The team is investigating changes to the index creation process.
Root Cause
The incident was caused by a deployment of one service that ran a database migration. The team has identified that this migration contained faulty code and as a result lead to database overload when attempting to deploy the service. As a result, the service was partially down until the deployment was reverted. During this time, all scans were processed with lower than expected performance.
Root Cause
The incident was caused by a deployment of one service that ran a database migration. The team has identified that this migration contained faulty code and as a result lead to database overload when attempting to deploy the service. As a result, the service was partially down until the deployment was reverted. During this time, all scans were processed with lower than expected performance.
The issue has been resolved, and we are continuing to monitor the situation.
The issue has been resolved, and we are continuing to monitor the situation.
GitHub component: Pull Requests
Original GitHub incident: https://stspg.io/sm1tp7kfm4vj
GitHub component: API Requests, Pull Requests
Original GitHub incident: https://stspg.io/j5c80shxqm53