(1/2)
1. Summary
On October 5, 2026, starting at 15:55 IDT, customers on our US platform saw slow responses, intermittent errors, and delays in pull request checks and CLI scans. Some customers whose repositories require Cycode pull request checks couldn't merge until scanning resumed.
The issue started when a background maintenance process in our primary database fell behind and stopped completing, and the database stopped saving new data. Restoring service took several hours because the standby database first had to process a large backlog of changes.
The database was back and the platform was available again by 19:43 IDT. Delayed processing then caught up, and the incident was resolved at 23:24 IDT. Pull requests that were missed during the incident were scanned.
2. Key Timeline (IDT)
• October 5, 2026, 12:55 IDT: A routine database maintenance process starts running much slower than normal. No customer impact yet.
• October 5, 2026, 14:53 IDT: The next maintenance cycle starts and doesn't complete.
• October 5, 2026, 15:55 IDT: The database stops saving new data, and platform performance degrades.
• October 5, 2026, 16:03 IDT: The database reaches its connection limit. Customers start seeing errors.
• October 5, 2026, 16:21 IDT: Incident declared and response team engaged.
• October 5, 2026, 16:24 IDT: First update posted to the status page.
• October 5, 2026, 16:49 IDT: Switch to the standby database started.
• October 5, 2026, 17:00 IDT: Support case opened with our cloud infrastructure provider.
• October 5, 2026, 18:24 IDT: Alternative recovery path started in parallel. Workaround published for customers blocked by required pull request checks.
• October 5, 2026, 19:43 IDT: Database available again, and the Cycode API and Cycode Platform are accessible.
• October 5, 2026, 19:54 IDT: Pull request scanning resumes.
• October 5, 2026, 20:20 IDT: Processing capacity increased to clear delayed events.
• October 5, 2026, 20:45 IDT: High-volume workloads paused to reduce database load.
• October 5, 2026, 21:56 IDT: Pull requests missed during the incident identified and scanned.
• October 5, 2026, 23:09 IDT: All services fully operational.
• October 5, 2026, 23:24 IDT: Incident resolved.
3. Root Cause
Our database runs a routine background process that regularly saves recent changes to permanent storage. Two conditions contributed to that process slowing down:
• High-volume short-lived data operations: some of our processing workloads were constantly creating and removing large numbers of short-lived data structures.
• A delayed background sync: a data-sync process had fallen far behind, which stopped the database from cleaning up after itself.
The maintenance process slowed down and eventually stopped completing. The database could then no longer save new data, and requests started to queue. Our services retried those requests and opened many new connections, which used up the database's connection capacity. Customers saw this as errors and slow responses.
We switched to our standby database. Before it could serve traffic, it had to process all changes made since the last completed maintenance cycle, about four hours of activity. That extended the outage.
4. Actions Taken
• Reduced load on the database by temporarily scaling down non-essential processing.
• Started a switch to the standby database.
• Opened a support case with our cloud infrastructure provider and worked with them on an escalated basis.
• Prepared an alternative recovery path in parallel.
• Shared a temporary workaround for customers blocked by required pull request checks.
• Once the database recovered, increased processing capacity to clear delayed events.
Resolved
(1/2)
1. Summary
On October 5, 2026, starting at 15:55 IDT, customers on our US platform saw slow responses, intermittent errors, and delays in pull request checks and CLI scans. Some customers whose repositories require Cycode pull request checks couldn't merge until scanning resumed.
The issue started when a background maintenance process in our primary database fell behind and stopped completing, and the database stopped saving new data. Restoring service took several hours because the standby database first had to process a large backlog of changes.
The database was back and the platform was available again by 19:43 IDT. Delayed processing then caught up, and the incident was resolved at 23:24 IDT. Pull requests that were missed during the incident were scanned.
2. Key Timeline (IDT)
• October 5, 2026, 12:55 IDT: A routine database maintenance process starts running much slower than normal. No customer impact yet.
• October 5, 2026, 14:53 IDT: The next maintenance cycle starts and doesn't complete.
• October 5, 2026, 15:55 IDT: The database stops saving new data, and platform performance degrades.
• October 5, 2026, 16:03 IDT: The database reaches its connection limit. Customers start seeing errors.
• October 5, 2026, 16:21 IDT: Incident declared and response team engaged.
• October 5, 2026, 16:24 IDT: First update posted to the status page.
• October 5, 2026, 16:49 IDT: Switch to the standby database started.
• October 5, 2026, 17:00 IDT: Support case opened with our cloud infrastructure provider.
• October 5, 2026, 18:24 IDT: Alternative recovery path started in parallel. Workaround published for customers blocked by required pull request checks.
• October 5, 2026, 19:43 IDT: Database available again, and the Cycode API and Cycode Platform are accessible.
• October 5, 2026, 19:54 IDT: Pull request scanning resumes.
• October 5, 2026, 20:20 IDT: Processing capacity increased to clear delayed events.
• October 5, 2026, 20:45 IDT: High-volume workloads paused to reduce database load.
• October 5, 2026, 21:56 IDT: Pull requests missed during the incident identified and scanned.
• October 5, 2026, 23:09 IDT: All services fully operational.
• October 5, 2026, 23:24 IDT: Incident resolved.
3. Root Cause
Our database runs a routine background process that regularly saves recent changes to permanent storage. Two conditions contributed to that process slowing down:
• High-volume short-lived data operations: some of our processing workloads were constantly creating and removing large numbers of short-lived data structures.
• A delayed background sync: a data-sync process had fallen far behind, which stopped the database from cleaning up after itself.
The maintenance process slowed down and eventually stopped completing. The database could then no longer save new data, and requests started to queue. Our services retried those requests and opened many new connections, which used up the database's connection capacity. Customers saw this as errors and slow responses.
We switched to our standby database. Before it could serve traffic, it had to process all changes made since the last completed maintenance cycle, about four hours of activity. That extended the outage.
4. Actions Taken
• Reduced load on the database by temporarily scaling down non-essential processing.
• Started a switch to the standby database.
• Opened a support case with our cloud infrastructure provider and worked with them on an escalated basis.
• Prepared an alternative recovery path in parallel.
• Shared a temporary workaround for customers blocked by required pull request checks.
• Once the database recovered, increased processing capacity to clear delayed events.
Resolved
(2/2)
• Paused the workloads that were creating high volumes of short-lived data operations, to reduce database load while delayed events cleared.
• Scanned the pull requests that were missed during the incident.
4b. Action Items
• Change how large batches of data are written to reduce database load.
• Move our highest-volume workload to its own dedicated database, so a problem there can't affect the rest of the platform.
• Add alerts and deeper monitoring for database connection usage and database health.
• Give each service its own dedicated database credentials.
• Process incoming events more efficiently to speed up recovery after a disruption.
• Improve monitoring of processing delays and of internal event alerts.
• Review database capacity and performance settings to add more headroom.
Investigating
The system is back to being fully operational
Investigating
Last remaining backlog continues being processed at a steady rate.
Our estimations show that last parts backlog should be fully processed in about 60-90 minutes (20:30 UTC).
We see the majority of pull request scans (60% at the moment of this update) are being completed in below 15 minutes. and this number is constantly growing at a steady rate.
Other than that system looks operational.
We believe application/UI are fully operational.
Investigating
The US platform is improving following the earlier RDS/PostgreSQL connection exhaustion.
AWS has confirmed that the original PostgreSQL instance has returned to an operational state, and we’re seeing PR scans being processed again.
The main remaining degradation is delayed GitHub-driven processing due to a large webhook backlog, which is continuing to drain.
We’ll continue monitoring recovery and provide another update as processing returns to normal
Investigating
Database remains fully operational.
We are working on increasing processing speed beyond regular to process all backlogs as soon as possible.
Investigating
We are seeing database is restoring its operational capabities.
We see first Pull Request scans being processed.
We are monitoring the system and increasing the throughput.
Investigating
AWS has confirmed that the affected PostgreSQL instance is currently in an internal recovery state.
The expected Multi-AZ failover did not occur as designed, and AWS is actively investigating why.
In parallel, we are continuing work to restore the affected services using an alternative database path to accelerate recovery
Investigating
The team continues to work with AWS, which escalated the issue internally.
In parallel, we are working to restore functionality by moving to another database.
Any blocked pull requests waiting for Cycode status can be unblocked by removing Cycode scans from expected checks directly in your SCM (Source Code Management). Reach out to Your Cycode representative to help with streamlining the process.
Investigating
The team is actively working with AWS and investigating what appears to be a database failover issue affecting the US environment.
We’ll share another update as soon as we have more information.
Investigating
We’re continuing to investigate a major degradation in the US platform caused by PostgreSQL connection exhaustion. Customers may experience slow response times, elevated 500 errors, and slower PR workflows and CLI scans.
We’ve reduced database load across several services and initiated an RDS failover. The team is actively working to restore normal performance.
Investigating
We have noticed degraded performance in prod-us
Flows like Pull Request or CLI scanning can be slower than usual