Resolved
The system is back to being fully operational.
Investigating
We’re currently experiencing some delays in GitHub scan processing. Our team is investigating the issue and working to restore normal processing times.
Resolved
Summary
Problem: We observed a delay in processing status updates for GitHub checks due to a backlog in our event processing system.
Impact: A small number of pull requests (PRs) experienced delays in status check updates. While the majority of the system remained functional, a specific segment of the processing queue was affected, causing a lag for those specific updates over several hours.
Cause: The delay was caused by a high volume of retries generated by unrecoverable permission errors on a limited number of requests, which blocked a portion of our processing capacity. This occurred during a period of high event volume, which led to the database reaching maximum capacity and slowing down the overall processing workflow.
Resolution: The issue has been resolved. We scaled up our database capacity, increased the number of processing units to handle the higher volume, and implemented logic to stop retrying updates that fail due to permanent permission errors. All backlogs have been cleared, and system performance has returned to normal for all pull requests.
Key Timeline (IDT)
August 21, 2026, 06:30 IDT: An incident was declared as delays in status updates for a subset of requests became visible.
August 21, 2026, 06:50 IDT: A public status page was published to inform customers of the degradation.
August 21, 2026, 07:15 IDT: The database was scaled up to address processing bottlenecks.
August 21, 2026, 07:45 IDT: Processing capacity was doubled by increasing the number of active queue partitions.
August 21, 2026, 08:10 IDT: A corrective update was deployed to stop unnecessary retries on unrecoverable permission errors.
August 21, 2026, 08:45 IDT: The event backlog was fully drained, and processing returned to real-time.
August 21, 2026, 09:00 IDT: The incident was officially resolved.
Root Cause
The incident was caused by a combination of high event volume and a lack of "fail-fast" logic for permanent errors. Specific requests with incorrect permission settings generated a massive volume of retry attempts. Because the system treated these permanent permission errors as temporary, it repeatedly retried them, which blocked a specific processing partition and prevented updates for a small number of pull requests from being processed. This was further exacerbated by the database reaching its resource limits, creating a temporary slowdown in the update workflow.
Actions Taken
Scaled Database Resources: Increased the capacity of the primary database to handle the increased load and improve processing speed.
Increased Processing Parallelism: Doubled the number of active partitions in the event queue to ensure all available processing units were utilized.
Deployed Retry Exclusion Logic: Updated the system to identify unrecoverable permission errors and stop retrying them, preventing queue blockages.
Monitoring
Scan processing delays have been draining quickly, and most scans have not been affected. A small number of pull requests may still experience delayed scan processing. We’re continuing to monitor recovery and will share another update as we learn more
Identified
GitHub scan processing and status-check updates remain delayed, affecting all scan types. We have made changes to improve processing capacity and are monitoring recovery as the backlog decreases.
Investigating
We’re currently experiencing some delays in GitHub scan processing. Our team is investigating the issue and working to restore normal processing times.