Summary
PR secret scans experienced processing lag, causing delays in processing pull requests.
The issue was caused by timeouts when calling the model validation service. The model validation service was experiencing elevated latency and errors because compute nodes were scheduled on a subnet that had exhausted its available IPs.
This slowed PR secret scan processing and caused consumer lag to build up.
To resolve the issue, we temporarily disabled the feature that produces new PR secret scans so the queue could drain. We also resolved the underlying infrastructure issue by moving nodes to subnets with available free IPs.
Customer Impact
Customers experienced delayed PR secret scans.
Other scan types remained operational.
Root Cause
The secret scanner calls the AI model validation service to classify detections during PR scans.
The model validation call had a 30-second timeout configured. When the model validation service became slow or unresponsive, each failed request could block a scan consumer for the full 30 seconds before timing out.
The underlying issue was in the model-serving infrastructure. One of the private subnets had run out of available IPs, and new nodes were created in that exhausted subnet, leaving model-serving pods in a pending state.
This resulted in elevated latency and errors in the model validation service and reduced the processing capacity of PR secret scans.
Contributing Factors
Model validation timeouts. The model validation service timeout was 30 seconds. When the service was degraded, scan consumers waited for the full timeout before the request failed, which created a processing bottleneck.
No fallback to bypass AI validation for PR scans. There was no mechanism to skip AI model validation specifically for PR secret scans while allowing the rest of the scan to continue. The available kill switch stopped producing PR secret scans entirely. This meant the mitigation required a tradeoff between allowing processing lag to continue or temporarily stopping PR secret scans.
Customer communication gap. Internal alerts for PR scan lag did fire, but the status page was not updated promptly to reflect the degradation. As a result, there was a period in which we had internal visibility into the issue while customers did not have the same visibility through the status page.
Mitigation and Recovery
A pre-existing feature flag was used to stop new PR secret scans from being produced.
This stopped new messages from being added to the PR secret scan queue and allowed the existing backlog to drain.
The backlog was cleared shortly after the mitigation was enabled.
In parallel, the underlying model-serving infrastructure issue was resolved by moving nodes to subnets with available free IPs, restoring model-serving capacity.
Once the infrastructure issue was resolved and the backlog had drained, PR secret scanning was restored.
Learnings and Corrective Actions
Review the AI model validation timeout. The 30-second model validation timeout meant that when the model-serving layer degraded, each scan consumer could remain blocked for the full timeout before failing. We are reviewing the timeout and retry behavior. This work must also account for the risk that shorter timeouts and faster retries could increase request volume against an already degraded model-serving service.
Add a graceful degradation path for AI validation. The system currently has a kill switch for PR secret scanning, but does not have a graceful degradation mechanism for the AI validation step. We are working on a mechanism that can disable AI model calls specifically for PR scans, allowing scans to continue while avoiding the dependency on a degraded model-serving layer. The goal is to allow scans to continue with reduced functionality rather than stopping PR secret scanning entirely.
Improve alerting and incident communication. The alerting infrastructure detected PR scan lag, but the status page was not updated promptly. We are improving the process for escalating service degradation and communicating customer-facing impact through the status page.
Improve model-serving infrastructure resilience. The underlying infrastructure issue occurred because nodes were scheduled on a subnet with no available IPs. The immediate issue was resolved by moving nodes to subnets with free IPs. We are continuing to improve the resilience of the model-serving infrastructure to reduce the likelihood that infrastructure capacity issues can affect PR scan processing.
Conclusion
The incident was caused by degradation in the AI model validation service due to an infrastructure issue where nodes were scheduled on a subnet with exhausted IP capacity.
The resulting model validation timeouts slowed PR secret scan processing and caused consumer lag to build.
We mitigated the incident by temporarily stopping new PR secret scans, allowing the backlog to drain, and restoring model-serving capacity by moving nodes to subnets with available free IPs.
We are following up with improvements to model-serving resilience, timeout and retry behavior, runtime configuration, graceful degradation, alerting, and customer communication.
Resolved
Summary
PR secret scans experienced processing lag, causing delays in processing pull requests.
The issue was caused by timeouts when calling the model validation service. The model validation service was experiencing elevated latency and errors because compute nodes were scheduled on a subnet that had exhausted its available IPs.
This slowed PR secret scan processing and caused consumer lag to build up.
To resolve the issue, we temporarily disabled the feature that produces new PR secret scans so the queue could drain. We also resolved the underlying infrastructure issue by moving nodes to subnets with available free IPs.
Customer Impact
Customers experienced delayed PR secret scans.
Other scan types remained operational.
Root Cause
The secret scanner calls the AI model validation service to classify detections during PR scans.
The model validation call had a 30-second timeout configured. When the model validation service became slow or unresponsive, each failed request could block a scan consumer for the full 30 seconds before timing out.
The underlying issue was in the model-serving infrastructure. One of the private subnets had run out of available IPs, and new nodes were created in that exhausted subnet, leaving model-serving pods in a pending state.
This resulted in elevated latency and errors in the model validation service and reduced the processing capacity of PR secret scans.
Contributing Factors
Model validation timeouts. The model validation service timeout was 30 seconds. When the service was degraded, scan consumers waited for the full timeout before the request failed, which created a processing bottleneck.
No fallback to bypass AI validation for PR scans. There was no mechanism to skip AI model validation specifically for PR secret scans while allowing the rest of the scan to continue. The available kill switch stopped producing PR secret scans entirely. This meant the mitigation required a tradeoff between allowing processing lag to continue or temporarily stopping PR secret scans.
Customer communication gap. Internal alerts for PR scan lag did fire, but the status page was not updated promptly to reflect the degradation. As a result, there was a period in which we had internal visibility into the issue while customers did not have the same visibility through the status page.
Mitigation and Recovery
A pre-existing feature flag was used to stop new PR secret scans from being produced.
This stopped new messages from being added to the PR secret scan queue and allowed the existing backlog to drain.
The backlog was cleared shortly after the mitigation was enabled.
In parallel, the underlying model-serving infrastructure issue was resolved by moving nodes to subnets with available free IPs, restoring model-serving capacity.
Once the infrastructure issue was resolved and the backlog had drained, PR secret scanning was restored.
Learnings and Corrective Actions
Review the AI model validation timeout. The 30-second model validation timeout meant that when the model-serving layer degraded, each scan consumer could remain blocked for the full timeout before failing. We are reviewing the timeout and retry behavior. This work must also account for the risk that shorter timeouts and faster retries could increase request volume against an already degraded model-serving service.
Add a graceful degradation path for AI validation. The system currently has a kill switch for PR secret scanning, but does not have a graceful degradation mechanism for the AI validation step. We are working on a mechanism that can disable AI model calls specifically for PR scans, allowing scans to continue while avoiding the dependency on a degraded model-serving layer. The goal is to allow scans to continue with reduced functionality rather than stopping PR secret scanning entirely.
Improve alerting and incident communication. The alerting infrastructure detected PR scan lag, but the status page was not updated promptly. We are improving the process for escalating service degradation and communicating customer-facing impact through the status page.
Improve model-serving infrastructure resilience. The underlying infrastructure issue occurred because nodes were scheduled on a subnet with no available IPs. The immediate issue was resolved by moving nodes to subnets with free IPs. We are continuing to improve the resilience of the model-serving infrastructure to reduce the likelihood that infrastructure capacity issues can affect PR scan processing.
Conclusion
The incident was caused by degradation in the AI model validation service due to an infrastructure issue where nodes were scheduled on a subnet with exhausted IP capacity.
The resulting model validation timeouts slowed PR secret scan processing and caused consumer lag to build.
We mitigated the incident by temporarily stopping new PR secret scans, allowing the backlog to drain, and restoring model-serving capacity by moving nodes to subnets with available free IPs.
We are following up with improvements to model-serving resilience, timeout and retry behavior, runtime configuration, graceful degradation, alerting, and customer communication.
Monitoring
PR Secrets scanning is operating normally, the backlog has been fully processed, and scan latency has returned to expected levels.
The incident was caused by elevated latency in our AI model-serving infrastructure. A mitigation has been applied, and we will continue monitoring the system to ensure stability.
Monitoring
The backlog of PR Secrets scans has been fully processed, and scan latency has returned to normal levels. The issue was caused by elevated latency in our AI model-serving infrastructure. We have applied a mitigation and are continuing to monitor the system.
We temporarily disabled PR Secrets scanning between 4:24 PM and 4:50 PM CEST and allowed the affected checks to pass to prevent PRs from being blocked. During this period, PR Secrets scans were not created and therefore will not appear on the PR Scans page.
PR Secrets scanning has now been re-enabled.
Identified
We have identified the root cause of the issue and are actively working on a fix. In the meantime, affected PR Secret detections will not be marked as stuck.
Investigating
We are investigating degraded performance affecting Secrets scanning for Pull Requests in the US environment. Our team is actively working to identify the root cause.