Summary
Between 16:00 and 17:30 UTC on May 28, we observed an issue where integration connections were intermittently reported as invalid. During this period, the processing of pull requests was paused until connection tokens were successfully refreshed. As a result, some pull requests may have experienced delays or remained in a stuck state. The issue is now resolved, and processing has resumed normally.
Key Timeline (UTC)
May 28, 16:00 UTC: A high rate of integration connections were reported as invalid, and pull request processing was paused.
May 28, 17:00 UTC: The external service began returning successful responses, and the system started marking integrations as valid.
May 28, 17:30 UTC: Connection tokens were successfully refreshed and pull request processing resumed.
Root Cause
The issue was caused by an external service provider returning "no access" responses during a routine validity check. This caused the system to mark integrations as invalid and pause processing. The failures occurred simultaneously across many unrelated accounts and were resolved without any changes on our side, indicating a service disruption on the provider's side.
While the observed behavior clearly points to the external provider, we are continuing to work with GitHub to better understand the underlying cause of this transient failure.
Actions Taken
Restored expected processing behavior once connection tokens were successfully refreshed.
Monitored the system to ensure all integrations returned to a valid state.
Action Items
Improve monitoring to detect similar external service anomalies.
Add alerting on simultaneous validity-check failures across unrelated accounts.
Resolved
Summary
Between 16:00 and 17:30 UTC on May 28, we observed an issue where integration connections were intermittently reported as invalid. During this period, the processing of pull requests was paused until connection tokens were successfully refreshed. As a result, some pull requests may have experienced delays or remained in a stuck state. The issue is now resolved, and processing has resumed normally.
Key Timeline (UTC)
May 28, 16:00 UTC: A high rate of integration connections were reported as invalid, and pull request processing was paused.
May 28, 17:00 UTC: The external service began returning successful responses, and the system started marking integrations as valid.
May 28, 17:30 UTC: Connection tokens were successfully refreshed and pull request processing resumed.
Root Cause
The issue was caused by an external service provider returning "no access" responses during a routine validity check. This caused the system to mark integrations as invalid and pause processing. The failures occurred simultaneously across many unrelated accounts and were resolved without any changes on our side, indicating a service disruption on the provider's side.
While the observed behavior clearly points to the external provider, we are continuing to work with GitHub to better understand the underlying cause of this transient failure.
Actions Taken
Restored expected processing behavior once connection tokens were successfully refreshed.
Monitored the system to ensure all integrations returned to a valid state.
Action Items
Improve monitoring to detect similar external service anomalies.
Add alerting on simultaneous validity-check failures across unrelated accounts.
Investigating
Summary Between 16:00 and 17:30 UTC, we observed a high rate of GitHub installations becoming invalid. During this period, PR processing was paused until tokens were successfully refreshed. As a result, some pull requests may have become stuck and not been processed in a timely manner.
Impact
GitHub installations intermittently reported as invalidPRs opened or updated during this window may not have been processedSome PRs may remain in a stuck state and require a manual re-trigger (e.g., a new commit or re-opening the PR)Current Status Token refreshes have completed successfully and PR processing has resumed normally.