Auto-Resolution of GitLab Enterprise Repos

Summary:

On March 13, 2025, repositories were unexpectedly removed for multiple customers in the Cycode Platform using GitLab Enterprise. The issue was traced to incorrect handling of the state tracker in the repository sync process, which caused the system to believe that repositories did not exist, which triggered the deletion of these repositories from the system.

This led to unnecessary resync operations, significantly increasing system load and causing delays in repository synchronization and violation updates.

Timeline:

13.03.25 05:13 AM - Customer opened a support ticket and reported that the number of repositories in the inventory tab filtered by GitLab in the Cycode platform was dropped significantly.
13.03.25 09:13 AM- A recent commit is suspected to be the cause.
13.03.25 09:17 AM - The issue was identified. Incorrect state tracker usage in repository sync led to unintended repository deletion events. The periodic metadata sync process removed repositories incorrectly, leading to missing data and resolved violations.
13.03.25 09:24 AM - Only customers using GitLab Enterprise installations with nongroup integration were affected.
13.03.25 09:30 AM—We understood that since a repository event was published to the system for each customer repository, all the violations were automatically resolved, the repositories were deleted from the projects, and they were missing in the inventory and the RIG.
13.03.25 09:31 AM - Action plan established: Revert the commit, restore affected violation statuses, restore repositories data, restore project data.
13.03.25 09:59 AM - Repository sync retriggered for affected customers.
13.03.25 10:30 AM - Repositories started gradually reappearing as resync progressed.
13.03.25 01:59 PM - The system is highly loaded due to repository resyncs and violations updates. Which causes delays until the RIG and Risk score is updated.
14.03.25 06:31 PM - The Incident was resolved, but monitoring continued to verify data integrity.

Root Cause Analysis:

• Incorrect usage of the state tracker in repository sync caused the system to believe repositories did not exist, triggering unnecessary resyncs.
• Continuous repository resyncs placed an excessive load on the system, leading to significant delays, syncs, RIG, risk score and violation updates.
• Affected customers experienced repository deletions, missing violations, and project corruption.

The incident has been fully resolved, and all data is now aligned. We will provide the RCA next week.

We have identified the issue and reverted the problematic code. We are now working on fully restoring the original status of the violations. Please note that this only impacts GitLab Enterprise (self-hosted) integrations. If you are not using GitLab Enterprise, you are not affected.

A recent code deployment deployed yesterday triggered an unintended event in our system, causing false events of repository deletions (in Cycode) from GitLab Enterprise (self-hosted). As a result, associated violations were auto-resolved by mistake. NO REPOS WERE DELETED FROM THE SCM

Current Status:

• We have identified the root cause and are actively reverting the faulty code.
• The affected violation statuses are being restored.
• No customer data was lost or permanently affected.

Next Steps:

• We will provide a full Root Cause Analysis (RCA) once the issue is fully resolved.
• If you notice any discrepancies in your violations, please reach out to support.

We apologize for the inconvenience and appreciate your patience as we work to resolve this.