Summary
On March 13, 2025, repositories were unexpectedly removed for multiple customers in the Cycode Platform using GitLab Enterprise. The issue was traced to incorrect handling of the state tracker in the repository sync process, which caused the system to believe that repositories did not exist, which triggered the deletion of these repositories from the system.
This led to unnecessary resync operations, significantly increasing system load and causing delays in repository synchronization and violation updates.
Timeline:
13.03.25 05:13 AM - Customer opened a support ticket and reported that the number of repositories in the inventory tab filtered by GitLab in the Cycode platform was dropped significantly.
13.03.25 09:13 AM - A recent commit is suspected to be the cause.
13.03.25 09:17 AM - The issue was identified. Incorrect state tracker usage in repository sync led to unintended repository deletion events. The periodic metadata sync process removed repositories incorrectly, leading to missing data and resolved violations.
13.03.25 09:24 AM - Only customers using GitLab Enterprise installations with nongroup integration were affected.
13.03.25 09:30 AM - We understood that since a repository event was published to the system for each customer repository, all the violations were automatically resolved, the repositories were deleted from the projects, and they were missing in the inventory and the RIG.
13.03.25 09:31 AM - Action plan established: Revert the commit, restore affected violation statuses, restore repositories data, restore project data.
13.03.25 09:59 AM - Repository sync retriggered for affected customers.
13.03.25 10:30 AM - Repositories started gradually reappearing as resync progressed.
13.03.25 01:59 PM - The system is highly loaded due to repository resyncs and violations updates. Which causes delays until the RIG and Risk score is updated.
14.03.25 06:31 PM - The Incident was resolved, but monitoring continued to verify data integrity.
Root Cause Analysis:
• Incorrect usage of the state tracker in repository sync caused the system to believe repositories did not exist, triggering unnecessary resyncs.
• Continuous repository resyncs placed an excessive load on the system, leading to significant delays, syncs, RIG, risk score and violation updates.
• Affected customers experienced repository deletions, missing violations, and project corruption.
No components marked as affected
Resolved
Summary
On March 13, 2025, repositories were unexpectedly removed for multiple customers in the Cycode Platform using GitLab Enterprise. The issue was traced to incorrect handling of the state tracker in the repository sync process, which caused the system to believe that repositories did not exist, which triggered the deletion of these repositories from the system.
This led to unnecessary resync operations, significantly increasing system load and causing delays in repository synchronization and violation updates.
Timeline:
13.03.25 05:13 AM - Customer opened a support ticket and reported that the number of repositories in the inventory tab filtered by GitLab in the Cycode platform was dropped significantly.
13.03.25 09:13 AM - A recent commit is suspected to be the cause.
13.03.25 09:17 AM - The issue was identified. Incorrect state tracker usage in repository sync led to unintended repository deletion events. The periodic metadata sync process removed repositories incorrectly, leading to missing data and resolved violations.
13.03.25 09:24 AM - Only customers using GitLab Enterprise installations with nongroup integration were affected.
13.03.25 09:30 AM - We understood that since a repository event was published to the system for each customer repository, all the violations were automatically resolved, the repositories were deleted from the projects, and they were missing in the inventory and the RIG.
13.03.25 09:31 AM - Action plan established: Revert the commit, restore affected violation statuses, restore repositories data, restore project data.
13.03.25 09:59 AM - Repository sync retriggered for affected customers.
13.03.25 10:30 AM - Repositories started gradually reappearing as resync progressed.
13.03.25 01:59 PM - The system is highly loaded due to repository resyncs and violations updates. Which causes delays until the RIG and Risk score is updated.
14.03.25 06:31 PM - The Incident was resolved, but monitoring continued to verify data integrity.
Root Cause Analysis:
• Incorrect usage of the state tracker in repository sync caused the system to believe repositories did not exist, triggering unnecessary resyncs.
• Continuous repository resyncs placed an excessive load on the system, leading to significant delays, syncs, RIG, risk score and violation updates.
• Affected customers experienced repository deletions, missing violations, and project corruption.