Inconsistencies in coverage levels
Incident Report for Fluid Attacks
Postmortem

Impact

An unknown number of users experienced inconsistencies in the coverage levels tables within progress reports. The issue started on UTC-5 23-07-28 18:09 and was proactively discovered 6.3 months (TTD) later by analysts who noticed the inconsistency in the reported coverage level metrics. The problem was resolved in 2.4 days (TTF) resulting in a total impact of 6.4 months (TTR).[1][2].

Cause

Some files were being incorrectly marked as verified. The method of comparing the oldest allowed attack date caused some files to be erroneously marked as verified, even if they did not meet the condition of having all affected lines [3].

Solution

The method for comparing the date of the ToE line attack with the oldest allowed date was corrected [4].

Conclusion

A unit test was added, and a script was created to check metrics using data from the group being verified [5]. These measures aim to avoid similar issues in the future by ensuring accurate verification processes. INCOMPLETE_PERSPECTIVE < MISSING_TEST

Posted Feb 12, 2024 - 10:56 GMT-05:00

Resolved
The incident was resolved, and the reports are now free of inconsistencies.
Posted Feb 09, 2024 - 11:02 GMT-05:00
Update
The report delivery was affected due to a database schema update, and the team is currently ensuring smooth execution.
Posted Feb 08, 2024 - 08:52 GMT-05:00
Update
The logic problem led to all files being incorrectly marked as 100% verified. The issue only affects the analysts' progress reports.
Posted Feb 07, 2024 - 19:32 GMT-05:00
Update
The cause has not yet been identified.
Posted Feb 07, 2024 - 17:56 GMT-05:00
Update
Still investigating the root cause.
Posted Feb 07, 2024 - 17:00 GMT-05:00
Update
The assigned developer is investigating the root cause.
Posted Feb 07, 2024 - 15:54 GMT-05:00
Identified
Anomalies detected in analysts' progress reports regarding coverage levels.
Posted Feb 07, 2024 - 14:54 GMT-05:00
This incident affected: Platform.