Impact

An unknown number of users experienced inconsistencies in finding indicators. The issue started on UTC-5 24-03-19 17:52 and was reactively discovered 1.7 days (TTD) later by a customer who reported through our help desk [1] that the indicators appeared with incorrect states. The problem was resolved in 3.6 hours (TTF) resulting in a total impact of 1.8 days (TTR) [2].

Cause

A new indicator was added that was producing unexpected values (a string instead of an integer) for some vulnerabilities and was not being converted correctly [3].

Solution

The team added type conversion and implemented a more comprehensive exception handler to address the issue [4].

Conclusion

The incident underscored the difficulty in testing and securing the Streams service compared to other products, exacerbated by peer review failure and the absence of tests. Implementing a global exception handling mechanism will enhance system stability, focusing on modular adjustments to prevent broader system collapse. UNHANDLED_EXCEPTION < INCOMPLETE_PERSPECTIVE < MISSING_TEST