Impact

An unknown number of users found problems related to outdated, missing, and incorrectly displayed data in various parts of the Platform, such as ToDo tabs, vulnerability search, indicators, and ToE (at UTC-5 22-08-02 17:00 to 23-11-16 22:26 | Time to recover was 8.16 days) The incident was discovered reactively (at UTC-5 23-11-08 18:38 | Time to detect was 15.2 Months) by a customer who reported through our help desk [1] that he found inconsistencies with findings' indicators.

Cause

The Platform uses advanced filters that rely on a search engine fueled by data from the database through specific software. This implementation's approach proved unreliable, causing multiple failures and periods of service disruption [2].

Solution

The engineering team opted to implement a change of approach based on AWS Lambda [3].

Conclusion

The issue arose in production due to the initial adoption of an approach that eventually proved suboptimal, failing to integrate seamlessly with our repository's continuous deployment dynamics. With the new implementation operating smoothly, the team is actively investing efforts into refining the technical decision-making process. This aims to anticipate such scenarios and effectively address similar changes in the future. INCOMPLETE_PERSPECTIVE