Vulnerabilities are showing up with the wrong status
Incident Report for Fluid Attacks
Postmortem

Impact

At least six groups encountered vulnerabilities with the wrong STATE. The issue started on UTC-5 23-08-24 13:15 and was reactively discovered 1.2 hours (TTD) later by a user who received two vulnerability notifications. Upon attempting to inspect them within the related findings, the user was unable to access them because they were erroneously listed as vulnerabilities in DRAFT status and reported this to our support team [1]. The problem was resolved in 12.8 days (TTF) resulting in a total impact of 12.8 days (TTR).

Cause

The incident stemmed from a scalability and synchronization issue between real-time indicators and the storage of their values in secondary warehouses. There were two contributing factors:

  1. The Platform's growth exceeded the processing capacity limit allocated for the secondary warehouses. Every time the Platform receives a new change in production, the latest version is deployed in a new Machine, and the old Machine is shut down. When this happens, the old Machine receives an instruction that tells it that it has 30 seconds to finish any task it has in execution.
  2. A bug in the configuration prevented the old Machine from receiving the shutdown instruction during deployments, causing new tasks to be abruptly canceled when the Machine was shut down.

Solution

The infrastructure capacity for secondary data stores was increased.

A patch was applied to correct the bug in the application responsible for synchronization [2][3][4][5], preventing abrupt shutdown during each new deployment and allowing the old machine time to complete pending activities.

Conclusion

No testing was available for this part of the infrastructure as the migration was in progress. IMPOSSIBLE_TO_TEST

Posted Sep 18, 2023 - 13:01 GMT-05:00

Resolved
The engineering team fixed the problem and now you can see the vulnerabilities with the correct status.
Posted Sep 06, 2023 - 10:22 GMT-05:00
Identified
Vulnerabilities with the wrong status were identified.
Posted Aug 24, 2023 - 14:21 GMT-05:00
This incident affected: Web.