Impact

At least six groups got vulnerabilities with the wrong STATE (UTC-05 23-08-24 13:15 to 23-09-06 10:22: 13.1 days -time to recover-). The incident was detected reactively (at UTC-5 23-08-24 14:21: 1.1 hours -time to detect-) by a user who received two vulnerability notifications, however when trying to inspect them within the related findings, he was unable to access them because they were erroneously listed as vulnerabilities in DRAFT status, then the user reported it to our support team [1].

Cause

The incident was caused by a scalability and synchronization problem between the real-time indicators and the storage of their values in secondary warehouses; This was caused by two reasons.

The growth of the platform exceeded the limit of processing capacity that was destined for the secondary warehouses.

• Context: Every time the Platform receives a new change in production, the new version is deployed in a new machine, and the old machine is shut down, when this happens the old machine receives an instruction that tells it that it has 30 seconds to finish any task it has in execution.
• Bug: There was a configuration that prevented the old machine from receiving the shutdown instruction, therefore it continued running new tasks which were abruptly canceled when the 30 seconds elapsed and the machine was shut down.

Solution

The infrastructure capacity for secondary data stores was increased.

A patch was applied to correct the bug in the application in charge of synchronization [2][3][4][5], thus avoiding an abrupt shutdown during each new deployment and giving the old machine time to complete the pending activities.

Conclusion

No testing was available for this part of the infrastructure, as the migration was in progress. IMPOSSIBLE_TO_TEST