Problem at listing groups for some organizations
Incident Report for Fluid Attacks
Postmortem

Impact

At least three organizations could not list the groups in the groups' section. The issue started on UTC-5 23-09-15 13:02 and was reactively discovered 2.8 days (TTD) later by a user who reported through our help desk [1] that when entering the list of his organizations, it did not load. The problem was resolved in 3.6 hours (TTF) resulting in a total impact of 2.9 days (TTR).

Cause

The problem was generated by a merge request [2] made to display real-time indicators, which operates by making individual requests to the database, degrading performance and causing 504: Timeout error errors.

Solution

The merge request that triggered the issue was reverted [3], and the groups were verified to load correctly.

Conclusion

As it was a performance issue, it could not be tested before occurring in production with all associated data. IMPOSSIBLE_TO_TEST

Posted Sep 18, 2023 - 14:37 GMT-05:00

Resolved
The problem was solved and the groups are now listed typically.
Posted Sep 18, 2023 - 18:32 GMT-05:00
Identified
Cases have been identified in which the groups of certain organizations are not listed.
Posted Sep 18, 2023 - 10:23 GMT-05:00
This incident affected: Web.