Impact

An unknown number of users found the Fluid Attacks Platform unavailable (at UTC-5 23-11-17 11:25 to 23-11-17 12:23 | Time to recover was 16 minutes). The incident was discovered proactively (at UTC-5 23-11-17 12:07 | Time to detect was 44 minutes) by the staff of the product team, who encountered an error screen when logging in to Platform during their typical workflow.

Cause

While updating the operational rules for our primary servers to limit unnecessary access, a crucial permission necessary for the Platform's functioning was inadvertently removed, leading to an unavailability after logging in [1].

Solution

A rollback was performed, returning to a previous version that did not include the error restoring the typical access to the Platform [2].

Conclusion

These errors enter the production environment due to limitations in thorough testing within the development phase. The team is actively working to bridge the gap between the development and production environments, aiming to prevent such issues in the future [3]. MISSING_ALERT < IMPOSSIBLE_TO_TEST