Platform is not generating group reports
Incident Report for Fluid Attacks
Postmortem

Impact

At least three users experienced indefinite waiting times when generating and exporting executive and technical reports. The issue started on UTC-5 23-07-31 09:43 and was reactively discovered 23.2 hours (TTD) later by a user who reported through our help desk [1] that the reports he generated the reports through the platform never arrived in his email. The problem was resolved in 3.6 hours (TTF) resulting in a total impact of 1.1 days (TTR).

Cause

A job definition in our infrastructure code was deleted under the impression that it was not in use [2]. However, this job was crucial for the batch processing that produces the reports. Its absence prevented the reports from being generated, as they were left waiting for the job to be deployed.

Solution

The engineering team added the job definition and generated the queued reports [3].

Conclusion

An important job for generating reports was eliminated, partly because its relevance in the infrastructure was not detected, and there were no tests to identify the problem before going to production. MISSING_TEST

Posted Sep 18, 2023 - 21:52 GMT-05:00

Resolved
The problem was resolved and reports are being delivered normally.
Posted Aug 01, 2023 - 17:38 GMT-05:00
Identified
Executive, commercial and export reports are not being delivered.
Posted Aug 01, 2023 - 15:36 GMT-05:00
This incident affected: Platform.