Survey Response Processing Outage
Incident Report for Hope Trust
Postmortem

Our survey integration operates on realtime processing of survey information. As survey responses are submitted, a data representation of the survey data is fetched and stored for future generation.

Within the last several weeks a fatal bottleneck was introduced into our survey partner’s software. This bottleneck would not allow us to fetch our data representations, which in turn limited our ability to store data for future generation.

A waterfall effect of issues was seen in our PDF generation when attempting to generate full documents. Sections of survey output were missing due to data not being fetched after responses were submitted.

Our survey partner has since solved this bug, an issue with escaping parameters in the JSON representation of a survey was the root cause.

Posted Jul 21, 2021 - 00:10 UTC

Resolved
We are seeing resolution of this issue across all environments. Survey responses are now processing in real time, PDF generation has recovered, our survey partner is reporting full functionality. We will continue to monitor this issue.
Posted Jul 21, 2021 - 00:06 UTC
Update
We are continuing to monitor for resolution.
Posted Jul 16, 2021 - 17:37 UTC
Monitoring
Our survey partner has prioritized resolution of this issue. We will continue to monitor for resolution.
Posted Jul 15, 2021 - 19:05 UTC
Update
We are continuing to work to fix this issue.
Posted Jul 14, 2021 - 23:51 UTC
Identified
We have identified an issue across our survey processing. Survey data is being saved without issue, the processing of surveys once completed is being effected, the output of generated survey data in PDF form is being effected.
Posted Jul 14, 2021 - 19:33 UTC
This incident affected: Surveys, PDF Generation and Alchemer Managed (Alchemer Response Processing - US).