[Email] Incorrect Azure recipient counts and exclusion rules not working properly

Incident Report for Staffbase

Postmortem

We have completed our investigation of the incident and are now ready to provide you with more detailed information about what occurred.

Summary of the issue:
Between 1st April at 04:20 PM CET and 3rd April at 12:30 PM, the synchronization from Azure Entra ID was not functioning correctly, causing the Azure synched lists to display 0 contacts.

Description of the Incident:
A misconfiguration introduced a bug within the release to update the Contact API on Thursday, March 28th, at 12:00 PM CET, resulting in synchronization failures with Entra ID. We fixed the bug and started a re-synchronization which has resolved the issue.

Impact:
Azure Active Directory lists that were supposed to sync on March 31st showed 0 contacts. Due to this, all CSV imports did not show the correct number of contacts. All CSV uploads performed prior to March 28 at 12 PM had to be re-uploaded as a result. Unfortunately, the lists uploaded between March 28 12PM and and April 1 at 5 AM CET days could not be restored.

Future Improvements:
To prevent similar incidents in the future, we have implemented additional checks in our contact services updates to prevent misconfigurations. We have also identified the need for improved measures to promptly recover data in case of emergencies and will now implement these successively.

Thank you for your patience and understanding as we strive to enhance our services and prevent future incidents. If you have any questions or concerns, please feel free to contact our customer support team at support@staffbase.com.

Posted Apr 12, 2024 - 22:51 CEST

Resolved

This incident has been resolved. All syncs are working again as intended.

If you have performed a CSV upload between March 28th 12 PM and April 1st 5am CET, please start a re-upload of those files.

We are sorry for the trouble this has caused.

Please contact us at support@staffbase.com should you experience any further issues. We’re happy to help!

Posted Apr 03, 2024 - 16:27 CEST

Monitoring

A fix has been implemented regarding Azure sync'd lists not showing the correct number of contacts. We are currently monitoring the results.

If you have any questions regarding this topic, please reach out to our support at support@staffbase.com.

Posted Apr 02, 2024 - 19:40 CEST

Update

We have found the cause of the issue with exclusion rules not working properly, and are now working to put a fix in place.

Posted Apr 02, 2024 - 16:47 CEST

Investigating

Customers are reporting that their Azure sync'd lists are not showing the correct number of contacts, and that their total Contact count is much higher than expected.

Our developers are working to identify the root cause and deployed a fix to resolve the issue. We will continue to monitor the systems until we are sure the issue is fully resolved.

We apologize for the inconvenience. If you have any questions regarding this topic, please reach out to our support at support@staffbase.com.

Posted Apr 01, 2024 - 16:49 CEST

This incident affected: Employee Email (all regions) (Web Application (US) [SLA], Web Application (Germany) [SLA]).

Welcome to Staffbase Status