We continuously monitor outages, slowdowns, and other issues. Should there be any interruptions in service, we'll provide status updates here.
Now that our investigation of the incident is complete, we would like to share more detailed information on what happened.
Summary of the issue:
Between February 20th at 09:21 PM CET and February 21st at 12:00 PM contact sync with Entra ID and Workday was no longer working, and all contacts were deleted from the existing dynamic and static distribution lists.
What happened:
Due to a misconfiguration in the test environment, a bug within the release to update the Contact API was introduced on Tuesday, February 20th, at 09:21 PM CET, leading to failures in the Entra ID and Workday contact syncs.
Impact:
The bug resulted in Entra ID and Workday syncs failing, causing outdated contact data in the contact directory. This deleted synced contacts from the contact directory and distribution lists. The subsequent fix didn't reinstate static lists or re-add missing contacts. This was remedied on Tuesday, February 27th.
Future improvements:
To avoid future incidents, we added additional checks to our updates of the contact services and have made adjustments to our test environment to avoid misconfigurations in the future.
Thank you for your patience and understanding while we work to improve our services and prevent future incidents. If you have any questions or concerns, please do not hesitate to reach out to our customer support team at support@staffbase.com.