Welcome to Staffbase Status

We are continuously monitor outages, slowdowns, or other issues. Should there be any interruptions in service, we'll provide status updates here.

Service disruption in the German datacenter
Incident Report for Staffbase
Postmortem

The good news first: We were able to identify and fix the cause of the incident on our media server. As we reported on our status page at 2:46 p.m. (CET), the app's performance should have stabilized by now and media files should be displayed again.

We would like to use this message to share the technical background of the outage, as well as our next steps.

This morning, at around 8:40 a.m. (CET), our cache storage for media content was partially unavailable on the German system. This storage is used to allow media content to be retrieved and uploaded. At this point, the incident was taken over with the highest priority by a developer team, so that a complete outage could be prevented. However, the availability of individual content was unfortunately slower or could no longer be guaranteed.

During the investigation, our development team then decided to remove and recreate the relevant media server instances so that a prolonged outage could be prevented. This resulted in media content and some areas of the app being unavailable for a short period of time. Unfortunately, we were unable to identify the initial cause of the outage at this time.

At 2:20 p.m. (CET), the new services finally ensured that the content was accessible again.

We will continue to monitor our systems accordingly. As a next step, we will start an internal post-mortem process to analyze the exact cause and initiate the necessary measures to avoid such incidents in the future.

I would like to apologize to you on behalf of the entire Staffbase team. It is very important to us that you, your team, and your employees can rely on a trouble-free use of the app also during these times. Therefore, we apologize that we have not always been able to meet this standard in recent days.

We are informing you about current developments and the exact technical background in real-time on our status page. As your trust means a lot to us, our Customer Care team is always at your disposal in case you have any questions or comments regarding this incident. Therefore, please do not hesitate to contact us.

Posted Dec 16, 2020 - 15:41 UTC

Resolved
The issue has been resolved. We apologize for the inconvenience.

If you have any questions regarding this topic, please reach out to our support via support@staffbase.com.
We will also update the customer tickets as soon as possible with additional information.
Posted Dec 16, 2020 - 14:21 UTC
Identified
The cause has been identified and we have implemented a solution. The services are currently recovering and the application, as well as media files, should be reachable again.

We are closely monitoring the systems and will post additional updates as soon as possible. If you have any questions regarding this topic, please reach out to our support via support@staffbase.com.
Posted Dec 16, 2020 - 13:46 UTC
Update
We are still investigating this issue.

Currently, our team has mitigated one root cause, however, we have still not been able to fully resolve the incident. Currently, the application and media files might be loading slower than usual and media uploads might not work.

We are really sorry for the inconvenience caused by this incident - we know how important the app is to you and your users at this time.

It is our highest priority to provide a resolution for this incident for you as soon as possible.
Posted Dec 16, 2020 - 12:25 UTC
Update
We are continuing to investigate the issue. We will post additional updates as soon as possible.
Posted Dec 16, 2020 - 11:08 UTC
Investigating
We are currently experiencing a service disruption in the German media center. Mediafiles (images, videos, attachments) might load slowly or not at all at the moment.

Our team is working to identify the root cause. We will post additional updates as soon as possible.
Posted Dec 16, 2020 - 09:35 UTC
This incident affected: Infrastructure (Germany) (Media API (Germany)).