2021-05-27: 10:49 – 17:35 CEST
Millions of users rely on Klarna to keep their personal information safe. Security and the protection of personal information is our highest priority. We have numerous safeguards in place to protect against information leaks. Despite this, on May 27 2021, a number of our users saw some of their information exposed to other users.
The root cause of this incident is detailed below. To summarize, one of our development teams made a configuration update which included an unintentional change to how information is cached between our backend and our app. The result of this configuration error was that information belonging to different users were cached and visible to other users. We detected that something was wrong within 9 minutes, and identified the cause and disabled the app within 31 minutes.
In addition to mitigating the current incident, we have implemented a number of changes to prevent recurrence and reduce the impact of this type of issue, and continue to work on additional safeguards to protect personal information.
All times CEST
- 2021-05-27 10:49 We deployed the problematic configuration update
- 2021-05-27 10:58 We began to investigate multiple issues based on incoming user reports
- 2021-05-27 11:01 We identified the situation as extremely serious, raised it as a critical incident and escalated it to senior management
- 2021-05-27 11:15-11:20 We disabled app features known to be affected by the incident to prevent further information leaks
- 2021-05-27 11:18 We identified the configuration update as the probable root cause and reverted the update
- 2021-05-27 11:20 We completely disabled the app in all markets
- 2021-05-27 16:08 We began to bring the app back online in all markets
- 2021-05-27 17:35 The app was fully online again in all markets
On May 27, Klarna experienced a configuration error causing Klarna app users to see some of each other’s information. For more information about how users were affected see the main blog post.
As soon as we got reports of issues, we assembled an incident management team that represented all relevant parts of the Klarna development organization. They immediately started to evaluate all recent changes made to our production environment. One of the first updates identified as a potential cause was a change to our Content Delivery Network (CDN) configuration.
Soon after the app was disabled, we had confirmed the following:
- The CDN configuration update had included a line that changed the cache configuration.
- The observed erroneous behavior started right after the CDN configuration update was rolled out.
- Verified in logs that calls to our backend had been cached in an unexpected way during the time period.
- Consulting documentation and experts, it was clear that the cache change would without a doubt result in the observed erroneous behavior.
This had us conclude that the CDN configuration update was the root cause, and the actions we took during the afternoon were based on that now confirmed hypothesis.
During the continued analysis work we made the following findings:
- A Klarna team had prepared a CDN configuration update as part of a project that aimed to improve the visibility for the user of where a particular device / account was used.
- This configuration change had been written by an authorized engineer in one team, consulting but misinterpreting vendor documentation. The change was then reviewed by three other authorized engineers in three separate teams, according to normal processes. None of them spotted the mistake in the configuration update.
- Automated tests did not cover verifying the cache configuration. The configuration change was an unintended side effect and part of a larger change.
- The change had been deployed into an internal testing environment on May 17, and had been part of validation tests numerous times between May 17 and May 27. No erroneous behavior had been observed in the test environment.
- Our app is configured specifically to avoid inadvertent caching through the use of cache control directives found in the HTTP standard (no-cache, no-store, must-revalidate). However, the CDN configuration can at times override the app configuration. This would normally not lead to an information leak since additional protections including user-specific cache keys are enforced. The inadvertent cache configuration update unfortunately disabled those protections.
On Friday morning, May 28, based on the confirmed root cause, we deployed a new test specifically designed to check for this issue in our test environment. This test did provoke the same error, but intermittently and only after an extended period of time. This explains why our automated testing had not detected this error.
The engineers working on this update were experienced, consulted documentation and did follow established processes. The error was hard to detect in a testing environment due to reasons outlined above. All remediations will therefore have to address this from a process standpoint.
We have numerous safeguards in place to protect personal information. Even so, keeping these safeguards strong requires continuous improvement and optimization. We have identified areas of improvement that reduce the risk and impact of similar incidents in the future.
Additional monitoring and anomaly detection processes have been implemented. These will allow us to react to unexpected changes even faster. We have also introduced several changes to ensure that cache configurations are handled with extra care. To address this, we have added several new restrictions and warnings when making configuration changes, which will force additional review before committing similar changes in the future.
We will also implement additional checks in the app to validate responses from our backend. This will help prevent information disclosure, as well as mitigate the effect of unexpected changes in the environment.
We take the protection of our users’ personal information very seriously and sincerely regret that this incident ever occurred. We will do everything we can to learn from this experience while working tirelessly to regain our customers’ trust.