Major service outage
Incident Report for Composer Resolver Cloud
Postmortem

There have been stability issues with the service over two weeks which were all caused by multiple different problems.

First of all, Contao Manager 1.3.0 was released which contained an issue that caused too many requests to the Composer Cloud Resolver. Combined with the fact that Contao published security releases, the demand was so high that it reached around 250 requests per second (!). A number the serviced was simply not able to cope with.

The number of requests have since been greatly reduced, thanks to new versions of the Contao Manager. On top of that, we’ve added caching headers and put the service behind a caching proxy and a CDN to reduce the number of requests actually reaching the service.

However, there was another issue introduced with autoscaling which caused loads of jobs to be dropped and thus seemed like they were running endlessly. This has been resolved by disabling autoscaling.

We are now actively working to migrate to a new hosting provider.

Posted Oct 20, 2020 - 10:07 CEST

Resolved
All services are currently down, we're investigating the issue.
Posted Sep 23, 2020 - 10:03 CEST