Amazon outage affecting some LearnUpon services Tuesday 28th February 2017 17:50:00

UPDATE 19:20 UTC: Amazon have identified the root cause and work is underway for a full recovery. We are standing by with a "Plan B" in case that recovery takes too long.

A significant outage at our infrastructure provider, Amazon Web Services, is affecting some LearnUpon components, though the application itself remains online.

Affected services include:

We apologize for the inconvenience this causes, and are readying alternatives in the event that the Amazon outage turns out to be an extended one.

Affected LearnUpon services have all been restored. Please contact support if you are seeing any remaining issues. All emails and processes that had been delayed as a result of the Amazon AWS outage have now been processed.

All of us at LearnUpon would like to apologize for the interruption. It started at 17:35 UTC, but became widespread by 17:55. At which point we were working on recovery options if the AWS outage was to extend past a number of hours, in accordance with our Disaster Recovery Policy. Given that LearnUpon itself was still available, we chose not to invoke those recovery options, preferring to wait for AWS to return to service, as any such steps would have meant further interruption and would have required customer action.

Email notifications were the most affected, with only a handful of notifications being processed from 17:45 to 22:05 UTC. As we use a content delivery network (CDN), which was unaffected by the Amazon outage, most content would have been available to learners. Uploading of new content, or displaying of recently updated content was affected up to 21:55 UTC.

Rest assured all data is intact, and this interruption just affected email and a small percentage of courses.

Amazon AWS continues to return to normal. Most services are back up and running; although performance might still be degraded for a while.

Affected LearnUpon course content should now play as expected, and new content uploaded (docs/video/SCORM) should now work also.

LearnUpon will work to shift the backlog of emails and processing over the next 30-45 minutes, and will continue to closely monitor AWS performance / error rate.

Again, our apologies for this interruption.

Amazon AWS services are starting to come back online now. With the scale of this outage, it will take some time to restore to normal service. Once that has been done, we will re-enable affected services, and start to clear the mail notification backlog.

Thank you for your patience, and understanding during this time. This path to resolution is quicker, and less disruptive than alternative paths to resolution (which would involve customer actions to remedy, etc.).

We'll update again when normality is closer. Right now we expect that to be within approximately an hour to 90 minutes.