Absorb Performance Issue (Unscheduled Maintenance) - Date: August/16/2021

Follow

Performance Degradation:

Absorb Support became aware of an issue on August/16/2021, at Approximately 12:05 PM MDT that was impacting a portion of our clients in the US environment (.com). Clients were experiencing performance delays while accessing and navigating the Absorb LMS.

  • As of approximately 12:35 PM MDT, service was restored to affected clients.

 

Unexpected Maintenance:

All clients in the US environment (.com) have then been impacted shortly after this by additional unexpected interruptions to access and would have encountered Maintenance pages while attempting to access the LMS environments. These secondary performance and access issues began at Approximately 12:53 PM MDT.

  • As of approximately 1:12 PM MDT, service was restored to affected clients.

 

We sincerely apologize for any inconvenience that this may have caused for your users and your organization.

  • Please follow this announcement for updates and an official Root Cause Analysis.
Published on
Have more questions? Submit a request

2 Comments

  • 0
    Avatar
    Luke Lawson

    Update: Below is an official Root Cause Analysis regarding the above announcement. Thank you for your continued patience throughout this investigation.

    What Happened:

    On August/16/2021, at Approximately 12:05 PM MDT our developers became aware of one U.S. web node that was encountering reduced response times resulting in performance degradation for a subset of clients on ".COM" environments. The high usage and reduced response times for the web node was caused by course duplication issues within the LMS. To increase performance, our developers added additional web nodes. This deployment was misconfigured and resulted in an unintended outage for all ".COM" clients, and incorrectly displayed a maintenance page when attempting to access the LMS.

    Client Impact:

    US based clients may have experienced slow response times on August/16/2021 from approximately 12:05 PM MDT to 12:53 PM MDT (services were improved at approximately 12:35) and all US clients experienced a full outage from 12:53 MDT to 1:12 PM MDT.


    Resolution:

    Our development team continued to monitor the situation and in order to regain performance, additional nodes were added.


    Mitigation:

    The root cause has been identified and the duplicated course tied to the initial performance degradation has been identified/addressed. Product enhancements focused on removing the ability to disrupt services in this manner is being scoped and merged, this update is a priority, though as the corrective action is still being scope, an ETA is not available at this time. A future update will be provided in reference to this correction once the time frame is known.

    Additional controls surrounding the deployment process of web nodes during times of high usage or other performance degradation have been reviewed and applied to standard operating procedures.

  • 0
    Avatar
    Luke Lawson

    Update - Sept/7/2021

    Mitigation:

    To address previous performance issues and prevent the further disruption of services due to course duplication issues, a temporary cap was introduced to this feature.

    Impact:

    As a result of this temporary cap, administrators across environments were unable to duplicate courses that were above these restrictions.

    Improvements:

    The cap was increased, on August 26, and is currently set to allow duplication of courses up to 7500 files and 2.1 GB within a course.

    Our Development Team will continue to monitor the situation and continue to adjust the cap to ensure no further impact of this to clients while continuing to mitigate any service disruptions which resulted from duplicating large course files.

    Once permanent improvements to these processes around duplication are complete, we aim to remove the cap entirely.
     
    Please note: That the above steps and information is subject to change as the investigation progresses.
Article is closed for comments.