Higher Logic Online Community (OC) site-specific theming CSS was unavailable for approximately 3 hours; this impacted those Online Community customers which used this feature.
Timeline - all times in EST on January 23
10:22 AM: Weekly OC code deployment for production tenants starts
01:04 PM: Customer Experience/Support teams relay reports of error responses to page requests for generated CSS to technical staff and management
01:10 PM: Decision made to roll back the code release
01:12 PM: Roll back begins
01:39 PM: Roll back completed
01:55 PM: Validation of rollback completed; error responses have stopped.
Root Cause
A broad code update was deployed which did not properly address the delivery of site-specific theming CSS.
Details
Site-specific theming CSS is a feature which can be enabled on a per customer basis; not all customers make use of this feature. This feature was deployed to a subset of customers on Monday, January 22nd and no reports were received of problems. This was likely due to none of these customers having site-specific theming CSS enabled.
A contributing factor to this issue was that the method for delivering site-specific theming CSS in the development environment could be adjusted to use a method inconsistent with the production environment allowing the defect to evade detection with QA testing.
Corrective Actions
The code which caused the issue was rolled back; resolving the issue.
New code was added to the impacted service to properly handle the code change and successfully deployed on January 24.
Implement application startup and health logging and create health check endpoints for the impacted service to detect and log issues during startup.
Change how the service is used within the development environment to allow identification of issues prior to production deployment.