Service Temporarily Unavailable Errors

What Happened:

On Wednesday morning all our Online Community sites started returning 504 Gateway Timeout and 503 service unavailable errors. This also affected Informz and Real Magnet admin portals.

Root Cause:

A backend in the Inside Proxies configuration was not responding, making the proxy config invalid. This caused the proxy servers to error on startup. They continued to cycle until no inside proxy servers were available.

Details:

We determined that some clean-up work in DNS removed an A record for the backend because it was pointing to a non-existent private IP. Without this A record present, the proxy config considered this an invalid backend. This invalid backend caused startup errors to occur in all our inside proxy servers resulting in 504 Gateway Timeout and 503 service unavailable errors.

Corrective Actions:

1. The invalid backend entry has been removed from the proxy config.
2. A ticket will be created to review all our proxy configurations to ensure all backends are valid and have rules. If not, they will be safely removed, along with the corresponding A record in DNS