The following documents an intermittent disruption to the Master Tour cloud that occurred on Wednesday, January 26th, 2022. The following times are in Central Standard Time.

Impact

From 11:48 am to 12:51 pm, users of the Master Tour cloud experienced intermittent connectivity issues interacting with the web portal, Master Tour desktop syncing, and Master Tour mobile. At this time, we have no reason to believe that any data loss occurred as a result of the disruption.

Wednesday, January 26th, 2022

• 11:48 am - Two of Master Tour’s message broker servers reached maximum memory capacity, causing API HTTP requests to not be fulfilled efficiently, thus causing an increase in response time.
• 11:52 am - Eventric’s automated notification system alerts technical staff to a potential issue.
• 11:56 am - Eventric staff confirms degraded performance on the platform, updates status.eventric.com accordingly and the team begins troubleshooting.
• 12:06 pm - Eventric staff increases server pool size to decrease average response time and increase successful HTTP transactions.
• 12:24 pm - Message broker servers with maximum memory identified and are begun to be replaced with healthy instances.
• 12:51 pm - Eventric staff confirms system integrity restored.

Root Cause

The root cause of the disruption of service was due to two servers of the Master Tour server pool reaching maximum memory capacity and, as a result, denying connections. The servers in question had been operating nominally for 1097 days and 20 hours and had not previously displayed any reason for concern.