MINDBODY Services on April 5, 2018
Incident Report for MINDBODY
Resolved
April 5 Outage Analysis


Dear MINDBODY Customers,

We understand how important our service is to your business, and we take our system reliability and security very seriously. As a result, our team has been working non-stop since recovering from last week’s outage to fully understand the root causes and minimize the chances of it happening again. Our root cause analysis is in progress and we can share the following:

What happened

For the past several months we have been upgrading and improving our system to enable new products and accommodate future growth. One of these upgrades, installed the evening of April 4th, caused a cascading series of events that brought down our service as load came up the next morning.

The entire MINDBODY system is designed and built with dual redundancy, so that any single system failure will not bring our service down. However, this unprecedented event overloaded both redundant components of a critical network system, preventing the redundant function from working. Our Network Operations team is on duty 24/7/365, and they responded immediately. Within minutes our network engineering team and executive leadership were alerted and also responded. Unfortunately, diagnosis and full recovery proved exceptionally difficult.

What we are doing

We are absolutely dedicated to delivering industry-leading system reliability and security to your business, and the following actions will improve both:

1. Capturing and applying lessons learned.

Tough events are powerful teachers, and our system and people are already better than they were a week ago.

2. Accelerating planned network upgrades.

We will be increasing system resiliency and capacity in the weeks and months ahead. Most of this will occur behind the scenes without you noticing. A few of these upgrades will require pre-announced system maintenance outages. We will work hard to ensure you know when they are coming, and they are as short and non-impactful as possible.

3. Developing unprecedented backup capabilities.

The truth of any cloud software is that when we cannot access it we lose the benefit immediately and without warning. This happens not just when our software provider goes down, but when our local internet provider has a hiccup, or the cable modem in the back closet locks up at 6am. These events have become rare in our modern life, but are painful when they do occur. That's why the MINDBODY Product Team is working on simple but powerful features that will allow you and your staff to accomplish critical tasks – access your schedule, look up a client, check in a class, process a credit card – even if your internet or our service is completely down. You will hear more from us on this in the months ahead.

All of these improvements are designed to enable your businesses to run more seamlessly, so that you and your staff can focus on doing what you love, helping your clients live healthier, happier lives. That is our purpose and that is why we are here.

Thank you for being MINDBODY customers. We love what you do for your communities and we appreciate your business. We will never stop improving and evolving to serve you better.

Sincerely,

Rick

Rick Stollmeyer
MINDBODY CEO and co-founder
Posted 4 months ago. Apr 05, 2018 - 12:00 PDT