Tuesday Double Whammy

March 30, 2010

Sorry sorry sorry

Yesterday an unannounced DNS change apparently made our mail server go incognito to the rest of the world. The consequences of this came sneaking over night as the changes propagated through the DNS network. Whammy.

On top of this our upstream internet provider late last night PST (early morning CET) experienced a failure that prevented our servers from reaching external destinations. Web access was not affected but email, widget, targets, basically everything that relied on communication from our servers to the outside world were. Double whammy.

It took too long time to realize that we had two separate issues at hand. We kept focusing on the former as root cause for the latter. And it took unacceptably long to determine that we had a network outage.

During the interruption our customers have experienced that no tickets were created from emails. And no email notifications were sent out. It also affected widgets, targets and other integrations. Some end-users may have received a soft or hard bounce when attempting to email your Zendesk help desk email address, but most emails have safely arrived at Zendesk’s servers and the backlog has been processed.

We try to do our best to communicate about these things when they are going on using the @ZendeskOps Twitter account. But this is also a process we can improve. And we will. We can not apologize enough for this mishap. We’ve learned an important lesson and will do our best to ensure that no 3rd parties can take us down like this again.

Thank you for your continued support and patience.