
"we are meeting to finalize the 2009 budget. We need to decide on X, Y, and Z. Nancy will email the latest TPS report, John is bringing the bagels"
"we are meeting to finalize the 2009 budget. We need to decide on X, Y, and Z. Nancy will email the latest TPS report, John is bringing the bagels"
Everybody screws up occasionally. When it happens the apology is important. There is only one right way to say I'm sorry and it should go something like this: "I'm sorry for what I did. I know it caused you pain. I promise it will never happen again." Notice that there is no "but." Anything that begins, "I'm sorry, but..." is not an apology.
Earlier this week one of the web services we use was performing horribly. Nobody from our company complained immediately, some of us even assumed the problem was on our end. Today, however we got the following note form the vendor (the name of the service has been changed.)
The last two days have been rough. HAL9000's performance was terrible. I would like to explain what happened, what we did to fix the problem, and how we are making sure that it doesn't happen again. Your time is valuable, and spending minutes waiting for every HAL9000 operation (or getting kicked out after entering data) is plain not good.
First, the slowdown was a performance issue not a security one. Our database was stuck in a logic loop. The root of the problem was the monthly snapshot that HAL9000 makes of each clients data for use in comparative reporting. The December 1st snapshot was the first one to include the new 2009 regulatory standards. We made a mistake in the computer code, so that the database made a full snapshot with EACH new standard. Needless to say, that's a lot of snapshots. The size of that request (measured in Terabytes, not megabytes or gigabytes) caused a cascade of safety features to kick-in. Our tech team has spent the past two days resetting those safety features and modifying the snapshot code. Short-term problem resolved.
To make sure this does not happen again, we have done two things: first we have upgrade the diagnostic systems that monitor our data base from industry standard to cutting edge, and second we have engaged a consulting company to analyze and optimize our database structure. While this performance issue was the first for HAL9000 in 2.5 years, we agree that even once is too much.
Thank you for your understanding, and we are sorry for the inconvenience this has caused. If we can help enter data that has accumulated over the past days, please forward them to your account manager. As always, contact me directly if you would like more gory tech details.
Sincerely,
Jon