menuA while back one of our apps had a performance problem. Over time a few of our key database tables grew too large and the query and update actions started bogging down. We never noticed it in our lower test environments because there was no other load on the database so the queries performed acceptably. But we definitely had a problem.
Unfortunately it took us way longer than it should have to learn about the error. Even though we have pretty good relationships with our customers, the performance issue just kind of built up over time so the customers kind of got used to it. Finally we started seeing some occasional timeout errors coming through the Error Logging and were receiving reports from customers that things were no longer acceptable. By this time our customers had been dealing with slow performance for weeks, eroding their satisfaction with our app and burning out some of our precious good will.
The fix ended up being a combination of some additional indexes and some query restructuring. Once we figured it out we had a solution out within a week and everyone was happy, except for us. We had introduced friction to our customers and didn't want that to happen again.
So, we introduced Performance Monitoring into JustLog.IT. Using the same RESTful API we use to collect errors, we log timing metrics for actions we care about. Typically these include:
Basically any type of action can be timed and tracked.
Normally introducing this type of logging is a catch-22 because introducing the logging slows down overall performance. But with our web API it is easy to spin this logging off into a separate thread to mask any noticeable impacts. Even better, if you use our web library the performance logging from the web pages is super lightweight, using what are known as "ping" drops to push the data out. Super light-weight.