JustLog.IT is the latest evolution of our Forward Logic logging service. This service has been in place for a number of years, capturing information from our products into a centralized location and alerting us to issues happening in production environments. Here's a quick summary of how JustLog.IT evolved into the current service we provide.
To start with we needed a way to capture and report errors. We build a web service to capture the error messages and record them in a way where we could see and troubleshoot the errors. Not long after that was built we added things like near-real-time alerts when errors hit the system, telling us about errors almost the moment they happened.
In addition to the standard error information we realized enhancing the error with additional metadata such as location and user information allowed us to better troubleshoot and resolve errors.
As our software matured and was in production environments for longer periods of time we encountered the common problem of growing performance issues. Specifically for us it was database query issues for the majority of our products. As our db's hit millions of rows we started to see performance hits for a number of queries that slowed down the system and backed things up.
Like most people we were typically unaware of the issue until it was too late, especially with our software that ran on-premise for customers. Things would be going smoothly and then we'd start to get reports of slowness and some other issues. People become unhappy and we had to figure out the issue and get a resolution in place asap. It wasn't fun.
Then we evolved our logging system so we can feed in performance metrics. We started doing this for the db calls around heavier traffic parts of the systems. While most of this information was noise, we did start to look at trends around the performance of specific calls and could see items that seemed to become slower over time. We were able to work those results into our backlog so we could troubleshoot and provide resolutions before users started to complain. Way better system.
The final piece of the system (so far) has been to capture what our users do in the system. This piece made it in last because our primary offering has been a service with minimal user interaction to date. However, some of our side projects are websites and getting the logs of what their users are doing was extremely helpful. Basically we created a simple to use Google Analytics which extra hooks that allow us to tie the actions back to specific users.