Monitoring: the Holy Grail

Holy Grail - Monty Python Holy Grail DVD Cover Art.

What is the ultimate goal of Monitoring? From my many years within the IT industry I have seen this evolved over time. From simply knowing when something doesn’t ping to a full synthetic traversal of an application and all of its components. At the core though it has never changed. The ultimate goal is to utilize technology to automate and reduce man hours so that the human factor is reduced in the equation. Ultimately the human factor in the equation of monitoring is removed.

This is the Holy Grail of monitoring, to completely remove a human from having to watch a screen for an issue to occur. To put it simply, we only want to know when there is an issue the needs action from a human to resolve it. For example if a server reboots in a maintenance window we do not need to wake up everyone to tell them that a critical server/application is having issues. Or the application owner wants to know when his services didn’t start correctly after that reboot. So we build additional layers of intelligence into our software. Simple logic at first, applying maintenance windows, thresholds, etc. and these worked really well to reduce noise and free up a persons time. We then began to engage in higher logic, such as “If X happens then run Z script”. We also built into the software hierarchical structures that would allow issues up-stream to be inherited to those down-stream so you could see a better picture of what was affected by an outage. These types enhancements , among others, have brought us so much closer to the Holy Grail.

The latest advancements have brought in great automation, in being able to join different systems together to remove even more human interaction with the monitoring.  For example, the monitoring system detects an issue which kicks off chain reaction of automatons between multiple different Operations systems. A ticket is opened with the Service Desk that automatically fills out and assigns to appropriate individuals.  Then a set of actions are performed in an attempt to resolve the issue. Success or failures of said actions are updated to the tickets without human intervention.  This great automation and significantly reduces the the need for human interaction and makes things move more smoothly.

For those of us who manage such systems we often have the challenge of setting up the logic to address the possible scenarios. This can be difficult if you do not have the full co-operations of key personal. I will get straight to he point. The key to achieving the Holy Grail is co-operation with those individuals and teams that would benefit most from having amazing systems that have been developed. Teamwork, if you will, is the fundamental part of achieving any goal within a group of individuals. I have had great success’s and monumental failures with implementing monitoring solutions. In both cases Teamwork was what allowed either to happen. All have to be willing to put in the effort to make the system work and if any one part does not, it will fail. This is why when I setup a new system, start a new project, or even minor upgrade, the most of my efforts is ensuring that I have full support of all parties to have the best chance for success.

More often than I would like, friction occurs and sometimes you cannot make the solution work. People have their own preferences and ideals and no matter how much customization you do, you cannot turn an apple into an orange. In those cases, I have found it optimal to provide the greatest solution you can, (best apple in the bushel) and present it to them in the manor that makes it the most appetizing. You can’t make them eat it,  but you can make it as irresistible as possible. This way if they still reject it, you know that you did your best and others will too. This will insulate you a bit from reprisals of failure.

As a child I would sit in my Great Grandmothers garden and watch the humming bird feeders. I kept seeing all kinds of insects around the feeders in addition to the birds. I asked my grandmother why “Everything” loved the humming bird feeders more than the followers. Her response stuck with me my entire life, “You will always catch more with something sweet, than with something sour.” This is, of course, a variation on the old saying ” You will catch more flies with honey than vinegar” and has rang true my entire life. I have found that in difficult situations where its a matter of preference, you will always get more acceptance from going the extra steps than by having a rule, law, or edict to accomplish the goal.

The Holy Grail of Monitoring will be amazing when it is accomplished. However, it will take a great deal of effort from all parts of an organization to make it a reality. Making a system that doesn’t require human interaction to resolve even 75% of the issues would be a great achievement, and this would be the beginning of what an organization would be able to accomplish if they worked that well together. Because no matter how much you would like to, you cannot remove the Human Factor from the equation.  The human factor is what gives purpose and HEART to any project. We can never forget that the purpose of any system is make things better for those who use the system.

Happy Holidays!


~ by Josh on December 23, 2013.

%d bloggers like this: