Benchmarking Project

I've been thinking about the idea of benchmarking, in various forms, for a very long time - sad person that I am! The basic idea is very simple. Companies subscribe to a central server that provides them with the metrics that they need, their results are then fed back to the server which uses the combined results to produce statistics showing how various industries and companies at different stages of maturity manage the job of getting Service Management (in this case) to work.

This allows managers to get an objective view of exactly how well they are doing relative to their peers so that their Service Improvement Plans (SIPs or CSIPS, for continuous) set appropriate goals with properly defined objectives.

That's the theory!

When I was consulting to Samsung Semiconductor in Seoul, South Korea, a couple of years ago, I was very exercised with the problems. That was when I wrote the metrics that became the book 'Metrics for IT Service Management'. I realised then that, without a comprehensive and properly thought out set of metrics the project wouldn't even get off the ground - back then there were virtually no metrics for ITIL or other Service Management disciplines. It is true that Cobit provided a good set of audit tools to see if the processes were working - but these didn't help that much in saying exactly how to get them working.

In other fields some companies are already doing this. With, it has to be said, varying success. Many have foundered on the practical difficulties.

The difficulties are easy to list:

- Having the metrics is the first; putting together a set of metrics requires deep analysis of both the technical processes and the management and business requirements they have to satisfy. It is easy to either produce a high-level wish list for the business that doesn't work in practice or to produce a highly technical set of metrics that are useless for day to day management. I was lucky in having many years experience in providing consulting to companies on both the process management and the technical side - having a B.Sc in Maths and Physics also helped me with the more technical matters. This work has now been done - though it will, of course, require much change, over time, as various businesses use the metrics and, through the SIP process, identify weaknesses and improvements. To capture all these improvements is one job of the benchmark project. I hope that the second edition of the book will include many improvements!

- Security and confidentiality is the next; The results of metrics are part of a company's intellectual property, they give an intimate picture of exactly how well, or badly, the company is doing. No organisation would be prepared to expose this to the outside world - particularly to their close competitors! Consequently the benchmarking central site has to emply utterly reliable (and how many software projects are utterly reliable?) security measures. The consolidated data must be well enough stratified to help companies in the same field or state of maturity, but never precise enough to identify any particular company. This is a very strong requirement that can't be dodged.

- Comparability, for want of a better word, is the final major obstacle. If two companies are using the same metrics, but measureing different processes, then their results are not comparable, they're like apples and oranges. Most companies, rightly, tailor their processes to meet their business needs and to conform with existing processes. ITIL, and other Service Management disciplines, do not recommend uprooting everything the company does well and replacing it with some externally derived fixed process. This is a difficult circle to square. If companies are not prepared to follow the same process, at least for those processes that are being benchmarked, then the project won't get off the ground. Users of the benchmark system have to be persuaded that it is worth their while to use standard processes. Once they have agreed to this (a difficult task in itself) that isn't the end of the matter. The SIP requires that processes are changed to improve them once improvement activities are identified. With a central benchmarking system this SIP has, somehow, to be coordinated across different companies. This isn't just a technical difficulty. If one company, through its SIP, discovers a really valuable improvement, do they have an incentive to share this discovery with other memmbers of the benchmarking community?

- Finally, an important technical problem is that there are many different tools used to implement processes in different companies. The benchmarking project has to find a way to both standardise processes across these, measure the metrics from different sources and ensure that these different processes are comparable. This is no mean feat - particularly as Service Management tools are known for their incompatibility at the moment. The incompatibility isn't deliberately worked into the products, but there is little incentive for a vendor of tools to have them too compatible as that can lead to their product being migrated out of the solution in favour of a competitor. We do live in the real world!

So, with all these very significant problems, is it worth trying to do this?

A few years ago, I'd have said (and most people would have agreed) certainly not! Now, though, I think that the industry has matured to a level where it is feasible, or, at least, worth a shot.

Many of the problems that I mention above are not unique to benchmarking. The Holy Grail (as some have called it) of an enterprise wide configuration management database (CMDB) is eagerly sought by many companies who are serious about Service Management. The vendors, and the ITIL refresh project (that is updating the ITIL at the moment) are working hard to understand how to produce an inter-operable distributed CMDB.

There is an active project to explore this need to inter-operate that was started by BMC, CA, IBM, Fujitsu, Hewlett-Packard, all major players in this field. I'm hoping that we will have an interim report soon.

The tools needed for the job are also being produced. The acronym SOA (Service Orientated Architecture) is being bandied about by marketeers for all the major players. This is a vision of web-based interoperability based on services. It isn't a new idea, years, back in the '80s, ago something called EDI (Electronic Data Interchange) was the great hope for solving this problem. Then, much more recently, came CORBA (Common Object Request Broker Architecture) and HP produced a simplified version called E'speak. These all went some way to solving the problem but the end result was much less than ideal.

The problem has become more urgent and recent developments have been coming thick and fast, mainly from the W3C now. It is all based on XML and derivations from it. Much of the failure of earlier attempts was in getting a common language to describe interactions between companies, something known as a name space or, now, an ontology.

To define how exactly processes work, so that a generic service (it doesn't matter what language it might be working in, though Java has been a favourite) can supply the necessary support to them is the problem. OWL and OWL-S are languages that allow somebody to define an ontology that can be used for this. Some big projects have found them too slow and a language developed at Hewlett-Packard laboratories called Jini has been used in at least one large project to define and use a SOA.

This does have the feel, if you're used to this sort of thing, of a bleeding-edge technology. It's true, it is. It might be sensible to wait a few years until the dust has settled and start then. It is like buying a new PC, though. You know that, if you buy one now, you'll be disappointed in a few months time to see one that is cheaper with better specifications - but you do have a job to do today, so you have to take the plunge when your need is matched by the technology.

I think that we are at that time now. Benchmarking is a vital need. I've seen lots of interest around the world in getting this project off the ground. The Netherlands has been at the forefront of ITIL and Service Management development for many years and people there are eager to get started. I was at the itSMF annual conference in Sydney a couple of months ago, giving a presentation on Metrics and Benchmarking and the enthusiasm for it was palpable. This is something whose time has come.

Obviously I'm involved in a lot of technical and commercial detail at the moment, so I have to, as in this entry to the blog, confine myself to generalites. I hope that, if you've read this far and are interested, you'll contact me with your ideas. I'll try to keep this blog updated as things develop. I think that later posts will be shoter than this one! I hope they'll be informative though...