Bona Iqiniso Application Board

The motivation to make this white paper is the surprising result we achieved in our organization while introducing metrics and measurements.

Usually, measuring how people work results in resistance because of the bad reputation metrics (and their consequences). In our case, the original reason to start measuring was to increase confidence in the work and quality delivered by the testing department.

This paper will focus on the following points:

• the first measurements as part of an assessment process,
• The reasons why we selected these metrics,
• The reasons to build a tool to measure and process these metrics
• the reasons to hand over responsibility to the people whose work was being measured.
• How we handled the initial resistance and
• how measurements led to the introduction of a number of “company good practices”, more motivation, and increased passion for testing.
• The positive changes at the organizational level as a result of the first successes,
• the creation of a test improvement task force giving direct support to project teams and
• the quest to define and introduce more “good practices”.

As the projects progressed, we also saw our testers becoming more passionate about measurements and starting to lead the quest for “good practices” and their metrics.

Success depended highly on the culture of the company, its learning ability, and its willingness to change. However, once the initial resistance was gone, using the metrics had more positive effects than negative effects.

This makes us conclude that numbers can lead to more “testing” passion and also that even skeptics can get a real passion for numbers.

Why metrics?

We introduced  measurements and metrics because management was setting three goals for the projects:

• deliver more quality
• improve the planning of software development activities
• improve traceability and prove that testing covered all requirements and risks.

Our first consideration, on how to reach our goals, was the fact that management liked results to be presented in balanced scorecards, key performance indicators (KPIs), or in short, one simple “score” to represent complex matters.

Our second consideration was the opinion of management about their testers, which was in fact a very positive opinion:

• they are professionals
• they have the intention to do their best
• they know what they have to do
• they are individualists but they function very well in a team
• they want quality

In fact, the management’s opinion was:

• “our testers are passionate about their job”.

However, this opinion also revealed the management’s expectations:

• Testers must be professionals.
• It is only common sense they do their best.
• They must know what to do
• They work for one company, our company
• it’s normal testers want quality

So in fact what management is saying is:

• “our testers must be passionate”.

So, on one hand, management wants a number to confirm what they think, expect, and say. On the other hand, management wants a number to point them to possible issues, even before these issues become problems. In fact, they want confirmation and evaluation. They want to know what the skill and maturity of their testers are and what they can expect.

The psychology of numbers

The danger with numbers is that one can get the strangest reactions to numbers from completely no reaction at all to furious resistance.

Most of the time I like numbers, so why don’t you like numbers?

If I would tell you the lottery numbers from five months ago, what would your sentiment be? What would be your reaction?

Well, probably nothing. Not interested, because it’s something that is already 5 months ago. Your reaction would be different if it were the numbers for next week’s lottery.

The reason why people react differently to numbers is that numbers mean different things to different people in different situations.

However, one thing is needed to have ‘any’ reaction: Passion or interest!

We can observe this very well in a classroom where you have a group of children that want to perform well in school and another group of children that don’t care at all.

Only the group that is interested in ‘school’, will react to the numbers they get for their tests. But, numbers hold a risk for this group of children. The risk is that numbers will raise resistance.

Resistance

•  35% on my test? But, I studied? I don’t understand it! The teacher must think I am stupid!! My teacher will not let me pass. In that case, why should I do the effort to study? It has no meaning! And, by the way, that 35% does not reflect the effort I put into studying. I’m just not good at doing tests. For me, this is reason enough to stop studying.

In this example, this child is passionate about his results therefore he will react on low points. This child, however, will raise resistance.

What we can conclude, not only when we look at this little example, is that numbers only result in resistance if the people are passionate about what they are doing.

But of course, we also could ask ourselves the question: “why don’t all people go into a positive loop, instead of a negative loop?”

Why resistance instead of improvement?

• 35%? The teacher knows I studied. I will study more. I need to study more than others because I’m not good at doing tests. The next test I will pass. I will be a winner.

This child is determined that he has to do better to get a better grade. He believes he can change his grades to something far better. It’s only a matter of having a different perspective. Nobody said it better than Franklin D. Roosevelt:

But how can we make people change their minds in a positive, constructive way?

It’s true. Our management wants ‘a number’ to confirm and evaluate the testing department.

And again true, if that number is not good, it could de-motivate people and raise resistance.

The right support

The same conflict of interest is also present in the following quote from Elias Canetti (Nobel prize winner of literature – 1981):

People want to do a good job. They just don’t want others to tell them they are not doing a good job. The number, which is a result of the evaluation, is considered an accusation!

Passionate people want to do better. They welcome any help to do better as long as they don’t consider this help as an accusation. That’s why the only way of support that can please both management as well as testers, is a tool.

A tool will help passionate people to do a self-evaluation, resulting in self-improvement. Let’s face it, passionate testers know their job, and they know what to do, but circumstances can make it difficult for them to perform to their maximum. Showing them a number, showing them lists to improve, and giving them the chance to self-improve helps testers to become more mature testers.

Also for our management, there were some additional benefits. A tool gives the ‘number’ they want. A tool is impersonal and uniform. Best practices can be built in and you can keep track of history. It can be used as evidence for management and it is fairly easy to build in triggers in the tool. Triggers to make the tool and hence also the improvement, a success.

Triggers? Let’s face the truth about numbers:

• plus is positive, and minus is negative.
• 100% is better than 0%.
• A ‘tick’ is better than a cross.
• Green is ‘go’, red is ‘stop’
• Just like Pavlov’s dogs, we humans are also conditioned to react in a specific way to certain signals, colors, sounds, symbols, etc….

The tool

In a period of 2 years, we developed 4 prototypes of tools in Excel and/or MS Access.

• The ‘Quality Center metrics tool’ to help/prove traceability, show coverage and analyze defect handling in Quality Center)
• The ‘trace tool’ to help/prove traceability between business requirements and test requirements
• The ‘baseline tool’ to help/prove on baselines are consistent and 100% traceable.
• The ‘cleaning tool’ to help/prove cleaning redundant ‘test preparation in a large program

On the QC metric tool, we started with 5 metrics. 5 numbers to prove our management we were able to measure the metrics they were looking for.

6 months later we had 5 groups of metrics, each telling a story. A total of 43 measurements and a total of 43 improvement lists.

5 metric groups for 5 practices are considered ‘good company practices’ by management. Not only by management but also by passionate testers:

• Traceability from test requirements to test cases to test executions.
• Traceability from defects to test executions, test cases, and test requirements.
• Coverage of test requirements by test executions.
• Coverage of test cases by test executions.
• Defect recognition and resolution.

Another 6 months later, the ‘trace tool’ extended the help/proof on traceability outside Quality Center, towards the business requirements, validation rules, business processes, etc. The latter is contained in word documents.

Another 6 months later, the ‘baseline tool’ extended the help/proof on traceability to configuration management.

This was for management the final proof of the added value of this way of working. Using the ‘baseline tool’ we were able to prove that 40% of the requirements from the past were no longer valid or had changed, while the linked tests were unchanged.

Another 6 months later the ‘cleaning tool’ was a fact.

Reasons for success

Why was this a success? Because we made sure the test managers had to include final evidence in their test summary report. In other words, we told them to run the tool(s) once, at the end of the project. We provided them with a demo and ran the tool the first time together with them, analyzing the results. We showed the improvement lists, knowing that this would interest them the most.

They were not forced to produce a weekly report. They could use the tool(s) whenever they wanted and they had a dedicated person to assist them when needed.

The result was amazing. We observed a minimum of resistance and even received hints for simplification and requests for more numbers to add to our tool.

We received a lot of positive feedback and the resistance we encountered was the kind of resistance we expected.

Things like, “my project has no test levels”, “my project has no test requirements”, or “my project’s test cases are in Excel”.

We considered this as positive resistance because it showed us gaps in the test team’s approach to testing.

It allowed us to give those teams hands-on training. This was not fundamental resistance against the measurements themselves. It was more a ‘crying out’ for help, more training, and more coaching.

Conclusion

Is this way of working usable in a generic way? To be honest, I don’t know. But, I know that it is worth a try. But you also have to be aware that using this way of improving is not for free, because you’ll need people dedicated to the cause.

What will the future bring?

• Four prototypes of tools are being used.
• Documentation is under development.
• The development of the final tool has to begin but has not yet started.
• We would like to have some partners.
• The decision is already made that it will become an open-source tool.

 It is also clear that we will be delivering four versions:

• In the cloud
• Desktop application: single user
• Web application: multi-user
• Mobile: iPhone and iPad.

For the last three years I have been using this kind of improvement with success and it is up to the future to show whether this way of improving will stay a success.