Test Estimation – a complex problem

Introduction

At the start of any project I am always asked ‘How long will the testing take’. For years I used the rule of thumb of 25%; that is if development takes 100 hours then I will need 25 hours to test it to a reasonable level. Then I changed company and worked with a totally different product and the rule of thumb didn’t hold anymore. Now I had to spend ages setting up the tests, configuring machines, ages generating test results and then only a couple of hours analysing to work out if there was a defect or not. Then I changed company and product again and had to spend 6 months building up my experience so I could set my rule of thumb anew.

Now I am working as a consultant and suddenly I don’t have six months to learn the product in order to produce decent estimates. Now I need to quickly assess the situation and produce an estimate which will be used as a basis for contract offers. They have to be in the right ballpark or the entire contract becomes economically risky.

So I have spent some time wandering the web to see what other people have thought and not found a lot, which means that I have had to think for myself. The rest of this paper is a summary of this thinking so far.

Aim

There needs to be a way to produce ball park accurate estimates for testing situations where there is little in depth knowledge and experience. There needs to be a way to do this that is relatively easy to apply so that I and my colleagues have the energy to apply the method i.e. it has to be simple and fast. Finally it has to give reproducible results ; my colleague should reach the same value as I do and if I do it twice I should get the same answer both times.

Size is not important

There are four main tasks in testing: Design of the test case, creation of the test data, setting up the test and analysing the results. Each task will take time and this will depend on how difficult the task is. Difficulty is dependent on how complex the problem is.

When you first start thinking about complexity it seems natural to assume that the number of steps will contribute to the complexity, but this is not necessarily true. For instance, in the case where you are completing a questionnaire on line: If each question is independent of the others then adding questions does not add complexity only length and thanks to cut and paste it is not a large job to extend the number of steps. Similarly if the test is to generate lots of transactions and see when the receipt comes out wrong then the test steps are many but repetitive.

Instead I think there are two key aspects to consider:

a)      The number of paths through the test case

b)      The amount of test data inputs or variables

Number of paths

By paths I mean the different options and outcomes that are possible. For instance, when testing log in, you can end logged in or not. In questionnaires it is usual to answer one question which means the next five questions are then omitted (for instance in a health assessment if you answer ‘Man’ to gender, then there shouldn’t be any questions related to breast cancer).

 

 

 

 

 

 

 

 

 

 

The above diagram illustrates this concept; there are is one main path and multiple alternative paths through the test case. The more paths there are the more complex the test.

Number of variables

Basically, a large amount of test data or variables is more difficult to deal with than a small amount. The data has to be defined and then created and often version controlled alongside the corresponding test cases. In running a search query the test complexity arises from the amount of criteria you can search on and how they can be combined. With log in there are two paths, logged in or not, but there are loads of different ways to be not logged in (I am up to sixteen combinations of username and password and application status so far).

Test set up& analysis

Then there is a little complication; how difficult is it to prepare for execution of test case? In my experience it can take a whole day to set up some embedded software (in terms of configuring) and then only an hour to execute the actual test. If you have a test database that has to be imported then it is considerably less time consuming than creating all the rows of data that you need in the database or fetching a dump from production database that then needs to be neutralised to hide sensitive data.

I think that test set up is an overhead, a fixed amount of time which varies on the kind of testing that is being executed and I can’t see a good way of estimating the time required without knowledge of the testing environment.

Similarly in some tests it is the generation of masses of results that is required, and then it is the analysis of the results that is time consuming. So I think that analysis is also extremely hard to estimate without expert knowledge.

Conclusion so far

I think a method for assessing test complexity is the way forward when it comes to building up decent estimates in a relatively unknown testing situation. The next step is to develop a suitable scale for this assessment. For instance what do we mean by a simple test or a complex test. Then, the next step is to work out how to go from this assessment to an estimate.

I also think that by coming up with a value for complexity we will be able to compare different kinds of testing; system testing is a different animal from unit testing but can still have the same level of complexity. This kind of common value may be useful when trying to set priorities and resourcing.

So I will keep thinking.

About Marie Kyletoft

Jag är en konsult på Softronic med uppdrag som testledare. Jag ramlade in i testbranschen över 15år sedan, via TQM, ISO9000, processförbättring osv. Sedan dess har jag arbetat med funktionstestning och acceptanstestning, med embedded och web applikationer, i projekt med PRINCE och med SCRUM. Ganska varierad men det finns mer att lära sig därute.