“Just because a test is automatable it does not mean it should be automated” – says Elfriede Dustin.
Automation testing begins with an analysis of what is feasible to automate, taking into account the budget, resource, schedule and available expertise. Given limited resource and tight deadlines we first need to prioritize what is to be automated. The effort required can be measured with the help of Automation Index.
Automation Index is the ratio of number of test cases that are feasible to be automated against the total number of test cases.
AI = TFA / TC
AI = Automation Index
TFA = Tests feasible to be automated
TC = Total number of Test Cases
A checklist to help in identifying the tests that are feasible to automate:
- Is the test executed frequently
- Does the test cover the critical areas
- Does the test cover high-risk areas
- Is the test difficult to perform manually, such as concurrency test, performance test
- Are there time critical components that are best automated
- Are the test dependant on various data combinations – i.e. running the same test with different set of data
- Are the expected results a constant i.e. the expected result does not vary with each run
- Does the test need to be verified on multiple environments – OS, browsers
Tests that are yes for the above are good candidates for automation.
Factors that are to be considered in addition to the above are:
- Is the automation effort is very high, if so its reuse potential needs to be assessed
- Is the automation on an area that is still under design or not stable or in a state of flux