There’s an old joke that in software development management there are three objectives – to deliver the software on time, on (or under) budget, and with quality. The joke says to pick any two….
While this may be an old joke, the reality is that it does a large number of people a disservice. Generally, it implies that the Project Manager is unable to manage his or her project efficiently to get the best outcome for the solution delivery, and is unable to manage the Triple Constraint (sometimes also referred to as the Iron Triangle); specifically for a Test Manager it implies that given the constraints of time and budget, the test manager is unable to derive an effective test strategy to deliver the optimum quality in the solution. There may be PM’s and TM’s out there that fit the joke, but I’ve worked with an overwhelming number of people in both roles for whom this would simply be insulting.
The joke says that you should pick any two, but this is not even remotely factually correct where quality is concerned. In any sensible project does selecting Time and Budget mean that there is zero quality? Of course not – even poor quality does not mean zero quality. For a test manager (as for a project manager), all three points of the Triple Constraint triangle are important and influence each other, and none of the points are as binary and black-and-white as they first appear – there are various tweaks that can be made to optimise each point as best as possible for an outcome that satisfies your stakeholders.
Where there is a possibility that one of the points will fail, we raise a Risk in our log.
Once we know and understand that Quality is a word that means different things to different people and that it’s impossible to deliver a defect-free system, achieving some form of acceptable quality within the bounds of the triangle becomes an exercise in risk management and risk-based testing. And once we accept that testing is risk-based and we are aware of the risks inherent in what we are testing and what we are not testing, together with the PM and business teams we can influence and tweak Time and Budget where necessary to deliver a solution optimised for the maximum amount of risk that the business is prepared to accept.
(Note: most PM versions of the triangle put it as Time, Cost, and Scope. As the context of this blog is towards testing rather than project management I’ve used “Quality” as the third point – to me, reducing the testing scope is a reduction in the Quality of the delivered product and so can be used as synonyms for our purposes.)
Regardless of whether you’re planning the first schedule or replanning the day before execution, there are many factors that you need to take into account when determining the amount of testing that can be completed within the given timeframe, including:
- risk-based selection of scenarios and sequencing of transactions/scenarios into logical groups
- late delivery of key configuration or developments, or last minute critical defects that prevent deployment to testing
- delays in or incomplete data loads that require manual remediation in the test client
- tester replacement for less experienced testers that need longer to execute transactions
- sickness or other unavoidable unavailability, with no immediate replacement
- last minute changes in location(s) that mean longer handovers between tasks
Many of these can be mitigated by building a contingency into the original schedule to provide a buffer against delays, but sometimes the extent of the potential delay means that the entire schedule needs to be replanned at the last minute.
Referring back to the Triple Constraint triangle, we’ve got three clear points to use as starters for replanning:
So how can we play with each of these to get the test schedule planned/replanned?
It would be nice if we could sometimes alter the laws of time to allow more time for testing, but until we can we’ve got to find other ways to replan:
- Reschedule so that available developments and configuration can be started. Your original plan may have called for a sequential Procure – Store – Make – Sell – Deliver, but in order to get started on testing and not be sitting around waiting on transports you may need to split the scenarios into “Front of House” and “Back of House” for example, and start with Sell – Deliver and only move back into Procure – Store once the config is ready. Depending on your test strategy you may have some retesting to do but that’s better than delaying testing altogether. You’re not increasing the overall amount of time to test that’s available, but you’re using the time that you have wisely.
- Overlap other phases. Related to the above, consider how far your testing can extend into overlapping the next phase without overly impacting it. You might be extending your integration testing into the user acceptance testing, or extending UAT into the cutover preparation phase. Determine what is the drop-dead date for when your testing needs to be completed, taking into account any shared SAP clients, interfaces, testers, locations, etc
- Use global time wisely. If you’re utilising offshore resources in a different timezone, investigate using the time differences to your advantage by redistributing the test tasks so that they flow between zones. One zone can create prerequisite transactional data overnight ready for the second zone to complete the functional testing, for example.
- Adjust the daily working times of your teams to fit the functional flow. If you’re testing the functional integration between (say) Make – Store – Sell, then get the Manufacturing and Logistics teams in earlier and plan for the Sales and Finance teams to start and finish later.
- Be flexible on the location of your testers. If you need access to hardware (RF scanners, label printers), or are testing a highly integrated scenario, or need floor-walking support, and this has already been set up then a late change of location may not be possible. But for certain testing you may be able to consider testers working from home or moving to a more central location to save on travelling time.
- Increase your working day/week. Nobody likes to plan for extended working hours (and nobody should plan this way) but at certain times in projects it’s not uncommon to spend periods of time testing past an 8 hour day and, if necessary, on weekends. At all times you’ll need to be mindful of the needs of your testers (for example, childcare arrangements that can’t be changed at late notice), site unions, and local labour laws in each state/country.
- Request an extension to the test phase. If all else fails, you’ll need to sit down with the PM and request an extension to the time available to test. This is usually a last resort (though not one that’s left to the last minute to discuss!). By this stage you will already have discussed every option including the ones under Budget and Quality with the PM, rejected the ones that are not possible, implemented the ones that are, and if you still can’t meet your exit criteria in the time provided then you will need to take a serious look at extending the test phase. As this has knock-on impacts on later test phases, cutover, and any Go Live this is not an option to be used lightly, and generally accepted PMI practice is that you can only use this option once.
Adjusting the project budget for testing is difficult at the best of times, but there may be several options available:
- Look at additional tool support, and optimise the tools and processes that you currently have. Are your test management tools and processes slowing the testing down, and can they be either replaced, optimised, or fully supported by the tool vendors or internal experts? Can you create automated workflows or batch jobs to support your testing?
- Utilise available project members as testers or shadow support. If possible, are there functional consultants or business analysts that can rotate into a test role for a short period? At a minimum can they shadow the testers to provide immediate support for any process questions and defect support to get testing moving quicker? This moves people from a functional role to a test or test support role, so the budget impact is cross-functional but important from a “what have we allocated to Design – Build – Test” perspective.
- Procure additional testers. This may mean bringing on more external testers, or bringing in more business testers to get UAT completed on time. Either way there is a cost, although whether that cost is borne by the project or business may differ. Bear in mind that depending on their skills and how early you can source them there will be an amount of training required to get them up to speed with the solution. Also bear in mind that the additional testers need to be balanced to where the resource issues are, so you will need to be able to show on your execution plan where the requirement is and what type/function/location your tester(s) need to be.
Testers and business users aren’t too keen to compromise on quality in the systems, so the focus is to minimise the risk to testing and the system under implementation:
- Minimise test management overhead as far as possible. Consider the minimum documentation and information that you need to fulfil internal and external audit requirements, to be able to successfully handover integrated scenarios between functional areas, and to be able to report test results and progress. Be realistic about what you can cut out – it’s no good removing formal test results altogether if your testers need to spend that time updating you in person – but also mindful that ECC contains a perfectly good document flow so if the testers record for example their Sales Document number you can easily review the test results directly in the system rather than in a screenshot.
- Balance out the scripted and unscripted testing. Many of the large system integrators insist on every test being a full test script and fully documented, but this takes a large amount of time to set up and in my experience the scripts are often generic and frequently incorrect, which wastes the tester’s time as they try to figure out whether the issue is with the system or the test script. While full test scripts have their place, there is also room for testers (particularly in discrete functional areas) to run through a simple Excel checklist of tests and to record their results as a simple Pass/Fail with the document numbers. Examples might be in the warehouse for the multiple combinations of internal movements, or in sales documents for checking combinations of sales doc types, customers, materials, and pricing.
- Optimise the defect management process and get the Development and Data teams on board. Frequently, delays in testing can be caused by the testers waiting for critical defects or master data to be fixed. Ensure that critical defects that stop testing are promoted quickly to the right team, and that they prioritise them accordingly. Get their reps in to your daily standups so even without a formal defect they are aware of what areas you are testing – this works both ways, as they can then provide feedback on what issues you might encounter.
- Optimise your scope using risk-based analysis. Focus on the key areas of risk (either technical risk from the development/configuration teams, or business risk from the functional scenarios you are testing). Find the big ticket defects quickly, and focus testing on problem areas. You’re looking for broad coverage across the system quickly, followed by in-depth deep dives into specific areas of risk. Create a functional risk matrix or complete a functional risk assessment as part of your test preparation to help you to focus your scope and remove lower-risk scenarios.
- In your later test cycles, focus on time-based scenarios. If you’re running two or more cycles within a test phase, consider prioritising scenarios based on what is Day One critical, what is Week One, what is Month End, etc with the proviso that within those criteria you should also rank them on how quickly any issues in Production would need to get fixed. For example, a Day One manufacturing scenario would be critical for testing. A Month End scenario might appear initially to be less critical priority-wise, but consider that if a defect is found in Production in your Month End processes how quickly would it need to be fixed? Scenarios with legal or regulatory implications also receive a higher risk rating, regardless of time.
Hopefully this has given you a few ideas on how and where you can look to replan the test schedule to keep your risk manageable and deliver on time, on budget, with quality.