During your SAP implementation the test manager should be thinking a fair bit about regression testing. It’s reasonable to assume that there will be some changes to your system in the future, as a result of:
- rollouts to additional parts of the business
- change requests and incident fixes
- service packs and upgrades
In order to ensure the existing system is not impacted by these changes you’ll need an effective regression testing strategy. For the purposes of this blog post we’ll assume that you know how to create an effective test strategy in a general sense, and also that you are aware of the purpose of regression testing and that it is necessary in your system (in other words, that a charter for setting up a regression test project exists).
Setting up an effective regression testing strategy is not difficult, but it does take time and effort (like riding a bike, it gets easier the more times you do it!). The strategy should take into account the standard project management knowledge areas, and I’ve outlined some key questions in each of the areas below.
Note that this is not sequential – your test planning could start with identifying the test tools and automation framework, for example, before whether or not you outsource all or some of your testing. More likely though, both of these things (plus others) will need to be considered together rather than one by one.
At a high level you should identify the various strategies that you need, to accomplish your goals. This is probably the single most important thing that needs to be done – defining your general approach to regression testing, and a roadmap to achieving that vision.
In short, ask yourself what a quality regression test programme would look like in your organisation, and then create the roadmap of activities and tasks that will get you there. If you have no idea about your destination or the path that you will take to reach it then you are liable to be spending considerable amounts of time and money in creating a regression test strategy that is ineffective and wasteful.
- Are you going to run a risk-based approach to defining your scope and strategy, by selecting a pool of scenarios but only running the ones that have been impacted? Or a general “run everything once, even if it hasn’t changed” approach? Or a combination of the two, by targeting your testing towards the changes and also running a light touch over other areas?
- How will you ascertain the business impact of changes, and the level of detail that you will be informed about? For example, will the level of detail of change only rest at a transaction level (“VA01”), or down to a functional level (“pricing procedure ZA10”)? Or a mix of both?
- What type(s) of tests do you need to run? For example, you could create different regression suites to cover different types of changes – a large regression suite with full coverage for upgrades, and a smaller suite to target individual changes. Consider having a pool of regression scenarios to select from, but only running scenarios on a risk-based approach.
- Automation or manual (or a mix)? Are you going to consider automating part or all of your suite?
- Protoyping? If you have a selection of possible test tools and test teams, consider building in to your plan the time required to prototype so that you can make your final decision.
- What are your business-critical processes?
- Do you have a method and process for prioritising scenarios and determining their criticality? Day One, Week One, Month End, Other? Change Impact analysis?
- Do you have a method and process for rationalising your scenarios? Leading plants? Coverage and transactional analysis matrixes?
- Have you assessed your coverage in a global sense to ensure your critical processes are not needlessly duplicated across plants? (This can be covered at a later stage, by standardising scenarios to be run for multiple plants from data sheets and by injecting variant scenarios into the final regression suite).
- How often will you plan on running your regression suite? This may impact your final scope decisions, so for example if you are planning on weekly deployments of small changes then you would need a smaller weekly regression suite (and perhaps some “lights-out” automated scenarios), complemented by larger monthly or quarterly packs.
- Can you refine your scope so that you create greater coverage with data-driven tests?
- How long have you got to develop your regression suite? In the short term it may be messy as you develop and refine your pack, do you have a high-level timeline to show the suite creation and refinement?
- Do you have a framework in place to speed up the suite creation? Reusable tcodes, repeatable scenarios, etc
- Do you have or require buy-in from functional experts to help speed up the functional handover?
Cost will generally be driven by the parameters of the other knowledge areas, but you may find in some cases that it’s the other way around – that cost (your total budget) drives your decisions on tools, people, scope, etc.
- Do you have a framework in place for automation, and who is checking and validating this?
- Who is checking that the scenarios are correct, and match your business processes?
- Once built, who is validating that the automation matches your business processes?
- What are your procedures for checking failed steps? Is this an invitation for your testers to explore further?
Do you have a RACI to clearly show the responsibilities across teams?
Do you go for an outsourced solution, or run all testing in-house (either by an internal test team or by seconded business testers)? Or a hybrid of both?
- if you are considering outsourcing, consider that you should not be transferring the overall responsibility for testing outside of the business – either yourself as test manager, or the Director / Head of testing within the business, should still remain with overall responsibility for the test results. This means that it is up to you to ensure that the outsourced testers are suitably experienced, that they are following the test strategy, and that they are actively engaged in testing when they say they are.
- If you are running testing in-house, consider that regression testing is usually only a part-time role; any regression testing strategy will need to fit in with the wider aims and objectives of the business.
- Have you communicated the overall vision and roadmap to all stakeholders, and achieved buy-in for the mission?
- Has the plan been communicated to the business representatives, and are they aligned with the timetable, scope, risks, etc?
Like any project, creating a regression suite will present risks and issues. Create a RAID log as you progress through the project to capture risks, issues, assumptions, and decisions.
There are may test tools on the market these days, but only a few stand out as market leaders for SAP testing. Your regression test strategy may also be constrained by existing organisational assets – for example, your business may already have licences for HP ALM for your test suite and defects.
SAP tends to recommend HP ALM + SAP TAO + HP QTP, but there are other possibilities including Selenium, IBM Rational Functional Tester, Worksoft Certify, Panaya, TestComplete, and others.
Buy-in of key stakeholders is critical to any project. Ensure that you identify all stakeholders and that they are aware of the purpose and planning behind your regression project.
Worksoft Publications – scroll down to “Success Stories”