Choosing the right automation approach, strategy and following industry best practices, guidelines will help us to faster time to market in SAP
project deliveries
Many organizations use SAP software in their various departments to take care of everyday business processes. The variety of configurations and
extensions possible increases the need for testing such business processes and mission-critical applications before releasing new SAP functionality
to potentially thousands of users.
As Fred Brooks, the infamous author of The Mythical Man-Month was once quoted,
“Nine women can’t make a baby in one month.” Beyond a certain point, you cannot practically shorten cycle time simply by adding more testers.
Communications overhead, training, license management and data management all come into play.
Why we have to carry out Automation Testing on SAP applications?
In this dynamic and competitive industry, immediate implementation of crucial business can help us to reduce cost and increase market share.
It’s true that test automation tools are designed to dramatically cut costs, manual labor, and shorten testing duration.
• Limited time and budget for development and QA cycle
• Huge number of test cases for execution to ensure robust testing for critical business processes
• Large number of changes in system with each release of an application
• Distributed and diverse legacy applications
• Risk associated with interfaces involved
• Manual testing requires large number of testing team, results in higher number of errors, provides less test coverage, offers inconsistent
results and is time-consuming
How Automation can address these Challenges
• Reduced cost of failure by early detection of bugs and issues during testing
• Reduced cost, by completing the QA cycle quicker
• Automated testing is consistent and repeatable
• Automation routines can be executed for data setups like creation of master data & variants
• Reporting and critical defect detection inputs for impact analysis
• Automated tests can extract variable data from one application and pass it to a test to run in another module, ensuring different
application modules and interfaces work well together
• Once created and benchmarked, the automation suite needs minimum manpower to execute tests during each cycle, eliminating the
need for experts in all SAP modules
• Automation is the way to cost-effectively and consistently achieve adequate test coverage. It provides proof of compliance to law with
strict quality regulations and audit trail standards
• Automation ensures reliability of mission-critical business processes like financial month-end closings, bank reconciliation, production
planning, sales order entry, etc
• As new versions of R/3 are implemented and “hot-packs” applied, automated tests can be re-executed for regression testing for critical
business processes to be performed without error
• Test suite can be used for regression testing while legacy system software, networks, server hardware, and communication components
are upgraded to prevent a negative impact
• Automation ensures uniformity in the testing process each time a test is executed
• Scripts can be executed at night or on weekend’s unattended – huge value in time crunch
• Test scripts can run independently with minimal human intervention in multiple sessions
• Given the challenges, it is clear that automated regression testing is not a luxury but a necessity. Having identified these challenges and
decided that automation is the way to go, it becomes imperative to develop the correct roadmap. The first step in this path is choosing
the right test tool and automation framework.
Know how to automate SAP tests
Testing SAP releases and upgrades is a challenging, time consuming and complex activity. Many organizations would like to automate their SAP
testing but often don’t know the best way to approach this.
Usually this results in the effort required to maintain automated Test Scripts being as high as that required to maintain SAP itself which is why over
70% of test automation initiatives fail to meet their anticipated Return on Investment. There are a number of reasons for this: Lack of experienced
automation testers, custom automation frameworks that do not create the necessary efficiencies, incorrect choice of test tool, and management
overhead are a few but there are many more. Our solution overcomes these challenges in an innovative and unique way.
Approach:
• Is proven to be a faster and more efficient method to automate tests
• is based on a world leading platform
• increases the coverage and speed of regression testing your existing functionality
• reduces the likelihood of mistakes being released into your production system
• eliminates the need to waste time and money creating a custom automation framework
• provides consistent, low level tests step documentation for the automated tests
• Integrates with ALM or Test Management tools
Providing accurate effort estimation:
Well defined SAP Automation/manual estimation templates, up on entering just few key values, estimates are automatically generated as they are formula driven.
Choosing the right framework:
Key success of automation is choosing the right framework for the tool to get maximum benefits with less time.
Component/Function/Method Driven: built up with various reusable functions and called in test scripts on need basis
Pros: Quick & easy to implement, popular, supported by HP UFT, SAP TAO, SAP CBTA, Worksoft Certify, and other tools.
Cons: Not as robust as Keyword Driven framework and coding expert is required
Keyword Driven: requires pre development of keywords and selecting keywords to make test script ready
Pros: Robust framework, can be deployed independent of test tool and application under test
Cons: Detailed steps for each test condition need to be documented, more initial time required
Hybrid Driven: most frameworks evolve into the hybrid framework – a combination of the above two frameworks which brings together their
strengths and mitigates their weaknesses
Component/Function/Method driven framework is best for SAP applications and it is quick and easy to implement with maximum ROI with short
span of time
Identifying the best Test management tool:
Test management tool should be open to integrate with internal and external automation tools to execute entire project management, it may be
adding Cycles, releases, requirements, manual and automation test scripts, results and reports, defects and dashboards..etc.
Ideally HP ALM (formally known as HP Quality Center) best for test management tool
Identifying right business scenarios for regression automation testing
Analyzing the system along with SMEs and stack holders and finding out key business scenarios where it is required to execute such scenarios
repetitively on each release is an important activity before kicking off automation testing.
• Is automation feasible for the selected scenarios/ test cases?
• Test cases are required to execute repetitively on each release/cycle multiple times
• No manual intervention during script execution
• Able to run script independently or in batch
• Always ROI focused
• Identify areas of ERP that have been customized as customizations can have an adverse effect on functionality and performance that
must be tested with every build
• Concentrate testing on interfaces with communication between SAP & legacy system
• Scenarios/transactions which need to be tested with different data sets/values
• High path frequency: Scenarios/process paths used with high degree of frequency when the software is running in full production.
Examples include creating customer records, invoicing and other high volume activities where software failures can occur frequently.
Performing Feasibility analysis (FA) and Proof of Concept (PoC):
Carrying out FA and PoC is really required for any automation testing project. It will definitely provide immediate feedback on our automation
testing capabilities and success ratio of our automation regression pack which we supposed to automate
E2E Test Management with Solution Manager:
With SAP Solution manager we can successfully carry out End to End test management as show in below picture. This includes various automation
tools called CBTA, HP UFT/QTP, Worksoft Certify, SAP TAO..etc
Easy to identify changes in SAP system for automation:
Transporting mechanism happens in SAP system in vary controlled way, each and every change is linked to executable object and added to
transport, so that at the end of transport, we come to know changes/additions happens in new SAP application
To control the transport and objects we have SAP BPCA (Business Process Change Analyzer) in it. So it is recommended to perform transports with
BPCA to track the changes in system. BPCA has an option to find out which test scripts got impacted because of certain transport, and this will give
us direct solution where to fix the script
Benefits with automation approach:
• Good amount of returns on our investments
• Saving substantial amounts of staff time previously spent on running regression tests
• Improving the quality of services by catching data-related defects before production
• Speeding up testing and application deployment cycles to create greater business agility
• Fostering collaboration between IT and business experts to streamline technology projects
• Double/Tripling the percentage of business processes automated
• Operational efficiency
• Reduced risk of application failure
• Improved customer experience
• Foster time to market
Key Considerations for Developing Automation Scripts Architecture for SAP
• To automate all critical business end-to-end scenarios, each scenario is split into test conditions, and each test condition broken down
into unique SAP T-Codes that must be executed
• Each unique SAP transaction is scripted as a reusable action under the Common Reusable library.
• Each reusable function/action is called in different scenarios. All the required parameters to execute the reusable function/action are
passed through the driver script
• Error handling, common functions also built into reusable common library; included in each script
• Using test management tool, test sets are created for each business scenario; test conditions are added to test set to complete testing &
validation of one end-to-end business process
• All scenarios required to be tested in each regression cycle are set for execution using individual test sets driven by test management tool
• The structure of having of one test script per test condition provides great modularity and ease of maintenance and debugging. If a test
condition fails, it is easy to detect which test condition and corresponding test script in the complete business scenario caused the issue
and broke the execution
• Separate the test data from the scripts. Maintain all the required Master & Transactional data in separate files. These files input the
required data to the scripts during execution
• Estimation & planning of effort required to completely build the regression test suite becomes easy. Calculate the effort required to build
script for each unique T-Code, common reusable library functions, error handling functions, developing of calling scripts and integration
of individual test conditions into scenarios
• Proper coding & commenting guidelines should be documented; all team members must follow the same
• Peer review process to ensure all scripts are well reviewed
• Ensure all the required test data (Master and Transactional data) are current and valid.
• Perform dry run to integrate and run scripts to test a complete business transaction
• Proper documentation for each test script
Preparing Test Suites with batch of automation test scripts
It is always good practice to create batches of test scripts as below, so that you will quickly get to know the feedback of the system quality
• Sanity pack
• Integration Pack
• Regression Pack
• CR Specific Pack
Generating Test Results:
Generating user friendly test results plays an important role here, as test results files can be used as evidences for future purpose.
Script Health Checkup:
We have to consider Automation testing as typical application development process; otherwise we may end up with failure at the end. As part of
this, making sure scripts are working fine on regular intervals is essential’
• Scripts evaluated and reviewed against checklist of following parameters periodically to ensure regression suite is up to date and ready
during regression testing of the SAP release:
• Apart from maintenance, scripts are executed periodically in batches overnight and test results analyzed. In case of failures, scripts are
updated to correct errors.
• Test data for each test script valid and current
• Test environments (SAP, other systems, databases & interfaces) up and running
• All required changes suggested incorporated and script up to date
• Test environment. setup, connectivity, tool license, configuration and tools available
• Test set and test scripts pulled in Test Director before execution starts
• Test cases are up to date and scripts reflect current steps as per test case
• Required results & defects reporting templates in place
• Access to SAP system & transactions and other systems/databases available
• ‘All clear’ in test results report and no issues identified with scripts during dry run
