As SAP ERP is huge in nature, every software development group tests its products, as the delivered product contains defects. Test Engineers strive to catch them before release of the product with the best manual testing processes but they end up in vain. Automated Testing is the best way to test the product with increase in efficiency. Automated tests are created and can be easily repeated to perform the tasks impossible with manual testing. Automated testing is considered critical for big software organizations or too expensive for small companies to implement. Automated software testing improves the accuracy, test coverage within short interval of time at low cost. Test automation has specific advantages for improving the long-term efficiency of a software team’s testing processes at low cost and time.
Many organizations use SAP software in their various departments to take care of everyday business processes. The variety of configurations and extensions possible increases the need for testing such business processes and mission-critical applications before releasing new SAP functionality to potentially thousands of users.
Why we have to carry out Automation Testing on SAP applications?
In this dynamic and competitive industry, immediate implementation of crucial business can help us to reduce cost and increase market share.
It’s true that test automation tools are designed to dramatically cut costs, manual labor, and shorten testing duration.
Challenges faces by Organizations in Typical SAP Releases
• Limited time and budget for development and QA cycle
• Huge number of test cases for execution to ensure robust testing for critical business processes
• Large number of changes in system with each release of an application
• Distributed and diverse legacy applications
• Risk associated with interfaces involved
• Manual testing requires large number of testing team, results in higher number of errors, provides less test coverage, offers inconsistent results and is time-consuming
An organization dealing with the above challenges can benefit greatly from automated regression testing if it is implemented properly using the right strategy, processes, guidelines and approach.
How Automation can address these Challenges
• Reduced cost of failure by early detection of bugs and issues during testing
• Reduced cost, by completing the QA cycle quicker
• Automated testing is consistent and repeatable
• Automation routines can be executed for data setups like creation of master data & variants
• Reporting and critical defect detection inputs for impact analysis
• Automated tests can extract variable data from one application and pass it to a test to run in another module, ensuring different application modules and interfaces work well together
• Once created and benchmarked, the automation suite needs minimum manpower to execute tests during each cycle, eliminating the need for experts in all SAP modules
• Automation is the way to cost-effectively and consistently achieve adequate test coverage. It provides proof of compliance to law with strict quality regulations and audit trail standards
• Automation ensures reliability of mission-critical business processes like financial month-end closings, bank reconciliation, production planning, sales order entry, etc
• As new versions of R/3 are implemented and “hot-packs” applied, automated tests can be re-executed for regression testing for critical business processes to be performed without error
• Test suite can be used for regression testing while legacy system software, networks, server hardware, and communication components are upgraded to prevent a negative impact
• Automation ensures uniformity in the testing process each time a test is executed
• Scripts can be executed at night or on weekend’s unattended – huge value in time crunch
• Test scripts can run independently with minimal human intervention in multiple sessions
• Given the challenges, it is clear that automated regression testing is not a luxury but a necessity. Having identified these challenges and decided that automation is the way to go, it becomes imperative to develop the correct roadmap. The first step in this path is choosing the right test tool and automation framework.
Know how to automate SAP tests
Testing SAP releases and upgrades is a challenging, time consuming and complex activity. Many organizations would like to automate their SAP testing but often don’t know the best way to approach this.
Usually this results in the effort required to maintain automated Test Scripts being as high as that required to maintain SAP itself which is why over 70% of test automation initiatives fail to meet their anticipated Return on Investment. There are a number of reasons for this: Lack of experienced automation testers, custom automation frameworks that do not create the necessary efficiencies, incorrect choice of test tool, and management overhead are a few but there are many more. Our solution overcomes these challenges in an innovative and unique way.
Approach:
• Is proven to be a faster and more efficient method to automate tests
• is based on a world leading platform
• increases the coverage and speed of regression testing your existing functionality
• reduces the likelihood of mistakes being released into your production system
• eliminates the need to waste time and money creating a custom automation framework
• provides consistent, low level tests step documentation for the automated tests
• Integrates with ALM or Test Management tools
Choosing the right framework:
Key success of automation is choosing the right framework for the tool to get maximum benefits with less time.
Component/Function/Method Driven: built up with various reusable functions and called in test scripts on need basis
Pros: Quick & easy to implement, popular, supported by HP UFT, SAP TAO, SAP CBTA, Worksoft Certify, and other tools.
Cons: Not as robust as Keyword Driven framework and coding expert is required
Keyword Driven: requires pre development of keywords and selecting keywords to make test script ready
Pros: Robust framework, can be deployed independent of test tool and application under test
Cons: Detailed steps for each test condition need to be documented, more initial time required
Hybrid Driven: most frameworks evolve into the hybrid framework – a combination of the above two frameworks which brings together their strengths and mitigates their weaknesses
Component/Function/Method driven framework is best for SAP applications and it is quick and easy to implement with maximum ROI with short span of time
Frameworks:
Design robust automation framework which uses industry available best framework features and made Hybrid automation framework ready.
Identifying the best Test management tool:
Test management tool should be open to integrate with internal and external automation tools to execute entire project management, it may be adding Cycles, releases, requirements, manual and automation test scripts, results and reports, defects and dashboards..etc.
Ideally HP ALM (formally known as HP Quality Center) best for test management tool
Identifying right business scenarios for regression automation testing
Analyzing the system along with SMEs and stack holders and finding out key business scenarios where it is required to execute such scenarios repetitively on each release is an important activity before kicking off automation testing.
• Is automation feasible for the selected scenarios/ test cases?
• Test cases are required to execute repetitively on each release/cycle multiple times
• No manual intervention during script execution
• Able to run script independently or in batch
• Always ROI focused
• Identify areas of ERP that have been customized as customizations can have an adverse effect on functionality and performance that must be tested with every build
• Concentrate testing on interfaces with communication between SAP & legacy system
• Scenarios/transactions which need to be tested with different data sets/values
• High path frequency: Scenarios/process paths used with high degree of frequency when the software is running in full production. Examples include creating customer records, invoicing and other high volume activities where software failures can occur frequently.
Performing Feasibility analysis (FA) and Proof of Concept (PoC):
Carrying out FA and PoC is really required for any automation testing project. It will definitely provide immediate feedback on our automation testing capabilities and success ratio of our automation regression pack which we supposed to automate
E2E Test Management with Solution Manager:
With SAP Solution manager we can successfully carry out End to End test management as show in below picture. This includes various automation tools called CBTA, HP UFT/QTP, Worksoft Certify, SAP TAO..etc
If we are committed to use CBTA, SAP TAO then it is recommended to use Solution manager, as it doesn’t add additional cost to us
Similar kind of test management can be performed in HP ALM too, so based on project nature we can pick up test management tool.
Easy to identify changes in SAP system for automation:
Transporting mechanism happens in SAP system in vary controlled way, each and every change is linked to executable object and added to transport, so that at the end of transport, we come to know changes/additions happens in new SAP application
To control the transport and objects we have SAP BPCA (Business Process Change Analyzer) in it. So it is recommended to perform transports with BPCA to track the changes in system. BPCA has an option to find out which test scripts got impacted because of certain transport, and this will give us direct solution where to fix the script
Benefits with automation approach:
• Good amount of returns on our investments
• Saving substantial amounts of staff time previously spent on running regression tests
• Improving the quality of services by catching data-related defects before production
• Speeding up testing and application deployment cycles to create greater business agility
• Fostering collaboration between IT and business experts to streamline technology projects
• Double/Tripling the percentage of business processes automated
• Operational efficiency
• Reduced risk of application failure
• Improved customer experience
• Foster time to market
Key Considerations for Developing Automation Scripts Architecture for SAP
• To automate all critical business end-to-end scenarios, each scenario is split into test conditions, and each test condition broken down into unique SAP T-Codes that must be executed
• Each unique SAP transaction is scripted as a reusable action under the Common Reusable library.
• Each reusable function/action is called in different scenarios. All the required parameters to execute the reusable function/action are passed through the driver script
• Error handling, common functions also built into reusable common library; included in each script
• Using test management tool, test sets are created for each business scenario; test conditions are added to test set to complete testing & validation of one end-to-end business process
• All scenarios required to be tested in each regression cycle are set for execution using individual test sets driven by test management tool
• The structure of having of one test script per test condition provides great modularity and ease of maintenance and debugging. If a test condition fails, it is easy to detect which test condition and corresponding test script in the complete business scenario caused the issue and broke the execution
• Separate the test data from the scripts. Maintain all the required Master & Transactional data in separate files. These files input the required data to the scripts during execution
• Estimation & planning of effort required to completely build the regression test suite becomes easy. Calculate the effort required to build script for each unique T-Code, common reusable library functions, error handling functions, developing of calling scripts and integration of individual test conditions into scenarios
• Proper coding & commenting guidelines should be documented; all team members must follow the same
• Peer review process to ensure all scripts are well reviewed
• Ensure all the required test data (Master and Transactional data) are current and valid.
• Perform dry run to integrate and run scripts to test a complete business transaction
• Proper documentation for each test script
Preparing Test Suites with batch of automation test scripts
It is always good practice to create batches of test scripts as below, so that you will quickly get to know the feedback of the system quality
• Sanity pack
• Integration Pack
• Regression Pack
• CR Specific Pack
Generating Test Results:
Generating user friendly test results plays an important role here, as test results files can be used as evidences for future purpose.
Summary Results as part of A2 framework with QTP Detailed Results Generated by TAO and CBTA
Script Health Checkup:
We have to consider Automation testing as typical application development process; otherwise we may end up with failure at the end. As part of this, making sure scripts are working fine on regular intervals is essential’
• Scripts evaluated and reviewed against checklist of following parameters periodically to ensure regression suite is up to date and ready during regression testing of the SAP release:
• Apart from maintenance, scripts are executed periodically in batches overnight and test results analyzed. In case of failures, scripts are updated to correct errors.
• Test data for each test script valid and current
• Test environments (SAP, other systems, databases & interfaces) up and running
• All required changes suggested incorporated and script up to date
• Test environment. setup, connectivity, tool license, configuration and tools available
• Test set and test scripts pulled in Test Director before execution starts
• Test cases are up to date and scripts reflect current steps as per test case
• Required results & defects reporting templates in place
• Access to SAP system & transactions and other systems/databases available
• ‘All clear’ in test results report and no issues identified with scripts during dry run
Many organizations use SAP software in their various departments to take care of everyday business processes. The variety of configurations and extensions possible increases the need for testing such business processes and mission-critical applications before releasing new SAP functionality to potentially thousands of users.
Why we have to carry out Automation Testing on SAP applications?
In this dynamic and competitive industry, immediate implementation of crucial business can help us to reduce cost and increase market share.
It’s true that test automation tools are designed to dramatically cut costs, manual labor, and shorten testing duration.
Challenges faces by Organizations in Typical SAP Releases
• Limited time and budget for development and QA cycle
• Huge number of test cases for execution to ensure robust testing for critical business processes
• Large number of changes in system with each release of an application
• Distributed and diverse legacy applications
• Risk associated with interfaces involved
• Manual testing requires large number of testing team, results in higher number of errors, provides less test coverage, offers inconsistent results and is time-consuming
An organization dealing with the above challenges can benefit greatly from automated regression testing if it is implemented properly using the right strategy, processes, guidelines and approach.
How Automation can address these Challenges
• Reduced cost of failure by early detection of bugs and issues during testing
• Reduced cost, by completing the QA cycle quicker
• Automated testing is consistent and repeatable
• Automation routines can be executed for data setups like creation of master data & variants
• Reporting and critical defect detection inputs for impact analysis
• Automated tests can extract variable data from one application and pass it to a test to run in another module, ensuring different application modules and interfaces work well together
• Once created and benchmarked, the automation suite needs minimum manpower to execute tests during each cycle, eliminating the need for experts in all SAP modules
• Automation is the way to cost-effectively and consistently achieve adequate test coverage. It provides proof of compliance to law with strict quality regulations and audit trail standards
• Automation ensures reliability of mission-critical business processes like financial month-end closings, bank reconciliation, production planning, sales order entry, etc
• As new versions of R/3 are implemented and “hot-packs” applied, automated tests can be re-executed for regression testing for critical business processes to be performed without error
• Test suite can be used for regression testing while legacy system software, networks, server hardware, and communication components are upgraded to prevent a negative impact
• Automation ensures uniformity in the testing process each time a test is executed
• Scripts can be executed at night or on weekend’s unattended – huge value in time crunch
• Test scripts can run independently with minimal human intervention in multiple sessions
• Given the challenges, it is clear that automated regression testing is not a luxury but a necessity. Having identified these challenges and decided that automation is the way to go, it becomes imperative to develop the correct roadmap. The first step in this path is choosing the right test tool and automation framework.
Know how to automate SAP tests
Testing SAP releases and upgrades is a challenging, time consuming and complex activity. Many organizations would like to automate their SAP testing but often don’t know the best way to approach this.
Usually this results in the effort required to maintain automated Test Scripts being as high as that required to maintain SAP itself which is why over 70% of test automation initiatives fail to meet their anticipated Return on Investment. There are a number of reasons for this: Lack of experienced automation testers, custom automation frameworks that do not create the necessary efficiencies, incorrect choice of test tool, and management overhead are a few but there are many more. Our solution overcomes these challenges in an innovative and unique way.
Approach:
• Is proven to be a faster and more efficient method to automate tests
• is based on a world leading platform
• increases the coverage and speed of regression testing your existing functionality
• reduces the likelihood of mistakes being released into your production system
• eliminates the need to waste time and money creating a custom automation framework
• provides consistent, low level tests step documentation for the automated tests
• Integrates with ALM or Test Management tools
Choosing the right framework:
Key success of automation is choosing the right framework for the tool to get maximum benefits with less time.
Component/Function/Method Driven: built up with various reusable functions and called in test scripts on need basis
Pros: Quick & easy to implement, popular, supported by HP UFT, SAP TAO, SAP CBTA, Worksoft Certify, and other tools.
Cons: Not as robust as Keyword Driven framework and coding expert is required
Keyword Driven: requires pre development of keywords and selecting keywords to make test script ready
Pros: Robust framework, can be deployed independent of test tool and application under test
Cons: Detailed steps for each test condition need to be documented, more initial time required
Hybrid Driven: most frameworks evolve into the hybrid framework – a combination of the above two frameworks which brings together their strengths and mitigates their weaknesses
Component/Function/Method driven framework is best for SAP applications and it is quick and easy to implement with maximum ROI with short span of time
Frameworks:
Design robust automation framework which uses industry available best framework features and made Hybrid automation framework ready.
Identifying the best Test management tool:
Test management tool should be open to integrate with internal and external automation tools to execute entire project management, it may be adding Cycles, releases, requirements, manual and automation test scripts, results and reports, defects and dashboards..etc.
Ideally HP ALM (formally known as HP Quality Center) best for test management tool
Identifying right business scenarios for regression automation testing
Analyzing the system along with SMEs and stack holders and finding out key business scenarios where it is required to execute such scenarios repetitively on each release is an important activity before kicking off automation testing.
• Is automation feasible for the selected scenarios/ test cases?
• Test cases are required to execute repetitively on each release/cycle multiple times
• No manual intervention during script execution
• Able to run script independently or in batch
• Always ROI focused
• Identify areas of ERP that have been customized as customizations can have an adverse effect on functionality and performance that must be tested with every build
• Concentrate testing on interfaces with communication between SAP & legacy system
• Scenarios/transactions which need to be tested with different data sets/values
• High path frequency: Scenarios/process paths used with high degree of frequency when the software is running in full production. Examples include creating customer records, invoicing and other high volume activities where software failures can occur frequently.
Performing Feasibility analysis (FA) and Proof of Concept (PoC):
Carrying out FA and PoC is really required for any automation testing project. It will definitely provide immediate feedback on our automation testing capabilities and success ratio of our automation regression pack which we supposed to automate
E2E Test Management with Solution Manager:
With SAP Solution manager we can successfully carry out End to End test management as show in below picture. This includes various automation tools called CBTA, HP UFT/QTP, Worksoft Certify, SAP TAO..etc
If we are committed to use CBTA, SAP TAO then it is recommended to use Solution manager, as it doesn’t add additional cost to us
Similar kind of test management can be performed in HP ALM too, so based on project nature we can pick up test management tool.
Easy to identify changes in SAP system for automation:
Transporting mechanism happens in SAP system in vary controlled way, each and every change is linked to executable object and added to transport, so that at the end of transport, we come to know changes/additions happens in new SAP application
To control the transport and objects we have SAP BPCA (Business Process Change Analyzer) in it. So it is recommended to perform transports with BPCA to track the changes in system. BPCA has an option to find out which test scripts got impacted because of certain transport, and this will give us direct solution where to fix the script
Benefits with automation approach:
• Good amount of returns on our investments
• Saving substantial amounts of staff time previously spent on running regression tests
• Improving the quality of services by catching data-related defects before production
• Speeding up testing and application deployment cycles to create greater business agility
• Fostering collaboration between IT and business experts to streamline technology projects
• Double/Tripling the percentage of business processes automated
• Operational efficiency
• Reduced risk of application failure
• Improved customer experience
• Foster time to market
Key Considerations for Developing Automation Scripts Architecture for SAP
• To automate all critical business end-to-end scenarios, each scenario is split into test conditions, and each test condition broken down into unique SAP T-Codes that must be executed
• Each unique SAP transaction is scripted as a reusable action under the Common Reusable library.
• Each reusable function/action is called in different scenarios. All the required parameters to execute the reusable function/action are passed through the driver script
• Error handling, common functions also built into reusable common library; included in each script
• Using test management tool, test sets are created for each business scenario; test conditions are added to test set to complete testing & validation of one end-to-end business process
• All scenarios required to be tested in each regression cycle are set for execution using individual test sets driven by test management tool
• The structure of having of one test script per test condition provides great modularity and ease of maintenance and debugging. If a test condition fails, it is easy to detect which test condition and corresponding test script in the complete business scenario caused the issue and broke the execution
• Separate the test data from the scripts. Maintain all the required Master & Transactional data in separate files. These files input the required data to the scripts during execution
• Estimation & planning of effort required to completely build the regression test suite becomes easy. Calculate the effort required to build script for each unique T-Code, common reusable library functions, error handling functions, developing of calling scripts and integration of individual test conditions into scenarios
• Proper coding & commenting guidelines should be documented; all team members must follow the same
• Peer review process to ensure all scripts are well reviewed
• Ensure all the required test data (Master and Transactional data) are current and valid.
• Perform dry run to integrate and run scripts to test a complete business transaction
• Proper documentation for each test script
Preparing Test Suites with batch of automation test scripts
It is always good practice to create batches of test scripts as below, so that you will quickly get to know the feedback of the system quality
• Sanity pack
• Integration Pack
• Regression Pack
• CR Specific Pack
Generating Test Results:
Generating user friendly test results plays an important role here, as test results files can be used as evidences for future purpose.
Summary Results as part of A2 framework with QTP Detailed Results Generated by TAO and CBTA
Script Health Checkup:
We have to consider Automation testing as typical application development process; otherwise we may end up with failure at the end. As part of this, making sure scripts are working fine on regular intervals is essential’
• Scripts evaluated and reviewed against checklist of following parameters periodically to ensure regression suite is up to date and ready during regression testing of the SAP release:
• Apart from maintenance, scripts are executed periodically in batches overnight and test results analyzed. In case of failures, scripts are updated to correct errors.
• Test data for each test script valid and current
• Test environments (SAP, other systems, databases & interfaces) up and running
• All required changes suggested incorporated and script up to date
• Test environment. setup, connectivity, tool license, configuration and tools available
• Test set and test scripts pulled in Test Director before execution starts
• Test cases are up to date and scripts reflect current steps as per test case
• Required results & defects reporting templates in place
• Access to SAP system & transactions and other systems/databases available
• ‘All clear’ in test results report and no issues identified with scripts during dry run
Hello,
ReplyDeleteThe Article on Effective Way of SAP Automation Testing is nice give detail information about it.Thanks for Sharing the information about Effective Way of SAP Automation Testing.
Software Testing Company
Thanks for sharing such a great blog Keep posting.
ReplyDeletemarketing automation tools India
sales automation
relationship intelligence
b2b database providers in India
database provider