Test Automation: Investment Today Pays Back Tomorrow
- 2. ! ! ! ! !!!!! ! !!!! Al Wagner is currently a testing evangelist with IBM, driving thought leadership, strategic initiatives, and tangible solutions with a specific focus on quality management, test automation, and service virtualization. In his fifteen years of practical IT field experience, Al has assisted, supported, mentored, and enabled teams with their IT application infrastructure, development, implementation, and operations challenges. A frequent conference speaker on software quality principles and techniques, Al has coauthored the IBM Service Virtualization for Dummies book and authored numerous papers on quality management and test automation. ! ! Al Wagner IBM
- 3. 5/26/2015 1 © 2013 IBM Corporation Continuous delivery of software innovation on hybrid cloud Test Automation: Investment today pays back tomorrow @Al_Wagner IBM Testing Evangelist [email protected] 1 Today’s reality for Testers Testers want to spend MORE time ibm.biz/TestingSurveyResults
- 4. 5/26/2015 2 2 Are we simply chasing Rainbows and Unicorns? 3 Test Automation is not a “Silver Bullet”!
- 5. 5/26/2015 3 4 Not all tests can or should be automated Where test automation makes sense Highly repetitive tasks Well understood test cases or scenarios Stable areas of the application under test Where test automation doesn’t make sense Tests which need that human component • Exploratory tests “One off” tests Experimental tests 5 A dream doesn’t become reality through magic; it takes sweat, determination and hard work. -Colin Powell
- 6. 5/26/2015 4 6 There is no “Easy” button Test automation is a Software Development project Needs funding Needs resources Needs planning Automated tests need to be designed, developed and tested Build a reusable code base – framework Take an iterative approach – learn and improve Modularize your automated tests - reuse Test automation is a Development Project that does not end Maintenance Ongoing development & refactoring Continuous improvement 7 Test automation tools are only robots Test Automation tools are not the same as human Testers… They do not think, ask questions, challenge results They do not get tired They are not impacted by other interruptions Test Automation tools excel at… Executing a series of steps the same way repeatedly Reporting on programed expected results Speeding up test execution (the “checking” of software) Test automation should not be an “excuse” to reduce head count!
- 7. 5/26/2015 5 8 Not all Testers are Automation Engineers understand how to best leverage the existing skills of your team Subject Matter Experts • Understand the needs of the business – Domain Knowledge • Aide in making sure software being developed is testable • Assist in designing and reviewing test cases Automation Engineers • Understand OO Concepts • Encapsulation, Abstraction, Inheritance, Polymorphism • Have a development background • knowledge of programming/scripting languages Java, Jscript, PHP, Ruby, Python, Jquery, etc… • Build and maintain test automation assets 9 With the right approach & skilled resources Test automation can… Help decrease the duration of a test cycle others have decreased their test cycle from months to hours Increase frequency of test execution being able to run faster means you can run continuously Improve level of testing by freeing up time planning and design; additional value-add tests Provide earlier detection and isolation of defects decrease project risk Improve confidence with software quality increase test coverage; improve the quality of testing itself
- 8. 5/26/2015 6 10 Automating tests – Historically speaking How users typically approach automation… - Record and playback Automated UI tests are… • the most fragile • the most difficult & time consuming to maintain as the application continuously changes. “Record & Playback” of the User Interface Integration Tests Testing messages exchanged between components Unit Tests Testing the lowest level of an application – a component and its methods 11 Automating tests – Is there a better way? What the user can see… - User Interfaces - Systems of Engagement What’s below the surface… - Integrations - Components - Objects/Classes - Methods - Systems of Record - Infrastructure - and much more UI Testing
- 9. 5/26/2015 7 12 Take a better approach to automating tests Exploratory Testing Component Layer Service / API Layer UI Layer Push testing down to focus on the highest areas of risk Automated integration testing increases coverage and confidence Service virtualization can be applied across all layers 13 Test what matters, test it earlier, test it continuously Earlier team collaboration focuses on prevention over detection Test execution shifts left with focus on testing the highest areas of risk Automated integration testing increases testing coverage and confidence Service virtualization can be applied at all phases – even allowing Integration Testing to begin in parallel with Unit Testing. UNIT SERVICE UI Automated Integration Testing Service Virtualization Traditional UI Testing Development Testing Staging Production C o l l a b o r a t i o n
- 10. 5/26/2015 8 14 Service / API testing (aka Integration Testing) Tests can bypass the UI and execute at the point of integration Test while your system is being built and at an accelerated rate Much greater coverage = much greater confidence 15 Comprehensive test automation Continuously Test in Multi-Channel, Complex and Distributed Environments Functional, Integration, UI Testing, and Mobile Testing Test across multi channel integrated systems Test across platforms and technologies Execute full scale, orchestrated performance testing, or test performance at the integration layer Host/Mainframe Client-Server Web/Desktop Mobile/Cloud
- 11. 5/26/2015 9 16 Testing the UI Separate what the test accomplishes from how it is executed Construct test scripts which are…durable, maintainable, resilient to change Keywords abstract the details of how to execute a logical unit of function Business scenarios compose keywords into a flow Reduce test maintenance Keyword behavior isn’t reproduced in multiple tests; update fewer places when application changes Reduce test cases because overlap becomes apparent when abstraction is raised to keyword level 17 Test Script 1) 2) 3) 4) Review shopping cart contents 5) Remove item from cart 6) Verify removal 7) 8) Logout Logon & Verify Click Order Entry Enter Order Details Click Home Test Script 1) 2) 3) 4) Review shopping cart contents 5) Remove item from cart 6) Verify removal 7) 8) Logout Logon & Verify Click Order Entry Enter Order Details Click Home Automating Tests – Maximize reuse / Minimize maintenance Test Script Test Script Test Script 1) Logon & Verify 2) Click order entry 3) Enter order details 4) Click process shopping cart 5) Enter billing info 6) Submit order 7) Verify order confirmation 8) Click Home 9) Logout Test Script 1) Logon & Verify 2) Click order entry 3) Enter order details 4) Click review shopping cart 5) Remove item from shopping cart 6) Verify removal 7) Click Home 8) Logout Test Script 1) 2) 3) 4) 5) Enter billing info 6) 7) Verify Order 8) 9) Keyword Repository Logon & Verify Click Order Entry Enter Order Details Process Cart Click Home Logout Logon & Verify Click Order Entry Enter Order Details Submit Order Click Home Process Cart Test Script 1) 2) 3) 4) Review shopping cart contents 5) Remove item from cart 6) Verify removal 7) 8) Logout Logon & Verify Click Order Entry Enter Order Details Click Home Logout Submit Order Modular ApproachTypical Approach Easier to create; Offers minimal reuse; Requires maximum maintenance Harder to create; Offers maximum reuse; Requires minimum maintenance
- 12. 5/26/2015 10 18 Automated Tests != Automated Testing 19 Automated testing enables continuous testing Automated Tests Automated Testing
- 13. 5/26/2015 11 20 Quality Management across the Lifecycle Build quality in with team collaboration Unify the team through real-time collaboration Minimize scrap and rework with lifecycle traceability Manage risk with smarter testing Focus your testing effort on the right 20% Apply automation where it matters Make confident decisions Take informed decisions and make proactive change with real-time analysis and actionable reporting Confidently deliver incremental quality improvements 21 Today’s reality for Testers Testers want to spend LESS time ibm.biz/TestingSurveyResults
- 14. 5/26/2015 12 22 Service virtualization Eliminate traditional project and environment contention Create virtual components that realistically simulate the behavior of missing components Test even when environments are unavailable, resources are unavailable – or before they have been built Change the fundamental way you approach testing Your system Interfaces to other systems (e.g. Billing) Substitute with virtual component 23 Public Cloud A public cloud is one based on the standard cloud computing model, in which a service provider makes resources, such as applications and storage, available to the general public over the Internet. Public cloud services may be free or offered on a pay- per-usage model. Private Cloud Private cloud is the phrase used to describe a cloud computing platform that is implemented within the corporate firewall, under the control of the IT department. Hybrid Cloud A hybrid cloud is an integrated cloud service utilizing both private and public clouds to perform distinct functions within the same organization. • Cloud is not a commodity. • Cloud computing doesn’t come out of the sky. • Clouds are physical hardware inside brick and mortar facilities connected by hundreds of miles of networking cable. Dynamic infrastructure computing and networking resources available on demand
- 15. 5/26/2015 13 24 Continuous deployment automating testing includes automating deployment Consistent deployments and rollbacks of applications Orchestration of changes across servers, tiers and components Configuration and security differences across environments Clear visibility: what is deployed where and who changed what • Multi-Tier Application Models • Scalable distributed automation • Quality gates and approvals Deployment Automation 25 Our “sample” application! Mobile Web UI Database SAP CICS Application Server Android, iOS IE, Firefox, Chrome, Safari MyFictiousCompany.com Dependent 3rd party service MyFictiousCompany App
- 16. 5/26/2015 14 26 The “many” environments of MyFictiousCompany Deployment Automation Mobile Env2 App Server 3rd Party Service Database CICSSAP Physical devices, emulators, devices clouds Rational Test Virtualization Server - virtualized services CICS Tx SAP External Services DB Service Virtualization Development Production Web UI Env1 DB App Server SAP CICS PaaS 3rd Party Service Test Customers DB App Server SAP CICS PaaS (private cloud) External Services SaaS (public cloud) Off premise Off premise On premise 27 Collaborative Development Continuous Feedback In a DevOps world Testing and test automation is critical to accelerating software delivery The Continuous Delivery Pipeline Development Testing Staging Production Business Owners Customers Continuous Monitoring Continuous Testing Dev CI At least 1/3 of is testing! Quality Management across the lifecycle Production like Test Labs Comprehensive Test Automation Orchestration from build to production
- 17. 5/26/2015 15 28 Continuous Customer Feedback & Optimization Collaborative Development Continuous Release and Deployment Continuous Monitoring Continuous Business Planning Continuous Testing Operate Develop/ Test Deploy Steer DevOps Continuous Feedback Accelerate software delivery – for faster time to value Balance speed, cost, quality and risk – for increased capacity to innovate Reduce time to customer feedback – for improved customer experience The Big Picture - DevOps & Continuous Testing Clients are achieving measurable business outcomes with DevOps! © 2014 IBM Corporation Enterprise capability for continuous software delivery that enables organizations to seize market opportunities and reduce time to customer feedback. 29 Payback is Real! Real Results from Real Projects
- 18. 5/26/2015 16 30 Results of adopting Test Automation & Service Virtualization A Major European Bank Reduced test cycle from 3 weeks down to 4 hours Increased project delivery capacity 100% (40 projects to 80 projects) Avoided hiring three additional FTE's, saving almost $600k. Reduced incidents found in production from 2.5 to .3 per project, saving $3 million Identified defects earlier and freed the team up to focus on new capabilities, saving over $560k Reduced consulting and third party testing fees with better test automation, saving $408k Eliminated the need for additional hardware and labor to set up test environments, saving $700k Generated new revenue for the business – estimated at tens of millions! 13x ROI in 3 years! Investment $433,149 Present Value $5,839,998 31 “Test automation reduces the time required to validate ‘rate filing’ by 94 percent, from 5,600 hours to 320 hours, and prior to this software, manual testing represented 67 percent of the total rate filing time.” An insurance provider Results from automating testing going where others have already gone
- 19. 5/26/2015 17 32 “The new framework helped to drive new efficiencies, cutting deployment timelines for new tools and reducing defective code within mobile-friendly systems by up to 90 percent.” IBM CIO Office Results from automating testing going where others have already gone 33 “Sandhata Technologies Ltd. cut costs by $100,000 per project by simplifying functional and regression testing for more rapid project delivery. Test automation accelerated development into production, lowered the risk of undertaking change and reduced the costs of running the project.” Sandhata Technologies Ltd. Results from automating testing going where others have already gone
- 20. 5/26/2015 18 34 “Companies testing their software through SQUALIO have been able to reduce their software testing costs by 30 percent due to improvements in infrastructure expenses and workflow efficiency.” SQUALIO Results from automating testing going where others have already gone 35 Summary Do something… It is better than doing nothing. What have you got to lose? Remember that automated tests are software… Automated test scripts and frameworks need on-going care Adopt a framework approach… Develop scripts which are easy to maintain and resilient to change Don’t be afraid to experiment… We learn from our mistakes Focus on pushing automating tests down…. To the component and service layers Automating tests != automating testing… They are not the same You are not the first… many have experienced success; be that next one!
- 21. 5/26/2015 19 36 Training accelerates Productivity Don’t let yourself be just another “fool” with a “tool”! Formal Training face to face virtual Mentoring Pairing and many more 37 Help is available!!! test
- 22. 5/26/2015 20 38 IBM Positioned As A Leader The Forrester Wave™: Modern Application Functional Test Automation Tools, Q2 2015 Ensure Quality Of Enterprise, Mobile, And Web Applications Without Sacrificing Delivery Speed “In order to support continuous quality and faster development, especially for those on an Agile and DevOps journey, testing must shift from a UI-only approach to an UI-and-API approach…” The Forrester Wave™ is copyrighted by Forrester Research, Inc. Forrester and Forrester Wave™ are trademarks of Forrester Research, Inc. The Forrester Wave™ is a graphical representation of Forrester's call on a market and is plotted using a detailed spreadsheet with exposed scores, weightings, and comments. Forrester does not endorse any vendor, product, or service depicted in the Forrester Wave. Information is based on best available resources. Opinions reflect judgment at the time and are subject to change. External Presentation Slide ibm.biz/ForresterWave2015Test 39
- 23. 5/26/2015 21 40