Building a Complete Pipeline: The Essential Components of Continuous Testing [webinar]

•

0 likes•703 views

Full webinar recording -- In an era when testing is no longer limited to end-of-line waterfall-type process -- but is an integral and substantial part of the end-to-end development process -- it is ever more pressing to look under the hood of the tools, technologies, processes, and best practices utilized by top-notch testing teams that release world-class software. In this webinar, we took an in-depth look at the technical aspects required to successfully scale continuous testing -- and discussed the tools and technologies used by leading engineering teams, including their approach to managing functional, performance and visual testing at scale. We also took a hands-on approach: via a live demo, we showed how all of these come together to create a complete end-to-end quality pipeline that supports continuous testing at scale.

More Related Content

Building a Complete Pipeline: The Essential Components of Continuous Testing [webinar]

1. Building a Complete Pipeline The Essential Components of Continuous Testing

2. Today’s Speakers 2 Eran Barlev Sr. Software Engineer Guy Arieli CTO

3. Agenda ● Continuous Testing Overview ● The Value of the Pipeline ● The Three Essential Components ○ Functional Testing ○ Performance Testing ○ Visual Testing ● Live Demo ● Recap and Q&A

4. Continuous Testing and the Value of CI/CD Pipeline • Quality + Velocity = Continuous Testing • Fast (~immediate) feedback to developers • Solid pipeline • Quality efforts aligned with business expectations • Prioritized list of actionable tasks • Outcome: Huge change in productivity Quality is the biggest challenge in most technology organisations Add ROI pillars

5. Development Engineer Fast Feedback with Continuous Testing Commit step deﬁnitions + feature ﬁles Trigger test execution Load app to cloud Tester + Test Engineer Pull request to the main branch Performance validation Continuous Integration reporttest build Pull request Approved? no yes Refuse pull request Debug / update test suite 1 Pull app source2 Build app3 Pull test source4 Build test project5 6 7 8 9 10 Git Hook Merge to master branch Visual validation

6. PERFORMANCE UX changes against a baseline The Three Essential Components FUNCTIONAL Unit / API / UI VISUAL Visual changes against a baseline

7. FUNCTIONAL TESTING

8. Functional Testing (Unit, API, UI) ●The most widespread type of functional testing ○ Automated with Selenium, Appium, Cypress, … ●Tests the end-to-end functionality of a system through its UI ●A test is a sequence of steps in which you ○ Simulate user inputs: navigation, clicks, swipes, keystrokes, etc. ○ Validate the system output on the page using assertions

9. PERFORMANCE TESTING

10. 10 Performance - User Perspective ● Duration: Long load time as 3 seconds can drive away half your customers ● Resources hogging: An application that hogs resources, CPU, battery, data traﬃc and memory is 10x more likely to be removed by customers App Store: While functional testing can get you to 3 stars, you need performance testing to get to 5 stars Performance is critical to customer user experience

11. Three Factors that Impact Performance 11 • Technology • Mobility • Coverage • Congestion Client • Code • Environment - Server / device resources - OS / Browser version - Other applications • Code • Environment - Architecture - Resources - Networking Back-end Performance Testing (Load, Stress, Spike, endurance..) Jmeter, Loadrunner, Network Virtualization (network types, mobility scenarios…) Device Monitors Back-end Network

12. Three Factors that Impact Performance 12 • Technology • Mobility • Coverage • Congestion Client • Code • Environment - Server / device resources - OS / Browser version - Other applications • Code • Environment - Architecture - Resources - Networking Back-end Performance Testing (Load, Stress, Spike, endurance..) Jmeter, Loadrunner, Back-end Network User Experience Performance Testing

13. Focusing on Transactions Transaction :An event in the UI that sends a request to the server, leading to a server response and another event in the UI A change of 2 seconds or 500 ms can be meaningful for a transaction and for customer experience, but not in a regular test duration that could take minutes Transaction Transaction only – no setup A single step - action and outcome only Duration – seconds, ms Enables granular measurement 5 UI Event Server

14. Gather Transaction Data as Part of Functional Testing Leverage existing eﬀorts and resources of functional testing for performance transaction testing Transactions Information Database Transactions Analytics Analysis and Debug Manual Tests Automated Functional Tests Transaction Info (Start/Stop) 7 REST API

15. UX Performance Testing - Example of a Report 15

16. 1 s 2 s 3 s 4 s 5 s 10% 10% 10% 10% 100% 10% 50% 90% 97% 100% What is Speed Index?

17. Network Traffic Analysis Evaluate the amount of traffic used by the application and its speed • HAR file and network transaction waterfall view

18. Transaction Performance - Analytics 18

20. VISUAL TESTING

21. Why are we talking about this? Catching visual bugs is not perceived as a priority. Nobody’s boss is asking for that.

22. What is the #1 priority? Everyone wants to increase functional test automation coverage to safely speed up releases

23. Types of Functional Testing

25. ●The most widespread type of functional testing ○ Automated with Selenium, Appium, Cypress, … ●Tests the end-to-end functionality of a system through its UI ●A test is a sequence of steps in which you ○ Simulate user inputs: navigation, clicks, swipes, keystrokes, etc. ○ Validate the system output on the page using assertions UI Testing

26. Test Required Login Details

27. A Cypress UI Test

28. What If the CSS Changes?

29. What If the CSS Changes?

30. …

31. How many functional assertions are needed here?

32. How many functional assertions are needed here?

33. When the UI Changes...

34. The Pain ● Most organizations struggle to succeed with UI testing ○ Every assertion increases the maintenance overhead - Is it really worth automating? ○ Coding skills required to create and maintain tests ○ Checks system functionality via the UI – Not the UI! ○ Can only catch expected bugs ○ Can’t assert on non-dom elements: image, canvas, css graphics, WebGL, etc. ○ Manual testing is still needed!

35. The Solution… UI Testing with Visual AI!

36. Adding a Visual Checkpoint

37. ●A single assertion for complete functional coverage ●Does not break when the UI changes ●No coding skills required to maintain baselines ●Root cause analysis ●Bonus: validates the UI Visual Assertions Advantages

38. Empirical evidence based on 278 completions (100 winners) ● Time to implement test suite: 6x faster (7.2x for winners) ● Total lines of code: 49% less code (72% for winners) ● Number of locators: 7.8x more stable (8.9x for winners) ● Bugs found: 100% vs 70%

39. UI vs. Visual Testing

40. DEMO

41. Thank You

Building a Complete Pipeline: The Essential Components of Continuous Testing [webinar]

Related slideshows

More Related Content

Building a Complete Pipeline: The Essential Components of Continuous Testing [webinar]