Automating Web Analytics

Bhumika S, Anand Bagmar (Thoughtworks) How many times do we test the same things at multiple layers, multiple levels, adding time to the build process and testing cycle, delaying the feedback? We know what to test and how to test, but what is the right place to test it? In this workshop, we will demonstrate how we, as QA’s, can identify which tests can be classified as unit tests, integration tests, and functional tests. Using a case study, we will see how each component can be tested as part of unit testing; the integration of different parts and the functioning of a software system as a whole, and how functional tests fit into this big picture. We will then bring all these tests together to understand and build the testing pyramid and how it enables us to build the right testing framework with fewer Selenium, i.e., functional tests.

Slides from the workshop I conducted on "Client-side Performance Testing". Abstract of the workshop: In this workshop, we will see the different dimensions of Performance Testing and Performance Engineering, and focus on Client-side Performance Testing. Before we get to doing some Client-side Performance Testing activities, we will first understand how to look at client-side performance, and putting that in the context of the product under test. We will see, using a case study, the impact of caching on performance, the good & the bad! We will then experiment with some tools like WebPageTest and Page Speed to understand how to measure client-side performance. Lastly - just understanding the performance of the product is not sufficient. We will look at how to automate the testing for this activity - using WebPageTest (private instance setup), and experiment with yslow - as a low-cost, programatic alternative to WebPageTest.

This document discusses the perils of using the page object pattern for test automation frameworks. It describes the page object pattern, how it can be used to model pages in code and reduce duplication. However, it also notes limitations like tests becoming too implementation focused rather than intent focused. It then introduces a business layer page object pattern that aims to address these limitations by validating business requirements and separating concerns with abstraction layers. This allows the test pyramid to remain balanced and makes maintenance and scaling easier.

Building a Test Automation Framework is easy - there are so many resources / guides / blogs / etc. available to help you get started and help solve the issues you get along the journey. However, building a "good" Test Automation Framework is not very easy. There are a lot of principles and practices you need to use, in the right context, with a good set of skills required to make the Test Automation Framework maintainable, scalable and reusable. Design Patterns play a big role in helping achieve this goal of building a good and robust framework. In this talk, we will talk about, and see examples of various types of patterns you can use for: Build your Test Automation Framework Test Data Management Locators / IDs (for finding / interacting with elements in the browser / app) Using these patterns you will be able to build a good framework, that will help keep your tests running fast, and reliably in your CI / CD setup!

Slides from the workshop conducted in ThoughtWorks, vodQA Gurgaon on - "What is BDT, and how can you use this technique to identify the 'right' regression suite for your product"

This document provides an overview of Cucumber, a behavior-driven development framework. It discusses what BDD is, the benefits it provides like usability, fewer defects, and living documentation. Popular BDD tools like Cucumber, Jasmine and JBehave are mentioned. Cucumber is introduced in more detail, explaining how it executes scenarios using a Given, When, Then format and supports multiple programming languages. Hands-on examples demonstrate features like data and table-driven execution, background, hooks, tags and reports. Suggested additional reading materials on BDD and Cucumber are also provided.

Behavior Driven Testing (BDT) is an evolved way of thinking about Testing. It helps in identifying the 'correct' scenarios, in form of user journeys, to build a good and effective (manual & automation) regression suite that validates the Business Goals.

This document discusses patterns for test automation frameworks. It begins by introducing common patterns like page objects, business layer, and factories. It then provides examples of page object and business layer page object patterns. The document also discusses test data patterns and different ways to specify test data and locators. Finally, it outlines advantages of using patterns like reduced complexity, reusability, and maintenance. The key message is that the best pattern depends on the specific test automation context.

The key objectives of Organizations is to provide / derive value from the products / services they offer. To achieve this, they need to be able to deliver their offerings in the quickest time possible, and of good quality! In such a fast moving environment, CI (Continuous Integration) and CD (Continuous Delivery) are now a necessity and not a luxury! There are various practices that Organizations and Enterprises need to implement to enable CD. Testing (automation) is one of the important practices that needs to be setup correctly for CD to be successful. Testing in Organizations on the CD journey is tricky and requires a lot of discipline, rigor and hard work. In Enterprises, the Testing complexity and challenges increase exponentially. I am sharing my vision of the Test Vision and Strategy required to make successful the journey of an Enterprise on the path of implementing CD. Learn via a case study of an Enterprise, what does CD really means, and how and what Testing needs to be done in order to enable CD.

This document discusses web analytics testing and the opportunities in testing IoT and big data solutions. It provides an overview of web analytics, why testing is needed, and common web analytics solutions like Google Analytics. It then describes challenges with manual testing and introduces WAAT (Web Analytics Automation Testing) as a solution to automate testing. WAAT supports various plugins and has Java and Ruby implementations. The document concludes by discussing how IoT and big data are generating large amounts of data and new opportunities for testing to validate data collection, analysis, and optimization of business value.

It is easy to measure code coverage when running unit tests. However, very frequently the following questions come up: - How can we measure API test coverage and e2e / UI test coverage? - Does e2e / UI test coverage add value? - If not, what other data can we look at to know if the e2e tests have good coverage? This session is about understanding the above questions, and finding solutions for the same.

Slides from my talk on how to Eradicate Flaky Tests from AppiumConf 2021 https://confengine.com/conferences/appium-conf-2021/proposal/15581/eradicate-flaky-tests

Slides from hands-on workshop at Agile India 2021. Machine setup instructions are available in the proposal - https://confengine.com/conferences/agile-india-2021/proposal/15552/next-generation-functional-amp-visual-testing-powered-by-ai In this workshop, using numerous examples, we will explore: Why Automated Visual Validation is essential to be part of your Test Strategy How Visual AI increases the coverage of your functional testing, while reducing the code, and increasing stability of your automated tests Potential solutions / options for Automated Visual Testing, with pros & cons of each How an AI-powered tool, Applitools Eyes, can solve this problem Hands-on look at Applitools Visual AI and how to get started using it

https://confengine.com/agile-india-2020/proposal/11065/visual-validation-the-missing-tip-of-the-automation-pyramid The top of the pyramid is our UI / end-2-end functional tests - which simulate end-user behavior and interactions with the product-under test. While Automation helps validate functionality of your product, aspects of UX validations can only be seen and captured by the human eye and is hence mostly a manual activity. This is an area where AI & ML can truly help. With everyone wanting to be Agile, make quick releases, the look & feel / UX validation, which is a slow, and error-prone activity, quickly becomes a huge bottleneck. In addition, with any UX related issues propping up cause huge brand-value and revenue loss, may lead to social-trolling and worse - dilute your user-base. In this session, using numerous examples, we will explore: Why Automated Visual Validation is essential to be part of your Test Strategy Potential solutions / options for Automated Visual Testing, with pros & cons of each How an AI-powered tool, Applitools Eyes, can solve this problem.

Slides from my talk on Measuring Consumer Quality - The Missing Feedback Loop - delivered at PSTC 2018

This document outlines an agenda for a Selenium workshop. It covers topics like getting started with Selenium, implementing page object models and design patterns, handling test data, using assertions, and responsive web design testing. Exercises are provided to demonstrate concepts like building page objects, running tests across browsers, and using test data. The goal is to help students learn Selenium concepts and address common automation challenges.

Workshop on "Getting started with Visual Testing using Applitools - @TPC (Test Practitioners Club, NCR), Feb2020"

Presentation on monitoring the web, including synthetic, UEUM, web analytics, interaction analysis. Given at www.meshconference.com/meshu on May 20, 2008

The document discusses creating semantic mashups by bridging Web 2.0 and the Semantic Web. It provides examples of how semantic data can be used flexibly from multiple sources to build applications and demonstrations. The key advantages of semantic data include increased utility of applications by helping users accomplish tasks more easily, and allowing for greater efficiency by reusing existing data sources where possible.

The document discusses creating semantic mashups by bridging Web 2.0 and the Semantic Web. It provides examples of how semantic data can be used flexibly across different domains and applications. The key benefits of semantic data include increased utility of applications by helping users complete tasks more easily, and allowing for greater efficiency by reusing existing data sources where possible.

WSO2Con EU 2015: An Introduction to the WSO2 Data Analytics Platform With Hadoop, we can easily process data from the disk, but this consumes a lot of time. The value of certain insights, such as a traffic alerts or heart attack alerts, degrades with time and handling this time sensitive data needs realtime technologies that can produce output within milliseconds. Moreover, some use cases need advanced analytics like machine learning. In this talk, we will discuss about the WSO2 Data Analytics platform that brings together all the technologies into one platform. It lets you collect data through a one sensor API, process it using batch, realtime or predictive technologies and communicate your results all within a single platform and user experience. Presenter: Srinath Perera Vice President – Research, WSO2

(reposting with clearer title) Performance tuning presentation from WindyCityRails 2010. Why performance matters The right way to approach it Front end testing tools Automated testing tools Common problems and the ways to solve them in Rails Rails specific tools bullet slim_scrooge rack bug request log analyzer rails indexes

Performance tuning presentation for Chicago Rails Conference. Focusing on Front end page improvements

This document provides an overview of the WSO2 Analytics Platform. It discusses how analytics is evolving from initial use cases to everyday, efficient use involving recovery, decision making, and other forms like real-time, interactive, and predictive analytics. The WSO2 Analytics Platform provides a single platform for all four forms of analytics with a consistent programming model. Key components include tools for collecting, analyzing, communicating, and storing streaming data. Real-world use cases demonstrate applications in health, transportation, smart cities, and more.

From Flourish 2009 conference in Chicago (http://flourishconf.com). Discussing the value of accessibility and universality for web UIs.

Modern software products tend to have a rich UI that supports many user workflows, all of which need to be covered in testing. Agile organizations quickly discover that manual end-to-end testing neither supports their velocity nor provides respectable regression coverage. A common progression is to move from fully manual testing to record/replay, then to Selenium IDE style tests, then to automation based on Selenium WebDriver, perhaps with a BDD overlay. Daria Mehra has practiced this approach and shares her experience. She compares the Selenium style of automation to an alternative—crowdsourced test execution— which provides all the benefits of test automation, including continuous integration. This approach allows non-coders to express test cases in natural language, avoids the dependency on UI implementation, adds a “human eye” advantage, and provides the ability to massively parallelize tests for fast execution. Learn the pros and cons of end-to-end, UI-based test automation via Selenium compared to crowdsourcing. Find out how you can assess the suitability of these testing approaches for your needs.

A talk about future technologies of the web, and how our time is equally well spent cleaning up the web of now.

Talk held at the conference Coding Serbia in Novi Sad. Performance of web applications is a crucial dissatisfier for users and thus an important quality criteria -- also used by Google to rank their result lists. As with other quality aspects, performance testing cannot be done at the end of a project but is an integral part of the development process. The practice presentations submitted explains web performance testing along practical examples in order to better understand and judge cause and effect of behavior observed. Usually few causes have a disproportionate effect on bad performance. In addition, it is important to understand diverse load and test scenarios to optimize application behavior. The presentation also introduces a methodology to systematically define and assess performance metrics of an application. The content is based on open source tools and the presentation includes live testing to illustrate the excellent cost benefit ratio of systematically white box testing of performance using an HTTP proxy.

I was meaning to put this talk up for grabs for some time now, but kept forgetting. I was invited to give the keynote speech for the Microstrategy World 2008 conference. The talk was very well received, so here it is.

How we used Customer Development at Dataloop.IO to get our initial customers and traction in the crowded monitoring space. Presentation from Hacker News London Meetup - 9th October 2014

You can find the video recording here: http://youtu.be/iG8Gnf5CD9s This talk given by Steve Caldwell, CEO/CTO of Strap, focused on how to add analytics to your Pebble app. Day 2 - Video 3

This document summarizes a talk given by Chris Heilmann at ForwardJS in 2015. Heilmann discusses the state of web development technologies and how developers have focused too much on experimental features that are not ready for production use. This has led to a fragmented web where browsers implement features differently. He argues developers should focus on standardizing and improving existing web standards rather than constantly introducing new technologies. ES6 is highlighted as a priority for improving existing JavaScript.

A talk from 2006 about securing Rails applications. Some information is outdated, but the core concepts should still be useful.

This document provides an overview of a presentation given by Nick Galbreath at DevOpsDays Tokyo 2013 about making operations visible. The presentation encourages organizations to expose more operational metrics and business data through systems like Graphite and StatsD to improve communication and collaboration between teams. It provides examples of how to collect and visualize different types of data from applications, systems, and business processes. The goal is to overcome excuses for lack of visibility and have organizations complete the "One Machine, One Day, One Person Challenge" to start capturing and sharing their key operational and business metrics.

This document provides an overview of a presentation given by Nick Galbreath at DevOpsDays Tokyo 2013 about making operations visible. The presentation encourages organizations to expose more operational metrics and business data through systems like Graphite and StatsD to improve communication and collaboration between teams. It provides examples of how to collect and visualize different types of data from applications, systems, and business processes. The goal is to overcome excuses for lack of visibility and have organizations complete the "One Machine, One Day, One Person Challenge" to start exposing all of their operational metrics.

This document discusses programming and the web development process. It begins by defining programming as a set of instructions to solve a problem, using the example of instructions for making a peanut butter and jelly sandwich. It then discusses why learning programming is important. The rest of the document outlines the typical steps in the web development process, including user experience design, information architecture, visual design, and development. It provides examples of each step and timelines for hypothetical web projects.

This document summarizes a presentation on performance testing. It introduces basic performance testing terminology and concepts like load testing, stress testing, and capacity testing. It discusses important performance metrics like response time and how to identify bottlenecks. The presentation then provides a real-world example of performance testing a web application hosted on Amazon EC2 and RDS. It discusses using the USE method of identifying resource utilization, saturation, and errors. The performance testing found that the RDS-based environment failed while the EC2-based one passed. The summary concludes by discussing next steps like automating performance testing.

Streamlining End-to-End Testing Automation with Azure DevOps Build & Release Pipelines Automating end-to-end (e2e) test for Android and iOS native apps, and web apps, within Azure build and release pipelines, poses several challenges. This session dives into the key challenges and the repeatable solutions implemented across multiple teams at a leading Indian telecom disruptor, renowned for its affordable 4G/5G services, digital platforms, and broadband connectivity. Challenge #1. Ensuring Test Environment Consistency: Establishing a standardized test execution environment across hundreds of Azure DevOps agents is crucial for achieving dependable testing results. This uniformity must seamlessly span from Build pipelines to various stages of the Release pipeline. Challenge #2. Coordinated Test Execution Across Environments: Executing distinct subsets of tests using the same automation framework across diverse environments, such as the build pipeline and specific stages of the Release Pipeline, demands flexible and cohesive approaches. Challenge #3. Testing on Linux-based Azure DevOps Agents: Conducting tests, particularly for web and native apps, on Azure DevOps Linux agents lacking browser or device connectivity presents specific challenges in attaining thorough testing coverage. This session delves into how these challenges were addressed through: 1. Automate the setup of essential dependencies to ensure a consistent testing environment. 2. Create standardized templates for executing API tests, API workflow tests, and end-to-end tests in the Build pipeline, streamlining the testing process. 3. Implement task groups in Release pipeline stages to facilitate the execution of tests, ensuring consistency and efficiency across deployment phases. 4. Deploy browsers within Docker containers for web application testing, enhancing portability and scalability of testing environments. 5. Leverage diverse device farms dedicated to Android, iOS, and browser testing to cover a wide range of platforms and devices. 6. Integrate AI technology, such as Applitools Visual AI and Ultrafast Grid, to automate test execution and validation, improving accuracy and efficiency. 7. Utilize AI/ML-powered central test automation reporting server through platforms like reportportal.io, providing consolidated and real-time insights into test performance and issues. These solutions not only facilitate comprehensive testing across platforms but also promote the principles of shift-left testing, enabling early feedback, implementing quality gates, and ensuring repeatability. By adopting these techniques, teams can effectively automate and execute tests, accelerating software delivery while upholding high-quality standards across Android, iOS, and web applications.

Slides from my talk on "Techniques to Eradicate Flaky Tests" at Testing Talks - The Reunion, Melbourne, Australia on 20th Oct 2022

This document discusses teswiz, an open source framework for automating real-user scenarios across multiple apps, devices, users, and platforms. It can simulate user actions and behavior to test web, mobile web, Android, iOS, Windows, MacOS and Linux applications. Teswiz uses Cucumber, Appium, Selenium and other tools and supports features like multi-device testing, visual testing with Applitools, and generating reports with ReportPortal. The document provides instructions on getting started with teswiz and links to its GitHub page.

The document discusses visual validation testing as a missing piece of the automation testing pyramid. Visual testing is important but challenging as it is typically done manually, which is tedious, error-prone, and difficult to scale. Automating visual testing can help by taking screenshots of expected user interfaces and comparing them to actual screenshots. However, automating also poses challenges around false positives/negatives, maintaining baselines, and accounting for product changes. The document promotes using artificial intelligence in visual test automation to help address these challenges and advocates for including visual testing in an organization's overall quality and automation strategy.

In this talk, we will talk about, and see examples of various types of patterns you can use for: - Build your Test Automation Framework - Test Data Management - Locators / IDs (for finding / interacting with elements in the browser / app) Using these patterns you will be able to build a good framework, that will help keep your tests running fast, and reliably in your CI / CD setup!