In software development, we understand very well the importance of testing. While writing unit tests is considered widely popular and common, unfortunately, they don’t provide the full coverage needed for the entire software.
The better approach is End-to-End testing as a part of the testing routine during development. In fact, End-to-End testing is considered the holy grail, the trophy in the Testing Trophy, or the top of the Testing Pyramid because they tend to be harder to implement than the rest of the testing types. But End-To-End tests can be quite complex and require a well defined infrastructure that is much harder to implement and maintain.
Today, I’m walking through the ins and outs of E2E testing and the lessons and takeaways we’ve learned on the ground here at Swimm.
Why R&D teams need End-to-End testing
As an early-stage startup, Swimm initially worked in one-week sprints that ended with a new release. The day of the release would start with manually QA-ing the entire app, doing all kinds of user flows to make sure everything worked and no regression bugs were discovered.
As Swimm grew, more developers joined the company, and as we began adding many new features, it was increasingly harder to keep up with the changes. The number of regression bugs that were created during each sprint grew in leaps and bounds; finding the cause of each bug became a nightmare; spending a whole day just for fixing regression bugs (out of a 1-week sprint) seemed like an inefficient use of our time – time that could be well spent on developing many more new Swimm features.
Our goal was to find a solution that would:
- Allow us to catch regression bugs during development
- Save us as much as possible manual regression QA time
End-to-End testing seemed to be the missing puzzle in our testing paradigm – it was time for E2E to rise and shine at Swimm.
Finding the best End-to-End testing devtool
At this particular point, it was game time for all of us at Swimm. Time, that is, to step up our game and write E2E tests that would replace the manual and tedious QA flows as much as possible.
It was time for us to research and find the best testing product.
We first needed to define the ideal End-to-End testing framework with the following considerations – that E2E tool be:
1. Easy – most developers don’t write E2E tests or have loads of extra time; the E2E testing devtool needed to be easy to use.
2. Intuitive – we already speak several languages; we needed an intuitive E2E testing tool without needing to learn a new language or complex API.
3. Maintainable – with no automation engineer on the team, we needed a product with minimum setup and no complex infrastructure.
4. Quick – we did not have a couple of months to build an infrastructure; we needed to see immediate testing wins.
Choosing Swimm’s End-to-End devtool
We began researching, considering some of the widely known and most popular frameworks for E2E testing for JavaScript based systems. We tested Jest + Puppeteer, Cypress, and WebdriverIO and had a POC for each.
Why we chose Cypress
We ultimately decided that Cypress was the best End-to-End testing devtool for Swimm.
Here’s why: with a great community of developers using Cypress, great docs, and many cool features such as automatic waiting, automatic screenshots and videos, and time travel, Cypress felt intuitive, was super easy to set up, and using its API applies for user behavior. Plus, we really enjoyed writing tests.
But no End-to-End test is perfect, Cypress included. Limitations include the lack of multi-tab support, difficulty with iFrames, and browser support (just Chrome-based browsers and Firefox). We ultimately decided that because these limitations would not be affecting us in the immediate future, the benefits outweighed those disadvantages.
Adding End-to-End testing to Swimm’s workflow
Game on!
It was time to write our first Swimm End-to-End tests using Cypress. Since we wanted to migrate many tests as quickly as possible, and since we wanted to engage all the developers right from the beginning with Cypress, we came up with the “E2E Heist” – a 1-day hackathon where we all got together, took a break from bugs and features, and migrated manual tests to automated ones.
And we all ate a lot of cookies.
At the end of the Heist, we had dramatically increased our test coverage, and all developers were onboard with the new testing framework. Plus, we got our hands dirty with the surprisingly fun yet no-that-simple testing methodology as most of us were brand new to End-to-End testing.
Sounds good, right?
But what happens when you add many new tests in the CI at once? The CI is most likely going to be red. And when the CI is always red, you might be inclined not to trust it and simply ignore it. And this, of course, kinda misses the purpose of a CI, right?
Heist #2 – Plan B and take 2
So, before we officially added Cypress to our daily development workflow, we realized that we needed a couple more days to make things a lot more stable, refactor some tests, and make sure our test suites were independent and deterministic. We decided that only after we got a green light (from the CI) would we add E2E tests as a gated step for all pull requests.
So Plan B: we started a 2nd End-to-End Heist, which was now a 3-day marathon, eating even more cookies and staying up all night with the nightly CI to make sure it’s green, every. single. time.
And voilà – we got a green light. At this point, Cypress was officially part of Swimm.
Challenges with End-to-End testing
The road to success with End-to-End testing is bumpy.
The wins: we identified and fixed countless number of bugs that would have affected Swimm customers; we saved an incredible amount of time repeating manual flows every week; we dramatically increased our stability and coverage.
The challenges: as always, with software development, End-to-End testing presented many challenges.
E2E testing challenge #1: human testing vs. machine testing
At this point, we had to decide whether we were going to dismiss our manual testing completely – even while moving forward with Cypress.
Before we dismissed the manual checks completely, we reviewed our manual testing user flows (written as tasks in ClickUp), similar to a simplified Test Plan, to ensure we covered everything we wanted to cover using Cypress automated testing.
Our initial plan was to simply convert each task into a test. But after a while, we realized that it was not so straightforward.
- Some operations were repetitive.
- Some checked arbitrary things that were not directly related to a certain suite.
- Some were harder than the others to test (for example, dragging and dropping elements in Cypress is cumbersome).
- Instead of manually testing where we could each create our own “environment” and do whatever we wanted, we now had to have a clean automation test environment before each test run. We also needed some test accounts, and we needed to cleanup after each suite.
We needed to change our mindset and “think like a machine” while still mimicking user behavior in tests.
E2E testing challenge #2: having no experience with End-to-End testing
We were not experienced with E2E best practices and didn’t have a lot of experiences to draw upon. For example:
1. Using well-defined selectors
Using CSS classes as selectors might be the easiest way to select elements during tests, but they are easily breakable. Any code refactor can break classes and therefore fail tests.
The better approach is to use data attributes on elements with a descriptive value and select them instead. When the code changes due to refactoring or design change, the data attributes will remain on the elements and tests won’t hurt. The most common convention used is data-testid.
2. Using commands for repetitive actions
The DRY (Don’t Repeat Yourself) is relevant for E2E as well. Navigating in the app, for example, can be as simple as cy.get('[data-testid="add-doc-button"]') but the overhead of finding the needed selector on every step of the test can be quite tedious.
Cypress allows you to define Custom Commands, where you can easily add repetitive actions just as you use other Cypress commands, for example: cy.addNewDoc(). Obviously we documented our custom commands in a Swimm Doc.
3. Timeouts, Timeouts Everywhere…
The default timeout for Cypress is 4 seconds, but as most Single Page Applications fetch data during rendering and test environments tend to be slower than production, it might be that 4 seconds is not enough.
So what do you do when you get a timeout error? Increase the timeout? No.
In trying to mimic user experience, we expect to see whatever the user sees in the UI. If we fetch data in the app and show a loader or a skeleton in the meantime, this is also what the test should expect to find.
So what did we do? We used a custom command that expects a loader to appear and then disappear. This solution allows us to avoid increasing the timeout whenever we do an “expensive” action.
E2E testing challenge #3: monitoring less reliable tests
So this is the variability factor: while an End-to-End test might initially seem stable, after a couple of runs, it sometimes fails, which means the test is not good enough for various reasons.
Sometimes having a flaky test in the CI may very well be worse than not having that test at all. Because as time goes by, you tend to lose trust in flaky tests. And the danger here is that when a real ‘fail’ happens, you might mistakenly think it’s just the flakiness factor and ignore it.
E2E testing challenge #4: how much time End-to-End testing takes
Writing tests take time, and it almost always takes more time than you’d think. This is true for all types of testing, not just for E2E.
Even though Cypress is developer friendly, intuitive, and fast – time is still required for End-to-End testing.
Is it worth it? Most definitely, yes. Testing of any kind is crucial for producing high-value, stable and robust features. Looking at the time we spent on manual QA, the time we spent on fixing regression bugs, and the time we wrote E2E – in the long run, is well worth it. It is significantly shorter to invest a couple of hours in writing an E2E test suite once, instead of committing regression bugs, trying to find the cause, fixing it every week.
Key takeaways
So many lessons learned on End-to-End testing. Here are my five takeaways.
1. Appoint E2E test ownership– It’s essential that there be an individual or team responsible for overseeing the E2E testing for every new feature being developed, no different from unit-testing.
2. You need a lot of patience – E2E testing is a process, and it takes time; the benefits may not be seen immediately, but they are well worth it. End-to-End testing is a long-term investment and requires a lot of patience.
3. Be sure to write stable tests – With E2E testing, before adding a new test to the CI, you’ll want to make sure it runs well with the other tests in the suite. Specifically, make sure it passes 10 out of 10 runs. If it fails once, it’s just not good enough.
4. Trust your tests – At some point, you have to either trust your tests or decide to go back to manual testing. Life is not perfect, and E2E testing is not fail-free. But it’s a step in the right direction, times a million. So at the end of the day, you have to trust the testing.
5. Stable tests = a stable product – The goal for companies is to create a stable product that users can trust. R&D teams need to prioritize testing as a requirement as companies mature and define their core values, and then move to the next level of quality assurance.
Bottom line
At Swimm, we have dramatically reduced the time we spend on regression bugs, and we have come a long way. Quality has increased, but End-to-End testing is not perfect, and to this day, we still encounter some stability issues with some specific tests. There are lots of places for improvements, but we are pushing forward and moving forward.
End-to-End testing is a continuous effort, but a well-paid off one for sure.
R&D teams, if you want to share feedback or have some ideas about End-to-End testing, we’d love to hear your thoughts. Reach out to our growing community of Swimm users on our Slack channel and learn more about creating and editing docs that are coupled with your code, auto-synced, and fully integrated in your workflow.
If you want to learn more about Swimm, sign up for a personalized 1:1 demo.
