Migration from Cypress to Playwright - hype or great?

There are several reasons why automated software testing is important, and before telling you about our migration from Cypress to Playwright let’s re-emphasize the importance of automated testing:

• Time efficiency: Automated testing allows you to run many tests in a short period of time, which can be especially useful for a startup with limited resources.

• Increased accuracy: Automated tests are repeatable and less prone to human error, which means they can help ensure a higher level of accuracy in your testing process.

• Improved coverage: Automated testing allows you to test a broader range of scenarios and configurations, which can help ensure that your software is reliable and works as intended in a variety of different environments.

• Early detection of defects: Automated tests can help identify weaknesses early in the development process, which can save time and resources by allowing you to fix the problem before it becomes more difficult and expensive to fix.

• Reduced risk: By thoroughly testing your software, you can reduce the risk of issues and defects that could impact your users or customers.

• Costs: Doing manual testing is expensive, and automated testing thus provides a huge cost-saving opportunity

Because of this, we adopted Cypress in 21RISK from the very beginning of 2018. In the time period from 2018 to 2023 we have undertaken a dozen refactors and stack changes:

• 21RISK started out on the  firebase platform

• Migrated to a simpler Node.js + Angular setup

• Adopted Auth0 for authentication

• We swapped  Angular for  Svelte

• Replaced  Auth0 for  WorkOS

• Adopted  Sapper , and finally  SvelteKit (think Next.js for Svelte) in 2022.

Going through all the above changes, Cypress was the stable factor that made sure everything was still working. Because of this, it was a big thing when we started to discuss moving away from Cypress.

The good and bad things about Cypress

With all the business and developer value Cypress has delivered through the years, the list has to start with all the amazing benefits Cypress has given us:

• A modern tool for developing and maintaining e2e tests

• Time-travel debugging makes it possible to debug advanced tests

• Good typescript support from the very beginning

• Lots of documentation and guides, making it easy to get help

• A dashboard service, providing photos + videos of tests in CI.

But our team also found several really frustrating things about Cypress

• The  chaining pattern of Cypress commands is really frustrating. It creates a weird mental barrier for developers because nowhere else in our codebase do we use this pattern. It also makes it difficult to use the normal ways of optimizing code and controlling parallelization like Promise.all and using simple variables and functions. Cypress offers some ways to work around this, for example using the  alias command . Our experience is that it’s not intuitive to make it work, without considerable effort.

• Refusing to start the web server . Cypress refuses to be involved in the process of starting your web server. This sounds like a small thing - but it is actually really annoying. With Playwright we experienced how extremely convenient it is to have a single command to run, instead of having to do 2 steps.

• Running Node.js in  Cypress tasks . It’s a very painful process to run a simple Node.js function to seed your database with Cypress. You have to write a task, and then execute it with cy.task . But there are many caveats before types will work with Typescript, and only serialized JSON values can be passed as input. All along doing this, you have to avoid hitting any ESM compiling errors. Compared to simply calling a function in Playwright, and well done, the contrast is astronomical.

• Cypress commands . Re-using commands in Cypress is done with custom commands, an important concept in the Cypress ecosystem. It’s easy to set up, but types have to be configured manually for Typescript to work properly - prone to errors. The biggest drawback is 100% that team members have to mentally learn the Cypress custom commands concept, instead of just re-using functions as we are used to when writing Typescript code in Playwright.

• The business model . Now we getting to some of the more serious problems with Cypress. To run tests in parallel you have to pay Cypress for the dashboard service. Yes - you read this right. You are even priced for the number of “it’s”, so if you write all your tests in a single file you get the lowest price. You can’t make this up. This fact also makes Cypress useless for API integration testing, something that Playwright is extremely good at (because here we need thousands of “it’s”). This pricing model incentives developers to write fewer “it’s”, something that should not be a driving factor when deciding on the right number of “it’s”.

• "Very slow developer feedback.  As a consequence of the business model, running tests in parallel required the Cypress dashboard service. In other words, it’s impossible to run Cypress tests in parallel locally. So the only way you can execute your e2e test suite locally is to run one test at a time. This is simply horrible, and worse than watching paint drying. Our team is using powerful computers while developing locally, typically MacBook Pro/Air laptops; capable of running many processes in parallel.

The migration to Playwright

We migrated the test suite manually to Playwright. It took approximately 2 weeks, and the results are profound (in no particular order):

• We wrote a  worker fixture , that starts our MongoDB in-memory database and SvelteKit webserver. It works like a charm.

• We can now run tests in parallel locally - it’s a game changer. No more having to wait for CI to make sure something would break.

• No tasks or commands, just simple function calls in Node.js sync and async. It 100% feels like a much simpler API. We are in love with pragmatic functions like  page.$eval that makes it simple to do JS manipulations to elements before taking visual snapshots, like await page.$eval('[data-cy="data-will-be-send-at"]', (el) => (el.innerHTML = "Aug 30, 2022")); - or just taking about how sweet it is to  iterate lists with “normal” for of loops.

• No need to pay for the Cypress Dashboard service 🤯! We simply run tests in parallel with Github actions with an action this simple npx -w=@eddy/e2e playwright test --shard=${{ matrix.shard }}/${{ strategy.job-total }} with 10 workers. The tests run so fast, that we month after month are within the  Github teams plan included CI minutes. Also did I mention, we don’t have to think about using many specific it’s now - because we don’t pay per it.

• Faster tests. Our tests simply run 1.5-2 times faster in CI, compared to using the Cypress Dashboard service.

• The debugging experience locally is superb, with a much more lightweight debugging tool that spins up the test framework faster.

We also found Playwright to be a phenomenal tool for API integration testing, it works great without having any browsers involved.

Playwright is truly great

The migration to Playwright is now done at 21RISK and we experience a test suite we find easier to maintain, faster to execute, less flaky, and cheaper to operate. 10 out of 10 would recommend.

Nothing is perfect, and we currently miss

• Watch mode is  currently missing from the Playwright test runner.

• We are planning to add component testing to our test suite, and  this feature is currently in beta at Playwright.

It’s never easy to compare tools, but I strongly recommend taking a look at Playwright if you are into automated testing. Knowing what we know now, I would 10/10 recommend migrating from Cypres to Playwright. I believe that Cypress needs to take drastic measures to meet the powerhouse Playwright, with my recommended first takes being:

• Cypress should embrace running the web server. It’s such an integrated part of running e2e tests, that it should not be ignored.

• Cypress should make parallelization core of the product. Forcing people to run only one process on a super-fast computer is wasting everyone's time.

• The dashboard service could still have a paid tier, where test results and analytics can be explored in a nice way.

• Refactor Cypress away from .then syntax, adopt async/await and let everyone enjoy using the skills they already have to learn.

• Make the core of Cypress much smaller and faster

Best of luck to everyone at Cypress, and thank you for revolutionizing testing. We are following the automated testing tools closely - things are moving at a high speed 🤩