Visual-Regression-Tests with Storyshots
Storybook.js is a wonderful environment to build and design components independently from the project. But what would be an environment without tests?
At sum.cumo, a Sapiens company, we use the official Storybook plugin "Storyshots" as a visual regression test. Storyshots is a Jest Snapshot addon that automatically tests every single story.
Sounds good for now!
But what is a visual regression rest anyway?
Let's imagine that we have a visual bug with a button in the current project. In just one place something has slipped or is not displayed correctly. But we already use this button dozens of times in the code. Now how can we make sure that the supposed fix in one place doesn't break something in a completely different place?
This is the "regression" idea. "Visual" then simply means that we test not only the functionality, but the interface, or the look and feel of the component - and this is best implemented using our well-maintained storybook!
So how does that work exactly?
First, we need a snapshot of every single story. These snapshots are the foundation for each test and are stored in our repository. The test has a standalone stage in our Gitlab CI and runs once on each push to the merge request. When the test runs, Storyshots uses Puppeteer to create new snapshots of all stories - based on the newly pushed code - and compares them to the ones we have stored in the project as a foundation.
If everything goes well and the snapshots match - the pipeline turns green. But what we are more interested in are errors when the snapshots do not match. After running the test, three folders are created in Gitlab, which are available for us to download in the form of CI artifacts:
- our default snapshots
- the new snapshots created during the test
- the diff snapshots - in case an error occurred.
The diff snapshots show us by some sort of heatmap pixel-exactly what doesn't match.
If this was an unintentional change, we can now look at how the change was made and correct it if necessary.
Or, if the change was intentional and we want to apply this change, the newly generated snapshot from folder 2 in the repository can simply replace the old snapshot and be used as the default snapshot on the next run of the pipeline.
What else can Storyshots do?
Of course, to get the most out of Storyshots, we still need the custom configuration. In our storyshots.js we can play with many parameters to make the best use of the test. Some useful settings would be:
Supported Devices:
This specifies whether "only" the desktop version of the stories should be tested or additional devices:
supportedDevices = new Set(['iPad', 'iPhone X', 'Wide'])
But be careful: With each additional device the number of snapshots to be generated increases accordingly, which lengthens the pipeline.
Failure Threshold:
Storyshots will raise an alarm as soon as even one pixel has slipped. For the human eye this is almost impossible to understand and in most cases can be neglected. Therefore it is useful to set an error threshold, so that you don't have to search for errors, which are, in most cases, none.
const failureThreshold = 0.001 // percent
Wait before screenshot:
This ensures that each story has finished loading before a snapshot is generated. For example, possible images or fonts have to be loaded by Storybook first.
const WAIT_BEFORE_SCREENSHOT = 500 // ms
But again, be careful: this time adds up in the pipeline, of course!
Visual regression tests are an essential part of our testing, because we don't only pay attention to the functionality of the component, but also to what the user experiences later in the interface.
