Visual regression testing: it's not about being pixel-perfect

Perry Mitchell / 2016-07-08 20:18:57
Visual regression testing: it's not about being pixel-perfect

Visual re­gres­sion oc­curs when parts of a UI change, even slightly, to a state that was un­in­tended by both de­vel­op­ers and de­sign­ers. It oc­curs every­where, and at least every front-end de­vel­oper has to deal with this on a very reg­u­lar ba­sis.

Regression of UI com­po­nents can be painful as even a pixel shift in any di­rec­tion or a colour off by just a few units is un­de­sir­able and can set-back re­leases and de­lay dead­lines. Many have cov­ered vi­sual re­gres­sion and have noted how easy re­gres­sion de­tec­tion soft­ware makes de­tect­ing these minute changes.

There’s a neat npm pack­age called im­age-diff which cal­cu­lates the dif­fer­ences be­tween an im­age and a con­trol to see if they dif­fer in any way.

The dif­fer­ence of the fol­low­ing two im­ages:

images to diff


Libraries like this make it easy to de­tect dif­fer­ences in im­ages, and can be adapted to help dif­fer­ence UI com­po­nents when us­ing tools like Selenium Webdriver to take screen­shots of loaded com­po­nents and pages.

When a com­po­nent mis­aligns or changes colour/​font etc., it’s quite easy to iden­tify that some­thing went wrong dur­ing the test­ing phase be­fore re­lease. What’s more im­por­tant, in my opin­ion, is the fact that vi­sual re­gres­sion helps catch much larger is­sues, like com­pletely miss­ing or bro­ken com­po­nents.

Take the fol­low­ing bar of but­tons, for in­stance:

component with buttons

If some­thing broke dur­ing the build process of your com­po­nent or ap­pli­ca­tion, and the last but­ton dis­ap­peared, then the user ex­pe­ri­ence could be af­fected. Not every­one has time to test each and every piece of UI through in­te­gra­tion tests or other means, but us­ing vi­sual re­gres­sion means that we’d catch cases like this by de­fault:

buttons difference

Testing in­di­vid­ual com­po­nents is a more ro­bust ap­proach to vi­sual re­gres­sion test­ing rather than dif­f’ing an en­tire web­page screen­shot, but if it’s eas­ier to do the lat­ter it’s def­i­nitely a step up from your reg­u­lar test strate­gies.

Static pages can be au­to­mat­i­cally tested by point­ing Webdriver to them once they’re in stag­ing or pro­duc­tion so de­fects can be found with­out user in­ter­ac­tion.

Regression test­ing ba­sic com­po­nents and pages makes sense, but try not to go over­board with the suite of items you test against. It’s easy to get car­ried away and just make headaches for you and your team. Remember that non-safe iframes won’t ren­der when tak­ing screen­shots from within the page con­text, but will when us­ing Webdriver.

There are al­ready some mostly-com­plete vi­sual re­gres­sion test­ing suites avail­able, like SC5′s Gulp vi­su­al­test:

visualtest by SC5

Be wary of mar­gins and padding, and have fun.