First visual regression testing at Kiosked with GUIRE

Perry Mitchell / 2016-07-27 20:51:04
First visual regression testing at Kiosked with GUIRE

Front-end ad-tech is pri­mar­ily about get­ting net­work place­ments in front of users as quickly as pos­si­ble, but there are some im­por­tant UI as­pects to con­sider when test­ing. Obviously ads need to be able to close us­ing some kind of close but­ton, and may in­clude some other in­for­ma­tion or logo re­gard­ing the ad or its provider. The place­ment it­self, with its shape and size, could also be con­sid­ered an im­por­tant part of its user in­ter­face.

By con­sid­er­ing what makes up an ad place­ment vi­su­ally, or what makes up any com­po­nent vi­su­ally, we can iden­tify the parts of our ap­pli­ca­tion that should be scru­ti­nised dur­ing test­ing as if they were to change, our ap­pli­ca­tion may no longer per­form its duty. A lot of my job at Kiosked is mak­ing sure our ads re­main con­sis­tent across sites and de­vices by de­vel­op­ing cross-plat­form code, but every de­vel­oper in this area knows that con­sis­tency is one of the most dif­fi­cult ideals to up­hold dur­ing con­tin­u­ous in­te­gra­tion.

QAs and good test­ing rou­tines can catch most re­gres­sion, but those edge-cases where ei­ther a com­po­nent is a pixel off or even not there at all can be eas­ily caught by us­ing au­to­mated test­ing soft­ware. Integration tests can help a lot for com­plex sce­nar­ios, but the boil­er­plate code that usu­ally comes with them make them hard to main­tain and un­wieldy.

My team and I have thought a lot about these as­pects of our code­base, and be­cause it’s con­stantly chang­ing and be­ing up­graded to out­per­form the com­pe­ti­tion, we’ve avoided any test­ing prac­tices that could po­ten­tially slow down our de­vel­op­ment speed and re­duce our abil­ity to pivot por­tions of the code­base. If we’re go­ing to cover that next base by adding vi­sual re­gres­sion tests into the mix, those tests need to be sim­ple and ver­sa­tile so oc­ca­sional main­te­nance is not a mas­sive headache.

We re­leased GUIRE very re­cently and are now start­ing to make it an in­te­gral part of our front-end test suite. GUIRE con­fig­u­ra­tions are pretty sim­ple:

const path = require("path");

let testURL = path.resolve("./index.html");

module.exports = {
    name: "test-name",
    url: `file://${testURL}`,
    waitForEl: "#myApp",
    components: [
        {
            name: "component-test-1",
            setupFn: function(done) {
                window.Library.initSomething.then(done);
            }
        }
    ]
};

Each sec­tion (whatever you de­cide this should be) of your app gets a test suite like this, and is then made up of com­po­nents. For each suite GUIRE cre­ates and launches a WebdriverJS in­stance to per­form a clean set of tests within. Once launched and ready (determined by check­ing wait­ForEl or se­tupFn), GUIRE ex­e­cutes each com­po­nent test in that ses­sion be­fore record­ing the re­sults and clos­ing the Webdriver in­stance.

Using GUIRE to test com­po­nents

There are a num­ber of dif­fer­ent ways to setup tests with GUIRE, but the test­ing en­gine ba­si­cally only needs 2 things: a JavaScript suite con­fig­u­ra­tion and a page to ex­e­cute the tests on.

This ex­am­ple demon­strates how a com­po­nent built with Webpack may be tested, though real-world cases may prove to be more com­plex:

module.exports = {
    name: "MyComponent",
    url: testURL,
    setupFn: setupFunction,
    components: [
        {
            name: "MyComponent-test-1"
        }
    ]
};

The url must be ei­ther a file URL (eg. file:///index.html”) or HTTP-served URL. Webpack makes it easy to gen­er­ate sta­tic files to test, but set­ting up the setupFn to pre­pare the page for in­ject­ing the script can be­come quite cum­ber­some:

let setupFunction = `var callback = arguments[arguments.length - 1];
let s = document.createElement("script");
s.src = "${bundleURL}";
document.body.appendChild(s);
setTimeout(callback, 1500);
`;

All-in-all run­ning a lo­cal server and test­ing the page via that may be the eas­ier op­tion. This func­tion­al­ity may find its way into GUIRE at some stage quite soon, as con­fig­u­ra­tion files are best left sim­ple and de­scrip­tive.

GUIREs Webdriver in­te­gra­tion launches Chrome by de­fault to ren­der the test page:

GUIRE rendering with WebdriverJS

If you’re test­ing dif­fer­ent facets of a com­po­nent or dif­fer­ent com­po­nents them­selves, you can use the setupFn func­tion on each com­po­nent de­f­i­n­i­tion to hide all oth­ers and ini­tialise the next item to test. Utility meth­ods like beforeEach and afterEach will also soon be­come avail­able for GUIRE.

Using GUIRE on a CI server

If you’re se­ri­ous about pro­ject sta­bil­ity you’re prob­a­bly run­ning a CI server. As Webdriver sup­ports test­ing in real browsers, run­ning GUIRE tests on a head­less server can some­times be tricky. Checkout GUIREs Travis con­fig­u­ra­tion to see how it uses X vir­tual frame­buffer (XVFB) to run Firefox and Chrome tests with­out a typ­i­cal win­dow man­ager. There’s even a plu­gin for Jenkins.