How to Ignore the Needle Docs

At PyCon 2014, I learned about a package called “needle” from Julien Phalip’s talk, Advanced techniques for Web functional testing. When I tried using it with a Django project, I immediately ran into problems:

1. The needle docs aren’t written for Django, so they don’t explain how to use NeedleTestCase with LiveServerTestCase.
2. I wasn’t using nose as my test runner, and didn’t want to start using it just to run Needle.

The first problem turned out to be easy; use both:

class SiteTest(NeedleTestCase, LiveServerTestCase):
    pass

The second problem wasn’t that bad either. If you examine the Nose plugin Needle adds, it just adds a  save_baseline attribute to the test case.

There were a lot of random hacks and tweaks I threw together. I think the best way to show them all is with an annotated example:

import os
import unittest

from django.test import LiveServerTestCase
from needle.cases import NeedleTestCase


# This is a configuration variable for whether to save the baseline screenshot
# or not. You can flip it by manually changing it, with an environment variable
# check, or monkey patching.
SAVE_BASELINE = False


# You should be taking screenshots at multiple widths to capture all your
# responsive breakpoints. Only the width really matter,s but I include the
# height for completeness.
SIZES = (
    (1024, 800),  # desktop
    (800, 600),  # tablet
    (320, 540),  # mobile
)


# To keep the test runner from running slow needle tests every time, decorate
# it. In this example, 'RUN_NEEDLE_TESTS' has to exist in your environment for
# these tests to run. So you would run needle tests like:
#
#     RUN_NEEDLE_TESTS=1 python manage.py test python.import.path.to.test_needle
@unittest.skipUnless('RUN_NEEDLE_TESTS' in os.environ, 'expensive tests')
class ScreenshotTest(NeedleTestCase, LiveServerTestCase):
    # You're going to want to make sure your pages look consistent every time.
    fixtures = ['needle.json']

    @classmethod
    def setUpClass(cls):
        """
        Sets `save_baseline`.

        I don't remember why I did it here. Maybe the timing didn't work when
        I put it as an attribute on the test class.
        """
        cls.save_baseline = SAVE_BASELINE
        super(ScreenshotTest, cls).setUpClass()

    def assertResponsive(self, scope, name):
        """Takes a screenshot for every responsive size you set."""
        for width, height in SIZES:
            self.set_viewport_size(width=width, height=height)
            try:
                self.assertScreenshot(
                    scope,
                    # include the name and browser in the filename
                    '{}_{}_firefox'.format(name, width)
                )
            except AssertionError as e:
                print(e)
                # suppress the error so needle keeps making screenshots. Needle
                # is very fickle and we'll have to judge the screenshots by eye
                # anyways instead of relying on needle's pixel perfect
                # judgements.
                pass

    def test_homepage(self):
        urls_to_test = (
            ('/', 'homepage'),
            ('/login/', 'login'),
            ('/hamburger/', 'meat'),
            ('/fries/', 'potatoes'),
            ('/admin/', 'admin'),
        )
        for url, name in urls_to_test:
            self.driver.get(self.live_server_url + url)
            self.assertResponsive(
                # for now, I always want the full page, so I use 'html' as the
                # scope for my screenshots. But as I document more things,
                # that's likely to change.
                'html',
                # passing in a human readable name helps it generate
                # screenshots file names that make more sense to me.
                name,
            )

Well I hope that that made sense.

When you run the tests, they’re saved to the ./screenshots/ directory, which I keep out of source control because storing so many binary files is a heavy burden on git. We experimented with  git-annex but it turned out to be more trouble than it was worth.

My typical workflow goes like this:

1. Make sure my reference baseline screenshots are up to date: git checkout master && grunt && invoke needle --make
2. Generate screenshots for my feature branch: git checkout fat-buttons && grunt && invoke needle
3. Open the screenshots directory and compare screenshots.

In that workflow,  grunt is used to generate css,  invoke is used as my test runner, and  --make is a flag I built into the needle invoke task to make baseline screenshots.

Now I can quickly see if a change has the desired effect for multiple browser widths faster than it takes to actually resize a browser window. Bonus: I can see if a change has undesired effects on pages that I would have been too lazy to test manually.

Next steps: I still haven’t figured out how to run the same test in multiple browsers.