Tag Archives: testing

Nerd

Django Nose without Django-Nose

Tycho Brahe

I’ve grown to dislike Django-Nose. It’s been over three months since Django 1.8 has been released and they still don’t have a release that fully supports it. These are the advantages they currently tout:

  • Testing just your apps by default, not all the standard ones that happen to be in INSTALLED_APPS
    • The Django test runner has been doing this since 1.6 https://docs.djangoproject.com/en/1.8/releases/1.6/#discovery-of-tests-in-any-test-module
  • Running the tests in one or more specific modules (or apps, or classes, or folders, or just running a specific test)
    • They all can do this, even the old Django test runner
  • Obviating the need to import all your tests into tests/__init__.py. This not only saves busy-work but also eliminates the possibility of accidentally shadowing test classes.
    • The Django test runner has this since 1.6
  • Taking advantage of all the useful nose plugins
    • There are some cool plugins
  • Fixture bundling, an optional feature which speeds up your fixture-based tests by a factor of 4
    • Ok, Django doesn’t have this, but you shouldn’t be using fixtures anyways and there are other ways to make fixtures faster
  • Reuse of previously created test DBs, cutting 10 seconds off startup time
    • Django can do this since 1.8 https://docs.djangoproject.com/en/1.8/releases/1.8/#tests
  • Hygienic TransactionTestCases, which can save you a DB flush per test
    • Django has had this since 1.6 https://docs.djangoproject.com/en/1.6/topics/testing/tools/#django.test.TransactionTestCase
  • Support for various databases. Tested with MySQL, PostgreSQL, and SQLite. Others should work as well.
    • Django has had this forever

So what if you need a certain nose plugin? Say, xunit for Jenkins or some other tooling? Well, you still have to use Nose because django-jux hasn’t been updated in 4 years.

Here’s a small script you can use that lets you use Django + Nose while skipping the problematic Django-nose:

Run it like you would Nose:

DJANGO_SETTING_MODULE=settings.test python runtests.py --with-xunit --with-cov

One choice I made is that I use Django 1.8’s --keepdb flag instead of the REUSE_DB environment variable, but you can see how to adapt it if you wanted it to feel more like Nose. Adapting the command above to reuse the database would look like:

DJANGO_SETTING_MODULE=settings.test python runtests.py --with-xunit --with-cov --keepdb

Case Study Nerd

How to Ignore the Needle Docs

At PyCon 2014, I learned about a package called “needle” from Julien Phalip’s talk, Advanced techniques for Web functional testing. When I tried using it with a Django project, I immediately ran into problems:

  1. The needle docs aren’t written for Django, so they don’t explain how to use NeedleTestCase with LiveServerTestCase.
  2. I wasn’t using nose as my test runner, and didn’t want to start using it just to run Needle.

The first problem turned out to be easy; use both:

class SiteTest(NeedleTestCase, LiveServerTestCase):
    pass

The second problem wasn’t that bad either. If you examine the Nose plugin Needle adds, it just adds a  save_baseline attribute to the test case.

There were a lot of random hacks and tweaks I threw together. I think the best way to show them all is with an annotated example:

import os
import unittest

from django.test import LiveServerTestCase
from needle.cases import NeedleTestCase


# This is a configuration variable for whether to save the baseline screenshot
# or not. You can flip it by manually changing it, with an environment variable
# check, or monkey patching.
SAVE_BASELINE = False


# You should be taking screenshots at multiple widths to capture all your
# responsive breakpoints. Only the width really matter,s but I include the
# height for completeness.
SIZES = (
    (1024, 800),  # desktop
    (800, 600),  # tablet
    (320, 540),  # mobile
)


# To keep the test runner from running slow needle tests every time, decorate
# it. In this example, 'RUN_NEEDLE_TESTS' has to exist in your environment for
# these tests to run. So you would run needle tests like:
#
#     RUN_NEEDLE_TESTS=1 python manage.py test python.import.path.to.test_needle
@unittest.skipUnless('RUN_NEEDLE_TESTS' in os.environ, 'expensive tests')
class ScreenshotTest(NeedleTestCase, LiveServerTestCase):
    # You're going to want to make sure your pages look consistent every time.
    fixtures = ['needle.json']

    @classmethod
    def setUpClass(cls):
        """
        Sets `save_baseline`.

        I don't remember why I did it here. Maybe the timing didn't work when
        I put it as an attribute on the test class.
        """
        cls.save_baseline = SAVE_BASELINE
        super(ScreenshotTest, cls).setUpClass()

    def assertResponsive(self, scope, name):
        """Takes a screenshot for every responsive size you set."""
        for width, height in SIZES:
            self.set_viewport_size(width=width, height=height)
            try:
                self.assertScreenshot(
                    scope,
                    # include the name and browser in the filename
                    '{}_{}_firefox'.format(name, width)
                )
            except AssertionError as e:
                print(e)
                # suppress the error so needle keeps making screenshots. Needle
                # is very fickle and we'll have to judge the screenshots by eye
                # anyways instead of relying on needle's pixel perfect
                # judgements.
                pass

    def test_homepage(self):
        urls_to_test = (
            ('/', 'homepage'),
            ('/login/', 'login'),
            ('/hamburger/', 'meat'),
            ('/fries/', 'potatoes'),
            ('/admin/', 'admin'),
        )
        for url, name in urls_to_test:
            self.driver.get(self.live_server_url + url)
            self.assertResponsive(
                # for now, I always want the full page, so I use 'html' as the
                # scope for my screenshots. But as I document more things,
                # that's likely to change.
                'html',
                # passing in a human readable name helps it generate
                # screenshots file names that make more sense to me.
                name,
            )

Well I hope that that made sense.

When you run the tests, they’re saved to the ./screenshots/ directory, which I keep out of source control because storing so many binary files is a heavy burden on git. We experimented with  git-annex but it turned out to be more trouble than it was worth.

My typical workflow goes like this:

  1. Make sure my reference baseline screenshots are up to date: git checkout master && grunt && invoke needle --make
  2. Generate screenshots for my feature branch: git checkout fat-buttons && grunt && invoke needle
  3. Open the screenshots directory and compare screenshots.

In that workflow,  grunt is used to generate css,  invoke is used as my test runner, and  --make is a flag I built into the needle invoke task to make baseline screenshots.

Now I can quickly see if a change has the desired effect for multiple browser widths faster than it takes to actually resize a browser window. Bonus: I can see if a change has undesired effects on pages that I would have been too lazy to test manually.

Next steps: I still haven’t figured out how to run the same test in multiple browsers.