{"id":793,"date":"2016-06-09T03:54:19","date_gmt":"2016-06-09T03:54:19","guid":{"rendered":"https:\/\/www.crccheck.com\/blog\/?p=793"},"modified":"2016-06-11T03:09:29","modified_gmt":"2016-06-11T03:09:29","slug":"requirements-txt","status":"publish","type":"post","link":"https:\/\/www.crccheck.com\/blog\/requirements-txt\/","title":{"rendered":"requirements.txt==2.0"},"content":{"rendered":"<p>There is a war going on. A war between those that say Python requirements should be explicit and those that say requirements should be implicit. Before I continue, I&#8217;m going to be talking about <code>requirements.txt<\/code>, not <code>setup.py<\/code>. The difference between explicit and implicit requirements comes down to whether the line says <code>Django==1.9.7<\/code> or <code>Django<\/code>, respectively. Going deeper, you could also say that adding dependencies of dependencies is explicit, and you could loosely pin like <code>Django&lt;1.10.<\/code><\/p>\n<p>The advantage of explicit requirements is you get a repeatable environment. Especially if you&#8217;re also specifying dependencies of dependencies. The advantages of implicit requirements are readability and automatic security upgrades.<\/p>\n<p>Here at TabbedOut, we&#8217;ve developed a technique that works very well I&#8217;d like to share: Use <code>pip-tools<\/code> to manage your requirements. You get the best of both worlds, at the expense of some extra boilerplate. Here&#8217;s how we do it:<\/p>\n<ol>\n<li>Be in a virtualenv<\/li>\n<li>Use our Makefile boilerplate (see below)<\/li>\n<li><code>pip install pip-tools<\/code><\/li>\n<li>Write a &#8220;sloppy&#8221; requirements.txt using implicit requirements, but name it <code>requirements.in<\/code><\/li>\n<li>Run <code>make requirements.txt<\/code><\/li>\n<li>Check all this into your codebase<\/li>\n<\/ol>\n<h2>Advantages<\/h2>\n<ul>\n<li><code>requirements.in<\/code> is easy to maintain<\/li>\n<li><code>requirements.txt<\/code> has pinned versions so your virtualenv matches your collaborators and production<\/li>\n<li>You automatically get patches and security fixes when you run <code>make requirements.txt<\/code>, and there are no surprises because it goes through your code review process<\/li>\n<\/ul>\n<h2>Tips<\/h2>\n<ul>\n<li>Try to loosely pin requirements in your <code>requirements.in<\/code>. Though it doesn&#8217;t matter that much because you&#8217;ll catch it when you see a major version change in <code>requirements.txt<\/code>.<\/li>\n<li>Specifying an exact version in <code>requirements.in<\/code> is an anti-pattern, and you should document why. Often it&#8217;s because there&#8217;s a bug or backwards-incompatible change.<\/li>\n<\/ul>\n<h3>Makefile boilerplate<\/h3>\n<p>Here&#8217;s what a Makefile might contain:<\/p>\n<pre><code>help: ## Shows this help\r\n\t@echo \"$$(grep -h '#\\{2\\}' $(MAKEFILE_LIST) | sed 's\/: #\\{2\\} \/\t\/' | column -t -s '\t')\"\r\n\r\ninstall: ## Install requirements\r\n\t@[ -n \"${VIRTUAL_ENV}\" ] || (echo \"ERROR: This should be run from a virtualenv\" &amp;&amp; exit 1)\r\n\tpip install -r requirements.txt\r\n\r\n.PHONY: requirements.txt\r\nrequirements.txt: ## Regenerate requirements.txt\r\n\tpip-compile --upgrade --output-file $@ requirements.in\r\n<\/code><\/pre>\n<ul>\n<li><strong>help:<\/strong> This is just a fast way of making your Makefile self-documenting.<\/li>\n<li><strong>install:<\/strong> Nowadays, you need Python and non-Python requirements. Putting it all in one make target makes it easier for developers to jump into a project.<\/li>\n<li><strong>PHONY:<\/strong> When you run <code>make requirements.txt<\/code>, you want it to run every time. Not just when <code>requirements.in<\/code> changes. That&#8217;s because new versions may have been uploaded to PyPI. I always group my PHONY with my target. Even though it adds more lines, your Makefile will be more maintainable because you&#8217;re not trying to keep a list off the screen up to date.<\/li>\n<li><strong>requirements.txt:<\/strong> Why <code>make requirements.txt<\/code> over <code>make requirements<\/code>? Because best practice dictates that if the output of a make target is a file, that file should also be the name of the target. That way, you can use the automatic variable <code>$@<\/code> and it&#8217;s explicit, even at the cost of needing the PHONY.<\/li>\n<li><strong>&#8211;upgrade:<\/strong> Without this, <code>pip-tools<\/code> doesn&#8217;t actually upgrade your dependencies.<\/li>\n<li><strong>&#8211;output-file $@:<\/strong> <code>pip-tools<\/code> does this by default, but explicit is better than implicit. I would prefer to do <code>pip-compile --upgrade requirements.in &gt; $@<\/code> but <code>pip-tools<\/code> 1.6 does a poor job of dealing with stdout (see below).<\/li>\n<\/ul>\n<h2>Caveats<\/h2>\n<ul>\n<li>When you change <code>requirements.in<\/code>, you do have to remember to run <code>make requirements<\/code>, but you could automate that with a git-hook or CI process. In practice, we&#8217;ve found that running <code>make requirements.txt<\/code> is fine.<\/li>\n<li><code>pip-tools==1.6<\/code> does not work with the latest pip (8.1.2). See <a href=\"https:\/\/github.com\/nvie\/pip-tools\/issues\/358\">#358<\/a><\/li>\n<li><code>pip-tools==1.6<\/code> has a poor understanding of how stdin and stdout are supposed to work. Hopefully this gets fixed soon but is only a minor annoyance. <a href=\"https:\/\/github.com\/nvie\/pip-tools\/issues\/362\">#362<\/a> <a href=\"https:\/\/github.com\/nvie\/pip-tools\/issues\/360\">#360<\/a> <a href=\"https:\/\/github.com\/nvie\/pip-tools\/issues\/353\">#353<\/a> <a href=\"https:\/\/github.com\/nvie\/pip-tools\/issues\/104\">#104<\/a><\/li>\n<li>The compilation step can depend on your platform. I&#8217;ve only noticed this with <code>ipython<\/code>, which needs packages for interacting with the terminal like <code>gnureadline<\/code>. It hasn&#8217;t been trouble for us, but it could be for you. A workaround is to run the process in a Docker container.<\/li>\n<\/ul>\n<h2>Sample Scenarios<\/h2>\n<p>If you need more convincing, here are some problems this approach solves for us:<\/p>\n<p><strong>I thought I was getting xyzpackage version 3, why is version 2 getting installed?<\/strong> Pip tools flattens all your requirements, and annotates which package specified what. So in <code>requirements.txt<\/code>, you&#8217;ll see <code>xyzpackage==2.4\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0 # via scumbagpackage<\/code> and see that <code>scumbagpackage<\/code> was responsible.<\/p>\n<p><strong>What packages am I actually using?<\/strong> In a large project, your <code>requirements.txt<\/code> will balloon as you run into bugs and start pinning dependencies of dependencies. Then one day, you&#8217;ll realize you don&#8217;t know what packages you&#8217;re actually using. With a much simpler <code>requirements.in<\/code>, there&#8217;s less to sort through and fully pinned packages stick out like sore thumbs.<\/p>\n<p><strong>It works for me<\/strong> Sometimes a project will work only for you. You check your installed versions against <code>requirements.txt<\/code> and they match. But what you didn&#8217;t realize is a dependency of a dependency broke something. Since <code>pip-tools<\/code> freezes <em>everything<\/em>, you&#8217;ll have the same version of every package installed. And if something does break, you&#8217;ll have history to trace down what changed.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>There is a war going on. A war between those that say Python requirements should be explicit and those that say requirements should be implicit. Before I continue, I&#8217;m going to be talking about requirements.txt, not setup.py. The difference between explicit and implicit requirements comes down to whether the line says Django==1.9.7 or Django, respectively.&hellip;<\/p>\n <a href=\"https:\/\/www.crccheck.com\/blog\/requirements-txt\/\" title=\"requirements.txt==2.0\" class=\"entry-more-link\"><span>Read More<\/span> <span class=\"screen-reader-text\">requirements.txt==2.0<\/span><\/a>","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"Layout":"","footnotes":""},"categories":[50],"tags":[48,82],"class_list":["entry","author-showmewhatyougot","post-793","post","type-post","status-publish","format-standard","category-best-practices","tag-python","tag-tabbedout"],"_links":{"self":[{"href":"https:\/\/www.crccheck.com\/blog\/wp-json\/wp\/v2\/posts\/793","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.crccheck.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.crccheck.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.crccheck.com\/blog\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.crccheck.com\/blog\/wp-json\/wp\/v2\/comments?post=793"}],"version-history":[{"count":8,"href":"https:\/\/www.crccheck.com\/blog\/wp-json\/wp\/v2\/posts\/793\/revisions"}],"predecessor-version":[{"id":801,"href":"https:\/\/www.crccheck.com\/blog\/wp-json\/wp\/v2\/posts\/793\/revisions\/801"}],"wp:attachment":[{"href":"https:\/\/www.crccheck.com\/blog\/wp-json\/wp\/v2\/media?parent=793"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.crccheck.com\/blog\/wp-json\/wp\/v2\/categories?post=793"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.crccheck.com\/blog\/wp-json\/wp\/v2\/tags?post=793"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}