{"id":292,"date":"2013-04-12T00:03:28","date_gmt":"2013-04-12T06:03:28","guid":{"rendered":"http:\/\/www.crccheck.com\/blog\/?p=292"},"modified":"2015-10-11T22:23:32","modified_gmt":"2015-10-12T04:23:32","slug":"dissecting-elevators-part-2-ez-scrape","status":"publish","type":"post","link":"https:\/\/www.crccheck.com\/blog\/dissecting-elevators-part-2-ez-scrape\/","title":{"rendered":"Dissecting Elevators part 2: EZ Scrape"},"content":{"rendered":"<blockquote><p><em>Intro<\/em>: This is part two of an eight part series looking at the <a href=\"http:\/\/elevators.texastribune.org\/\" target=\"_blank\">Elevator Explorer<\/a>, a fun data interactive mostly coded between the hours of 10 PM to 2 AM during the week leading up to April Fools&#8217; Day, 2013. I&#8217;m going to be looking at the things I learned, things I wish I could have done, and the reasoning behind my design choices. The code I&#8217;ll be referring to will be in this <a href=\"https:\/\/github.com\/texastribune\/tx_elevators\/tree\/2013-april-fools\">tagged release on github<\/a>.<\/p><\/blockquote>\n<h1>Scraping Data<\/h1>\n<p>Having worked on many data apps now, there&#8217;s always that initial excitement of figuring out how to import a dataset. But soon, the months (or hours) pass, and you forget how to do the import. Well, that&#8217;s no good. Elevators was my latest iteration on doing simple, repeatable data updates. I say latest iteration, because while I&#8217;ve managed to improve, there&#8217;s plenty to improve upon. For this post, I&#8217;ll cover my process up to the actual data import.<\/p>\n<h2>tx_elevators_data.git<\/h2>\n<figure id=\"attachment_422\" aria-describedby=\"caption-attachment-422\" style=\"width: 143px\" class=\"wp-caption alignright\"><a href=\"http:\/\/threewordphrase.com\/pardonme.htm\" rel=\"attachment wp-att-422\"><img loading=\"lazy\" decoding=\"async\" class=\"size-full wp-image-422\" alt=\"Because of Reasons\" src=\"http:\/\/www.crccheck.com\/blog\/wp-content\/uploads\/2013\/04\/because-of-reasons.png\" width=\"143\" height=\"184\" srcset=\"https:\/\/www.crccheck.com\/blog\/wp-content\/uploads\/2013\/04\/because-of-reasons.png 239w, https:\/\/www.crccheck.com\/blog\/wp-content\/uploads\/2013\/04\/because-of-reasons-232x300.png 232w\" sizes=\"auto, (max-width: 143px) 100vw, 143px\" \/><\/a><figcaption id=\"caption-attachment-422\" class=\"wp-caption-text\">by Ryan Pequin<\/figcaption><\/figure>\n<p>If your source data is in plain text and has line breaks, you should keep it in source control. This is an extra step, but there are reasons:<\/p>\n<ul>\n<li><em>There&#8217;s history.<\/em> I can look at the previous state of the data and see when they fixed a typo.<\/li>\n<li><em>Can quickly see what happened.<\/em> Browsing the diff gives you insight into how the data changes.<\/li>\n<li><em>Adds portability.<\/em> If I had to pull 12 different data sources, I can tell a coworker to do one pull from my git remote to get data in exactly the same state I have it.<\/li>\n<\/ul>\n<p>To do updates, I use Make. Mike Bostock has a good post on why to use Make called: <em><a href=\"http:\/\/bost.ocks.org\/mike\/make\/\" target=\"_blank\">Why Use Make<\/a><\/em>. To do source control, I use git. You <em>could<\/em> host it online at github or bitbucket, but that feels wrong. What I do instead is make a bare git repo in Dropbox as my remote <code>origin<\/code>. To share the makefile, I put a copy of it in the main tx_elevators repo. You can see it at <a href=\"https:\/\/github.com\/texastribune\/tx_elevators\/blob\/2013-april-fools\/data\/Makefile\">https:\/\/github.com\/texastribune\/tx_elevators\/blob\/2013-april-fools\/data\/Makefile<\/a>.<\/p>\n<h2>Keeping the Diffs Cleans<\/h2>\n<p>After my first update, it was apparent that I was getting the data back in no particular order and my diffs were going to be unusable. The first thing I tried was a sort:<\/p>\n<pre><code>mv $@ $@.orig\r\nawk 'NR==1; NR &gt; 1 {print $0 | \"sort -n\"}' $@.orig &gt; $@\r\n<\/code><\/pre>\n<p>It worked, except I discovered that my data had one record with a line break that broke many things. So I turned to csvkit. The current iteration uses csvkit&#8217;s <code>csvsort<\/code>:<\/p>\n<pre><code>mv $@ $@.orig\r\ncat $@.orig | csvsort &gt; $@\r\n<\/code><\/pre>\n<p>It&#8217;s significantly slower, but not terrible, and it doesn&#8217;t break the data. For reference, the whole command only takes 31 seconds on my old machine:<\/p>\n<pre><code>$ time make clean elevator_data_file.csv\r\n...snip...\r\nreal    0m30.641s\r\nuser    0m19.841s\r\nsys     0m1.588s\r\n<\/code><\/pre>\n<h2>Improvements<\/h2>\n<p>Maintaining two copies of the data importer&#8217;s makefile is annoying. I&#8217;m still trying to figure out that one, but I have an idea. I think I can adjust my\u00c2\u00a0 <code>make scrape<\/code> <a href=\"https:\/\/github.com\/texastribune\/tx_elevators\/blob\/2013-april-fools\/Makefile#L27-L29\">command<\/a> to assume there&#8217;s a\u00c2\u00a0 <code>.\/tx_elevators_data<\/code> directory alongside <code>tx_elevators<\/code>, and fallback to <code> tx_elevators\/data<\/code>. I should also change the csv filenames, keeping elevator_data_file.csv the original and making a new elevator_data_file.sorted.csv.<\/p>\n<h2>Conclusion and Next Time&#8230;<\/h2>\n<p>Once again, I&#8217;ve chosen to use Make to automate things. I was hoping to go into actually importing the data here, but then this post would be way too long. The gist of it is <code>tx_elevator<\/code>&#8216;s makefile calls\u00c2\u00a0 <code>tx_elevators_data<\/code>&#8216;s makefile to make sure it has fresh data before importing it. The entire process is condensed to one command:\u00c2\u00a0 <code>make scrape<\/code>. In addition to making the process simple and repeatable, it&#8217;s also important for the it to be easy to understand. Document everything as if you expect contributors, because you&#8217;re going to forget how everything worked.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Intro: This is part two of an eight part series looking at the Elevator Explorer, a fun data interactive mostly coded between the hours of 10 PM to 2 AM during the week leading up to April Fools&#8217; Day, 2013. I&#8217;m going to be looking at the things I learned, things I wish I could&hellip;<\/p>\n <a href=\"https:\/\/www.crccheck.com\/blog\/dissecting-elevators-part-2-ez-scrape\/\" title=\"Dissecting Elevators part 2: EZ Scrape\" class=\"entry-more-link\"><span>Read More<\/span> <span class=\"screen-reader-text\">Dissecting Elevators part 2: EZ Scrape<\/span><\/a>","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"Layout":"","footnotes":""},"categories":[50,21,4],"tags":[54,48],"class_list":["entry","author-showmewhatyougot","post-292","post","type-post","status-publish","format-standard","category-best-practices","category-case-study","category-technical","tag-makefiles","tag-python"],"_links":{"self":[{"href":"https:\/\/www.crccheck.com\/blog\/wp-json\/wp\/v2\/posts\/292","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.crccheck.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.crccheck.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.crccheck.com\/blog\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.crccheck.com\/blog\/wp-json\/wp\/v2\/comments?post=292"}],"version-history":[{"count":19,"href":"https:\/\/www.crccheck.com\/blog\/wp-json\/wp\/v2\/posts\/292\/revisions"}],"predecessor-version":[{"id":750,"href":"https:\/\/www.crccheck.com\/blog\/wp-json\/wp\/v2\/posts\/292\/revisions\/750"}],"wp:attachment":[{"href":"https:\/\/www.crccheck.com\/blog\/wp-json\/wp\/v2\/media?parent=292"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.crccheck.com\/blog\/wp-json\/wp\/v2\/categories?post=292"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.crccheck.com\/blog\/wp-json\/wp\/v2\/tags?post=292"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}