Tag Archives: javascript

Finish Writing Me Plz Nerd

Demo: Fuckin A

I’ve been learning Ansible off and on and off for the past year. I have so many complaints, but I still think it’s the least worst provisioning system out there.

It’s online at crccheck.github.io/fuckingansible/, but I’ve also embedded it below:

I’d go more into Ansible, just just thinking about it make me so angry. So instead, I want to go over some of the new (JavaScript) things I learned working on this project.

LibSass

I’ve converted some projects from Sass to LibSass before, but this is the first time I used LibSass from the start. Not having to mess with a Gemfile and Bundler is so freeing. It’s too bad there are still so many bugs in the libsass, node-sass, grunt-sass chain. They finally fixed sass2scss in the summer, I think Sass maps are still messed up. Despite this, I would definitely go with LibSass first.

Grunt-Connect

This is the first project I’ve used `grunt-contrib-connect`. Prior, I would have just used `python -m SimpleHTTPServer`. What I like about using Connect is how it integrates with Grunt, and how I have better control over LiveReload without having to remember to enable my LiveReload browser extensions.

Browserify

I’ve experimented with Browserify before with lazycolor.com, where I had previously experimented with RequireJS and r.js. I was very happy with my experience with it for this project:

  1. It found my node modules automagically. I didn’t know it did that. I just tried it and it worked.
  2. It made writing JavaScript tests so much easier. I did something similar with text-mix, but that used Mocha, and I did not have a pleasant experience setting that up. Seriously, why are there so many Grunt/Mocha plugins? And why do so many of them just not work? This time, I used Nodeunit, which was available as a Grunt contrib plugin and a breeze to set up.

If you don’t need a DOM, writing simple assertion tests is definitely the way to go. If it’s faster to write tests, you’re more likely to write them. And best of all, since Browserify runs in an IIFE, it doesn’t put `module` or `define` into the global scope and mess up everything.

Case Study Nerd

Dissecting Elevators Part 7: Search + D3

With almost 25 thousand buildings, having a way to search the database would be nice. And just like how I did geo searches client side, searching buildings was done client side too. The inspiration for the interface was The Texas Tribune’s Tribpedia landing/search page. The final building search page can be found at elevators.texastribune.org/building/ and the JavaScript source can be found at github.com/texastribune/tx_elevators/blob/2013-april-fools/tx_elevators/static/tx_elevators/js/search.js.

How it’s made

The one lonely chart on the home page isn’t the only place where D3 is used. The building search UI is built using D3. “But it looks like HTML, not SVG!” the astute reader might remark, and you would be correct. D3 is not just for SVG, it can be used for HTML too.

The first step was to build the NOSCRIPT version, which was just a list with links. This list also serves as a way to map building ELBI IDs to urls and as a map for search engines. This was partially because the original API didn’t have a way to return urls, and because I wanted to keep the json light. As a side note: When I moved to putting slugs in the building URLs, I had to go back and modify my geolocation API to return building URLs (and thanks to generators, it was not hard to do).

The first thing I did was bin each building by name. The important thing to note is that I split it up into several steps. The binData function organizes the raw data into bins, which almost parallels the structure of the final HTML. The binned data then gets passed to the prepNameData function which finesses that data to look exactly like the structure of the HTML. The next block of code, prepBinsHtml, is just basic D3 to to map the finessed binned data to the final HTML. The rest of the file is just jQuery for the user interaction. This demonstrates the power of D3: Once I had the code to map the one set of raw data to HTML, changing the data, either by filtering it for search or re-binning the data, just magically worked.

Performance details

I didn’t have a lot of time to make tweaks, or even do basic documentation, but I did make one optimization: I kept the number of dom elements to a minimum. I used divs for items to avoid ul>li nesting, and I didn’t use a tags because it would have doubled the number of elements needed for each building. And when you have nearly 25,000 buildings, it matters. The major downside is I had to fake a tag behavior, which is still incomplete. For example, you can’t middle-click to open in a new tab, and the hover interaction is only partially there. Much of that optimization was inspired by these slides by Jon Rohan of GitHub. And if you compare the interaction of the Tribpedia landing/search page with the building search page, you’ll see that this approach results in a much faster experience.

Case Study Nerd

Dissecting Elevators Part 6: GeoDjango? We don’t need no stinkin’ GeoDjango

Being able to locate the nearest elevator was a must have feature for me. Being able to deploy using the free tier on Heroku was a nice to have feature. Luckily, I managed to find a way to do both: do the calculations client side in JavaScript.

Lucky for me, I found a post, Calculate distance, bearing and more between Latitude/Longitude points, with some JavaScript algorithms on how to get distances between two lat/long points. There’s no such thing as perfect distance algorithm, but if you start making assumptions about the shape of the Earth, you can get a pretty good guess. I implemented four different ways to calculating distance; each a tradeoff between accuracy and speed. The four ways being: the Haversine formula, the spherical law of cosines, the equirectangular projection (Pythagorean), and rectilinear (Taxicab) distance. For my own reference, I’m putting a copy of the JavaScript algorithms here:

// distance approximators
/** Converts numeric degrees to radians */
if (!Number.prototype.toRad){
  Number.prototype.toRad = function(){
    return this * Math.PI / 180;
  };
}

var distance = {
  R: 6371,  // kilometers
  haversine: function(lat1, lng1, lat2, lng2){
    var dLat = (lat2-lat1).toRad();
    var dLon = (lng2-lng1).toRad();
    lat1 = lat1.toRad();
    lat2 = lat2.toRad();

    var a = Math.sin(dLat/2) * Math.sin(dLat/2) +
      Math.sin(dLon/2) * Math.sin(dLon/2) * Math.cos(lat1) * Math.cos(lat2);
    var c = 2 * Math.atan2(Math.sqrt(a), Math.sqrt(1-a));
    return distance.R * c;
  },
  spherical: function(lat1, lng1, lat2, lng2){
    lat1 = lat1.toRad();
    lat2 = lat2.toRad();
    lng1 = lng1.toRad();
    lng2 = lng2.toRad();
    return Math.acos(Math.sin(lat1)*Math.sin(lat2) +
      Math.cos(lat1) * Math.cos(lat2) *
      Math.cos(lng2 - lng1)) * distance.R;
  },
  pythagorean: function(lat1, lng1, lat2, lng2){
    lat1 = lat1.toRad();
    lng1 = lng1.toRad();
    lat2 = lat2.toRad();
    lng2 = lng2.toRad();
    var x = (lng2 - lng1) * Math.cos((lat1 + lat2) / 2),
        y = lat2 - lat1;
    return Math.sqrt(x * x + y * y) * distance.R;
  },
  taxicab: function(lat1, lng1, lat2, lng2){
    lat1 = lat1.toRad();
    lng1 = lng1.toRad();
    lat2 = lat2.toRad();
    lng2 = lng2.toRad();
    return (Math.abs(lat1 - lat2) + Math.abs(lng1 - lng2)) * distance.R;
  }
};

For more information about the first three distance metrics, you should check out the original post. You can see that the computational complexity for each algorithm decreases dramatically, from using trigonometry and a square root, to pure trig, to a simple trig and a square root, to basic arithmetic. The code for getting the ten closest buildings turned out pretty simple. Here, _data is a global array of all the buildings:

// Get the closest `Building`s to `lat` and `lng`.
//
// Modifies the global `_data` by storing the distance and also sorts it.
var closestBuildings = function(lat, lng){
  var metric = distance.spherical, x;
  for (var i = 0; i < _data.length; i++){
    x = _data[i];
    x.distance = metric(lat, lng, x.latitude, x.longitude);
  }
  // go ahead and sort in place.
  _data.sort(function(a, b){ return a.distance - b.distance; });
  return _data.slice(0, 10);
};

Evaluating the different distance metrics

Just to see what the difference was between the four different distance metrics, I compared the result of the same search. I picked an arbitrary point where there weren’t too many elevators so it would have to go out a long ways before getting 10 elevators. I like BBQ, so I picked a place in Lockhart, TX. To my surprise, not only did the Haversine, spherical law of cosines, and equirectangular projection give the same results in the same order, but they also gave the same distances to 4 significant digits (your results on the live site may look different from mine because of differences between the data on my local machine and the live site):

Closest Elevators to Lockhart, TX
Closest elevators to Lockhart

The results of the search using the taxicab metric were very not bad either. You can see that the taxicab metric punishes going diagonally:

Closest elevators to Lockhart using the taxicab metric
Closest elevators to Lockhart using the taxicab metric

They were even better (obviously) over shorter distances, as you can see in this comparison for downtown Austin:

Closest elevators, downtown Austin
Closest elevators, downtown Austin
Closest elevators, downtown Austin, using the taxicab metric
Closest elevators, downtown Austin, using the taxicab metric

You can see I got the same results in a slightly different order.

Trickery!

If you’re wondering how I got the map pins to go from A to J and match the labels on the legend, wonder no more. The answer is: I cheated. I created the map pins using the old Google charts API in my mapping code:

"http://chart.apis.google.com/chart?chst=d_map_pin_letter&chld=" +
        String.fromCharCode(65 + i) + "|" + pinColor,

If you vary i from 0 to 9, you get the letters A to J. The legend is an ordered list with the css: list-style: upper-alpha, which handles going from A to J on the map legend. Since both are built from the same source in the same order, they match up.

Conclusion

And there you have it. Basic closest point searches done client side. The nearly fully geocoded dataset with 23,205 buildings takes a megabyte to transfer, but it could be optimized. At launch, I only had a fraction of the buildings geocoded, so it was less than 150 kilobytes then. I may change the default distance metric from spherical to Pythagorean to get some more speed. It would be an interesting exercise to convert to GeoDjango and compare the distance results again.

Case Study Nerd

Dissecting Elevators part 4: Django Data Driven Documents

Intro: This is part four of an eight part series looking at the Elevator Explorer, a fun data interactive mostly coded between the hours of 10 PM to 2 AM during the week leading up to April Fools’ Day, 2013. I’m going to be looking at the things I learned, things I wish I could have done, and the reasoning behind my design choices. The code I’ll be referring to will be in this tagged release on github.

Hey everybody, come look at the D3

So first off, this is going to go over the sketchiest code in the project. It seemed to work well for Elevators, but I have no idea how well this would translate to other projects.

At the bottom of the landing page, there’s this chart:

Texas Elevators Chart
The distribution of age vs floors for almost every elevator in Texas.

If you examine the source, you’ll see that it’s just a DIV with data attrs that the JavaScript uses to create an IFRAME version of the real chart full of D3 magic.

Responsive Iframes

So why use an iframe? Why not just load the chart directly into the container div? The iframe may cost us a bunch of extra HTTP requests and make it harder to integrate with the page*, but they also give us an instant embeddable. But this is the era of responsive design, and iframes are even harder to make responsive than SVGs. My solution was to insert the iframe with JavaScript. I could have just operated directly on the IFRAME, but adding another layer (hooray more complexity) also adds the ability to serve different content based on feature detection (noscript, no svg, mobile) in the future. Why didn’t I use a preexisting library like NPR’s responsiveiframe? They were all too complicated to use (e.g. I couldn’t get them to work in 5 minutes) and I didn’t have to worry about same origin issues so I could write simpler JavaScript:

https://github.com/texastribune/tx_elevators/blob/2013-april-fools/tx_elevators/static/tx_elevators/js/tx_elevators.js#L327-L363

With this technique, adding an interactive visualization is as easy as adding an image. How it works is you put markup into the page similar to how you would do a  VIDEO tag:

<section>
  <h2>Distribution of Age and Floors of Elevators</h2>
  <div data-src="/chart/elevatorlist/">
    <noscript>
      Missing Figure
    </noscript>
  </div>
</section>

https://github.com/texastribune/tx_elevators/blob/2013-april-fools/tx_elevators/templates/tx_elevators/landing.html#L46-L50

And the JavaScript knows how to interpret that to insert the chart.

*Integration problems with iframes:

  • Tooltips can’t extend outside the bounds of the iframe.
  • Interactivity between the chart and other elements becomes difficult.

One chart, one view, one queryset

Doing iframes for charts also solves another headache: managing all the querysets that power each visualization. Powering the visualizations off a generic api is an admirable goal, but falls flat in reality. And mixing in all the querysets you need into your views makes for some massive views and maintenance headaches. Writing simple single-purpose views keeps the payload small and the code neat. You may end up with a maintenance problem of another kind, but I would rather have 10 simple small views than one unintelligible mammoth view. You can see what I ended up doing here:

https://github.com/texastribune/tx_elevators/blob/2013-april-fools/tx_elevators/chart_views.py#L9-L45

The same basic Django TemplateView handles both the chart HTML and the data needed for the visualization. Another feature is the ajax endpoint for the chart is always at the same relative URL, so for the visualization at: /chart/elevatorlist/, the data lives at /chart/elevatorlist/data.json. You don’t have to worry about moving information about your Django urlconf into JavaScript. And I’m foreshadowing here, but it’s at ./data.json and not ./data/ because it’s friendlier for wget. Another tip is to make sure you’re gzipping your json, especially for development. In my urls.py, you can see I’m using the gzip_page decorator. But you can also enable GZipMiddleware. This makes it much easier to know right away if your json is too large. For production, you should think about turning off gzipping in the application server and moving it up the stack.

Next steps

Like I said, this code was the sketchiest. There are many improvements to be made. Some big ones I can think of include:

  1. Making the chart loader JS capable of loading the chart directly into the container div instead of in an iframe.
  2. Adding a URL pattern helper for automatically setting up the routes for the chart template and data in one line.
  3. Making the API return an array of arrays instead of an array of objects to save bandwidth. It’s a little easier to throw into D3 this way too.

Live demo

Conclusion

So the title was a lie. I didn’t go into the D3 at all. Gotcha. If you’re curious, here it is. Next time, I’ll show you how a rarely used database from 1986 helped me geocode.

Nerd Portfolio

CodebaseHQ Tickets+ for FireFox

What’s this? Another release already? Well yes. I’ve been working on this project, and we use Codebase for project management. And the whole time, I kept thinking, “Why can’t I sort these tickets?!”. Apparently, I’m not the only one, because I overheard some other people say the same thing so I decided to make a Greasemonkey script.

While there is a mechanism to sort, it requires modifying your search query and a page refresh. My script not only does all that live, but it also turns the search box into a live-search, which makes finding a particular ticket fast. To accompany all this, I also de-paginate the results so all the tickets are in one page. This could potentially cause problems, because some projects might have thousands of tickets; but that’s a problem for another day.

While I have been on a jQuery spree recently, I made sure not to use it for this Greasemonkey script. Even though there is a lot of DOM manipulation, I rely on Xpath and JavaScript 1.6 Arrays. The script is simple enough where it’s easy enough to convert to jQuery if needed.

If you use FireFox, Greasemonkey, and CodebaseHQ, Give CodebaseHQ Tickets+ a shot.

Life=Boring Mental Note: Add Category Portfolio

Announcing allbfcards.com

So I’ve been playing this game called BattleForge, and there are a handful of websites that exist just to display information about these cards you can collect in the game (think Magic: The Gathering or Pokemon). People have also made Adobe AIR applications and Microsoft Access databases. So, naturally I thought I would pollute the web with another website.

At first, I was going to use xml + xslt to make it. I actually got a proof of concept working nicely. But once I tried using javascript, it got weird on me because it was XML instead of HTML. Then I changed it to plain HTML + an XHR request to get the XML file and got basic realtime filters working. Then someone pointed out that there was some google docs with the information I wanted so I switched from XML to JSONP.

Anyways, this is the limit of my attention span so I’ll paste the URL now. http://allbfcards.com/

Neato!

A new canvas demo

I finally got around to updating my canvas / sql visualization code. Interacting with the map is now much easier.

And I’ve posted a live demo of it here: http://crccheck.com/demo02

It’s not as refined as google maps, but it’s still pretty fancy. Navigation is done in the minimap, and you can draw on the large map with your mouse pointer.

This time, I made it work in FireFox and Chrome/Safari

Mental Note: Add Category Nerd Portfolio

Next adventure in HTML Canvas

This one comes with a live demo!

demo01

One of the biggest problems with my  last Canvas experiment is that if you put text labels over points of interest, they quickly clutter up and become difficult to read. In this demo, I take a list of labels and the coordinates they belong, then I treat each label as a like-charged ion and they automatically repel each other and find their own non-overlapping positions. I also put a spring force between each label (visualized by the red line) to its original position so they don’t stray too far. There are also some visual embellishments specific for the application.

In my next iteration, I’m going to try and properly attach events to each particle. I have rudimentary drag and drop now, but it’s not robust enough for other ideas I want to add. I also want to be able to add uncharged particles and have wind and gravity. So I can make the whole thing a proper particle playground.

Note: since the code was ripped from a Greasemonkey script, it’s FireFox only.

Update 18 July, 2010: I’ve made it a little more interactive, re-ported a more up to date version of the code, and made it somewhat compatible with Safari and Chrome.