You may not need d3

If you’re working on visualization on the web, the go-to choice is to use d3js (or a higher-level library). d3js is powerful and versatile. The best proof of that is the lead author of d3, Mike Bostock, worked until recently at the New York Times which many, including myself, consider the ultimate reference in terms of information graphics. At the NYT and elsewhere, d3js has powered breathtaking projects.

d3js was built as a successor to protovis. Protovis was a slightly higher-level library than d3js, more specialized in data graphics, and designed with the assumption that it could be used with little or no programming experience. And indeed this was true. When I started using protovis in 2009, my javascript skills were limited, and so I learned by deconstructing and recombining the examples. I kept on doing that when d3 came along – learning through examples. And the d3 examples, like those in protovis before, were short and standalone, demonstrating how easy it was to create something with a few lines of code.

d3js requires more technical knowledge than protovis did to get started, but not much. However, it is much more versatile – no longer constrained to charts and maps, it can handle a lot of the front-end functionalities of a web site. And so, it is possible to make visualizations by:

  • learning how to do stuff in d3js,
  • learning the essential javascript that you need to run d3js, as needed (i.e. variables, if-then statements, array methods…)
  • then, learning through doing, and obtain a feeling for how SVG, HTML and the Document Object Model works.

This approach is actually very limiting.

You would create much more robust code by:

  • learning javascript,
  • learning SVG/HTML/DOM,
  • then learning and appreciate d3 logic.

That would give you the choice when to use a d3js method, when to use another library (past, present or future), or when to use native javascript.

The point of this article is to go through the main functions of d3js and see how they can be replicated without using a library. As of this writing, I am convinced that using d3js is the most sensible way to handle certain tasks (though this may change in the future). In almost any case I can think of, even when it’s not the most efficient way to do so, using the d3js approach is the most convenient and the easiest from the developer’s perspective.  But I also believe it’s really important to know how to do all these things without d3js.

This will be a very long post. I’m trying to keep it as structured as possible so you can jump to the right section and keep it as a reference. This also assumes you have a certain familiarity with d3js and javascript – this is not an introductory post.

I’ve divided it in 4 parts:

  • Tasks that you really don’t need d3js for. And ok, you may want to use it regardless. But here’s how to do those things without it. (manipulating DOM elements)
  • interlude: a reflexion on d3js data binding. Do you really need that?
  • Tasks which are significantly easier with d3js (working with external data, animation)
  • Tasks where as of now, d3js is clearly superior to anything else (scales, array refactoring, map projections, layouts).

Selecting and manipulating elements

At the core of d3js is the notion of selection. In d3js, you select one or several elements which become a selection object. You can then do a variety of things:

  • join them with data, which is also an essential tenet of the d3 philosophy,
  • create or delete elements, especially in relation to the data you may have joined with their parent,
  • manipulate these elements by changing their attributes and their style, including in function of the data joined with each of them,
  • attach events listeners to those elements, so that when events happen (ie somebody clicks on a rectangle, or change the value of a form) a function might be triggered,
  • animate and transform those elements.

Selecting elements and parsing the DOM

d3 selection objects vs Nodes, HTML NodeList vs HTML LiveCollections

When you select something with d3js, using methods such as d3.select() or d3.selectAll(), this returns a d3 selection object. The selection object is a subclass of the javascript object Array – check here for all the nitty-gritty. The gist of it is that d3 selection objects can then be manipulated by d3 methods, such as .attr() or .style() to assign attributes or styles.

By contrast, the DOM is made of Node objects. The root of the DOM, the Document Object Model, is a special Node called Document. Nodes are organized in a tree-like structure: Document has children, which may have children etc. and encompass everything in the page. The Node objects are really the building blocks of a web page. When an element is added, be it an HTML element like a <div> or an SVG element like a <rect>, a new Node is created as well. So d3js has to interact with Node objects as well. d3 selection objects are tightly connected to their corresponding Node objects.

However, with “vanilla” javascript, you can directly access and manipulate the Node objects.

d3.select / d3.selectAll vs document.querySelector / document.querySelectorAll

In d3js, you parse the document using d3.select (to select one object) or d3.selectAll. The argument of this method is a CSS3 selector, that is, a string which is very similar to what could be found in a CSS style sheet to assign specific styles to certain situations. For instance, “g.series rect.mark” will match with all the rectangles of the class “mark” which are descendants of the SVG g groups of the class series.

When d3js was introduced in 2011, javascript didn’t have an equivalent syntax – instead you could select elements by class, or by id, or by tag name (more on that in a minute). Things have changed however and it is now possible to use CSS3 selectors using document.querySelector (which will return just one node) or document.querySelectorAll (which will return an HTML NodeList).

An HTML NodeList is a special object which is kind of like an array of Node objects, only it has almost no array methods or properties. You can still access its members using brackets, and get its length, but that’s it.

I wrote document.querySelectorAll, because you can use this method from the document, but you can use it from any Node. Those two snippets of code are parallel:

var svg = d3.select("svg"); // svg is a d3 selection object
var g = svg.selectAll("g"); // g is a d3 selection object
var svg = document.querySelector("svg"); // svg is a Node
var g = svg.querySelectorAll("g"); // g is a NodeList

 Getting elements by class name, ID, tag name, name attribute

d3js doesn’t have a special way to get all descendants of a selection of a certain class, of a certain ID, etc. The CSS3 selector syntax can indeed handle all those cases, so why have a separate way?

By contrast, javascript pre-2011 didn’t have a querySelectorAll method, and so the only way to parse the document was to use more specific method, like document.getElementsByClassName().

document.getElementsByClassName() retrieves all descendants of a certain class. document.getElementsByName() retrieves elements with a certain “name” attribute (think forms). documents.getElementsByTagName() gets all descendants of a certain type (ie all <div>s, all <rect>s, etc.).

What’s interesting about that is that what is returned is not an HTML NodeList like above with querySelectorAll, but another object called HTML Live Collection. The difference is that matching elements are created after, they would still be included in the Live Collection.

var svg = d3.select("svg");
svg.selectAll("rect").data([1,2,3]).enter().append("rect");
var mySelection = svg.selectAll("rect"); // 3 elements
mySelection[0].length // 3
svg.append("rect");
mySelection[0].length // 3
mySelection = svg.selectAll("rect"); // re-selecting to update it
mySelection[0].length // 4
var svg = document.querySelector("svg");
var ns = "http://www.w3.org/2000/svg";
var i;
for (i = 0; i < 3; i++) {
  var rect = document.createElementNS(ns, "rect"); // we'll explain creating elements later
  svg.appendChild(rect);
}
var mySelection = svg.getElementsByTagName("rect"); // 3 elements
var rect = document.createElementNS(ns, "rect");
svg.appendChild(rect);
mySelection.length // 4 - no need to reselect to update

How about IDs? there is also the getElementById (no s at elements!) which only retrieve one element. After all, IDs are supposed to be unique! if no elements match, getElementById returns null.

Children, parents, siblings…

Truth be told, if you can use selectors from the root, you can access everything. But sometimes, it’s nice to be able to go from one node to its parents or its children or its siblings, and d3js doesn’t provide that. By contrast, the Node object has an interface that does just that – node.childNodes gets a nodeList of child nodes, node.parentNode gets the parent node, node.nextSibling and node.previousSibling get the next and previous siblings. Nice.

However, most often you will really be manipulating elements (more on that in a second) and not nodes. What’s the difference? all Elements are Nodes, but the reverse is not true. One common example of Node which is not an Element is text content.

To get an Element’s children, you can use the (wait for it) children property. The added benefit is that what you get through children is a LiveCollection (dynamic), while what you get through childNodes is a NodeList (static).

var svg = document.querySelector("svg");
var ns = "http://www.w3.org/2000/svg";
var i;
for (i = 0; i < 3; i++) {
  var myRect = document.createElementNS(ns, "rect"); // we'll explain creating elements later
  svg.appendChild(rect);
}
// the variable myRect holds the last of the 3 <rect> elements that have been added
svg.childNodes; // a NodeList
myRect.parentNode; // the svg element
myRect.nextSibling; // null - myRect holds the last child of svg.
myRect.previousSibling; // the 2nd <rect> element
svg.firstChild; // the 1st <rect>. Really useful shorthand
svg.querySelector("rect"); // also the 1st <rect>.
svg.children; // a LiveCollection

Adding/reading attributes, styles, properties and events

Node, Element, EventTarget and other objects

In d3 101, right after you’ve created elements (to which we’ll come in a moment), you can start moving them around or giving them cool colors like “steelblue” by using the .attr and .style methods.

Wouldn’t that be cool if we could do the same to Node objects in vanilla javascript!

Well, we can. Technically, you can’t add style or attributes to Node objects proper, but to Element objects. The Element object inherits from the Node objects and is used to store these properties. There is also an HTMLElement and SVGElement which inherit from the Element object.

If you look at the Chrome console at an SVG element like a rect, you can see, in the properties tab, all the objects it inherits from: Object, EventTarget, Node, Element, SVGElement, SVGGraphicsElement, SVGGeometryElement, SVGRectElement, rect.

All have different roles. To simplify: Node relates to their relative place in the document hierarchy, EventTarget, to events, and Element and its children, to attributes, style and the like. The various SVG-prefixed objects all implement specific methods and properties. When we select a Node object as we’ve done above with svg.querySelector(“rect”) and the like, note that there’s not a Node object on one side, then an Element object somewhere else, a distinct SVGGeometryElement, and so on and so forth. What is retrieved is one single object that inherits all methods and properties of Nodes, Elements, EventTargets, and so on and so forth, and, as such, that behaves like a Node, like an Element, etc.

Setting and getting attributes

You can set attributes with the Element.setAttribute method.

var rect = document.querySelector("rect");
rect.setAttribute("x", 100);
rect.setAttribute("y", 100);
rect.setAttribute("width", 100);
rect.setAttribute("height", 100);

To be honest, I’m a big fan of the shorthand method in d3js,

var rect = d3.select("rect");
rect.attr({x: 100, y: 100, width: 100, height: 100});

Also, the Element.setAttribute method doesn’t return anything, which means it can’t be chained (which may or may not be a bad thing, though it’s definitely a change for d3js users). It’s not possible to set several attributes in one go either, although one could create a custom function, or, for the daring, extend the Element object for that.

Likewise, the Element object has a getAttribute method :

rect.getAttribute("x"); // 100

Classes, IDs and tag names

Classes, IDs and tag names are special properties of the Element objects. It’s extremely common to add or remove classes to elements in visualization: my favorite way to do that is to use the classed method in d3js.

d3.select("rect").classed("myRect", 1)

In vanilla javascript, you have the concept of classList.

document.querySelector("rect").classList; // ["myRect"]

ClassList has a number of cool methods. contains checks if this Element is of a certain class, add adds a class, remove removes a class, and toggles, well, toggles a class.

document.querySelector("rect").classList.contains(["myRect"]); // true
document.querySelector("rect").classList.remove("myRect");
document.querySelector("rect").classList.add("myRect");
document.querySelector("rect").classList.toggle("myRect");

How about IDs ? with d3js, you’d have to treat them as any other property (rect.attr(“id”)). In vanilla javascript, however, you can access it directly via the id property of Element. You can also do that with the name property.
Finally, you can use the tagName to get the type of element you are looking at (though you cannot change it – you can try, it just won’t do anything).

document.querySelector("rect").id = "myRect"; // true
document.querySelector("rect").name; // undefined;
document.querySelector("rect").tagName; // "rect"
document.querySelector("rect").tagName = "circle";
document.querySelector("rect").tagName; // "rect"

Text

Text is a pretty useful aspect of visualization! it is different from attributes or styles, which are set in the opening tag of an element. The text or content is what is happening in between the opening and closing tags of that element. This is why, in d3js, text isn’t set using attr or style, but either by the html method for HTML elements like DIVs or Ps, or by the text method for SVG elements like <text> and <tspan>.

Those have equivalent in the DOM + javascript world.

HTMLelements have the .innerHTML and outerHTML properties. The difference between the two is that outerHTML includes the opening and closing tags. innerHTML and outerHTML both return HTML, complete with tags and syntax.

SVG elements, however, don’t have access to this property, so they have to rely on the Node property textContent. HTML elements also have access to it, by the way. textContent returns just the plain text content of what’s in the element. All three properties can be used to either get or set text.

Style

In d3js, setting styles to elements is very similar to setting attributes, only we use the .style method instead of the .attr one. It’s so much similar that it’s a rather common mistake to pass as attribute what should be a style and vice-versa! Like with attributes, it is possible to pass an object with keys and values to set several style properties at once.

d3.selectAll("rect").style("fill", "red");
d3.selectAll("rect").style({stroke: "#222", opacity: .5});

In the world of DOM and vanilla javascript, style is a property of the HTMLElement / SVGElement objects. You can set style properties one at a time:

rect = document.querySelector("rect");
rect.style.fill = "red";
rect.style.stroke = "#222";
rect.style.opacity = .5;

Technically, .style returns a CSSStyleDeclaration object. This object maintains a “live” link to what it describes. So:

myStyle = rect.style;
rect.style.fill = "yellow";
myStyle.fill; // "yellow"

Finally, the window object has a getComputedStyle method that can get the computed styles of an element, ie how the element is actually going to get drawn. By contrast, the style property and the d3js style method only affect the inline styles of an element and are “blind” to styles of its parents.

myStyle = window.getComputedStyle(rect, null);
myStyle.fill; // "yellow"

Adding and removing events

In d3js, we have the very practical method “on” which let users interact with elements and can trigger behavior, such as transformations, filtering, or, really, any arbitrary function. This is where creating visualizations with SVG really shines because any minute interaction with any part of a scene can be elegantly intercepted and dealt with. Since in d3js, elements can be tied with data, the “on” methods takes that into account and passes the data element to the listener function. One of my favorite tricks when I’m developing with d3js and SVG is to add somewhere towards the end the line:

d3.selectAll("*").on("click", function(d) {console.log(d);})

Which, as you may have guessed, displays the data item tied to any SVG element the user could click on.

In the world of the DOM, the object to which events methods are attached in the EventTarget. Every Element is also an EventTarget (and EventTarget could be other things that handle events too, like xhr requests).

To add an event listener to an element, use the addEventListener method like so.

document
  .querySelector("rect")
  .addEventListener("click", function() {
     console.log("you clicked the rectangle!"
   }, false);

The first parameter is the type of event to listen to (just as in “on”), the second is the listener function proper. The third one, “use capture”, a Boolean, is optional. If set to true, it stops the event from propagating up and being intercepted by event listeners of the parents of this element.

There is also a “removeEventListener” method that does the opposite, and needs the same elements: in other words, yes, you need to pass the same listener function to be able to stop listening to the element. There is no native way to remove all event listeners from an element, although there are workarounds.

Creating and removing elements

Selecting and modifying elements is great, but if you are creating a visualization, chances are that you want to create elements from scratch.

Let’s first talk about how this is done in the DOM/javascript, then we’ll better understand the data joins and d3 angle.

Node objects can exist outside of the hierarchy of the DOM. Actually, they must first be created, then be assigned to a place in the DOM.

Until a Node object is positioned in the DOM, it is not visible. However, it can receive attributes, styles, etc. Likewise, a Node object can be taken from the DOM, and still manipulated.

To create an HTML element, we can use the document.createElement() method:

var myDiv = document.createElement("div");

However, that won’t work for SVG elements – remember in an earlier example, we used the createElementNS method. This is because SVG elements have to be created in the SVG namespace. d3js old-timers may remember that in the first versions, we had to deal with namespaces when creating elements in d3js as well, but now this all happens under the hood.

Anyway, in vanilla javascript, this is  how it’s done:

var svgns = "http://www.w3.org/2000/svg";
var myRect = document.createElementNS(svgns, "rect");

Warning, because document.createElement(“rect”) will not produce anything useful as of this writing.

Once the new Node objects are created, in order to be visible, they should be present in the DOM. Because the DOM is a tree, this means that they have to have a parent.

svg.appendChild(myRect);

Likewise, to remove a Node from the DOM means to sever that relationship with its parent, which is done through the removeChild method:

svg.removeChild(myRect);

Again, even after a Node has been removed, it can still be manipulated, and possibly re inserted at a later time.

Nodes don’t remove themselves, but you can write:

myRect.parentNode.removeChild(myRect);

In contrast, here is how things are done in d3js.

The append method will add one element to a parent.

d3.select("svg").append("rect");

The remove method will remove one entire selection object from the DOM.

d3.select("svg").selectAll("rect").remove(); // removes all rect elements which are children of the SVG element

But the most intriguing and the most characteristic way to create new elements in d3js is to use a data join.

d3.select("svg")
  .selectAll("rect")
  .data(d3.range(5))
  .enter()
  .append("rect");

The above snippet of code counts all the rect children of the svg element, and, if there are fewer than 5 – the number of items in d3.range(5), which is the [0,1,2,3,4] array – creates as many as needed to get to 5, and binds values to those elements – the contents of d3.range(5) in order. If there are already 5 rect elements, no new elements will be created, but the data assignment to the existing elements will still occur.

Data joins, or the lack thereof

The select / selectAll / data / enter / append sequence can sound exotic to people who learn d3js, but to its practitioners, it is its angular stone. Not only is it a quick way to create many elements (which, in vanilla javascript, takes at least 2 steps. Creating them, and assigning them to the right parent), but it also associates them with a data element. That data element can then be accessed each time the element is being manipulated, notably when setting attributes or styles and handling events.

For instance,

d3.selectAll("rect")
  .attr("x", function(d) {return 20 * d;});

the above code utilizes the fact that each of the rectangle have a different number associated with them to dynamically set an attribute, here position rectangles horizontally.

d3.selectAll("rect")
  .on("click", function(d) {console.log(d);})

A trick I had mentioned above, but which illustrates this point: here by clicking on each rectangle, we use the data join to show the associated data element.

Having data readily available when manipulating elements in d3js is extremely convenient. After all, data visualization is but the encoding of data through visual attributes. How to perform this operation without the comfort of data joins?

Simply by going back to the dataset itself.

Consider this:

var data = [];
var i;
for (i = 0; i < 100; i++) {
  data.push({x: Math.random() * 300, y: Math.random() * 300}); // random data points
}

// d3 way
var d3svg = d3.select("body").append("svg");
d3svg.selectAll("circle").data(data).enter().append("circle")
  .attr({cx: function(d) {return d.x;}, cy: function(d) {return d.y}, r: 2})
  .style({fill: "#222", opacity: .5});

// vanilla js way
var svgns = "http://www.w3.org/2000/svg";
var svg = document.createElementNS(svgns, "svg");
document.querySelector("body").appendChild(svg);
for (i = 0; i < 100; i++) {
  var myCircle = document.createElementNS(svgns, "circle");
  myCircle.setAttribute("cx", data[i].x);
  myCircle.setAttribute("cy", data[i].y);
  myCircle.setAttribute("r", 2);
  myCircle.style.fill = "#222";
  myCircle.style.opacity = .5;
  svg.appendChild(myCircle);
}

See the Pen eNzaYg by Jerome Cukier (@jckr) on CodePen.

Both codes are equivalent. Vanilla JS is also marginally faster, but d3 code is much more compact. In d3js, the process from dataset to visual is:

  • Joining dataset to container,
  • Creating as may children to container as needed, [repeat operation for as many levels of hierarchy as needed],
  • Use d3 selection objects to update the attributes and styles of underlying Node objects from the data items which have been joined to them.

    In contrast, in vanilla Javascript, the process is:

  • Loop over the dataset,
  • create, position and style elements as they are read from the dataset.

For visuals with a hierarchy of elements, the dataset may also have a hierarchy and could be nested. In this case, there may be several nested loops. While the d3js code is much more compact, the vanilla approach is actually more simple conceptually. Interestingly, this is the same logic that is at play when creating visualization with Canvas or with frameworks like React.js. To simply loop over an existing, invariant dataset enables you to implement a stateless design and take advantage of immutability. You don’t have to worry about things such as what happens if your dataset changes or the status that your nodes are in before creating or updating them. By contrast most operations in d3js assume that you are constantly updating a scene on which you are keeping tabs. In order to create elements, you would first need to me mindful on existing elements, what data is currently associated with them, etc. So while the d3js approach is much more convenient and puts the data that you need at your fingertips, the vanilla JS approach is not without merits.

Loading files

The first word in data visualization is data, and data comes in files, or database queries. If you’re plotting anything with more than a few datapoints, chances are you are not storing them as a local variable in your javascript. d3js has a number of nifty functions for that purpose, such as d3.csv or d3.json, which allow to load the files asynchronously. The trick in working with files is that it can take some time, so some operations can take place while we wait for the files to load, but some others really have to wait for the event that the file is loaded to start. I personally almost always use queue.js, also from Mike Bostock, as I typically have to load data from several files and a pure d3 approach would require nesting all those asynchronous file functions. But, for loading a simple csv file, d3js has a really simple syntax:

d3.csv("myfile.csv", function(error, csv) {
  // voila, the contents of the file is now store in the csv variable as an array
})

For reference, using queue js, this would look like

queue()
 .defer(d3.csv, "myFirstFile.csv")
 .defer(d3.csv, "mySecondFile.csv")
 .await(ready);

function ready(error, first, second) {
  // the contents of myFirstFile is stored as an array in the variable "first",
  // and the contents of mySecondFile are in the variable "second".
}

The way to do the equivalent in vanilla Javascript is to use XMLHttpRequest.

function readFile() {
  var fileLines = this.responseText.split("\n");
  var fields = fileLines[0].split(",");
  var data = fileLines.slice(1).map(function(d) {
    var item = {};
    d.split(",").forEach(function(v, i) {item[fields[i]] = v;})
    return item;
  })

  var request = new XMLHttpRequest();
  request.onload = readFile;
  request.open("get", "myFile.csv", true);
  request.send();

The syntax of loading the file isn’t that cumbersome, and there are tons of nice things that can be done through XMLHttpRequest(), but let’s admit that d3js/queue.js functions make it much more comfortable to work with csv files.

Animations

d3js transitions is one of my favorite part of the library. I understand it’s also one the things which couldn’t be done well in protovis and which caused that framework to break. It feels so natural: you define what you want to animate, all that needs to change, the time frame and the easing functions, and you’re good to go (see my previous post on animations and transitions). In native javascript, while you can have deep control of animations, it’s also, unsurprisingly, much more cumbersome. However, CSS3 provides an animation interface which is comparable in flexibility, expressiveness and ease of use to what d3js does. First let’s get a high-level view of how to do this entirely within JS. Then let’s get a sense of what CSS can do.

requestAnimationFrame and animation in JavaScript

JavaScript has timer functions, window.setTimeout and window.setTimeinterval, which let you run some code after a certain delay or every so often, respectively. But this isn’t great for animation. Your computer draws to screen a fixed number of times per second. So if you try to redraw the same element several times before in between those times, it’s a waste of resources! What requestAnimationFrame does is tell your system to wait for the next occasion to draw to execute a given function. Here’s how it will look in general.

function animate(duration) {
  var start = Date.now();
  var id = requestAnimationFrame(tick);
  function tick() {
    var time = Date.now();
    if (time - start < duration) {
       id = requestAnimationFrame(tick);
       draw(time - start / duration);
    } else {
      cancelAnimationFrame(id);
    }
  }
  function draw(frame) {
    // do your thing, update attributes, etc.
  }
}

 

See the Pen PqzgLV by Jerome Cukier (@jckr) on CodePen.

OK so in the part I commented out, you will do the drawing proper. Are you out of the woods yet? well, one great thing about d3js transitions is that they use easing functions, which transform a value between 0 and 1 into another value between 0 and 1 so that the speed of the animation isn’t necessarily uniform. In my example, you have (time – start) / duration represents the proportion of animation time that has already elapsed, so that proportion can be further transformed.

So yay we can do everything in plain javascript, but that’s a lot of things to rewrite from scratch.

CSS3, animations and transitions

(This is not intended to be an exhaustive description of animations and transitions, a subject on which whole books have been written. Just to give those who are not familiar with it a small taste).

In CSS3, anything you can do with CSS, you can time and animate. But there are some caveats.

There are two similar concepts to handle appearance change in CSS: animations and transitions. What is the difference?

  • with animations, you describe a @keyframes rule, which is a sequence of states that happen at different points in time in your transition. In each of these events, any style property can be changed. The animation will transform smoothly your elements to go from one state to the next.
  • in transitions, you specify how changes to certain properties will be timed. For instance, you can say that whenever opacity changes, that change will be staged over a 1s period, as opposed to happen immediately.

Both approaches have their uses. CSS3 animations are great to create complex sequences. In d3js, that requires to “chain transitions”, which is the more complex aspect of managing them. By contrast, going from one segment of the animation to another is fairly easy to handle in CSS3. Animations, though, require the @keyframes rule to be specified ahead of time in a CSS declaration file. And yes, that can be done programmatically, but it’s cumbersome and not the intent. The point is: animations work better for complex, pre-designed sequence of events.

Transitions, in contrast, can be set as inline styles, and work fine when one style property is changed dynamically, a scenario which is more likely to happen in interactive visualizations.

Here’s how they work. Let’s start with animations.

See the Pen vOKKYN by Jerome Cukier (@jckr) on CodePen.

Note: as of this writing, animation-related CSS properties have to be vendor prefixed, i.e. you have to repeat writing these rules for the different browsers. Here’s a transition in action.

See the Pen xGOqOR by Jerome Cukier (@jckr) on CodePen.

For transition, you specify one style property to “listen” to, and say how changes to that property will be timed, using the transition: name of property + settings. (in the above example: transition: transform ease 2s means that whenever the “transform” style of that element changes, this will happen over a 2s period with an easing function). 

One big caveat for both CSS animations and transitions is that they are limited to style properties. In HTML elements this is fine because everything that is related to their appearance is effectively handled by style: position, size, colors, etc. For SVG, however, color or opacity are styles, like in HTML, but positions, sizes and shapes are attributes, and can’t be directly controlled by CSS. There is a workaround for positions and sizes, which is to use the transform style property.

But wait: isn’t transform an SVG attribute as well? that’s right. And that’s where it can get really confusing. Many SVG elements are positioned through x and y properties (attributes). They can also have a transform property which is additive to that. For instance, if I have a <rect> which has an x property of 100 and a transform set at “translate(100)”, it will be positioned 200px right of its point of origin. But on top of that, SVG elements can have a transform style which affects pretty much the same things (position, scales, rotation…) but which has a slightly different syntax (“translate(100)”, for instance, wouldn’t work, you’d have to write “translateX(100px)”). What’s more, the transform set in the style doesn’t add to the one set in the properties, but it overrides it. If we add a “transform: translateX(50px)” to our <rect>, it will be positioned at 150px, not 200px or 250px. Another potential headache is that some SVG elements cannot support transform styles.
While any of these properties can be accessed programmatically, managing their potential conflicts and overlaps can be difficult. In the transition example above, I have used the transform/translateX syntax.

That said, a lot of awesome stuff can be done in CSS only. For scripted animations, the animation in pure CSS is definitely more powerful and flexible than the d3js equivalent, however, when dealing with dynamically changing data, while you can definitely handle most things through CSS transitions, you’ll appreciate the comfort of d3js style transitions.

See the Pen GJqrLO by Jerome Cukier (@jckr) on CodePen.

Now a common transformation handled by d3js transitions is to transform the shape of path shapes. This is impossible through CSS animations/transitions, because the shape of the path – the “d” – is definitely not a style property. And sure, we can use a purely programmatic approach with requestAnimationFrame but is there a more high level way?
It turns out there actually is – the animation element of SVG, or SMIL. Through SMIL, everything SVG can be animated with an interface, this includes moving an object along a path, which I wouldn’t know how to do on top of my head in d3js. Here is an extensive explanation of how this works and what can be done.

Data processing, scales, maps and layouts

For the end of the article let’s talk about all of which could technically be done without d3js, but in a much, much less convenient way. Therefore, I won’t be discussing alternatives with vanilla Javascript, which would be very work intensive and not necessarily inventive.

Array functions and data processing

d3js comes with a number of array functions. Some are here for convenience, such as d3.min or d3.max which can easily be replaced by using the native reduce method of arrays. When comparing only two variables, d3.max([a, b]) is not much more convenient than Math.max(a,b) or a > b ? a : b.

Likewise, d3js has many statistical functions, which saves you the trouble to implement them yourself if you need them, such as d3.quantile. There are other libraries who do that, but they’re here and it’s really not useful to recode that from scratch.

d3js comes with shims for maps and sets, which will be supported by ES6. By now, there are transpilers which can let you use ES6 data structures. But it’s nice to have them.

In my experience, d3js most useful tool in terms of data processing is the d3.nest function, which can transform an array of objects into a nested object. (I wrote this article about them).  Underscore has something similar. While you can definitely getting a dataset of any shape and size by parsing an array and performing any grouping or operations manually, this is not only very tedious but also extremely error prone.

Scales

Scales are one superb feature of d3js. They are simple to understand, yet very versatile and convenient. Oftentimes, d3 scales, especially the linear ones, are a replacement for linear algebra.

d3.scale.linear().range([100, 500]).domain([0, 24])(14);
((14 - 0) / (24 - 0)) * (500 - 100) + 100; // those two are equivalent (333.33...)

but changing the domain or the range of a scale is much safer using the scale than adhoc formulas. Add to this scale goodness such as the ticks() method or nice() to round the domain, and you get something really powerful.

So, of course it is possible (and straightforward, even) to replace the scales but that would be missing out one of the best features of d3js.

Maps

d3js comes with a full arsenal of functions and methods to handle geographic projections, ie: the transformation of longitude/latitude coordinates into x,y positions on screen. Those come in two main groups, projections proper that turn an array of two values (longitude, latitude) into an array of two values (x, y). There are also paths functions that are used to trace polygons, such as countries, from specially formatted geographic files (geoJSON, topoJSON).

The mercator projection may be straightforward to implement, But others are much less so. The degree of comfort that d3js provides when visualizing geographical data is really impressive.

Layouts

d3js layouts, that is special visual arrangements of data, were in my opinion one of the key drivers of protovis (where they originated) then d3js adoption. Through layouts, it became simple to create, with a few line of codes, complex constructions like treemaps or force-directed networks. Some layouts, like the pie chart or the dendogram, are here for convenience and could be emulated. Others, and most of all the force layout, are remarkably implemented, efficient and versatile. While they are called different names in d3js, geometry functions such as voronoi tessellation or convex hulls are similar functionally and there is little incentive in reproducing what they do in plain javascript.

Should I stop using d3?

d3js is definitely the most advanced javascript visualization library. The point of this article is not to get you to stop using it, but rather, to have a critical thinking in your code. Even with the best hammer, not everything is a nail.

To parse the DOM, manipulate classes and listen to events, you probably don’t need a library. The context of your code may make it more convenient to use d3 or jQuery or something else, but it’s useful to consider alternatives.
The concept of the data join unlocks a lot of possibilities in d3js. A good understanding of data join would lead you to implement your visualization much faster, using more concise code and replicable logic. It also makes trouble shooting easier. Data joins are especially useful if you have a dataset which is structured like your visualization should be, or if you plan to have interaction with your visualization that requires quick access with the underlying data. However, data joins are not necessary in d3js, or in visualization in general. In many cases, it’s actually perfectly sensible to parse a dataset and create a visual representation in one fell swoop, without attaching the data to its representation or overly worrying about updating.

Assuming you have d3js loaded, nothing prevents you from creating elements using d3js append methods instead of vanilla javascript. Or to listen to events using addEventListener rather than with d3js on method. It’s totally ok to mix and match.

Like data joins, transitions are a very powerful component of d3js, and, once you’re comfortable with them, they are very expressive. There are other animation frameworks available though, which can be better adapted to the task at hand.
Scale, maps, layouts and geometries are extremely helpful features however and I can think of no good reason to reimplement them.

Credit where it’s due

To write this I drew inspiration from many articles and I will try to list them all here.
The spark that led me to write was Lea Verou’s article jQuery considered harmful, as well as articles she cites (you might not need jQuery, you don’t need jQuery, Do you really need jQuery?, 10 tips for writing JavaScript without jQuery).

Most of the information I used especially in the beginning of the article comes more or less directly from MDN documentation, and direct experimentation. For CSS and animation, I found the articles of Chris Coyier (such as this one or this one) and Sara Soueidan (here) on CSS Tricks to be extremely helpful. Those are definitely among the first resources to check out to go deeper on the subject. Sara was also the inspiration behind my previous post, so thanks to her again!

Finally, I’ve read Replacing jQuery with d3 with great interest (like Fast interactive prototyping with d3js and sketch about a year ago). It may seem that what I write goes in the opposite direction, but we’re really talking about the same thing -that there are many ways to power front ends and that it’s important to maintain awareness of alternative methods.

 

d3: adding stuff. And, oh, understanding selections

From data to graphics

the d3 principle (and also the protovis principle)
d3 and protovis are built around the same principle. Take data, put it into an array, and for each element of data a graphical object can be created, whose properties are derived from the data that was provided.

Only d3 and protovis have a slightly different way of adding those graphical elements and getting data.

In protovis, you start from a panel, a protovis-specific object, to which you add various marks. Each time you add a mark, you can either:

  • not specify data and add just one,
  • or specify data and create as many as there are items in the array you pass as data.

.

How de did it in protovis

var vis=new pv.Panel().width(200).height(200); 
vis.add(pv.Panel).top(10).left(10)
  .add(pv.Bar)
    .data([1,4,3,2,5])
    .left(function() {return this.index*20;})
    .width(15)
    .bottom(0)
    .height(function(d) {return d*10;});
vis.render();

this simple bar chart in protovis
you first create a panel (first line), you may add an element without data (here, another panel, line 2), and add to this panel bars: there would be 5, one for each element in the array in line 4.

And in d3?

In d3, you also have a way to add either one object without passing data, or a series of objects – one per data element.

var vis=d3.select("body").append("svg:svg").attr("width",200).attr("height",200);
var rect=vis.selectAll("rect").data([1,4,3,2,5]).enter().append("svg:rect");
rect.attr("height",function(d) {return d*20;})
  .attr("width", 15)
  .attr("x",function(d,i) {return i*20;})
  .attr("y",function(d) {return 100-20*d;}
  .attr("fill","steelblue");

In the first line, we are creating an svg document which will be the root of our graphical creation. It behaves just as the top-level panel in protovis.

However we are not creating this out of thin air, but rather we are bolting it onto an existing part of the page, here the tag. Essentially, we are looking through the page for a tag named and once we find it (which should be the case often), that’s where we put the svg document.

Oftentimes, instead of creating our document on , we are going to add it to an existing <div> block, for instance:

<div id="chart"></div>
<script type="text/javascript">
var vis=d3.select("#chart").append("svg:svg");
...
</script>

Anyway. To add one element, regardless of data, what you do is:

The logic is : d3.select(where we would like to put our new object).append(type of new object).

Going back to our code:

var vis=d3.select("body").append("svg:svg").attr("width",200).attr("height",200);
var rect=vis.selectAll("rect").data([1,4,3,2,5]).enter().append("svg:rect");
rect.attr("height",function(d) {return d*20;})
  .attr("width", 15)
  .attr("x",function(d,i) {return i*20;})
  .attr("y",function(d) {return 100-20*d;}
  .attr("fill","steelblue");

On line 2, we see a different construct:

an existing selection, or a part of the page
.selectAll(something)
.data(an array)
.enter()
.append(an object type)

This sequence of methods (selectAll, data, enter and append) are the way to add a series of elements. If all you need to know is to create a bar chart, just remember that, but if you plan on taking your d3 skills further than where you stopped with protovis, look at the end of the post for a more thorough explanation of the selection process.

Attributes and accessor functions

At this stage, we’ve added our new rectangles, and now we are going to shape and style them.

rect.attr("height",function(d) {return d*20;})
  .attr("width", 15)
  .attr("x",function(d,i) {return i*20;})
  .attr("y",function(d) {return 100-20*d;}
  .attr("fill","steelblue");

All the attributes of a graphical element are controlled by the method attr(). You specify the attribute you want to set, and the value you want to give.
In some cases, the value doesn’t depend on the data. All the bars will be 15 pixels wide, and they will all be of the steelblue color.
In some others, the value do depend on the data. We decide that the height of each bar is 20 times the value of the underlying data, in pixels (so 1 becomes 20, 5 becomes 100 etc.). Like in protovis, once data has been attributed to an element, function(variable name) enables to return a dynamic value in function on that element. By convention, we usually write function(d) {…;} (d for data) although it could be anything. Those functions are still called accessor functions.
so for instance:

.attr("height",function(d) {return d*20;})

means that the height will be 20 times the value of the underlying data element (exactly what we said above).
In protovis, we could position the mark relatively to any corner of its parent, so we had a .top method and a .bottom method. But with SVG, objects are positioned relatively to the top-left corner. So when we specify the y position, it is also relative to the top of the document, not necessarily to the axis (and not in this case).
so –

.attr("y", function(d) {return 100-d*20;})

if we use scales (see next post), all of this will have no impact whatsoever anyway.
Finally, there is an attribue here which doesn’t so much depend on the value of the data, but of its rank in the data items: the x position.
for this, we write: function(d,i) {return i*20;}
Here is a fundamental difference with protovis. In protovis, when we passed a second argument to such a function, it meant the data of the parent element (grand parent for the third, etc.). But here in d3, the second parameter is the position of the data element in its array. By convention, we write it i (for index).
And since you have to know: there is no easy way to retrieve the data of the parent element.

Bonus: understanding selections

To add many elements at once we’ve used the sequence: selectAll, data, enter, append.
Why use 4 methods for what appears to be one elementary task? If you don’t care about manipulating nodes individually, for instance for animations, you can just remember the sequence. But if you want to know more, here is what each method does.

selectAll

the selectAll method
First, we select a point on which to add your new graphical objects. When you are creating your objects and use the selectAll method, it will return an empty selection but based on that given point. You may also use selectAll in another context, to update your objects for instance. But here, an empty selection is expected.

data

the data method
Then, you attribute data. This works quite similarly to protovis: d3 expects an array. d3 takes the concept further (with the concept of data joins) but you need not concern yourself with that until you look at transitions.
Anyway, at this stage you have an empty selection, based on a given point in the page, but with data.

enter

the enter method
The enter method updates the selection with nodes which have data, but no graphical representation. Using enter() is like creating stubs where the graphical elements will be grafted.

append

the append method
Finally, by appending we actually create the graphical objects. Each is tied to one data element, so it can be further styled (for instance, through “attr”) to derive its characteristics from the value of that data.

 

From protovis to d3

You’ve spent some time learning protovis only to find that its development is halted as authors have switched to work on d3. Have your efforts all been in vain? Fear not! This series of posts will help you adapt to d3 with a protovis background.

Before we go anywhere further, let me say that these posts won’t make you awesome at d3 (yet). We won’t be talking about how to do all amazing things you could never do in protovis. Rather, we’ll focus on enabling you to be as comfortable with d3 than you could have been with protovis. And once that’s done, nothing will prevent you from learning the more powerful aspects of d3.

Anyway, if you’re reading this, you are already awesome.

Why should I make the switch to d3?

Frankly, you don’t have to. Protovis is a fine framework and works well. Now you may want to switch to d3 for several reasons.

  • d3 is fast. d3 is better at handling scenes with hundreds or thousands of elements. So if you like scatterplots or network graphs, and who doesn’t, d3 has much stronger performance.
  • d3 does animation. There were workarounds to get animation in protovis but there were that. Workarounds. Animation and transitions are built in d3 and are a snap to implement.
  • More features. Just because development has stopped on protovis doesn’t mean that it has stopped elsewhere… for instance, d3 has more ready-to-use layouts, like voronoi tesselation or chords, and it has more methods and functions to make your life easier, to access and manipulate data for example.
  • Styling. In d3 it is possible to apply style sheets in CSS to graphical elements. This helps keeping the code and the format separate.

Yes but doesn’t everything change?

Short answer: no.

Less short answer: some things do change substantially. Most things stay the same. And then, some things look the same but have changed.

Things that stay the same

  • The general principle.Protovis is about transforming an array of data into the same number of graphical elements, with characteristics derived from that data. d3 does exactly this as well.
  • pv.Nest, which in my personal protovis experience has been the hardest to understand. Only, it’s called d3.nest now.
  • Methods that supplement the existing javascript array manipulation methods, like pv.min, pv.values, pv.entries etc. are also back (as d3.min, d3.values, d3.entries, but you’ve guessed it by now). Some, like pv.mean or pv.median, didn’t make it through but you could easily rewrite them, or continue using the protovis ones.

Things that look different, but which are largely the same

Protovis had a number of native graphical objects, or marks, that could be manipulated at will with methods.

var vis=new pv.Panel()
  .height(400)
  .width(400)
  .fillStyle("aliceblue")
  .lineWidth(1)
  .strokeStyle("steelblue");
vis.render();

In protovis, it is inherently different to set the height, the width or just any property of an object. This uses different methods.

var vis=d3.select("body").append("svg:svg");
  vis.append("svg:rect")
    .attr("height",400)
    .attr("width",400)
    .attr("fill","aliceblue")
    .attr("stroke","steelblue")
    .attr("stroke-width",1);

This produces essentially the same thing. We add a rectangle of a specified height, width, and colors. There are a few differences though. Here, controlling height, width or fill is essentially the same thing and uses the same method, .attr(). Notice also that we first created an svg document, then a shape within that document. And also, that we don’t need to use vis.render(); anymore.

The d3 approach looks longer. But if we define all the style information first we could make it much shorter, shorter than in protovis in fact!
For instance:

var vis=d3.select("body").append("svg:svg").append("svg:rect").attr("class", "myRect");

Much of the apparent differences between d3 and protovis come from using explicitly svg shapes (paths, polygons, ellipses, etc.) as opposed to native objects (pv.Panel, pv.Bar, pv.Dot, etc.), although – it’s essentially the same thing. Yes, you have to learn your SVG but it’s really on a need-to-know basis. In fact, if you’ve worked with protovis, or even if you’ve worked with HTML and CSS, you probably know more SVG than you thought.

SVG is more flexible than protovis objects. The flipside is that constructs which were once simple in protovis become less obvious in SVG. But for those cases, d3 has recreated some native objects, even if not as many as in protovis.

Things that look the same, but which are different

There have been some changes in methods that have kept the same name since protovis – some minor, some more substantial. In any case, the basic ways of using these methods (like scale, color, data…) doesn’t change much. It’s only their more exotic uses who do change.