Amelia Wattenberger talks with Joel Hooks about the book she wrote on data visualization with D3.
D3.js is the defacto library that people use to create custom data visualizations on the web today. It's powerful and flexible. You can do whatever you want with it. However, that kind of power and flexibility comes at the cost of complexity. You have to know what you're doing, and it takes a long time to learn.
There's existing content written on D3, but there's always room for another voice. With the help of Newline, Amelia wrote the book of over 600 pages, Fullstack D3 and Data Visualization. In this book, she teaches all the theory and application you need to know to make badass visualizations using D3.
Amelia also takes the unique approach of having you use your own dataset! Data that means something to you is going to be much more interesting than anything that could be provided.
As a React developer, Amelia wrote the excellent blog post Thinking in React Hooks. She says that you have to make a paradigm shift with hooks. You can't keep thinking about your components in terms of lifecycle, but instead, think about them in terms of data synchronization.
"Writing The Book On Data Visualization With Amelia Wattenberger" Transcript
Amelia: Hey Joel.
Joel: How are you doing today?
Amelia: I'm good. How are you?
Joel: I'm doing all right. I'm really excited to talk to you. You recently dropped this amazing post on React hooks that completely blew my mind and was probably one of the coolest presentations of not only React hooks but programming concepts in general. And I want to get into that. But first I am going to go right into this book that you've written Fullstack D3 and Data Visualization. It's a 600 page book. This is something that if you had a physical printed copy you could use to defend yourself. And I'm wondering, I want to talk about your book more, but I was also wondering what the experience was like for you writing something like this, writing something this comprehensive and how was that as a process?
Amelia: Yeah. Yeah. And the funny thing is we're actually working on a hardcover right now and these are very beefy books. I think they're three pounds or something. The funny part is I didn't ever start, I never thought to myself, "Hey, I want to write a book or I'm going to write a book." The way it started is that Fullstack, which is now New Line, but at the time they were Fullstack. They had a survey for people who wanted to submit book ideas just to see if it would be a good fit and I stumbled across this survey and I was originally thinking, "Hey, it'd be awesome to write a book on React." This is something I've been focusing on for awhile. I have a ton of opinions. I think it's hard to wrap your head around and I can provide a unique insight and then I realized that they already have one or two books on React.
Amelia: So I was racking my brains trying to think about what else would I be passionate about that I would want to think about for the amount of time that it would take to write a book. And I've been working as a front end developer and designer on dashboards for must be eight or nine years now. So data visualization is definitely something that has been something that I have to think about all the time.
Joel: Yeah. It's on your mind a lot.
Amelia: Yes, a lot. It's never actually been part of my job description or anything, but it's something I think about a lot. So I submitted the survey saying, "Oh, would you be interested in writing a book about data visualization?" Originally thinking it would be a theory book and thinking that they would say no because it's kind of out of left field for Fullstack since they write about frameworks a lot of the time. And they came back and they were like, "Yeah, that sounds interesting." And so it was just a very inner process of the next step was writing an outline. So we did some research into what are the biggest blockers that people run into and what is it like as a developer trying to visualize data or learn D3.JS, D3.JS is kind of the defacto library that people will use to create custom data visualizations on the web.
Joel: I would say it has a reputation also. Both as amazing software and also accepting that it takes a little bit of effort to learn, I think.
Amelia: Yeah, it definitely has a learning curve, that's for sure. I see it as there's a spectrum of, at the top of this or on the right of the spectrum, you have tools like Tableau or like very full feature charting libraries that will do most of the work for you and you just input a dataset and then you say, "I want a line chart or a bar chart."
Joel: Would you say that that literally puts you in a box though?
Amelia: Yes, yes. And I have a lot of opinions on that. There's pros and cons.
Joel: Yeah, yeah, for sure.
Amelia: And then all the way on the other side of the spectrum is D3 where it's not opinionated. You can do basically whatever you want with it. It's kind of just a library of tools that you can use to visualize data and it's all the way over here so it has the pros of being flexible, you can do whatever you want with it. And then the cons are you have to know what you're doing kind of and it takes a long time to learn. So I think that's the trouble that most people run into. And then a lot of people, they'll just end up moving more towards the middle of the spectrum where there's fairly low level charting libraries where you can, like if you're in Reacts, you can use the React charting library and stay in React and say, "Hey, I want to learn chart."
Joel: Built on D3 too. They're charting libraries that are abstractions on top of D3, which is kind of a low level data visualization tool.
Amelia: Yes. I think pretty much all charting libraries on the web use D# to some extent.
Joel: Yeah. It's a good place to start. Starting from scratch with your ... to get what D3 offers from scratch is, I would call it non trivial, I think.
Amelia: Yeah. Yeah. That's part of the reason. That's so true. And that was a big motivation behind writing the book is because I went down the same path that most people go down with D3 where they're like, "I work at this company and they need a dashboard or they need charts. They have this data they need to visualize. I have no idea what's going on. Let me Google bar chart in JavaScript. Look, I found an example, I'm going to copy and paste this code and then somehow it works or it doesn't work and I'm going to bang my head against it." But it's hard to know what every line of that code does. Because it's new concepts. Every single line is a new concept I've never run into before as a normal developer.
Joel: Yeah. You can get it done, you'll find it on Stack Overflow and you drop in your bar chart and it's like, "Okay, well this works. Just please don't touch it. Please don't."
Amelia: Yeah. You'll have comments like, "Don't touch this line. I don't know why it's working, but it does."
Joel: Do not change anything. This is it, yeah. So you mentioned your research process for the book and I'm wondering could you go into more depth? How do you even approach that and with Fullstack at the time. I know some of those folks that they work over there and I was wondering, are they supporting that process and helping you do the research or how did that work in terms of figuring out how you going to write something and kind of keeping the scope contained but also making it bountiful as well.
Amelia: Yeah. Yeah. Without them I 100% would not have written a book. They have good experience and processes. So doing research for the outline, we'd do things like look on Stack Overflow to see what are the biggest issues people have with D3. Find the watering holes too.
Joel: Sales Safari. That all comes from Amy Hoy. So you say watering holes. So Amy Hoy is where RA Lerner and myself were in the same graduating class of Amy Hoy's, how to start an internet business seven years ago. So Fullstack and Egghead actually started at the same time. So watering holes, can you explain what a watering hole is?
Amelia: I'm using jargon that I don't completely understand.
Joel: No, I think it's awesome actually that it comes up, that's amazing. It's really cool.
Amelia: As far as I understand it, a watering hole is where people within the community who would be reading the book, would go to discuss like, "Hey, I'm having this problem."
Joel: Or talk about their pain.
Amelia: Yeah, yeah. So that's the perfect place to go to understand for the people who would be reading the book, what would they need and that they're not getting online. Because there's tons of tutorials, there's tons of chart examples online and for some reason that doesn't do it. That doesn't get someone from not knowing how to use D3, to knowing how to create a chart. So figuring out where those holes are.
Joel: I think that's an interesting point too because a lot of times I hear this even down to the blog post level, where somebody will say, "The feeling is this has already been written." And I was wondering, did you feel that when you're starting out? Like there's already books on D3, there's already blog posts or did you have the confidence to say, "Well my opinions need to be expressed and I have a lot to say." Or how did that work for you?
Amelia: Oh my goodness. So much though, because I mean imposter syndrome is just really easy to have in the first place. And also when you're doing the research, you're also looking at prior art, existing books that have been written on the same thing you want to write a book about and you go through and look at the table of contents, you see what they wrote about just to figure out how someone else would break down this subject.
Joel: Yeah. Structurally right?
Amelia: Yeah, yeah. So I've seen all of the books on D3.JS and I at least skimmed over all of them before writing it. So it's hard to do that kind of research and then still feel like, "Hey, I'm going to write another book. I know there's already 20 books, but one more would be great." But I think, what I tell myself is that everyone has a unique perspective. So I think it's not that everyone's perspective is important, but there are going to be certain learning styles that work for some people and others that work for others. So you can't do wrong by providing more learning materials for something.
Joel: Nobody's ever going to say there are too many books on D3. We need less books on D3 or JavaScript or React or anything in my opinion, it's like you have a unique journey. You learned it a certain way. You are a professional and an expert and you have something to offer. And maybe not 650 pages of book, not everybody has that in them. But how do you compare that to, you blog and you write tutorials that way as well, how do you compare the book process to how you develop say a blog post?
Amelia: Yeah, that's a good question. I think it was actually a lot easier for me because of the work that was put in at the beginning. We did a lot of work for creating the outline and figuring out what each chapter would cover. And then what I really loved about writing this book is that each of at least the first six chapters is working through a code example. So the first chapter gets you through creating a line chart, I think. And then the second chapter is a bar chart. The third chapter is a scatter plot, while teaching different concepts throughout the tutorial through a specific chart, which is nice to keep it concrete that way. And it also made it nicer for me for writing it because I'm a developer by trade. This is what I have historically been doing and have a lot of experience at, so being able to ease into a chapter by working on the code first and then iterating on the code and taking notes on what did I struggle with or what are some gotchas that people might run into?
Joel: Trying to learn it yourself, getting back into that beginner's mind.
Amelia: Yeah, yeah, yeah. Which I think could be really hard because you're really trying to pull people up to where you are.
Joel: Yeah. So you approach an example and you're approaching it as if you ... like what do I need to do to learn this, is a different mindset than, what I need to do to get the job done and get the bar chart up for whatever at work. It's a different kind of process if you are actually wanting to analyze what it is you want to teach and show somebody else. A lot of people too, because there's a lot of people that are hopefully going to read the book and come behind you so they are all going to struggle at some level, I would assume.
Amelia: Yeah. Yeah totally. And that kind of approach made it, I think a lot easier to write each chapter of the book. The chapters, they just kind of like, I would sit down and just start writing and they would come pretty easily. But then sometimes with blog posts they're not well scoped and I am the queen of scope creep. I will scope croup the thing to death, so there's not those constraints. So I'll just sit down. This past blog post I sat down and I said, "I have like one statement to make." And then six, 10 hours later I had a lot more text on the page than I was expecting.
Joel: Do you consider blog posts done? When you write something like that is it finished or are you ... I've been lately saying I'm tending a garden in a lot of ways and trying to come back to things and think about them and not be so bent on making a complete tutorial when I write these things.
Amelia: Yeah, I definitely prefer that outlook. One, because it makes it easier to throw out into the world because they never truly feel finished with anything. And two, because I don't know, I guess the concept of the blog is kind of interesting, especially for a technical blog to say, "This is my newest blog posts." It doesn't matter when you wrote these things.
Joel: Date matters almost not at all.
Amelia: Yeah. Or it shouldn't. I would say for a good blog.
Joel: It's the number one way to devalue your blog posts to those folks listening, is to put a date on it because people will look at that day and go, "Oh, this is three years old. This doesn't matter anymore." And guess what? It still does. Technology does not move as fast.
Amelia: This has expired. This is stale. We can't use this anymore. It is nice sometimes when D3 went through a little bit of a syntax change for version four. So in that kind of thing the date would matter. But honestly-
Joel: The date or the version number matter more to me. to me it's like D3, D4 because we have some D3 content on Egghead and it's like even the old stuff is actually still valuable in terms of concepts and context. As long as you understand that if you try to use the newest library it's going to be broken.
Amelia: Yeah, totally, totally.
Joel: In a book like yours, there's a lot of interesting concepts in there and to me one of the coolest thing that ... I've read a few data visualization books and I've never seen somebody right up front encourage people to bring their own data and I thought that was amazing. You say, "Hey, go get a data set for your town and your weather." And I want to know why did you make that choice and where did that come from?
Amelia: Yeah, that was probably the hardest part of writing the book is figuring out what dataset to use because there's both too many and too few data sets out there.
Joel: Lot of data, too few good datasets maybe, I don't know how you'd phrase it.
Amelia: Yeah, a lot to dig through and then you have to figure out can I use it in a book? That kind of thing. And then is this something everyone will understand or are there too many metrics? Is it going to be overwhelming? And so what I ended up using was a weather dataset, which it was actually really hard to find one that people wouldn't need to go through too many steps to get their own weather dataset. I was pretty shocked about that. But it's ideally the last year of daily weather points. So for each day you have what was the precipitation rate and did it snow, what was the temperature, what was the min temperature. So it was just a good set of numerical data and quantitative data and qualitative data and a year's a good length to play with.
Joel: I think you don't have to explain it to anybody either too and that's important. Some data domains you're going to have to ... people are going to have to actually learn and understand the domain and with weather we all have to wear a coat or some shorts or whatever.
Amelia: Yeah, most people have some concepts of weather.
Joel: Yeah hopefully.
Amelia: And another thing that was really cool about it, well, so part of the reason that I like for readers to grab their own dataset is I could include, I think I include New York city data, but that might be very meaningless to someone who lives in South America. It's just numbers at that point. But for someone to go and get the weather data for where they are, it all of a sudden becomes meaningful. When they finish the first chart, which is max temperature over time, they get to actually see what it was for where they are, which kind of gives you more of the feeling that you will hopefully get when you visualize your data at work or your own personal data.
Joel: I think it's interesting too because when you mention it that way it gives a person ... you might remember that day, when you see that hottest point on your graph, you might remember that hottest day of the year or the coldest day of the year or that period where it was super frosty or whatever. And then you can relate to that data and start to understand. So we have the D3 part, which is its own ball of wax, and then you actually have the data visualization piece, which to me, you get into design and how do you do this and how do I make something useful and being able to contextually feel that like, "Oh that was when it was colder. I remember that." I think that's probably pretty important as well.
Amelia: Yeah, yeah. Yeah. We did a beta reader group, so before the book was finished and published, we had a group of people, it was a really great group. They would go through each of the examples and they would read through the chapters. And in my first drafts, I wanted people to plot where freezing was on the chart. And thankfully we did a beta reader group because someone in Africa said, "Hey this rectangle does not show up on my chart." Because their temperature never gets below freezing. And I was like, "Oh of course."
Joel: Yeah, that makes sense. So speaking of the data visualization side of this, how do people start learning the design principles you need to create some sort of data visualization? You need to be a designer or what sort of core skills can you learn to get started visualizing data and making meaningful visualizations that other people can use?
Amelia: Yeah, yeah, that's a good question. I figured when I first thought about this book, I thought I would focus pretty extensively on this because it's a huge question and it's something that especially developers struggle with. But we have one or two chapters that help you deal with, "Hey, I have this dataset. What do I do?" And I think that one of the biggest issues that people have is that they'll look at one of these chart taxonomy charts or they'll think of, "Oh, these are the basic chart types." And then they'll say, "Okay, which one fits my data? Oh, is this a bar chart? Is this a line chart? Is this a scatterplot?"
Amelia: And, I don't know, that frustrates me a lot because I would really prefer people to see it as, I have these metrics, let me figure out what's important. There's the different ways to display a metric, to visualize it, work better or worse based on what dimension you're displaying it in. So say I have temperature, I can show that as a vertical position, I can show that as a color, I can show that as the size of a circle and there's a ton of basic physical properties and facts about human perception that all come into play and it gets much more complicated when you try to understand data visualization on a fundamental level, but you're going to end up with better charts that are easier to read for your users than if you approach it from a chart picker kind of approach.
Joel: I think for me personally, and this is a long running subject and I'm not a very good ... I don't have a lot of practical experience, it's just an area I'm interested in. And I read The Visual Display of Quantitative Information by Edward Tufte and I was almost intimidated and discouraged from even trying because I'm looking at that and it's like, "I'll never get to this level." Which is probably true. I just don't have the time and energy. But do we have to go that far? Do we have to take it to that extreme to get useful information and data visualization that is interesting and useful to the people that are looking at it, which is probably the actual point.
Amelia: Yeah, yeah. Yeah. It really depends. It's interesting, I'm part of this Slack group for people who do data visualization and it's interesting the variety of backgrounds that people are coming from and the amount that they actually do data visualization in their day to day jobs. I feel like it's the new UX design or UX research.
Joel: It's definitely a subset of that for sure, I would think.
Amelia: Yeah. Yeah. And I found that a lot of my user experience design background has come into play. There's a lot of crossovers with UX design and data visualization and the way I like to look at it is visualizing data is really just communicating. It's just trying to show a reader what the shape is of the underlying data.
Joel: I liked what you said too, where it's, you have different extremes and different ways, these are all numbers, but you can apply color values or shapes or size and all these different ways and kind of almost play and explore and what's meaningful and what registers, to get the idea across to have the ultimate outcome of the dataset that you're trying to help people to understand. That's the true point. And that's the point of user experience. It's not, is this site pretty, it's, are we building something that's useful and gets the outcomes that people want.
Amelia: Yeah, exactly. There's the user experience design part and there's the UI design part of, I just want it to look nice and have my brand colors.
Joel: Yeah. And I mean, you can make nice looking charts that are totally meaningless, they're missing labels on their axes or they're like, "Wow, cool chart." But if it's not helping anybody and then not to say for the sake of art, you can't make cool charts. But generally speaking, we're trying to convey information and encourage specific thoughts.
Amelia: Yeah, yeah. There's this whole field called data art which is kind of like an offshoot of data visualization, which is just, we've made art from data and there's this underlying data that I promise you it's there but you can't understand it.
Joel: I love graph data with physics applied. You have big graphs and the physics go in and different weights for different data pieces. And it's kind of like what you're saying where there's colors and all this stuff. And even when it's meaningful data and you can spot patterns and do different stuff with it, I love it as a purely aesthetic, visual, like look at that data. And I might be a nerd, I don't know.
Amelia: Yeah, yeah. I also like data visualization because it's in this in between, between science and art. We try to create rules for data visualization, but also there are no rules.
Joel: Often interesting results are when they break the rules.
Amelia: Yeah, exactly.
Joel: Something like, well I would have never thought of that. And you see it in a new light.
Amelia: Yeah. There's this recent proliferation of bar chart races on Twitter. Which is just, it's a bar chart that changes. So I forget, the original was showing brands over time or something like that and it was, you have to sit there and watch it for a minute or two minutes, just sitting here. You're just sitting here watching something for two minutes when this could easily have been shown with a line chart. So there's something so weird about what is appealing to people, why am I willing to sit here for two minutes and watch these bars?
Joel: Almost like the entertainment value of it, to convey information as well. Engaging people that length of time. I really now I feel like I need to get more deep into data visualization, Twitter because it sounds like my kind of place.
Amelia: It's a great place.
Joel: Yeah, I imagine. So I want to talk about this blog post thinking in React hooks. I think it blew up. Was that safe to say that you had a lot of interest in this particular post?
Amelia: Yeah. It's so funny because I'm sitting here thinking like, "Yeah, I've made better things. It's okay."
Joel: It's neat though. And I think what blows people's minds is you scroll through this post and it's talking about hooks, which is interesting, anyway, it's kind of a new way of thinking about our React applications and everybody's kind of in the midst of, I need to learn this thing. And they've only been around for a year and it's this whole paradigm shift. But then you get in and it's colorful. The code examples are great. There's neat elements that happen on the screen to show me, don't do this and do this. And I feel like that sort of presentation is really, really great to help introduce something that's a very dense and kind of anxiety filling concept for a lot of people. Like, "Oh I got to like change my React app. What does this mean to me?" And you are really teaching people to literally think in hooks. And I was wondering, how did you go from not even knowing what a React hook was, to kind of having this delightful visual explanation of React hooks that you shared with the entire world?
Amelia: That's a good question. Yeah, actually, I was a late adopter of hooks, I kind of just ignored it for a while.
Joel: I like to wait ane see a little bit.
Amelia: Yeah, I'll let someone else guinea pig that one. I guess, this is the one I was talking about previously where I noticed myself starting to use because I think most people try to map hooks on to their existing knowledge of life cycle events with class components. And I started off that same way saying, "Okay, use effect is this." And it's component did mount and [inaudible 00:27:53]. And then I noticed myself starting to use them in a way that was more syncing data and it felt more natural. And when I had realized that it was kind of more of a paradigm shift, I was like, "Oh this would be good to write about because it's something that wasn't immediately obvious to me."
Amelia: So that's where it started. And then it was really just working through like, "Okay, put yourself back in the old mindset. What would you need to bridge the gap, to explain to you how you would need to change your mental model.' And then so I think what people find cool about the thinking in hooks, I post is that there's side-by-side old and new examples, which I had originally and then I posted on Twitter asking, I forget what I asked, but people responded with examples of git merge tools where they use those smooth curves in between like ...
Joel: The connecting lines.
Amelia: Yeah, the old code and the new code. And at first I was like, "No, that seems really hard."
Joel: It's true.
Amelia: Yeah. And then I finally talked myself into trying it and it actually ended up being easier than I had originally thought. Just using SVG curves.
Joel: SVG is amazing.
Amelia: It really is.
Joel: Yeah. Yeah. It's a wonderful, wonderful tool.
Amelia: Yeah. SVG is great. And I think not enough people understand it.
Joel: I always send people to Sarah Drasner's book. I think it's in terms of getting to know SVG and truly kind of understanding the potential, her book is really pretty awesome.
Amelia: Yeah. Yeah. She's great. I did a workshop at a conference recently, at a data visualization conference recently talking about these are the basics of this SVG and even researching that, I couldn't find any good resources other than the MDN docs for figuring out what are the basic shapes? So we're planning-
Joel: It's been around a long time too right?
Amelia: Yeah, yeah. It's been, I don't know, 10 years or something crazy. And one of the hard things with SVG is that the implementation is so inconsistent across browsers, that it can be frustrating. And then I can see that people would just try it and then like, "Oh, I looked at it in Firefox. It doesn't work. I don't know, I give up, let's get a graphic." So yeah, I think part of what's cool about this post is the SVG curves between the old and new code.
Joel: Again, certainly understandable and familiar. There's a comforting familiarity to it that makes it like, "Oh, okay, I see." And it's a diff and we kind of get that as developers, it's speaking the right language, but visually.
Amelia: Yeah, yeah, exactly. And I'm always trying to find ways to use web technology because I feel like we've just taken books and we've taken newspapers and we've put them in our computers and we're like, "Yep, this is the best it's going to get. This is the way people learn."
Joel: Lies.
Amelia: It's a lie.
Joel: Yeah, it's true. There's so much potential. And that's what blew my mind when I saw your post was it's just peeling it back. And Bret Victor's been talking about this for a long time and there's a lot of talk about the potential of learning on the web, but I don't feel like we've seen that. Egghead is videos and then we also have texts. But how do you combine all of this to give people a really truly engaging, effective, learning experience that helps them meet their goals. And there's a lot of work to do I think.
Joel: I'm so happy to see your style of presentation and pushing the envelope. And I think it's going to be ... it really is, it's inspiring us to push our work further and I think it's going to help people and it's really truly a great resource. And I wanted to thank you for creating that and kind of pushing the envelope because I think that takes a level of commitment and even bravery to get out there and do that sort of thing. So I appreciate that. Thank you.
Amelia: Thanks. I appreciate your appreciation. I feel like I partially don't deserve it because a lot of the motivation behind it is that I don't love writing.
Joel: Oh, so you can get away with not writing by, okay, well I'll spend 25 hours making visualizations instead of just writing words.
Amelia: Yes. I'm very good at procrastination.
Joel: That's a fine approach. And then I also love how ... and you give us a bunch of resources down at the bottom and you kind of amplify other voices and the whole thing is just, it would be like my starting point. Oh you want to learn hooks, here go jump into this and this should give you a good idea. And I also love how you did it collaboratively. You asked for feedback, you didn't try to hold this thing and do the big reveal at the end. It's like, hey. And how did that go for you actually? I was curious, kind of doing the writing in public and creating that thing. Was that a good experience?
Amelia: Yeah, I really love doing that for a few reasons. One is that Sharon along the way keeps me motivated because people will say like ... I'll work in the evenings and then I'll post a screenshot on Twitter and then I can wake up the next day and have people saying like, "Oh, this is great." Which is really nice because I'll creep the scope but then share along the way so then it'll keep you motivated even though-
Joel: Keep you in check too.
Amelia: Yeah, doing way too much. And people have great ideas on Twitter, the swoops in this blog posts were definitely not totally my idea and I don't know, I have the nicest Twitter followers, so people are generally really nice.
Joel: Yeah. Twitter for the most part is a pretty great place. That has helped me. If you can avoid the not great parts of Twitter, the great part to make the whole thing worth it.
Amelia: Yeah. Yeah. Still in that positive for me, although I hear once you pass 10 K followers, it gets ...
Joel: I just did. I sat at like 9,700 Twitter followers for probably six weeks and then in a week finally. It hasn't changed for me.
Amelia: Okay, good.
Joel: But I know everybody's experience is different. Amelia, thank you so much for hanging out with me this afternoon and chatting about this stuff. I really appreciate it. I think people should totally check out your book. It sounds like your Twitter is a good place to find you as well. Thank you again and I look forward to talking to you in the future.
Amelia: Yeah, thanks so much. This has been a ton of fun.