This weekend I made a video aimed at people just getting into d3. I use a super simple bar graph to illustrate the fundamental principle of selections with d3. At 13 minutes it’s a little over the 10 minute goal I had, but significantly shorter than my last video!
I hope this is helpful to some. My plan is to make more of these so if you have any requests I’ll try to make a video to address confusing topics. The code is available here. :)
My buddy Andrew Young at FSU DSC has made some serious progress with RTPS! He’s implemented rigid bodies with particles and they can interact with the fluid. Watch the video and be amazed as a block of particles splashes, floats and is tossed around by the fluid!
Rigid-body and Fluid interaction with velocity vectors from Andrew Young on Vimeo.
Things have been pretty intense at work the last couple weeks, we are writing hella code (you can see I’ve begun acclimating to the west coast) and a lucky few people have begun using what we’re making! This weekend was a much needed break from programming all day, so what did I do? Code all night.
What you see here is the square circle spiral optical illusion first made popular on the internets by a Dr. Richard Wiseman. What I’ve done is implemented the illusion using d3.js, a powerful javascript library for data visualization. I took it a step further and animated it:
You can get the code for the original illusion as well as a live demo. Here is the code for the animated version and a standalone demo here. Feel free to fork and come up with your own variations. Hint: there are several variables we can play with, and a little phase shift is all it takes ;)
So I’ve been diving head first into d3.js and I must say its a mighty fine library! I do feel like getting up to speed with it can take quite a bit of reading and playing, and now that I’ve figured out how to do a couple of things with it I wanted to share how I go about using it. So I made a screencast of myself building a bar chart.
The video is a little long at 40 minutes, and it’s just me, my terminal and the browser, but I do hope it helps people who are just starting get into a d3 frame of mind.
[d3cast] Simple Bar Chart Walkthrough from Ian Johnson on Vimeo.
You can follow along with annotated (commented) javascript or just look at the resulting html/css/javascript while I talk.
As an aside, the annotated javascript was generated with the awesome docco project.
Hope this was helpful, I plan to do more screencasts in the future as I get deeper into d3. Any feedback or suggestions are welcomed!
Now what I want you to do is visualize yourself… visualizing your data.
Can you see it? Maybe you need some help? Fret not, I’m working on it!
This week I moved from Florida to California to start work at visual.ly where we are building some awesome tools to help you visualize your data. I’m super excited to be in the mix with some of the most talented (and somehow humble) designers and hackers that I’ve ever met. I’m still struggling to accept the reality of a dream come true, after all those years of reading Hacker News I’ve actually gone two full work days hardly browsing it at all! I’ve been learning a lot, both technology and startup wise and it’s way cooler than I imagined.
To top it off, the office is a quaint old brick building in downtown San Francisco, with open space inside and Sushi Burritos right around the corner outside, it’s like I’m in office heaven.
So you wont see as many OpenCL posts out of me in the near future (c’mon WebCL!) but you will see more tutorials and thoughts on the latest in web technologies related to visualization! Right now I’m learning d3.js and backbone.js and all of the curly braces are making me itch for CoffeeScript.
Well, time to get back to coding!
Hello World!
Today I became a computational science ninja master! Well, maybe not so much the ninja part, but my vim skills are starting to feel like Jedi powers.
This morning I gave an hour long presentation on my work putting a real-time fluid simulator and particle systems into Blender with OpenCL. It’s a pretty high level overview of the whole project, but I do dive into a little bit of math and CS for a few minutes so hopefully there is something for everyone who is interested in the project.
[RTPS] Master’s Defense Presentation from Ian Johnson on Vimeo.
Here are the slides I used (not so great out of context, and I have demos that go with them in the video)
As I’ve now graduated and moving to a sweet new job (more on that in the next post) my Blender and OpenCL dev will have to take a back seat and be more of a hobby. Luckily there are other talented students working with the project and it will be used in the third installment of the Game Design class at Florida State University (where Blender and the Blender Game Engine are used as the primary tool).
Much love to the whole Blender community for all the help and support throughout this project! I can’t thank my advisor, Gordon Erlebacher enough for everything he’s taught me, including how to think like a scientist. It’s the end of an era for me, in one week school will not be starting for me like it has my whole life. I’m excited by what’s in front of me and I feel like I’ve been slingshotted into the future!
The summer months certainly fly by, especially when you are running in parallel…
Anyway, the project has culminated in some working code and interesting test runs on the Longhorn visualization cluster. We are using some actual flow data sets, for example this fishtank:
 |
| From LBNL FTLE |
or this “nimrod” simulation
 |
| From LBNL FTLE |
The algorithm was implemented as described in previous blog posts, and will be written up in a paper with the performance results and conclusions we draw. Hank Childs and David Camp made this project possible, helping with everything from design, data handling and constructing an MPI framework. I was also able to use code from my RTPS project to which Gordon Erlebacher is a major contributor.
We have run studies where we fill most of the 4GB of available GPU memory, which we can scale up to more nodes with big enough data sets. One thing I really like from this project which Hank introduced me to is Gantt charts.
 |
| From LBNL FTLE |
They are a way of showing how much time each process spends on different parts of the algorithm. This one isn’t labeled, I just like the way it looks. I think I will need to do a whole separate blog post on Gantt charts.
I’d like to thank everyone in the Vis group at LBL for a fun and very educational summer! I’m back at DSC ready to put some lessons to good use in the particle code :D
This is an essay addressed to my colleagues, those who make software for fun and profit and those who enjoy all the benefits technology brings to our lives. A recent Forbes article contained an amazingly illustrative anecdote revealing the corruption inherent in software patents, a dialogue between IBM lawyers visiting the then small company Sun Microsystems:
An awkward silence ensued. The blue suits did not even confer among themselves. They just sat there, stonelike. Finally, the chief suit responded. “OK,” he said, “maybe you don’t infringe these seven patents. But we have 10,000 U.S. patents. Do you really want us to go back to Armonk [IBM headquarters in New York] and find seven patents you do infringe? Or do you want to make this easy and just pay us $20 million?” After a modest bit of negotiation, Sun cut IBM a check, and the blue suits went to the next company on their hit list.
This is practically a scene from every mob movie and crime drama ever produced, except the numbers usually aren’t this big. I feel this anecdote speaks for itself and is a call to action for geeks like me who will be next on the current big boy’s hitlist, those of us who develop and distribute open source software, those of us who create and innovate in small businesses or consulting and those of us who feel blessed to live in a time where we can make a living with our minds. This is a real threat to our livelihoods, because it shows that the status quo has finally started understanding our turbulent times and are actively using their muscle to “protect” our neighborhood. So what do we do? My personal answer to this question is rooted in pop culture.
I have been extorted with threats of physical violence in my past, but in that case the balance of power was much less complicated than the situation we collectively find ourselves in, so I reach for the source of understanding I am most familiar with: pop culture. Whether its The Wire, the Godfather series, Sons of Anarchy, Brotherhood, Carlito’s Way, or any number of other criminal dramas the pattern is always the same. The “bad guys” use their power (money, guns or muscle) to control and extort members of their community for their own profit. Inevitably some of the small businesses fight back in desperation and are wiped out, other powers compete with the central characters and either destroy and replace them or negotiate power deals to maintain the status quo within some “reasonable” bounds. This always goes to show that the power wielded by the gangsters is real, and not playing by their rules has real consequences.
So if we have to play by their rules, how do we fight back? We don’t have the money, lawyers or violent inclinations to revolt, and even the mighty Google is besieged on all sides. There is one weapon that the gangsters have no defense against, the power behind the pen being mightier than the sword: pubic opinion. In all of these dramas, and in history too, the actions and crimes perpetrated by gangsters are tempered by public opinion. There are unspoken as well as spoken rules, codes of “honor” keeping thugs in line so that their actions do not bring attention and scrutiny on their organizations. A critical mass of public hatred for an organization is often the only way to remove their power and influence.
Why is this so? I’m no psychologist and I hope other’s will chime in, but it makes sense that a government of men (rather than laws) is vulnerable to the humanity of it’s rulers. Those humans have pride, ego, vanity and morals. They do not do unscrupulous deeds out of some sick pleasure, but rather to further their goals inside the world they have created for themselves with the resources at their disposal. We need to appeal to those human tendencies, or at the very least target their weaknesses.
I propose a new crime drama, one who’s context is the 21st century, the information age. The humanity of our generation has not changed, only the tools and environments in which we find ourselves. How do we convince people who care nothing about software that there is a threat to their way of life? By making them care about the characters who compose the story. How many people such as myself will never be the victim of small town violence yet eat up any compelling power drama? How thankful am I that the rule of law has progressed so I can feel secure in my station most places I go, and how thankful will the next generation be that we shined light on a mounting story of human greed and freed technological innovation from it’s shackles.
Let’s make them care, let’s tell a story.
So I’ve been up in the Berkeley hills for a few weeks now, coding by day and hacking by night. I’ve made some progress at work which I will detail here, which includes some things I’ve been wanting to do for my RTPS project (to be detailed in a post coming soon!).
 |
| From LBNL FTLE |
A good bit of my time was spent on cleaning up my code, getting everything into (relatively) simple to interface with classes so that next week I can integrate with a coworker’s MPI framework for communication. I finally ported NVIDIA’s OpenCL Radix sort to use my C++ wrappers as well as give key-value sorting functionality (their demo only had sorting of keys). This should mean a nice little speedup for RTPS when I plug it in!
In my last report I had already implemented particle advection and FTLE computation in OpenCL for one GPU. Since then I’ve been developing my code so that each GPU (let’s call it a process) will load a subset of the domain and advects its own set of “seed” particles in that domain. The major issue here is what happens when a particle goes outside of the process’ domain (lets call it subdomain)? Each particle needs to be advected for the same amount of time, and we need to advect them over the entire domain. So when a particle crosses the boundary from one subdomain to another, we need to send it over to the other process so it can continue being advected.
 |
| From LBNL FTLE |
Additionally, once a particle is done being advected, it needs to go back to its original process so that the FTLE can be computed (each process will do FTLE calculations using their original set of seed particles).
What I’ve implemented so far is a relatively efficient way of determining which particles need to go to which subdomains on the GPU. This is important because copying memory from the GPU to the CPU is expensive (takes a while) and since we want to run a whole helluva lot of particles we don’t want to burden the CPU with checking which particle goes where. If we can do all of our bookkeeping on the GPU (and in parallel) then we only need to copy the particles needed when we want to send them to another process.
I do this by using a technique called spatial hashing, which is when you overlay a (regular) grid onto your domain, and each cell of this grid has a one dimensional index which will be our hash. The simplest way to do this is with cartesian indexing, a topic I intend to make a tutorial post on.
So if each of our subdomains is one of these big grid cells, we can go through and calculate a hash for each of the particles. Then we sort our particles based on their hash values, which puts them into contiguous (next to each other) blocks by subdomain. Now we run a quick kernel that tells us the start and end indices into the particle array for each subdomain (so if the first 150 particles are in cell 0, the start index would be 0, the end index would be 150, or we might have start index 300 end index 1000 for cell 7). Now we can quickly check how many particles moved into each subdomain and copy them out directly using the start and end indices.
Besides moving into other subdomains, particles can also go totally out of bounds which means they wont go anywhere for the rest of the simulation. We can consider them finished, and speaking of finished we also need a way to deal with particles that have run their course. So the same hashing routine that labels the particles by their subdomain also sticks any finished or completely out of bounds particles in an extra hash (its just one more than the total number of subdomains) that our cell indices kernel will recognize. Then we copy those finished particles into a new buffer and sort it by each particle’s original index (a value we keep with each particle). This way we can send finished particles back to their original processes for the FTLE computations.
Next week we start sending the particles around and stomping out the inevitable bugs in my design, it’s like parallel in parallel baby!
 |
| LBNL FTLE |
This past weekend I attended Data Insight and hacked alongside a great group of people. Our team had a lot of fun and learned a lot by tackling a tough issue, child abuse. We used the statistics for 2002 from infochimps to visualize reported child abuse cases in the US. We wanted to do more than just show proportions by category and with some creative influence from excellent speakers we came up with this:
Check the original full width standalone version
You can mouse over the circles in the top left to view each category of abuse and see the warning signs. We hope that interacting with the visualization allows people to spend a little more time exposed to something we would otherwise rather not think about. Awareness is a large part of prevention, and we hope this can help.
Additionally we’d like to point out the lack of good data in this area. The reports available offer no insight into categories which would be useful to explore, such as breakdown by age, gender, race, family size or any other metric. There seem to be proprietary data sets which require actual mail to obtain. I particularly hope that events like data insight help show the importance and usefulness of good data. Check out the winning teams for some awesome examples
I also want to shout out to my team: Kristen, Jacob and Wei. We met on Friday and became friends by Sunday. We had a lot of fun in a short amount of time!
As always I’ve gotta get technical, so I want to say a few things about how this was made. It’s all processing.js with a nice Tween library from Quasipartikel. I implemented a 2D flocking particle system based on my colleague Myrna’s work. The “kids” each represent 1% of child abuse cases, so each category has however many kid particles as percent of cases. They react to the mouse with avoidance or goal oriented behavior depending on a variable set per category. They also have flocking rules which vary slightly by category to get for example more aggressive behavior or more spread out. All the code is on github so you can check it out. Keep in mind it was hacked together in one weekend!
@enjalot
+Ian Johnson