Once again, where are we in simple mere machine readable, standardized format, truly searchable data

Things are getting better all the time. It’s not fast. But there is slow and steady progress.

2009 was a big year. We saw the Senate start publishing votes in XML, the launch of data.gov and the Open Government Directive, and the GPO released XML for the Federal Register and Code of Federal Regulations. The House also began publishing its spending data electronically. There were also open standards laws passed in Vancouver and Portland.

2010 was a big year for posturing. We saw introduced in Congress H.R. 4983: Transparency in Government Act of 2010 (Quigley), H.R. 6289: To  direct the Librarian of Congress to make available to the public the bulk legislative… (Foster), and H.R. 4858: The Public Online Information Act of 2010. The Congressional Transparency Caucus was created (Quigley/Issa). An open data law was passed in San Francisco, and bills were introduced in New York City and New York State.

This year, an open data bill was introduced in New Hampshire (HB 310-FN), and POIA was reintroduced (Israel/Tester). I just noticed that GPO has last month added a bulk data download for Public Papers of the Presidents of the United States (2009). The new House rules package addresses public access to committee records and data formats, though I am not aware if the rules have had any practical consequences.

We owe Sunlight a lot of credit for pushing many of these things forward.

Besides all of this, there have been a number of “contests” lately (http://challenge.gov/) some offering prizes to use government data. Clay Johnson posted two new ones to the sunlightlabs mail list this week. The only thing I’ll say here about the strategy of the open government movement is that we haven’t taken the challenges seriously, and I think it’s a missed opportunity to show why open data matters. But it’s just one of many things to do.

UPDATE 1: Jan 9, 2011: I made a number of mistakes the first time around, including having the diagrams flipped.

There are a number of types of color blindness (see Wikipedia) but the most common are the absence or dysfunction of the “red”, “blue”, and “green” cones in the retina. The “red” and “green” cones affect up to 10% of men, the “blue” cones and all cones in women must more rarely. When a cone is absent, the individual can’t make distinctions among colors that vary only in the quantity of that color, if you think about colors as being a mix of red, green, and blue. (Actually the cones don’t respond to prototypical red, green, and blue but instead have a distribution of response over a range of color wavelengths not necessarily particularly close to the prototypical wavelengths.)

Until today I didn’t really understand the mechanics of color blindness and so it was difficult to understand how to choose good color spectra. Worse, the only simple guide I could find through Googling gave some examples of good color palettes to use without explanation and without relation to the various types of color blindness.

Here’s what I’ve learned this afternoon.

The CIE 1931 color space is a mathematical model for our perception of color based on the activity of the three types of cones. As with “normal” seeers who have three types of cones, the CIE 1931 model has three dimensions of color. And it has two powerful implications: first, it gives us a coordinate system that covers the gamut of colors that can be perceived. Second, it gives a mathematical model that can be used to understand what happens for color blind people. In particular, in the CIE 1931 flattened two-dimensional color space, color blindless is represented by radial lines emanating from a red, green, or blue “copunctal point”. Color blind individuals (of each type) cannot distinguish two colors if they fall on the same radial line. These are called confusion lines. (Also, this is an interesting way of understanding the reduction of one dimension of perception.)

CIE 1931 Color Space and Confusion Lines for "Red" Color Blindness, from http://www.colblindor.com

I found this fascinating.

Now let’s make this practical. For a web designer concerned about accessibility, avoid following radial lines! More on this in a moment.

Backing up from color blindness, it’s important that the choice of colors on a spectrum are spaced to correspond with our ability to distinguish nearby colors. One drawback of the CIE 1931 model is that, for instance, green gets an unfairly large region compared to other colors — that whole area up top just looks like the same green to me. A newer alternative color space called CIE 1976 (L*, a*, b*) (a.k.a. CIE LAB) is a transformation of the older CIE 1931 space so that equal distances in the color space represent equal amounts of perceptual distance. A color spectrum for a chart should be taken by drawing a line or curve through this type of color space. (In the images below, the color space is larger than the colored region but the black areas cannot be represented on computer screens because they do not fall in the RGB color space.)

Now we have to put these two together. The CIE 1931 color space gives a model for how to choose accessible colors: avoid the confusion lines. The best way to avoid a confusion line is to go perpendicular to confusion lines. But we want to go perpendicularly in CIE LAB space so that we make the most perceptually distinct step. (I’m assuming that perceptual distinctiveness between two points in CIE LAB space is unaffected by color blindness. It’s probably wrong but good enough.)

Let’s start with protanopia, the lack of “red” cones, as an example. The image below plots in the dark dotted lines the confusion lines for protanopia on the CIE LAB color space. Note that because CIE LAB space is a distortion of the CIE 1931 space, the confusion lines no longer appear to radiate from a point.  (This color space has a third dimension for lightness, L in 0-100, not shown. Here I’m choosing L=50.) For a “normal”-sighted individual, any path through the color space will be perceptually useful for a chart. For a protanopic, only paths that go perpendicular to the confusion lines will have maximal perceptible differences. If you follow confusion lines, the protanopic will not be able to tell the difference. As you can tell, going from red to green is not a good idea since it follows a confusion line. The perpendiculars are indicated by white arrows. Good gradients follow the perpendiculars.

Protan Spectrum Lines in CIE LAB L=50

Here’s the full set of images for protanopia (red, left), deuteranopia (green, middle), and tritanopia (blue, right), for different values of lightness:

As you can see, “red” and “green” cone color blindness is similar. “Blue” cone color blindness is totally different, in fact it’s practically a 90-degree rotation of the other two making it impossible to follow a line that is maximally perceptible by everyone.

Since the first two are similar and the prevalence of tritanopia and tritanomaly are considerably rarer compared to the other two, if we put tritanopia and tritanomaly aside (for now!) and design for the other two, we might be able to choose a single color spectrum that at least works reasonably well for those cases. A good color spectrum to use will be a vertical line that stays within the RGB boundary, either orange to blue or red to purple. That said, if we vary from the perpendiculars a little bit we might be able to satisfy everyone a little. Orange to turquoise and green to pink go diagonally across the color space and so might cover everyone.

That said, this is all theoretical. I’m not color blind so I don’t have any intuitions about whether this is right. Also, this is my first time getting into the math of colors so… maybe I got it wrong somewhere. In fact, in my first version of this I had numbers backwards and perpendicular lines that weren’t. Hopefully this is closer to the truth now. (And I appreciate the great explanations given by Daniel Flück at his blog.)

Finally, apparently everyone can see lightness, so the most accessible spectrum is just varying the lightness (and the color doesn’t matter).

These images were created with a Python script and the grapefruit, numpy, and matplotlib libraries. Here is the code:

# Usage: python plot.py L protan|deutan|tritan
#
# e.g. python plot.py 50 protan
# e.g. python plot.py 50 protan
#
# To generate all of the images in at a bash shell:
#    for L in {25,50,75}; do for b in {protan,deutan,tritan}; do echo $L $b; python plot.py $L $b; done; done

###########################################

import sys
from math import sqrt, atan2
from grapefruit import Color
import matplotlib.pyplot as plt
import numpy

w, h = (480, 480)
L = float(sys.argv[1])
bt = sys.argv[2] # blindness type

# According to http://www.colblindor.com/2009/01/19/colorblind-colors-of-confusion/
# These are points for each type of color blindness around which the dimensionality
# of the color space is reduced, in CIE 1931 color space.
copunctal_points = {
	"protan": (0.7635, 0.2365),
	"deutan": (1.4000, -0.4000),
	"tritan": (0.1748	, 0.0000)
	}

# Draw the color space.
colorspace = [[(0.0,0.0,0.0) for x in xrange(0, w)] for y in xrange(0, h)]
for x in xrange(0, w):
	for y in xrange(0, h):
		# Compute the CIE L*, a*, b* coordinates (easy, since our x,y coordinates
		# are just a translation and scaling of the LAB coordinates).
		a = 2.0*x/(w-1) - 1.0
		b = 1.0 - 2.0*y/(h-1)

		# Convert this into sRGB so we can plot the color, and plot it.
		clr = Color.NewFromLab(L, a, b)
		r, g, b = clr.rgb
		if r < 0 or g < 0 or b < 0 or r > 1 or g > 1 or b > 1:
			continue

		colorspace[y][x] = (r,g,b)

# Draw the confusion line or spectrum line gradient.
csegs = 15
contourpoints = {
	"x": [[0 for x in xrange(0, csegs)] for y in xrange(0, csegs)],
	"y": [[0 for x in xrange(0, csegs)] for y in xrange(0, csegs)],
	"spectrum": [[0 for x in xrange(0, csegs)] for y in xrange(0, csegs)],
	"confusion": [[0 for x in xrange(0, csegs)] for y in xrange(0, csegs)]
	}
for xi in xrange(0, csegs):
	for yi in xrange(0, csegs):
		# Compute pixel coordinate from grid coordinate.
		x = xi/float(csegs-1) * (w-1)
		y = yi/float(csegs-1) * (h-1)

		# Compute the CIE L*, a*, b* coordinates (easy).
		a = 2.0*x/(w-1) - 1.0
		b = 2.0*y/(h-1) - 1.0

		# Compute the corresponding CIE 1931 X, Y, Z coordinates.
		X, Y, Z = Color.LabToXyz(L, a, b)

		# Convert CIE 1931 X, Y, Z to CIE 1931 x, y (but we'll keep capital
		# letters for the variable names). The copuntual point is in CIE 1931
		# x, y coordinates.
		X, Y = (X / (X + Y + Z), Y / (X + Y + Z))

		contourpoints["x"][yi][xi] = x
		contourpoints["y"][yi][xi] = y

		# To compute the confusion lines, we plot a contour diagram where
		# the value at each point is the point's angle relative to the copunctal
		# point. Two points on the same confusion line will have the same angle,
		# and contour plots connect points of the same value.
		dY, dX = Y - copunctal_points[bt][1], X - copunctal_points[bt][0]
		contourpoints["confusion"][yi][xi] = atan2(dY, dX) # yields confusion lines

# To compute the spectrum lines, we want lines perpendicular to the
# confusion lines. In my first attempt at this, I computed perpendiculars
# in the CIE 1931 space by choosing the contour plot value at a point to
# be the *distance* from the point to the copunctal point. This plotted
# the concentric circles around the copunctal point, transformed to
# CIE LAB space.
#
# However this is wrong, because the perpendiculars should be computed
# in CIE LAB space, which is the perceptual space. To compute the
# perpendiculars, we compute the gradient of the matrix that underlies
# the confusion lines. Then the gradient is plotted with the quiver plot type.
contourpoints["spectrum"] = numpy.gradient(numpy.array(contourpoints["confusion"], dtype=numpy.float))
for xi in xrange(0, csegs): # normalize!
	for yi in xrange(0, csegs):
		d = sqrt(contourpoints["spectrum"][0][yi][xi]**2 + contourpoints["spectrum"][1][yi][xi]**2)
		contourpoints["spectrum"][0][yi][xi] /= d
		contourpoints["spectrum"][1][yi][xi] /= d

# Draw it.
plt.figure()
plt.axes(frame_on=False)
plt.xticks([])
plt.yticks([])
plt.imshow(colorspace, extent=(0, w, 0, h))
plt.quiver(contourpoints["x"], contourpoints["y"], contourpoints["spectrum"][1], contourpoints["spectrum"][0], color="white", alpha=.5)
plt.contour(contourpoints["x"], contourpoints["y"], contourpoints["confusion"], w/15, colors="black", linestyles="dotted", alpha=.25)
plt.text(0, 0, "~".join(sys.argv[2:]) + "; L=" + sys.argv[1], color="white")
plt.savefig("colorspace_" + "_".join(sys.argv[1:]) + ".png", bbox_inches="tight", pad_inches=0)

Here are histograms for horizontal and vertical resolutions based on visitors to my site GovTrack.us over the last month. The horizontal resolutions show that around 95% of visitors support at least 1024 pixels, but it drops off to only around 70% of visitors supporting a greater horizontal resolution. The 70% hangs out till about 1280 pixels (meaning, should we be designing for 1280 pixels now and make things harder for just the remaining 30%?). Then it drops again to a mere 35% for anything greater than 1280. And as for the standard wide-screen resolution of 1680, it’s just around 15%.

For reference, the iPad’s resolution (in its most popular orientation) is 768×1024.

With 1024 pixels horizontally still the resolution most widely supported, it’s not surprising that 780 pixels vertically is the point of a big drop off too, from around 95% down to less than 50% supporting anything greater. While 70% of visitors support 1280 pixels horizontally, only around 30% support its 4:3-corresponding vertical resolution of 1024 (probably as more people are using widescreens).

I don’t believe any of my transparency colleagues believe that there is a pervasive systematic problem with Hill staff & Members making literally corrupt decisions on a regular basis. However, I’m sure everyone here recognizes systemic selection biases (who can afford to get elected) and incentives (the revolving door) that are worthy of study.

The net effect of these biases and incentives is not clear, but there is certainly an effect. We know there have been a few bad apples and it is the public’s duty to be on the lookout for more. And we know that the biases and incentives affect policy results i.e. through who is elected, how committees are assigned, what lobbyists/advocates have access to Congress’s ears, and maybe in some more pernicious ways.

Now whether the net effect is in some sense good, neutral, or bad is something we’re disagreeing on. Tom is basically defending neutral, while most else would say bad.

Compared to what, and how would you know?

The problem with this discussion is that we can’t make up a hypothetical less-money-obsessed world world that we would all agree on. Take away money and some other aspect of the human condition is going to take its place. And even if we could imagine a world, how would you measure if that world was better off?

I’ll try to tie this back into the point I initially made:

Discovering bad applies through investigative, data-driven reporting is great for the country. It is actionable information. But while reporting on mere “correlation” establishes a *possible* bias or incentive, it neither indicates an actual effect on policy nor suggests any action that we could take that we could be reasonably sure would in fact improve policymaking.

Of course, correlations can be the beginning of an investigative project. Paul Blumenthal’s recent post “Incoming finance committee chairman relies on finance campaign contributions” raises a lot of concern over correlations. But its relevance is backed up by other observations and makes a good case that, at the very least, the media should be keeping a close eye on a Member of Congress who is being tempted by some very strong incentives. Hopefully he’ll resist the temptations.

For instance, last month Lisa Rosenberg wrote for Sunlight that without additional election spending disclosure we are headed toward the “corruption of our democracy by secret campaign spending.” And MAPLight.org describes its mission as being a watchdog for when “[e]lected officials collect large sums of money to run their campaigns, and they often pay back campaign contributors with special access and favorable laws.”

I certainly don’t doubt that money can corrupt, especially systemically. My favorite open secret is that Members of Congress are assigned to committee in part by how well they have fund-raised for the party (which to me sounds like a simple bribe).

But where I get worried is when an organization’s reporting arm gets caught up in reporting only on one side, making the body of evidence appear to support that corruption is wide-spread when in fact it is the exception rather than the rule, let alone a systemic problem.

What prompted me to write this was actually a blog post over at the Center for Responsive Politics that exemplifies exactly the type of reporting that is often missing. Megan Wilson writes on OpenSecrets today that “General Motors’ Political Committee Cut Big Checks to Lawmakers Who Voted Against Company’s Bailout.” Wison calls it “ironic,” that GM’s PAC seemed to be promoting candidates against its own interests. Well, not exclusively but at least two-to-one (“$63,500 to [congressmen] who voted against federal assistance for the company. That’s more than one-third of the overall amount GM gave to all House candidates this election cycle.”).

Ironic is one way to look at it. But more interesting to me is that of all times you might think we would see some easily understood evidence of a corrupting influence of money, lo and behold we see clearly that that’s not the case.

There is hope for our system after all.

I’m going to take a little shot at a post Wilson wrote in September, “Journalists, Media Professionals Donating Frequently to Federal Political Candidates this Election Cycle“. She wrote, “235 people … identified themselves on government documents as journalists, or as working for news organizations, who together have donated more than $469,900 to federal political candidates, committees and parties during the 2010 election cycle … with the median amount donated coming in at $500.”

As she noted, many of the donations came from those employed by “lighter fare” such as ESPN, or were employed in a non-reporting (i.e. business) role. She provided a spreadsheet of the numbers she used. When I looked over it, to me it appeared as if around half of the contributions were from individuals whose job description clearly indicated there was no conflict of interest: science writers, radio talk show hosts, etc.

If you’ll give me the benefit of the doubt here, or even if you don’t, we’re talking about about 100-200 people nation-wide who were journalists who might have made a conflict-of-interest mistake. That sounds pretty good to me! Where’s the reporting on the other 90,000 journalists that abstained from contributing to a candidate? Is there any substantial impact on reporting or on policymaking that resulted from any of these so-thought bad contributions? I doubt it. (If any of the contributions were in any sense nefarious, they are from people who do more political damage by what they report, rather than by who they give money to.)

So, those are my thoughts tonight.

The general idea is to use the phone to create a forwarded port from a laptop to some other machine with access to the web, and then to run a VPN connection over that port.

You will need:

• A phone running Android 2 or later (or maybe earlier).
• ConnectBot installed on the phone
• A “host” machine running Linux, with SSH and OpenVPN installed and on which you have root access, and which has a public IP address.
• Your Linux laptop that you want to tether, where you’ll need OpenVPN and root access again. And the Android SDK development tools (in particular, adb).

Setup on the host machine (i.e. the VPN server):

• Install OpenVPN. Start the server with the following command: sudo /usr/sbin/openvpn --proto tcp-server --local localhost --dev tun --ifconfig 192.168.2.1 192.168.2.2
• This will set up the server to run and listen for incoming VPN connections from the localhost only, using 192.168.2.1 for itself on the VPN and expecting your laptop to be configured for 192.168.2.2 (in the last step).
• The VPN itself is set up unencrypted since the VPN connection will be running over an encrypted line (see below).
• Obviously you’ll want to do this before you leave your home, though you also have an opportunity to start it where I’ve noted it below.

On the phone:

• In Settings -> Applications -> Development turn on USB Debugging, which allows us to forward TCP connections through the USB cable.
• Connect the phone to the laptop using the USB cable.
• In ConnectBot, set up a new connection to the host machine with a SOCKS (aka dynamic) port forward on port 1080.
• Start the connection and log in to the host machine.
• If you haven’t already, start OpenVPN on the host machine at the command prompt.

On the laptop:

• Start a port forward from the laptop to the phone using: sudo adb forward tcp:1080 tcp:1080
• Set up DNS options in an alternate /etc/resolv.conf.tether file. For instance, put “nameserver IPADDR” into the file where IPADDR is the IP address of a public nameserver (e.g. copy it from /etc/resolv.conf on the host machine).
• Start the VPN:
  sudo openvpn --proto tcp-client --dev tun --socks-proxy localhost --remote localhost --ifconfig 192.168.2.2 192.168.2.1 --route 0.0.0.0 0.0.0.0 --script-security 2 system --route-up "/bin/cp /etc/resolv.conf.tether /etc/resolv.conf"
• The VPN connection first connects to the standard SOCKS port of 1080 on localhost, which adb is forwarding to the actual SOCKS proxy server on the phone. Through the proxy OpenVPN is connecting to “localhost” which is passed through the proxy to the host machine where it resolves to the host machine itself.
• The route parameter sets up a default route to forward traffic on the laptop over the VPN. And the final option sets up DNS from your preconfigured options.

I haven’t tested this really. Just some notes for later.

I’m wary of limitations on political speech, especially when the limitations are uneven. As the ACLU wrote:

The [bill] includes an amendment obligating many advocacy organizations that wish to speak out on candidates and, in certain situations, political issues, to release the identities of many of their donors, while allowing a few large mainstream organizations to preserve the privacy of their donors. The amendment exempts organizations that have over 500,000 members, are over ten years old, have a presence in all 50 states and whose revenue from corporations and unions is less than 15 percent. By exempting larger mainstream organizations from certain disclosure requirements, the bill inequitably suppresses only the speech of smaller, more controversial organizations and compromises the anonymity of small donors.

The discussion of the Citizens United decision and the DISCLOSE Act in the blogosphere has taken it for granted that money is bad for politics. Or, perhaps we should say monetary inequality is bad for politics. (And if that’s the problem, exactly what does DISCLOSE do to rectify that? I guess it makes big spenders think twice about spending money on politics because they will have to put their name behind it. But what if they are proud to put their name behind a candidate? What if it even becomes good publicity to put your name behind a candidate?) In any case, monetary inequality is only part of the picture. What’s missing from the discussion is looking at how the money buys influence.

Interlude.

Back in the 1970s, when IBM dominated the world of computers with their high-cost mainframes, entrepreneurs — later to start Apple — uprooted the link between money and access to computational power by reinventing the computer in creating the personal computer (democratizing computational power!). In the 1990s, the big institutional newspapers and the newly dominating cable news channels ruled the media scene, but in the early 2000s bloggers started to uproot the power of the “MSM” by decentralizing public discussion of news (and in some impressive cases even taking on the role of investigative journalists). In the late 2000s with the emergence of the Open Government Data movement world-wide, we’re seeing an attempt to empower citizens by providing them with access to government information that only interests with well-paid lobbyists could access before.

End interlude.

We’re not powerless when it comes to the link between money and influence. This leads us to an alternative approach to decoupling monetary inequality with political inequality. Rather than essentially reducing monetary inequality by force of law (by discouraging corporate spending), we should be looking at uprooting the link between money and influence.

Money buys influence through advertising, because the only way citizens learn about candidates is through advertising. What if we could undermine that pathway? What if there were other ways for citizens to learn about candidates where there was no pay-to-play? Entrepreneurs and civic hackers have done this type of thing before. Perhaps it’s time to focus our skills on this problem.

I’m a new Android user on my HTC Incredible, and while I’ve read that the next version of Android is going to support tethering, it might take a while for this to be supported on existing HTC devices and I’m not sure I believe Verizon will let HTC ship it. So the hacker that I am, I found a relatively elegant way to tether. The only catch is that you need some other Linux machine with a public facing address on the Internet to serve as your “router”.

The only other good technique I found that works on Linux is to use an app that provides port forwarding or a SOCKS server. This isn’t really tethering as I understand it. SOCKS gets you only so far: it won’t do DNS, for instance, and it only works in applications that support SOCKS. The solution here provides what looks like a full Internet connection to the tethered device.

The gist is:

• The laptop you want to tether uses the socat tool to create basically a virtual ethernet device (“tun0″) that forwards packets over a TCP connection, rather than a physical cable. The tun0 device is set up as the default gateway for the computer so that its Internet connection goes through the virtual device.
• This TCP connection goes first to localhost, but an Andr0id SDK tool forwards the TCP connection over the USB cable to your phone.
• The Connect Bot app on your phone makes an SSH connection to the host computer and sets up port forwarding from the phone to the host. I’m not particularly concerned about encrypting the traffic, but the advantage of using a port-forward over SSH (versus using e.g. the Internet Sharer app by JADS to forward directly) is that on the host side we can make sure that we only provide Internet sharing for ourselves and not the whole world.
• The host computer is listening for the connection from localhost only and forwards packets from the connection back into a second virtual ethernet device (“tun0″ on the hots). Internet sharing (NAT) is enabled for the tun0 device so that the packets get routed to the Internet.

You will need…

• An Android phone. I’m testing on Android 2.1. No root access is required here.
• A Linux computer to tether. I’m testing on Ubuntu 9.04. Root access required.
• A Linux computer that has a permanent connection to the Internet with a public IP address on which you have root access. This is the host computer. I’m testing this one with Ubuntu 10.04.

Before you begin…

On the phone, install the free Connect Bot app from the marketplace. Enable USB debugging on the phone in Settings -> Applications -> Development -> USB debugging, which allows us to forward connections over the USB cable.

On the computer to tether, while you still have an Internet connection, download  the Android SDK for Linux. Extract it and find the adb tool. On both the computer to tether and the host computer, make sure you have socat installed. (In Ubuntu, apt-get install socat.)

And you should be good to go.

This was a solid 2-day project with less than 300 lines of code and it’s something that only recently became this easy to do. I used Amazon Web Services (AWS), Census TIGER/Line cartographic data in an AWS pubic data snapshot, OpenStreetMap for the street detail in an AWS snapshot prepared by MapBox.com, Mapnik to render the maps (pre-installed on an AWS machine image prepared by MapBox), and the Python modules osgeo (for OGR) and PIL.

Here’s what  I did. This took a lot of trial and error, but in the end the steps were relatively simple.

Setting up the EC2 instance and the OpenStreetMap (OSM) planet data:

• Start up a new Amazon EC2 Linux instance using the AWS machine image (AMI) prepared by MapBox linked above.
• Create Amazon Elastic Block Storage (EBS) volumes for the two data sets (OpenStreetMap and Census TIGER/Line) in the same availability zone as the EC2 instance. If you do it in the AWS console, you’ll just need to search for the snapshots by ID or name (see the links above).
• Attach the two volumes to the running EC2 instance as /dev/sdf (OSM) and /dev/sdg (TIGER).
• Log into the EC2 instance with SSH.
• Mount the two volumes: mkdir /mnt/osm; mount -t ext3 /dev/sdf /mnt/osm; mkdir /mnt/tiger; mount -t ext3 /dev/sdg /mnt/tiger
• Following the MapBox instructions, attach the OSM data to Postgres, change the Postgres configuration to remove password protection, and restart Postgres.

To set up Mapnik, I followed the OpenStreetMap wiki which shows how to reuse their map styling. Most of the steps can be skipped because the data has already been set up in Postgres by MapBox. That involved:

• Getting the OSM Mapnik files from their SVN repository.
• Downloading some extraneous boundary information.
• Create a new style definition that controls how map features are rendered based on the OSM defaults.
• Editing the defaults a) so it actually works, and b) so it looks good at high DPI for printing (increasing font sizes, removing some icons). This took a lot of trial and error since I didn’t understand what was going on and regenerating a map takes some time.

The last step was to write a Python script that invokes Mapnik for each congressional district and generates a high-resolution map image.

• The Census’s TIGER/Line cartographic data has a Shapefile-format file for each state containing the congressional districts in the state. The osgeo/OGR Python module can load the file and tell you the latitude/longitude bounds of the congressional district (among other things).
• Then the Mapnik Python bindings are used to create a new map with the given size, loading in the OSM street data.
• Additional layers are added from the TIGER/Line data for place names (CDPs and county subdivisions if you’re familiar with Census data), county names and borders, state borders (and shading of other states), and the boundaries of the congressional district itself and shading of other congressional districts.
• After rendering the map, which takes ~30 seconds, I used the Python Imaging Library module to add header and footer text with a nice translucent effect.

Generating the maps at three resolutions for all of the congressional districts (except districts at-large) took several hours. I let it run overnight. They’re stored on Amazon S3 (the s3cmd tool is really useful for that).

There’s still room for a lot of improvement. After playing with the style instructions I got too much local road detail that in some places just ruins the whole map at low resolution. And in many places the county names aren’t showing up. Maybe because there’s too much detail. It’ll take some more trial and error to fix.

The source code (which includes all of the preparation steps in detail) is posted here.

Honestly, I don’t really get what the big hubub is over this information. First, are corrupting influences on the administration really going to stop corrupting because of this? And, a corollary, who exactly is in a position to be reading over the records to make sure nothing bad is going on? Who are these visitors anyway?

But I always enjoy playing with data all the same. To do it with a little levity, I thought I would profile Robynn Sturm’s visitors. I met Robynn recently and certainly got the feeling that of all people to hold the title of Assistant Deputy Chief Technology Officer for Open Government for the United States of America, she seemed like a good person to hold the job.

Anyway, in September-October 2009, she had 35 visits. I only have the names and can’t be sure of who they are, but I’ll do my best to give Google search results that might be reasonable. Text comes from the pages I’ve linked to.

Ellen Alberding and Gretchen Sims are the president and education program manager, respectively, of the Joyce Foundation, which supports efforts to protect the natural environment of the Great Lakes, to reduce poverty and violence in the region, and to ensure that its people have access to good schools, decent jobs, and a diverse and thriving culture. Sims donated to the Democrats in the last two presidential elections. (I wouldn’t have mentioned it except that that’s how I found out where she worked.)

Ethan Batraski (@ethanjb): startup co-founder, mathematician, machine learning researcher, techanista, sharing thoughts on product management, startups, venture funding & semantic web

Marc Berejka worked in senior government affairs roles at Microsoft, including eight years as a lobbyist for the high-tech giant. Says Politico: “Opponents of the Obama administration’s position on patent reform say that David Kappos and Marc Berejka, who recently took top jobs in the Commerce Department, are wielding too much influence over a policy that stands to benefit both of their former companies.”

Lawrence Brandt, a co-editor of Digital Government, is a program manager within NSF.

Gerard Fiala is the staff director in the Senate HELP committee’s subcommittee on employment and workplace safety.

Seena Jon Ghaznavi (@sjgood) is a young actor in the movie Death of a President.

Michael Harding – This name is too popular.

Greg Horowitt and Victor Hwang are co-founders and Managing Directors of T2 Venture Capital, a venture fund focused on breakthrough technology spinning out of government and academia. Horowitt is also Director and Co-Founder of the Global CONNECT program based at the University of California, San Diego, and is a key thought leader in the field of ‘innovation systems’, and their relevant applications for sustainable regional economic development through technology commercialization.

Ester Lee might be the Ester Lee that works for AT&T. But maybe not.

Joseph Mancio – another somewhat popular name.

Dominic Mauro (@mynameisdom) is a TA for Internet Law at NY Law School (which, for reference, is the school that Beth Noveck comes from; Beth is Robynn’s boss).

Sara Mirsky is the American Constitutional Society’s NYLS chapter‘s co-president. (See note about about NYLS).

Courtney Patterson (@cnpatterson) is a obsessive-compulsive law student in NYC. (We probably know at which school.)

Gina Wells is another common name.

Phillip Wickham is President and CEO of the Kauffman Fellows Program at the Center for Venture Education in Palo Alto, CA. The mission of the Kauffman Fellows Program is to develop the next generation of leaders in venture capital.

John Bell is way too common a name.

Pamela Frugoli works for the Department of Labor.

Daniel Gomez might be a lawyer.

Melissa Sperry is too common a name.

Meredith Stewart is another popular name.

Haley van Dyck was a part of the Obama 08 campaign team, according to her step-mom’s LinkedIn page, who, btw, is proud of her.

Jing Vivatrat is either an m&a businesswoman or an FCC director, or both.