Once again, where are we in simple mere machine readable, standardized format, truly searchable data

Things are getting better all the time. It’s not fast. But there is slow and steady progress.

2009 was a big year. We saw the Senate start publishing votes in XML, the launch of data.gov and the Open Government Directive, and the GPO released XML for the Federal Register and Code of Federal Regulations. The House also began publishing its spending data electronically. There were also open standards laws passed in Vancouver and Portland.

2010 was a big year for posturing. We saw introduced in Congress H.R. 4983: Transparency in Government Act of 2010 (Quigley), H.R. 6289: To  direct the Librarian of Congress to make available to the public the bulk legislative… (Foster), and H.R. 4858: The Public Online Information Act of 2010. The Congressional Transparency Caucus was created (Quigley/Issa). An open data law was passed in San Francisco, and bills were introduced in New York City and New York State.

This year, an open data bill was introduced in New Hampshire (HB 310-FN), and POIA was reintroduced (Israel/Tester). I just noticed that GPO has last month added a bulk data download for Public Papers of the Presidents of the United States (2009). The new House rules package addresses public access to committee records and data formats, though I am not aware if the rules have had any practical consequences.

We owe Sunlight a lot of credit for pushing many of these things forward.

Besides all of this, there have been a number of “contests” lately (http://challenge.gov/) some offering prizes to use government data. Clay Johnson posted two new ones to the sunlightlabs mail list this week. The only thing I’ll say here about the strategy of the open government movement is that we haven’t taken the challenges seriously, and I think it’s a missed opportunity to show why open data matters. But it’s just one of many things to do.

I don’t believe any of my transparency colleagues believe that there is a pervasive systematic problem with Hill staff & Members making literally corrupt decisions on a regular basis. However, I’m sure everyone here recognizes systemic selection biases (who can afford to get elected) and incentives (the revolving door) that are worthy of study.

The net effect of these biases and incentives is not clear, but there is certainly an effect. We know there have been a few bad apples and it is the public’s duty to be on the lookout for more. And we know that the biases and incentives affect policy results i.e. through who is elected, how committees are assigned, what lobbyists/advocates have access to Congress’s ears, and maybe in some more pernicious ways.

Now whether the net effect is in some sense good, neutral, or bad is something we’re disagreeing on. Tom is basically defending neutral, while most else would say bad.

Compared to what, and how would you know?

The problem with this discussion is that we can’t make up a hypothetical less-money-obsessed world world that we would all agree on. Take away money and some other aspect of the human condition is going to take its place. And even if we could imagine a world, how would you measure if that world was better off?

I’ll try to tie this back into the point I initially made:

Discovering bad applies through investigative, data-driven reporting is great for the country. It is actionable information. But while reporting on mere “correlation” establishes a *possible* bias or incentive, it neither indicates an actual effect on policy nor suggests any action that we could take that we could be reasonably sure would in fact improve policymaking.

Of course, correlations can be the beginning of an investigative project. Paul Blumenthal’s recent post “Incoming finance committee chairman relies on finance campaign contributions” raises a lot of concern over correlations. But its relevance is backed up by other observations and makes a good case that, at the very least, the media should be keeping a close eye on a Member of Congress who is being tempted by some very strong incentives. Hopefully he’ll resist the temptations.

For instance, last month Lisa Rosenberg wrote for Sunlight that without additional election spending disclosure we are headed toward the “corruption of our democracy by secret campaign spending.” And MAPLight.org describes its mission as being a watchdog for when “[e]lected officials collect large sums of money to run their campaigns, and they often pay back campaign contributors with special access and favorable laws.”

I certainly don’t doubt that money can corrupt, especially systemically. My favorite open secret is that Members of Congress are assigned to committee in part by how well they have fund-raised for the party (which to me sounds like a simple bribe).

But where I get worried is when an organization’s reporting arm gets caught up in reporting only on one side, making the body of evidence appear to support that corruption is wide-spread when in fact it is the exception rather than the rule, let alone a systemic problem.

What prompted me to write this was actually a blog post over at the Center for Responsive Politics that exemplifies exactly the type of reporting that is often missing. Megan Wilson writes on OpenSecrets today that “General Motors’ Political Committee Cut Big Checks to Lawmakers Who Voted Against Company’s Bailout.” Wison calls it “ironic,” that GM’s PAC seemed to be promoting candidates against its own interests. Well, not exclusively but at least two-to-one (“$63,500 to [congressmen] who voted against federal assistance for the company. That’s more than one-third of the overall amount GM gave to all House candidates this election cycle.”).

Ironic is one way to look at it. But more interesting to me is that of all times you might think we would see some easily understood evidence of a corrupting influence of money, lo and behold we see clearly that that’s not the case.

There is hope for our system after all.

I’m going to take a little shot at a post Wilson wrote in September, “Journalists, Media Professionals Donating Frequently to Federal Political Candidates this Election Cycle“. She wrote, “235 people … identified themselves on government documents as journalists, or as working for news organizations, who together have donated more than $469,900 to federal political candidates, committees and parties during the 2010 election cycle … with the median amount donated coming in at $500.”

As she noted, many of the donations came from those employed by “lighter fare” such as ESPN, or were employed in a non-reporting (i.e. business) role. She provided a spreadsheet of the numbers she used. When I looked over it, to me it appeared as if around half of the contributions were from individuals whose job description clearly indicated there was no conflict of interest: science writers, radio talk show hosts, etc.

If you’ll give me the benefit of the doubt here, or even if you don’t, we’re talking about about 100-200 people nation-wide who were journalists who might have made a conflict-of-interest mistake. That sounds pretty good to me! Where’s the reporting on the other 90,000 journalists that abstained from contributing to a candidate? Is there any substantial impact on reporting or on policymaking that resulted from any of these so-thought bad contributions? I doubt it. (If any of the contributions were in any sense nefarious, they are from people who do more political damage by what they report, rather than by who they give money to.)

So, those are my thoughts tonight.

I’m wary of limitations on political speech, especially when the limitations are uneven. As the ACLU wrote:

The [bill] includes an amendment obligating many advocacy organizations that wish to speak out on candidates and, in certain situations, political issues, to release the identities of many of their donors, while allowing a few large mainstream organizations to preserve the privacy of their donors. The amendment exempts organizations that have over 500,000 members, are over ten years old, have a presence in all 50 states and whose revenue from corporations and unions is less than 15 percent. By exempting larger mainstream organizations from certain disclosure requirements, the bill inequitably suppresses only the speech of smaller, more controversial organizations and compromises the anonymity of small donors.

The discussion of the Citizens United decision and the DISCLOSE Act in the blogosphere has taken it for granted that money is bad for politics. Or, perhaps we should say monetary inequality is bad for politics. (And if that’s the problem, exactly what does DISCLOSE do to rectify that? I guess it makes big spenders think twice about spending money on politics because they will have to put their name behind it. But what if they are proud to put their name behind a candidate? What if it even becomes good publicity to put your name behind a candidate?) In any case, monetary inequality is only part of the picture. What’s missing from the discussion is looking at how the money buys influence.

Interlude.

Back in the 1970s, when IBM dominated the world of computers with their high-cost mainframes, entrepreneurs — later to start Apple — uprooted the link between money and access to computational power by reinventing the computer in creating the personal computer (democratizing computational power!). In the 1990s, the big institutional newspapers and the newly dominating cable news channels ruled the media scene, but in the early 2000s bloggers started to uproot the power of the “MSM” by decentralizing public discussion of news (and in some impressive cases even taking on the role of investigative journalists). In the late 2000s with the emergence of the Open Government Data movement world-wide, we’re seeing an attempt to empower citizens by providing them with access to government information that only interests with well-paid lobbyists could access before.

End interlude.

We’re not powerless when it comes to the link between money and influence. This leads us to an alternative approach to decoupling monetary inequality with political inequality. Rather than essentially reducing monetary inequality by force of law (by discouraging corporate spending), we should be looking at uprooting the link between money and influence.

Money buys influence through advertising, because the only way citizens learn about candidates is through advertising. What if we could undermine that pathway? What if there were other ways for citizens to learn about candidates where there was no pay-to-play? Entrepreneurs and civic hackers have done this type of thing before. Perhaps it’s time to focus our skills on this problem.

This was a solid 2-day project with less than 300 lines of code and it’s something that only recently became this easy to do. I used Amazon Web Services (AWS), Census TIGER/Line cartographic data in an AWS pubic data snapshot, OpenStreetMap for the street detail in an AWS snapshot prepared by MapBox.com, Mapnik to render the maps (pre-installed on an AWS machine image prepared by MapBox), and the Python modules osgeo (for OGR) and PIL.

Here’s what  I did. This took a lot of trial and error, but in the end the steps were relatively simple.

Setting up the EC2 instance and the OpenStreetMap (OSM) planet data:

• Start up a new Amazon EC2 Linux instance using the AWS machine image (AMI) prepared by MapBox linked above.
• Create Amazon Elastic Block Storage (EBS) volumes for the two data sets (OpenStreetMap and Census TIGER/Line) in the same availability zone as the EC2 instance. If you do it in the AWS console, you’ll just need to search for the snapshots by ID or name (see the links above).
• Attach the two volumes to the running EC2 instance as /dev/sdf (OSM) and /dev/sdg (TIGER).
• Log into the EC2 instance with SSH.
• Mount the two volumes: mkdir /mnt/osm; mount -t ext3 /dev/sdf /mnt/osm; mkdir /mnt/tiger; mount -t ext3 /dev/sdg /mnt/tiger
• Following the MapBox instructions, attach the OSM data to Postgres, change the Postgres configuration to remove password protection, and restart Postgres.

To set up Mapnik, I followed the OpenStreetMap wiki which shows how to reuse their map styling. Most of the steps can be skipped because the data has already been set up in Postgres by MapBox. That involved:

• Getting the OSM Mapnik files from their SVN repository.
• Downloading some extraneous boundary information.
• Create a new style definition that controls how map features are rendered based on the OSM defaults.
• Editing the defaults a) so it actually works, and b) so it looks good at high DPI for printing (increasing font sizes, removing some icons). This took a lot of trial and error since I didn’t understand what was going on and regenerating a map takes some time.

The last step was to write a Python script that invokes Mapnik for each congressional district and generates a high-resolution map image.

• The Census’s TIGER/Line cartographic data has a Shapefile-format file for each state containing the congressional districts in the state. The osgeo/OGR Python module can load the file and tell you the latitude/longitude bounds of the congressional district (among other things).
• Then the Mapnik Python bindings are used to create a new map with the given size, loading in the OSM street data.
• Additional layers are added from the TIGER/Line data for place names (CDPs and county subdivisions if you’re familiar with Census data), county names and borders, state borders (and shading of other states), and the boundaries of the congressional district itself and shading of other congressional districts.
• After rendering the map, which takes ~30 seconds, I used the Python Imaging Library module to add header and footer text with a nice translucent effect.

Generating the maps at three resolutions for all of the congressional districts (except districts at-large) took several hours. I let it run overnight. They’re stored on Amazon S3 (the s3cmd tool is really useful for that).

There’s still room for a lot of improvement. After playing with the style instructions I got too much local road detail that in some places just ruins the whole map at low resolution. And in many places the county names aren’t showing up. Maybe because there’s too much detail. It’ll take some more trial and error to fix.

The source code (which includes all of the preparation steps in detail) is posted here.

Honestly, I don’t really get what the big hubub is over this information. First, are corrupting influences on the administration really going to stop corrupting because of this? And, a corollary, who exactly is in a position to be reading over the records to make sure nothing bad is going on? Who are these visitors anyway?

But I always enjoy playing with data all the same. To do it with a little levity, I thought I would profile Robynn Sturm’s visitors. I met Robynn recently and certainly got the feeling that of all people to hold the title of Assistant Deputy Chief Technology Officer for Open Government for the United States of America, she seemed like a good person to hold the job.

Anyway, in September-October 2009, she had 35 visits. I only have the names and can’t be sure of who they are, but I’ll do my best to give Google search results that might be reasonable. Text comes from the pages I’ve linked to.

Ellen Alberding and Gretchen Sims are the president and education program manager, respectively, of the Joyce Foundation, which supports efforts to protect the natural environment of the Great Lakes, to reduce poverty and violence in the region, and to ensure that its people have access to good schools, decent jobs, and a diverse and thriving culture. Sims donated to the Democrats in the last two presidential elections. (I wouldn’t have mentioned it except that that’s how I found out where she worked.)

Ethan Batraski (@ethanjb): startup co-founder, mathematician, machine learning researcher, techanista, sharing thoughts on product management, startups, venture funding & semantic web

Marc Berejka worked in senior government affairs roles at Microsoft, including eight years as a lobbyist for the high-tech giant. Says Politico: “Opponents of the Obama administration’s position on patent reform say that David Kappos and Marc Berejka, who recently took top jobs in the Commerce Department, are wielding too much influence over a policy that stands to benefit both of their former companies.”

Lawrence Brandt, a co-editor of Digital Government, is a program manager within NSF.

Gerard Fiala is the staff director in the Senate HELP committee’s subcommittee on employment and workplace safety.

Seena Jon Ghaznavi (@sjgood) is a young actor in the movie Death of a President.

Michael Harding – This name is too popular.

Greg Horowitt and Victor Hwang are co-founders and Managing Directors of T2 Venture Capital, a venture fund focused on breakthrough technology spinning out of government and academia. Horowitt is also Director and Co-Founder of the Global CONNECT program based at the University of California, San Diego, and is a key thought leader in the field of ‘innovation systems’, and their relevant applications for sustainable regional economic development through technology commercialization.

Ester Lee might be the Ester Lee that works for AT&T. But maybe not.

Joseph Mancio – another somewhat popular name.

Dominic Mauro (@mynameisdom) is a TA for Internet Law at NY Law School (which, for reference, is the school that Beth Noveck comes from; Beth is Robynn’s boss).

Sara Mirsky is the American Constitutional Society’s NYLS chapter‘s co-president. (See note about about NYLS).

Courtney Patterson (@cnpatterson) is a obsessive-compulsive law student in NYC. (We probably know at which school.)

Gina Wells is another common name.

Phillip Wickham is President and CEO of the Kauffman Fellows Program at the Center for Venture Education in Palo Alto, CA. The mission of the Kauffman Fellows Program is to develop the next generation of leaders in venture capital.

John Bell is way too common a name.

Pamela Frugoli works for the Department of Labor.

Daniel Gomez might be a lawyer.

Melissa Sperry is too common a name.

Meredith Stewart is another popular name.

Haley van Dyck was a part of the Obama 08 campaign team, according to her step-mom’s LinkedIn page, who, btw, is proud of her.

Jing Vivatrat is either an m&a businesswoman or an FCC director, or both.

After five years of run­ning GovTrack.​us, I saw a need to move be­yond data. As we’ve seen over at the OpenCongress.org blog, reporting is an important part of following the legislative process, staying ahead of the game, and understanding the ramifications of procedural events. GovTrackInsider.​com puts a focus on grass-roots reporting of the legislative process.

GovTrack Insider has hired a small team of paid free­lance re­porters to cover con­gres­sion­al com­mit­tee meet­ings. This is the ear­li­est point in the leg­isla­tive pro­cess that we can ob­serve di­rect­ly, and it’s going to help us get an inside look like we’ve never had before. We’ll also be syndicating coverage from OpenCongress and elsewhere. In all of the articles we run, the focus will be on legislation and policy. We’ll be leaving politics and gossip to the old media.

In­sid­er is an open-access on­line news­pa­per, but one with some un­usu­al pages. For ar­ti­cles on many top­ics you’ll find on the right side a topic dash­board. There you can con­nect with other read­ers in a va­ri­ety of ways, such as a Q&A system, link submission, and a community notebook (a wiki).

In time we’ll be in­te­grat­ing In­sid­er more with GovTrack.​us so that if you’re track­ing a bill over on Gov­Track you’ll get a no­tice when­ev­er it’s men­tioned in an ar­ti­cle in GovTrack Insider.

I hope you’ll join us on this ex­per­i­ment. Happy read­ing.

(GovTrack Insider was built with the help of Josh Sulkin (of FlyOnTime.us fame).)

NJSP analyst Dean Baratta, who along with his boss joined us in Philadelphia on the first day of our work, has this to say today:

The amount of work these guys did in two days is really incredible given the shape of the underlying data. It was fine for researchers and academics but not suitable at all for the general public. . . . Not long ago, I mentioned on another site that the democratization of information and the ability of people to collaborate who (in the pre-internet days) would not even known of each others existence was one of the global issues which makes me feel optimistic. This was the type of work that I was thinking of when I wrote that. Check out the work these guys did and then start asking your local and state law enforcement agencies why they don’t make information like this available to the public.

This is a work in progress. We accomplished all we could in two days. This included a large amount of work learning about the survey, normalizing the survey data, linking it to geospacial data, and learning Django and GIS web technology so we could create a mapping website rapidly. The site displays gang statistics for the municipalities in New Jersey and maps of the locations of the major gangs present in the state.

The NJ Gang Survey Viewer was created by volunteer civic hackers Nick Cazoneri (Delaware Valley Regional Planning Commission), Robert Cheetham (Avencia), Don Coleman (Chariot Solutions), David Middlecamp (Avencia), and Joshua Tauberer (Civic Impulse), and from the New Jersey State Police Dean Baratta, intelligence analyst, and Peter Lynch. We were hosted for the weekend in the Avencia office in downtown Philadelphia. Here we are (besides me behind the camera and David who came a bit later):

We mostly didn’t know each other before the weekend, but we all got along great. Our weekend work was the Philadelphia version of the Great American Hackathon 2009, a network of similar civic technology meet-ups held throughout the nation and loosely organized by Sunlight Foundation, a DC-based government transparency nonprofit foundation.

David, one of our team members, says:

I enjoyed getting to know the group and working on something that normally doesn’t get the technical attention that it should.  Learning a new platform and contributing to the greater good, a fun weekend.  :)

If you are interested in working with New Jersey gang and crime data, you can contact me. The NJSP would like to continue this effort.

This gang survey data is relevant to a wide range of New Jersey residents and visitors to the state. The current perception of ‘gang threat’ is frequently one that is primarily urban and particularly violent. This has implications for both government and society at large. The belief that gangs are someone else’s problem — and someone else’s tax burden — could potentially reduce public support for anti-gang initiatives that go beyond an initial impulse to “lock ‘em all up.” Gangs are reported present in dozens of rural and suburban municipalities throughout the state. Almost seven out of every ten New Jerseyans live in a municipality where gangs can be found. Clearly, gangs can not be considered an exclusively urban phenomenon in any part of New Jersey.

Our work this weekend was a great experience and is proof that technologists are willing and able to work with government agencies on a volunteer basis to develop civic applications that the agencies don’t have the mandate or funding for themselves.

Last week I reviewed the House’s Statement of Disbursement electronic document along the dimensions of open government data (which unfortunately has the same abbreviation). The OGD talks about how agencies should go about the process of opening data. Here is a review of what the OGD says, organized by open data principle.

I’ll put the conclusion up front: The OGD addresses nearly all of the open government data principles that have been put forward, and even adds two of its own: being pro-active about data release and creating accountability by designating an official responsible for data quality (more on these at the end). So from this perspective, the OGD is pretty spot-on. It is very strong in public input, public review, and interagency coordination, which are normally the weakest spots of government data (but, on the other hand, this isn’t data, this is a goal, so the proof will be in the pudding). It could have been stronger in the areas of machine processability & promoting analysis, and explaining what is appropriate for data licensing (ideally, none).

Here are the details:

Information is not meaningfully public if it is not available on the Internet for free.

The OGD says “each agency shall take prompt steps to expand access to information by making it available online in open formats.” The OGD itself doesn’t say free, but executive branch policy already requires that public information not be sold to the public at more than the marginal cost of distribution — which is about as good as one might expect. So we’ll count this principle as asserted by the OGD.

— GOOD

Data Should Be Primary. Primary data is data as collected at the source, with the finest possible level of granularity, not in aggregate or modified forms.

In the OGD’s appendix where it outlines further details for agencies, it says agencies should release data “as granular as possible”.

— GOOD

Timely.

The OGD days, “Timely publication of information is an essential component of transparency. Delays should not be viewed as an inevitable and insurmountable consequence of high demand.”

— GOOD

Accessible. Data are available to the widest range of users for the widest range of purposes, meaning use an open standard, with a bulk download, and with documentation.

Machine processable: Data are reasonably structured to allow automated processing.

The OGD specifically defines “open format” — which is the subject of the directive — as something that is platform independent and machine readable. Now, here the OGD slips a little because it redefines “open” but actually leaves out open standards. I don’t think that was intentional, so we’ll give the OGD credit for mentioning open standards even though it didn’t exactly. It mentions “downloadable” but not in bulk, and there is no mention of documentation in the OGD. We can’t tell what the OGD meant by “machine readable” — I think of this term now a sloppy form of “machine processable”. It would have helped if the OGD specifically noted that the point is to support analysis and reuse of the data.

I used to use “machine readable” until someone corrected me that really any format can be read by a machine. The question is what the machine can do with it: to what degree can the data be meaningfully processed by a machine? So now I use machine-processable.

– WEAK

Non-discriminatory: Data are available to anyone, with no requirement of registration.
Non-proprietary: Data are available in a format over which no entity has exclusive control.
License-free. Dissemination of the data is not limited by intellectual property law or other terms.

The OGD says data must be “made available to the public without restrictions that would impede the re-use of that information.” Here we could have really benefited from some simple but concrete guidance.

— WEAK

Promote analysis: Data published by the government should be in formats and approaches that promote analysis and reuse of that data.

There is a sense in which this is implicit in the OGD, but maybe it is the goggles through which I read it. The OGD fails to say explicitly that analysis is the whole point of open government data.

— FAIL

Public input: The public is in the best position to determine what information technologies will be best suited for the applications the public intends to create for itself.

Public review: There should be a means for the public to interact with the data publisher during and after the data has been made. The public may have questions or may find errors. The process of creating the data should also be transparent.

These principles are perhaps the least commonly addressed, and yet it is one of the most prominent aspects of the OGD. The OGD requires agencies to allow the public to give feedback on data quality, data prioritization, and other aspects of the agency’s OGD plan. In fact, the OGD says, “Each agency shall respond to public input received on its Open Government Webpage on a regular basis.”

In addition, the OGD will form a working group (described next) that will discuss “ideas to promote
participation and collaboration, including how to … take advantage of the expertise and insight of people both inside and outside the Federal Government, and form high-impact collaborations with researchers, the private sector, and civil society.”

In the appendix where it outlines further goals for agencies, the OGD says, “Your agency should also identify key audiences for its information and their needs, and endeavor to publish high-value information for each of those audiences in the most accessible forms and formats.”

— EXCELLENT

Interagency coordination: Interoperability makes data more valuable by making it easier to derive new uses from combinations of data. To the extent two data sets refer to the same kinds of things, the creators of the data sets should strive to make them interoperable.

The OGD will establish a working group lead by the Deputy Director for Management at OMB, the Federal Chief Information Officer, and the Federal Chief Technology Officer to provide a forum to share best practices for data collection, aggregation, validation, and dissemination throughout the government, to coordinate implementations of federal spending transparency, and to provide a forum for sharing best practices for participation.

— EXCELLENT

Provenance and trust: Published content should be digitally signed or include attestation of publication/creation date, authenticity, and integrity.

Permanent Web Address: The file should have a stable location.

Safe file formats: Government bodies publishing data online should always seek to publish using data formats that do not include executable content.

Globally Unique Identifiers: This concept, important on the world wide web, is that any document, resource, data record, or entity mentioned in a database, or some might say every paragraph in a document, should have a unique identification that others can use to point to or cite it elsewhere.

Linked Open Data: This is a method for publishing databases in a standard format for interconnectivity with other databases without the expense of wide agreement on unified inter-agency or global data standards.

These get into some of the more precise details of data format. I might have liked to see provenance & trust addressed, but I am not sure whether I would really expect these principles to be included in a high level 120-day plan, at least not at this point. So their absence is not something I hold against the OGD. Still:

— FAIL

Other Notes

The OGD talks about being proactive with data release.

The OGD also adds accountability: “Each agency … shall designate a high-level senior official to be accountable for the quality and objectivity of, and internal controls over, the Federal spending information publicly disseminated.”

The best thing I see so far is the documentation provided on disbursements.house.gov. There is a nice explanation of the reporting process, a FAQ, and a glossary. There is also a table of transaction codes found in the document, and these all are crucial for anyone reading or analyzing the information. This is one of the best examples of documentation I’ve seen for government data of this kind.

Here’s an evaluation of the disclosure based on standards I’ve drawn from others and outlined here.

In summary, many of the goals are met. The important ones not met are machine processability, public input, and public review. Machine processability is a very important one and the fact that this goal was not met seriously undermines much of the reason for publishing the information in the first place.

Information is not meaningfully public if it is not available on the Internet for free.

— GOAL ACHIEVED.

Data Should Be Primary. Primary data is data as collected at the source, with the finest possible level of granularity, not in aggregate or modified forms.

We can evaluate this because the documentation actually describes how the House Clerk receives the information. It talks about some degree of aggregation taking place, such as a $10,000 travel record not necessarily being for one trip. But by and large we’re seeing the level of detail that I think is expected.

– GOAL ACHIEVED

Timely.

The SOD is published quarterly, and hopefully this will include the online/electronic version going forward. I don’t think we can have too much higher of an expectation here. Ten years from now perhaps I would like to see real-time expense reporting, but not today.

– TO BE SEEN (if the electronic version is published as often as the print version in the future)

Accessible. Data are available to the widest range of users for the widest range of purposes.

This goal from the “8 Principles” refers to:

Data Format: PDF is an open standard, and therefore a good choice among the data formats for the purpose of publishing a document. (See more below.)

Bulk Data: The 3,400-page document is provided in a single PDF file, rather than hundreds or thousands of separate downloads. This satisfies the goal of bulk data.

Documentation: Documentation is excellent. There is an explanation of the reporting process, a FAQ, a table of codes, and a glossary.

– GOAL ACHIEVED

Machine processable: Data are reasonably structured to allow automated processing.

This is the first goal which is not addressed at all by the data release. While PDF is good for documents, it is bad for tabular information. It does not support sorting, transforming, or other analysis, and it only marginally supports search. A spreadsheet format of any sort would be useful here, some formats better than others.

Considering the size of this data set, without the help of computers to process this information it is far less useful than it could be. To be given a barely-searchable 3,000-page file is only a small step up from being mailed several reams of paper.

Furthermore, there is no indication on the disbursements website that this will be considered in the future.

– GOAL NOT MET

Non-discriminatory: Data are available to anyone, with no requirement of registration.
Non-proprietary: Data are available in a format over which no entity has exclusive control.
License-free. Dissemination of the data is not limited by intellectual property law or other terms.

– GOALS ACHIEVED

Promote analysis: Data published by the government should be in formats and approaches that promote analysis and reuse of that data.

This goal (and the next two) comes from the Association of Computing Machinery’s Recommendation on Open Government and is similar to the Machine Processable goal above. So, see above.

– GOAL NOT MET

Safe file formats: Government bodies publishing data online should always seek to publish using data formats that do not include executable content.

PDF is, relatively speaking, a safe file format.

– GOAL ACHIEVED

Provenance and trust: Published content should be digitally signed or include attestation of publication/creation date, authenticity, and integrity.

According to the disbursements website, the files are digitally signed. I haven’t verified that the signature process was done correctly.

– GOAL ACHIEVED

Public input: The public is in the best position to determine what information technologies will be best suited for the applications the public intends to create for itself.

I am sure members of our community have been in touch with the Clerk’s office. However, there was no public discussion on how these files ought to have been made available, and therefore I am going to not count this goal as having been met.

– GOAL NOT MET

Public review: There should be a means for the public to interact with the data publisher during and after the data has been made. The public may have questions or may find errors. The process of creating the data should also be transparent.

This is a goal rarely given any attention. The documentation gives significant insight into this process. But there is no contact person for this data set that is made known to the public.

– GOAL NOT MET

Interagency coordination: Interoperability makes data more valuable by making it easier to derive new uses from combinations of data. To the extent two data sets refer to the same kinds of things, the creators of the data sets should strive to make them interoperable.

There is a potential to link the names of Members of Congress to their ID numbers provided in, say, the XML voting records.

– GOAL NOT MET

Permanent Web Address: The file should have a stable location.

– GOAL ACHIEVED (provided it is kept there)

Globally Unique Identifiers: This concept, important on the world wide web, is that any document, resource, data record, or entity mentioned in a database, or some might say every paragraph in a document, should have a unique identification that others can use to point to or cite it elsewhere.

– GOAL NOT MET

Linked Open Data: This is a method for publishing databases in a standard format for interconnectivity with other databases without the expense of wide agreement on unified inter-agency or global data standards.

– GOAL NOT MET