With an executive order and a new Memorandum on Open Data Policy today, the focus on entrepreneurship remained at the forefront of federal data policy. This focus began with last year’s Digital Government Strategy, and these days weather data and GPS signals are the examples of choice. That said, the policies set in the new memorandum are quite good for the classic use of this data (transparency, accountability, and civic education) even if “transparency” is only barely mentioned in passing.

Defining Open Data: How well does it do?

This new Open Data Memorandum presents the most detailed definition to date of “open data” by the federal government. It included many of the principles that our community has reached consensus on, but it gets one severely wrong.

As I wrote many years ago, the 2009 Open Government Directive itself already adopted some of the principles of open government data including: online, primary, timely, public input, and public review. It also added two principles of its own: being pro-active about data release and creating accountability by designating an official responsible for data quality.

Comparing to my list of open government data principles in my book, the new memorandum’s definition of open data covers:

• Principle 1: Information should be online (to quote the Memorandum: “retrieved, downloaded”)
• Principle 2: Primary (the Memorandum even uses language from the 8 Principles; interestingly the memorandum places this under the heading of “Complete,” which was a different principle from the original 8 Principles).
• Principle 3: Timely.
• Principle 4: Accessible (the Memorandum repeats the language from the 8 Principles, “available to the widest range of users for the widest range of purposes” and the use of “multiple formats” where necessary, and for documentation says the data should be “described”).
• Principles 5 and 10: Analyzable (“machine readable”).
• Principle 6: Non-discriminatory
• Principle 7: Non-proprietary (open) data formats
• Principle 14: Public review (“A point of contact must be designated to assist with data use and to respond to complaints about adherence to these open data requirements.”)

Its definition also states that open data has a presumption of openness. (Principles 2-7 and 14 are from the 8 Principles of Open Government Data. Principle 1 is from the Sunlight Foundation.)

Elsewhere in the memorandum it addresses:

• Principle 13: Public input (“engage with customers” for prioritizing what data should be made available and how to make it available)
• Principle 15. Interagency coordination (“interoperability”)

It also asks agencies to create data catalogs to include datasets “that can be made publicly available but have not yet been released” at agency.gov/data URLs. And it says agencies must consider the needs of open data at all stages of the information collection lifecycle. In other words, data should be collected in such a way as to promote public dissemination of open data later on.

The Memorandum misses the principle that data should be license-free, which is a core principle and a grave mistake. It also misses the peripheral principles of permanence, the use of safe file formats, and practices of provenance and trust (e.g. digital signatures). (These last two are ACM principles.)

“Open licenses” presume access is closed by default!

Rather than requiring open data to be license-free, which was a core part of the 8 Principles of Open Government Data, it instead promotes the use of “open licenses.” This is a subtle but important distinction. Licenses presume data rights. Open licenses, including open source licenses and Creative Commons licenses, create limited privileges in a world where the default is closed. These licenses create possibilities of use that do not exist in the absence of the license because copyright law, or other law, creates an initial state of closedness.

Most open licenses only grant some privileges but not others, and some privileges come along with new requirements. The GPL and Creative Commons Attribution License, for instance, rely on copyright law so that restrictions on data use intended by the open license (GPL’s virality clause, or the restriction that users must attribute the work to the author) are enforceable in court.

Federal government data is not typically subject to copyright law, and in this case a license is not needed for the data to be open. Thus the application of a license suggests a change from the open-by-default state of this data to a closed-by-default state where a license is required to open it up. While the memorandum requires “an open license that places no restrictions on their [the dataset's] use,” the term “open license” is typically understood to presume a default closed state. This policy opens the door (so to speak) to agencies applying licenses (i.e. new contractual agreements) to data that serve only to restrict use.

Federal government data not subject to copyright cannot be free if a license is applied. The license-free principle of the original 8 Principles says open government data cannot be limited in this way.

When data may be subject to copyright protection (copyright law is murky and there are many gray areas), or when copyright law definitely applies (such as to documents produced originally by federal government contractors), then a public domain dedication such as the Creative Commons CC0 statement or the Open Data Commons Public Domain Dedication and License (PDDL) (both of which combine a waiver and a license) is appropriate. A public domain dedication differs from an open license in that it disclaims copyright and other protections, whereas, again, an open license implies that such a limitation on use is already present. The CC0 statement was successfully used by the Council of the District of Columbia to disclaim copyright over data files containing the DC Code.

What’s the definition used for?

While the definition of open data is otherwise quite strong, the definition is used just once in the whole memorandum. The memorandum does not mandate that government data be open data under its definition, at least as far as I could see. The only use of the open data definition is in its request for agencies to create roles for staff to ensure data released to the public are open. That is, staff should promote open data, but open data itself is not required.

Although the definition itself is not used much, there are independent provisions that repeat some of the same principles. Agencies must use “machine-readable and open formats,” existing standards, and metadata. And information collection should be done in a way to support information dissemination: “[A]gencies must design new information collection and creation efforts so that the information collected or created supports downstream interoperability between information systems and dissemination of information to the public.”

It also requires the use of open licenses:

“Agencies must apply open licenses, in consultation with the best practices found in Project Open Data, to information as it is collected or created so that if data are made public there are no restrictions on copying, publishing, distributing, transmitting, adapting, or otherwise using the information for non-commercial or for commercial purposes.”

As I mentioned, federal-government-created data needs no license to be open, although the memorandum implies that all agency data should have an open license. (That’s either legally impossible or it means something usual.) For other data, it appears that the memorandum intends to create a public-domain-like state. But it is qualified, for contracts may only use “existing clauses” (i.e. standard contract terms already approved by OMB) to implement terms of open licensing. Looking over those terms, I don’t see the necessary legal framework to do it. And a nearby footnote confusingly says that a data user who modifies the data “is responsible for” describing the change. Does that mean an “open license” can require users to describe modifications? The qualifications make it very difficult to know what an acceptable implementation of open licensing looks like.

Conclusion

While the goals of the Memorandum in defining open data and using open licenses are laudable, the implementation does not meet the 8 Principles’s requirements of open government data, at least under the usual understanding of “open license,” and the use of the definition to promote open data is very limited.

PS. As Derek Willis points out over Twitter, the “mosaic effect” paragraphs in the memorandum are also somewhat concerning. The mosaic effect is hard to quantify and therefore difficult to limit, and this creates a big hole for keeping data government out of public reach.

UPDATE 5/10/2013 #1:

Rufus Pollock points out that the Open Data Commons Public Domain Dedication and License (PDDL) is similar to CC0 and would also be appropriate. I agree.

Eric Mill notes that for data already in the public domain, the Creative Commons Public Domain Mark, which is basically an icon/badge, would be appropriate. Agencies should definitely mark public domain data as such.

UPDATE 5/10/2013 #2:

I added a few paragraphs to the section now called “What’s the definition used for?”.

The TL;DR version goes like this:

Tom MacWright wanted to build his own version of the DC Code website. The DC Council couldn’t share its electronic copy of the Code because it contained intellectual property owned by West. This became a little and very geeky controversey (spurred by Carl Malamud). But Zvenyach — the general counsel — recognized the value of making the law open and did it. He removed the West IP from their electronic copy of the Code (I helped), posted the file on the Council’s website, and even included a CC0 public domain dedication

The last bit all happened within a matter of days, and it was one of the easiest open data success stories I’ve been a part of. Tom recapped the events here and began hacking the code immediately. He held a hacakthon on April 14 which he wrote about here (and Eric Mill wrote about here).

DC is setting an example for other jurisdictions. In terms of the 10 Principles of Law.Gov, DC’s bulk law download — achieved within only a few days of work — satisfies principles of no-charge to access (1), no copyright or terms of use (2), data in bulk (3), and, to some extent, machine processability (8).

Here’s the longer version:

This all began a few months ago when DC-based civic hacker Tom MacWright took an interest in making local law more accessible. Intending to import the DC Code into Waldo Jaquith’s State Decoded project, he ran into a small problem: he couldn’t get a complete copy of the law. Intellectual property issues prevented the DC Council from simply emailing over their copy of the Code.

Many states, like the District, contract out the codification and code-publishing work to a third-party like West (owned by the Canadian-owned Thomson Reuters) or Lexis (owned by the Amsterdam-based Reed Elsevier). DC had previously contracted out to West, and last year switched to Lexis. Neither likes to share. DC’s official website to read the Code — which has been run by West — is free to the public, but copying any part of the Code off of that website might violate West’s copyright or terms of service, or both. Sharing the law might have been illegal.

In the case here in DC, the DC Council had Word documents containing the Code, given to them by their contractor West, but the documents contained West’s logo. The DC Council could not share the documents with West’s logo intact. And it wasn’t easy to take those logos out (more on that later). Informally speaking, West owned the DC Code.

I had met Zvenyach, the general counsel, before. He is very technologically savvy and has been trying to modernize the office he took over only a few years ago. We had even talked about holding a hackathon to help him do it. (As a DC resident, I’m also interested in DC law.)  But his office, like all of government, is bound by limited resources and much work to do. When Tom brought the issue onto Zvenyach’s radar, I don’t believe there was any point at which Zvenyach didn’t want to make the files available. It was, as far as I’ve observed, merely a matter of time and resources.

Tom asked Carl Malamud to get involved. Carl has been working on this issue in other states, like in Oregon, where the State of Oregon itself claimed copyright over their laws. Carl bought (for quite some money) a physical copy of the DC Code, digitized it, and mailed thumb drives in the shape of famous presidents containing the digitized code to various important people. This was a spin on a tactic that Carl began in the 1990s when he opened the SEC’s corporate filings data: get the data online, pressure the government to put the data online themselves, and then help the government take over that responsibility.

The media and bloggers caught on, beginning I think with Corey Doctorow on March 27, followed by DCist on March 28, The Washington Times on March 31, Steve Schultze on April 1, and Think Progress on April 3. The files themselves went up on April 4, so little more than a week from the first media blog post about it, and the decision to put the files up with a CC0 license was made in any case some days earlier. It really did not take much pressure at all. (Tom also wrote a post on Greater Greater Washington on March 19.)

Carl had noticed early on that the DC Council asserted copyright over the Code. Some of the media reports focused on that. As Zvenyach explained in The Washington Times article, the rationale was to protect DC from West, by making sure West could not claim copyright over the same Code, not to limit access to the law. Whether or not state codes can be copyrighted was mostly besides the point, and the focus on this issue turned out to be a red herring. It was resolved quickly with the choice of the Creative Commons CC0, a public domain dedication.

I went in to Zvenyach’s office on April 3 to help them take West’s logo out of the Word documents. There was one document per title of the Code, or about 50 documents, many in the 50-megabyte size range. The West logo was in the header, but the header was specified independently for each section of the code, so in reality there were thousands of logos to take out. We also took out a DC copyright line from the documents, which was also repeated in each section.  It took about 4 hours for Microsoft Word to process all of the files, and 1 hour for us to figure out how to do it so “quickly.”

When I left Zvenyach’s office that evening, Zvenyach pointed out the presidential thumb drive still sitting on his desk that he received from Carl — unfortunately I forget if it was a little George Washington or a little Abraham Lincoln. I have a feeling that thumb drive will be around for a while.

Now, there is a bigger issue here. There’s no plan for updating the public files. DC’s contract with Lexis going forward doesn’t require Lexis to provide DC with an electronic copy of the code. Perhaps after this they’ll refuse to do so. But we’ll tackle this another time.

—–

I write to urge the subcommittee to expand funding for legislative transparency.

I am the president of Civic Impulse LLC, which operates the free legislative tracking service GovTrack.us. Our website has become an authoritative source for legislative information:

• More citizens turn to GovTrack.us for information about the status of legislation than the Library of Congress (LOC)‘s THOMAS and Congress.gov websites. [See compete.com.]
• Hundreds of House and Senate staff use GovTrack.us each day.
• More than 70 congressmen use GovTrack services to display congressional district maps and their voting record on their official website.

Why is this? GovTrack.us has become the de facto authoritative source for legislative information because the Congress does not publish enough “bulk legislative data.” In 2004 we stepped in to fill the vacuum created by the lack of information coming from the Congress. It is long past due for the House to correct this problem.

When the Committee released a draft report last year indicating it intended to have legislative branch agencies publish less bulk data, The Washington Post picked up on the story and wrote:

“At Congress’s ’90s-vintage archive site, there’s no way to compare bills side by side. No tool to measure the success rate of a bill’s sponsor. And there’s certainly no way to leave a comment. Congress makes it hard for outside sites to do any of this, either, by refusing to give out bulk data on its bills in a user-friendly form.” (“Congressional data may soon be easier to use online,” The Washington Post, June 8, 2012.)

Soon after, the Speaker and Majority Leader formed the “Bulk Data Task Force.” Since the formation of the task force, new bulk data projects have been completed at the Government Printing Office (GPO) including bulk bill text and at the House Clerk (committee schedules and documents and bulk floor action data).

“Bulk data” is a core component of any government information dissemination program. The House Clerk publishes roll call vote results as bulk XML data. In 2009, the Government Printing Office began offering bulk data for bill text, the Federal Register, and other publications. The Office of Law Revision Counsel publishes the United States Code in multiple bulk data formats. Bulk data can be produced at a fraction of the cost of other information dissemination methods, such as colorful websites.

Yet much information about the Congress remains out of public view. There is no public bulk data for the status of legislation (the LOC “BSS” database), amendments, or committee votes. I believe that eventually all official artifacts of the legislative process should be available online, free, in real time, and as structured bulk data. [See Recommendations to the Bulk Data Task Force.]

And, sadly, proposals for cost-reduction threaten the public’s access to the law itself. A 2013 congressionally-funded report by the National Academy of Public Administration (NAPA) called for the Congress to consider charging the public fees to read the law online at GPO’s website. NAPA’s report is severely out of touch. There is no dispute that it is a moral imperative for Congress to fund programs that provide broad access to the law and other parts of the public record.

GovTrack.us is a demonstration that bulk data creates broad public access and that bulk data is also the most cost-effective way to create access. Since 2004, GovTrack.us has reached tens of millions of individuals at a cost of less than $1 million.

The Committee can advance broad public access to legislative information by providing adequate funding for:

• Publishing the LOC legislative status (“BSS”) database as bulk data. [See Recommendations to the Bulk Data Task Force.]
• Enhancing GPO’s highly successful FDSys system.
• Creating bulk data program officers at GPO, LOC, and under House Clerk.
• Evaluating the cost and impact of legislative transparency by an organization that believes in the public’s right to primary legal documents (i.e. not NAPA).

Thank you for the opportunity to submit comments on legislative branch appropriations for FY 2014.

Joshua Tauberer

President, Civic Impulse LLC

Over 150 developers, data scientists, social entrepreneurs, government employees, and other open data enthusiasts participated in our event, first at a kickoff Friday night at Google’s DC headquarters and then at the Saturday session at The World Bank. Participants worked on local DC issues, global open source mapping, world poverty, and open government. Here are some quick links:

Videos: One | Two – Photos: One | Two

Eric’s Recap | Sam’s Recap | Tumblr | Storified Tweets

Press coverage is listed at the end.

Our approach to the hackathon was a little different than many others. Our goals were to strengthen the open data community, to foster connections between people and between projects, and to emphasize problem statements over prototypes and solutions. There was no beer or pizza at our hackathon, no competitions, and no pressure to produce outputs. Participants came motivated and stayed focused without needing to be treated like brogrammers. This created a positive, welcoming, and highly productive environment.

In the morning Eric Mill (Sunlight Foundation/@konklone) ran a several-hours-long tutorial on open data for about 40 participants. Some were new to coding. Others were project managers (inside and outside of government) who wanted to learn more about what open data is all about from the ground up. Eric walked the participants through exploring APIs through the web browser and using command-line tools to process CSV files — a very concrete way to explain the benefits of adding structure to data.

Several projects focused on local DC issues: mapping zoning restrictions (more), graphing public and charter school enrollment and (other education data), mapping trees by species, and building a database of social service providers.

A large team of map hackers worked on mapping Kathmandu in Open Street Map to aid disaster response, and with their collaborators around the world mapped over 7,000 building footprints.

Global poverty and international development was the focus of several other projects, from building APIs for international development project performance data to measuring poverty in real time using Twitter.

The open government projects worked on adding semantic information to legislative documents, comparing legislative documents for similarity, extracting legal citations, cataloging our government representatives at the local level, and building “devops” tools for rapid deployment of VMs that might be useful in government or for open data researchers.

And there were other projects that don’t fit into any of those categories, like building Python tools for creating education curricula,

The event was organized by me (Josh Tauberer/GovTrack/@JoshData), Eric Mill (Sunlight Foundation/@konklone), Katherine Townsend (USAID/@DiploKat), Dmitry Kachaev (Presidential Innovation Fellow/Millennium Challenge Corporation/@kachok), Sam Lee (The World Bank/@OpenNotion), and Julia Bezgacheva (@ulkins/The World Bank).

Thanks to The World Bank especially, and to Google, the participants that helped out with registration in the morning, and to everyone who came!

This was DC’s second open data day. Our first was on Dec. 3, 2011 and was co-hosted by POPVOX (Josh Tauberer) and Wikimedia DC (Katie Filbert). See what we did on the post-event recap at https://www.popvox.com/features/opendataday2011. Participants then worked on improving access to U.S. law, scanning federal spending for anomalies following Benford’s Law, understanding farm subsidy grants, building local transit apps, and keeping Congress accountable. Only about half of the participants were programmers, buteveryone found a way to be involved.

It was also DC’s second international development data day. The last one was held on December 9, 2012 in the lead-up to the Development DataJam hosted by White House’s Office of Science & Technology. Those events primarily served as ideation jams to bring together issue area experts and data experts to develop new ideas and partner for new solutions. Experts were sought out to inform the discussions, but anyone with an interest in open data in development were welcomed and participated.

Press coverage

DCist: Hack D.C.: Hackers Put Open Data to Use to Help Improve Local Government

The Atlantic Cities: Is There a Link Between Walkability and Local School Performance?

Greater Greater Washington: How school tiers match up with Walk Score

Greater Greater Education: Community of civic hackers for education takes shape

Ludlow’s Position

Ludlow depicts the war as tug-of-war between two ends of the spectrum. On the one end is what we generally call cyber crime, the sort of “hacking” portrayed in movies. The other end, in Ludlow’s description, is a “less sinister” and more generic activity. An example he gave is putting wool sweaters on trees, whatever that is. Ludlow also indicates that he believes this form of hacktivism has no “positive affect.” Ludlow’s analysis is fundamentally incorrect. There is no spectrum on which a war is occurring. And the other sort of hacktivism most certainly has a positive affect.

The Meanings of Hack (n.)

“Hack” has two at least three distinct meanings as a noun. It’s a homograph. Just like “mouse” and “keyboard” are (think rodents and pianos). A lot of jargon is like this. And “gay”. “Gay pride” is not an attempt to tug the definition of “gay” away from “happiness”. Maybe decades ago it was. It isn’t today. “Hack” is the same way. One meaning is more or less the same as cyber crime — that much Ludlow got right.

Another meaning is the sense of hack in “party hack” or “hack journalist.” (A hack journalist is someone who takes the side of whoever their employer is at the time.) There is no “hacktivist” in this sense, but this meaning demonstrates the plausibility of the argument that I’m making: that “hack” isn’t the object of lexical warfare but instead has multiple unrelated meanings. (Thanks to Neville Ryant for reminding me of this meaning.)

The Good Sort of Hacking

The last meaning of hack is hard to pin down, and I can’t claim to define it, but it’s roughly the perverting of something’s original purpose to solve a new problem. Rube Goldberg machines are hacks. The use of the lunar lander to bring the Apollo 13 crew home was a hack. Putting folded-up newspapers under table legs to stop tables from shaking is a hack. Hacks are often creative uses of technology. Hacks are usually applauded. They’re positive, creative, even artistic.

In my neck of the woods, “civic hacking” is a term for creative, often technological approaches to solving civic problems like how to get more people to register to vote or making beautiful city maps. It has nothing to do with crime. Sometimes it has nothing to do with computers. It’s about solving real world problems.

Hacking is by no means some sort of jargon specific to the tech-nerd culture either. There’s a website devoted to hacking IKEA products called IKEA Hackers. Its creator defined hacking too:

IkeaHackers.net is a site about modifications on and repurposing of Ikea products. Hacks, as we call it here, may be as simple as adding an embellishment, some others may require power tools and lots of ingenuity.

An example is turning a pillow into a child’s Haloween custom. (Cute, and of course not criminal!) From IKEA furniture to computer systems, there’s a shared hacker culture around repurposing, creativity, and solving problems.

If you’re a journalist writing about hacking or hacktivism, take a moment to think about which type of hacking you mean.

Is It Warfare?

So let’s compare now: cyber crime and solving problems.  This is not a natural spectrum. Not that there can’t be overlap. That’s how historically the words are related (to the best of my knowledge), like if you look back in the 1980s when the term was first coming into mainstream use. There’s a reason the two meanings shared a single word: Using technology for unintended reasons is often illegal. But it’s not because it’s hacking (in the positive sense of hacking) but because technology can do so much that it’s easy to run up against the boundary of the law. God forbid you use a copy machine (or ipod?) to copy something without permission! Stuff like that.

Is there a case of lexical warfare here? Ludlow defines what he means:

“Lexical Warfare” is a phrase that I like to use for battles over how a term is to be understood.

If lexical warfare is a battle over the single meaning of a term, that is not the case here. Civic hackers don’t particularly care that “hack” is used to refer to cyber crime. We lost that battle before I was born a long, long time ago. And cyber criminals don’t care about what civic hackers are up to, so far as I have seen.

There’s more evidence from how the verbs are used. You can “hack a server” (i.e. break in) and “hack the weather” (solve weather-related problems), but while one “hacks into” systems, one “hacks on” problems. “Hacking into voter registration” and “hacking on voter registration” mean something different. The choice of preposition (“into”/”on”) depends on which type of hack you mean, and it is evidence that the two meanings are distinct. (As a verb, by the way, “hack” has even more meanings, some totally unrelated to any of the meanings of the noun so far. Related to the problem solving meaning, some use “hack” to mean the same as to do computer programming.)

Derivations

The words “hacker,” “hacktivist,” and “hacktivism” all share the same ambiguity that derives from the meanings of “hack.”

A “criminal hacktivist” is roughly someone who does ”criminal hacking,” like denial of service attacks, for political purposes.

A “problem-solving hacktivist” is roughly someone who builds websites to motivate the public toward a public policy goal. The IT guys at nonprofits are problem-solving hacktivists (among many other groups of people).

In one of the articles Ludlow cites, the one in Infosecurity Magazine, hacktivism is said to be defined by Wikipedia as “the use of legal and/or illegal digital tools in pursuit of political ends.” This conflates the two meanings into one. This definition incorrectly includes anyone who emails their representatives in government, for instance. Such an action is not hacktivism because it is neither criminal hacking nor a creative or technological solution to a problem.

A “hackathon” — a hacking marathon — for the problem-solving type is when a bunch of optimistic people gather in a room and try to solve some problems.  Often with computer code. Often open-source and for the public good. Not always.

If you don’t know me, I’m a civic hacker and I’ve got a degree in linguistics. The title of this post of course refers to the famous Eminem song.

On 1/19/2013 I updated the post to include a third meaning of hack, “party hack.” Thanks Neville. On 1/20/2013 I added an the examples “hacking into” and “hacking on” and discredited the Wikipedia definition. On 2/28/2013 I added the section on IKEA Hackers.

What we’re seeing with the bills bulk data project is how the wave of culture change is moving through government. Over the last two years the House Republican leadership has embraced open government in many ways (my 112th Congress recap | the new House floor feed). With this bills XML project, we’re seeing more legislative support agencies being involved in how the House does open government.

This isn’t a technical feat by any means, but it is a cultural feat. The House and GPO worked together to institutionalize a new way for the House to publish bulk data.

Because of the way Data.gov is managed in the executive branch, we’ve become accustomed to big announcements. The bills bulk data project and the other recent projects show that the House is taking a different approach, an incremental approach, to open government data: publish early and often, gather feedback, then go on to bigger projects. This is something open government advocates have been asking for.

As I mentioned, the tech side itself is not much. They took files they and the Library of Congress already make available (and in some sense already in bulk) and zipped them up into up to 16 ZIP files. (4 files now, but that will probably grow to 16 by the end of the Congress.) So there’s no new data here, and thus not the data that the bulk legislative data advocates have been asking for. But it’s on the road to that. The files involved in this project have the text of legislation but not bill status, which is what the bulk data advocates have been asking for.

There is one thing crucial missing from this, and that’s that there is no feedback loop with the users of this data. The incremental approach can’t work unless the users of the data have a way to tell GPO what is and is not working. There is no public point of contact for these files, and I don’t even know a private point of contact at GPO.

But that doesn’t detract from the fact that this is a good step forward.

When the open gov tech community coalesced at the start of the 107th Congress in 2007 Democrats had just regained control of Congress after a series of ethics scandals in 2006 brought the Republican Party’s commitment to ethics into question. But despite Speaker Pelosi’s call for transparency at the start of the Democrat’s control, honestly very little happened over the following four years (the launch of HouseLive.gov and the availability of disbursements PDFs come to mind).

In fact, when calls for transparency persisted in the House — that is, Republicans asking Democrats for more transparency — we would often chalk that up to transparency being used by the minority party as a delay tactic.

But when the Republicans took over in 2011, they kept at it. With mixed success, of course. Some promises, like 72-hour delays before votes, were not taken even remotely seriously. But that shouldn’t detract from what they got right:

• They began a moratorium on earmarks, which was somewhat successful.
• They launched Docs.House.gov, which gave the public a heads-up about what would be happening on the floor up to a week in advance. Prior to Docs.House.Gov, (UPDATED) there was essentially no advanced notice whatsoever there was no structured data about the House calendar. (Thanks to Eric Mill for correcting my apparent exaggeration.)
• They held a “hackathon” in December 2011, during which transparency and technology activists in the public had a chance to talk with House staff and get to understand the complexities of the House better.
• They held a legislative data and transparency conference in February 2012, the first conference of its kind.
• They promised legislative data, and after public outcry they formed a task force to consider it. (On the downside, we had to have an outcry.)
• They centralized committee video webcasting and archiving infrastructure, leading to much more of committee proceedings being available over the web.
• At the very end of the 112th Congress they made any committee documents sent to GPO available electronically by default (update: link posted)
• The Clerk’s official list of members got a new column of bioguide IDs.
• They began the creation of data standards for committees which lead to significant updates on Docs.House.Gov on the first day of the 113th Congress.
• (UPDATE) They passed the DATA Act.

The Sidebars: Style and Media Types

The sidebars — the extra media besides the text of the story — were the focus of most of the innovation in the presentation of this story. I say “sidebar” loosely, since some of the sidebars were more like between-bars or beneath-bars. Here’s more on that:

The sidebars were limited to a few distinct styles and media types, which created a sense of continuity throughout the piece. Sidebars were positioned in just five ways: full-wide interrupting the text, inside the text column floated right, the same but floated left, inside the right-column, and roughly 3/4ths-wide cutting into the text column and extending to the right edge of the window. (During the ski map feature in part three, all bets were off on sidebar style.)

There were seven media types: full-wide videos, video mini-features (in the text column, or 3/4ths wide), photos (full wide montage and 3/4ths wide singles and pairs), slideshows (3/4ths wide), mug shots (inside the right column), auto-play mini-videos (inside the right column), and audio-only clips (inside the text column but floated left). Styles were highly consistent, although there was some variation in the placement of photo captions. Each media type brings its own short learning curve: how to start a movie, what does a slideshow do, why doesn’t this movie have a restart button, etc. If anything, I might have dropped the two media types that had the least value as distinct media types: slideshows and the auto-play mini-videos, both of which could have been done in the format of video mini-features.

The types of media besides the body text were introduced gradually, both within parts and across parts. This again helped with a sense of continuity, or at least limited the amount of surprise, confusion, and sense of visual clutter you feel when you encounter a new type of media. In the first part of the series, five types of sidebar media are used but half of the types appear for the first time only in the second half of the page. Media types mostly repeat between a new type and a type the user has seen before. The second part in the series introduces the full-wide photo montage toward the middle and the auto-play mini-video near the end (more on that later), but otherwise sticks to the same types of media found in the first part. Part five added audio-only clips.

Most full-wide sidebars actually extended to the width of the browser, beyond the main 1024px wide body area that other sidebars are confined to, if your browser happens to be open wider. This near-full-screen effect adds to the incredibility of the splash videos and photos especially. At first I thought the fixed-positioned ski map in part three was a right-column sidebar (more on that below), but when I maximized my browser it turned out to have more in common with full-wide sidebars. It fills the browser width just like full-wide sidebars, and has a fixed-positioned effect like the intro title. But unlike the others, the now 1/4-wide text column, which appears beside it at low browser window widths, actually runs over the sidebar at large browser window widths. The text has a solid white background (i.e. it’s not a transparent overlay) and it appears not so unlike the ski paths that carve through the mountain on the map beneath it.

The mug shot sidebars are notably low-key. The photos are tiny (55×55) and are next to usually just two lines of text, the person’s age and occupation. But on hover, the whole thing becomes a link to a slideshow for more images of that person. What surprised me most was that the mug shots in the first part of the series were not placed next to the first introduction of a person in the story. Instead they were grouped together toward the end of the page. In later parts, mug shots were more often next to first mentions.

Sidebars videos and mug shots (but not other types of sidebar content) all have an anchor in the text. For mug shots the anchor is the person’s name. For videos, a few words or a sentence near the video were the anchor. The anchor text had a grey background and a video or slideshow icon at their left side. This suggests to the reader this is a good stopping point to try the video, or if not there then maybe at the end of the paragraph. Hovering over the highlighted text causes the play button on the video to get bigger, letting you know what is going to happen. Clicking the text starts the video.

Full-wide videos and what I called the auto-play mini-videos started automatically, which I found annoying. (In the second part of the series the mini-video was an animation of how some avalanche protection gear worked.) But while the full-wide videos had start-over buttons, the mini-videos were an attempt at simplicity. The mini-video in part two had no border and a white background and no play control, so it blended well into the background, but the “way” to play it again if you missed it the first time was guess that you could scroll up and then scroll back down to it.

The Ski Map Sidebar and the Avalanche Video

Then there is the fixed-positioned ski map in part three, part sidebar part interactive graphic. It slides up as a 3/4ths-wide sidebar (or so I thought) but remains fixed in place for the remainder of part three. It forces the text to its left to stay at 1/4th-width (more on that above). The map evolves as you scroll down, with different ski paths highlighted every few paragraphs according to what is happening in the story. Mug shots appear on top of the map as you scroll, to indicate that the ski paths currently highlighted on the map were the paths those individuals took. But oddly while the map was fixed, the mug shots layered above it continued to scroll with the body text. It was an interesting (but perhaps not entirely successful) effect. While the ski map was up on the right, video mini-features appeared in a smaller size wholly inside the small text column.

I don’t have much to say about most of the videos, but the Tunnel Creek Avalanche: In Real Time video deserves a few notes. This was a video similar in design to the 3D fly-over of the mountain from part one. It’s a computer generated view of the mountainside, sort of photo-realistic, and from what I read constructed from scratch using elevation data and satellite imagery. The video follows the path of the avalanche, which is represented by a sort of highlight over the image.

It’s also a multimedia diagram. Contour lines cover the surface of the mountainside: thick lines every 2,000 ft of elevation (some but not all labeled), thin lines every 250 ft. And, incredibly, a synthesized drum beat counts off the feet, a light tap every 125 ft, a strong tap on the 2,000 ft lines, giving a multi-modal and actually emotional experience of the change in speed of the avalanche. The audio track also has the sound of wind. White dots highlight the locations of skiers (before the avalanche starts, tastefully not how they were carried away by the avalanche).

Focus and Color

There’s a splash screen at the start of each part in the series. And it’s huge. Depending on your window size, there may not be any body text visible at all before you scroll down, which I usually think is a design problem but didn’t mind here. At some window sizes, the page would scroll down gently automatically to reveal the first two lines of body text. The splash screen is a beautiful movie of snow and clouds at the start of four parts in the series (including of course the first part), and images with some text or CSS animation in the second and third parts.

On the main landing page for the series, part one, navigation to other parts in the series at the top doesn’t appear until you scroll past the splash movie. (And once it appears it never goes away.) It keeps the focus on the series title and movie.

As you scroll past the splash movie and it slides away (getting covered up from below), the title and byline begin to fade out — as if to force you to look away toward the body text.

As you scroll into the Tunnel Creek 3D fly-over video in part one, and similarly the Tunnel Creek Avalanche video in part four, in the area above the video the background color fades to match the background color at the top of the video, so that by the time you are scrolled to the video the page feels coherent as a single presentation and not as body content plus a sidebar.

Sidebar videos inside the text column start greyed-out with no text or play controls. Once it reaches a certain height on the page, it fades in to full color and the title of the sidebar and the play button fade in.

The bottom navigation to move on to the previous or next part in the series only appears when you have scrolled to almost the very bottom, which is similar to how regular NYT articles nag you to read more when you get to the bottom. But it did not feel like a nag here.

As you scroll past the splash movie and some other full-wide videos, the media stays fixed while the content below covers it up from below. For the Tunnel Creek video, I found this confusing. Because it would scroll up when you’re going into it, but it does not scroll away.

Layout, Structure, and Font

By and large, the body area was confined to 1024px, centered horizontally at larger window sizes, similar to the NYT site as a whole. But as I noted, some full-width sidebars extended horizontally to the size of the browser window (and preserved aspect ratio, so got taller too).

There are two levels of structure to the piece, the six parts in the series with navigation across the top and <h3>’s throughout the body text.

The body text size is 15px (22px/147% line height), which is the same as other NYT articles. The width of the column is 626px, which is larger than the 600px in typical NYT stories. I changed the font on my blog to match!

In the first few screenfuls of the body, there is only plain text. No links. No ads. Nothing in the right column. The only bit of fancy is the enlarged first letter of the article.

At the end, the story offers itself to be viewed as an ebook or to be watched in documentary form. And it concludes with half a screenful of credits.

Advertisements

Much of the success of the layout in this series comes from the limited advertising. There is one advertisement per section, I think. For comparison, there were about 10-15 sidebars per section. That isn’t necessarily a sustainable practice for a newspaper.

The first advertisement is 3/4ths the way down the first part in the series, and similarly positioned in some of the other parts. Advertisements all had the same distinct style that could not be confused with sidebar media: full-wide, labeled Advertisement really clearly, and separated from the editorial content with a black background (#333333) and large padding.

Parliaments, like our Congress here, never seem to listen to citizens as much as we the citizens would like. One of the largest problems in public trust is the perception that Congress only listens to moneyed lobbyists. But a recommendation to listen more mostly misses the reasons for the lack of sufficient communication, and so is not likely to be an effective remedy.

There are two very different types of input the public can provide to parliaments. One is sentiment, the other is expertise. When a member of parliament votes against a popular position in his or her district, it is a problem of sentiment. When a member of parliament introduces a flawed bill, it is a problem of expertise.

In my experience here, Members of Congress are eagerly interested in knowing the sentiment in their district. That’s because they don’t want to be voted out next election. Congressional offices here each have staffs dedicated solely to processing incoming mail. So what’s the problem? Well, representatives here have constituencies of up to 1 million individuals (that’s Nevada’s 3rd congressional district; the average size of a district is 710,000) and senators up to 37 million (California). And a lot of those individuals write their representative, which means Congress gets an enormous amount of mail. They couldn’t possibly read it all.

Asking members of parliaments to engage in more forms of communication isn’t helpful. They’re reading as much as their budgets will allow them to hire staff to do that. A better ask could be: allocate more funding to processing mail. But even better is: develop innovative ways to handle communication more efficiently based on whatever particular factors are at work in your parliament. Efficiently means processing mail faster, for instance by making sure citizens submit messages that are shorter, clearer, aggregated, or otherwise easier to process in an automated way. This was a significant component of our approach at POPVOX, a platform for constituent communication that I co-founded several years ago. Yes, that means in some cases boiling messages down to Support or Oppose and  having less actual reading of constituent messages by humans (more by computers).

Of the dozen forms of constituent communication recommended by the Declaration, the vast majority make constituent communication less efficient, and therefore less effective. Email and Facebook, for instance, are significantly less efficient than an online poll.

That said, there are times when members of parliament might find it convenient to ignore the sentiment they are hearing. In those cases, it’s important for the constituent to have the option to make his or her sentiment public (but typically anonymously). You might want to have your parliament handle the openness part, or you might want to ask your parliament to  support the technical infrastructure so a third party can handle the openness part (as in our case at POPVOX).

The second type of communication — expertise — is entirely different. Expertise needs to be communicated in a different channel from sentiment. The reason is that while everyone thinks their own expertise is important, it’s just not true. The reason lobbyists have so much influence is a combination of several factors related to expertise, some of which are: they are experts on a particular subject, they are experts on the current law and the  legislative process and thus can communicate effectively with legislative staff, and they are willing to do whatever additional research the legislator needs to make a decision. Citizens don’t have all three. If they did, they’d be lobbyists.

Better communication of citizen expertise is a conundrum.  While we know what efficient communication of sentiment looks like (e.g. online polls, petitions, or what we did at POPVOX), there are not yet any proven methods of constituent communication of expertise. Some examples to draw from, however, are the MADISON project (collaborative editing of bills run out of the office of a congressional committee here) and especially Peer to Patent.

I want to emphasize why it is a conundrum, to be clear. Obviously we want the expertise that legislators have access to to be correct, comprehensive, unbiased, etcetera. But those aren’t the only factors legislators have to consider. The information has to be readily available, it has to be from a source they trust, and it has to be in language they understand. The communication of expertise is therefore hurt by widening the base of people who can participate if there is no mechanism for determining who can be trusted, who communicates well, and so on.

My suggestion for the Declaration would be not to recommend simply greater engagement, but instead to recommend implementing a two-track system of taking constituent input. One track would be optimized for sentiment: it would encourage concise, easy-to-process messaging with publicly displayed aggregate totals and an opt-in to make messages public. The other track would be optimized for expertise and would focus on establishing trust with citizens who are willing to put in the time to navigate public policy and the legislative process.

I readily admit that “open government” and “open data” is vague already. Justin Grimes, Harlan Yu, and I held a panel at Transparency Camp a few weeks back about how vague the terms are and how that can lead to trouble when we aren’t clear about what we mean. Di Maio runs right into this problem, placing the burden of a successful open data movement on “mythical ‘application developers’” (sorry, I don’t exist?). Transparency is only one of a dozen or more reasons why open government data is a good thing, and these reasons cannot all be judged by the same rubric.

Focusing on open data for transparency, Di Maio argues that “[t]he more the data, . . . the more specialized are the skills and resources required to process that data,” or in other words that open data can actually exacerbate a digital divide rather than close it. I’ll be one of the first to say that open data doesn’t necessarily mean better government (see the link to my book above), but Di Maio’s statement that I quoted can easily seen to be simply wrong.

One has to evaluate open gov data with everything else held equal. in other words, if there is open government data in the hypothetical, the comparison to make is to another world where the same government processes exist — they’re just not published in a machine-processable format. Now, you tell me, which world requires more skill to understand the data? Clearly the second, because every skill you need in the open-data-enabled world you need in the open-data-denied world, but you *also* need some other skills just to get the information in the second world.

There are certainly unintended consequences of open data. In my book I discuss two cases where legislative data affected the behaviors of the legislators in a way that’s probably not good. But let’s stay on fact and, for that matter, on logic.