Public Data
In the worlds of politics, big business, and finance, information has the power to change everything. When it comes to converting this information into meaningful insights, however, the devil is truly in the details. Crucial information is often obscured in a cloud of arcane data formats and documents. Without the right tools, we end up lost at sea, and without answers.
I've spent a good portion of my recent years helping to build web apps that address this problem. Here are the highlights.
Vigilant
In 2015, along with Mike Phillips and Zac Palin, I co-founded Vigilant, a comprehensive public records search and monitoring platform. I architeched and built the initial core technologies behind the platform, and structured much of the data ingestion across what has become thousands of sources. I also handled design for both UI / UX and branding / marketing. Vigilant's first customer was the Clinton presidential campaign, and has since been employed regularly by prominent organizations in politics, finance, and journalism.
More About the Platform
There's an abundance of valuable public data out there, but it’s buried in siloed, balkanized government databases where it lies largely unusable. Vigilant unlocks that information, make it actionable, and help customers leverage it to make critical decisions. Customers span financial services, politics, public affairs, and media, and the product suite includes a universal search layer for public data, and sophisticated monitoring systems.
Lobby Search
California Lobby Search is a pet project of mine, the beta version of which I soft launched in late-2018. It is an open-source public records research tool that connects bills from the California state legislature with lobbying activity filings from the California Secretary of State office.
You can search all lobbying activity for a bill or bill keyword within a given time range. You can also search by company name — both employers or lobby firms. Ideally, your searches will quickly answer the question, "Who is interested in this bill and how interested are they?"
Rationale and Hopes
Why build such a thing? Because though lobbying data in California is technically available, it is not particularly actionable.
In California, online access to state public records is famously antiquated (though in all fairness, it is significantly further along than most states). Despite all lobby activity filings being posted on the Secretary of State website, it is only possible to search filings by the name of the company. Moreover, the filings themselves only refer to bill names (e.g. "AB 300") and not to bill titles (e.g. "Internet Privacy").
This makes broad-based questions about lobbying functionally impossible. For example, suppose I wanted to know which companies or interest groups are lobbying on bills about nutrition this year. In order to answer this, I would have to manually go through every filing of every single company, plus translate all of the bill names found in each.
At the time of this writing there are over 575,000 filed lobbying activities. This would make too long a night for a very unlucky poli-sci intern!
It is my hope that the tool will enable journalists and researchers to ask questions that were previously impossible to answer practically, and that those new answers can ignite new stories and illuminate new understandings about the wetworks driving California state government.