Nick Diakopoulos, a fellow at the Columbia School of Journalism’s Tow Center for Digital Journalism, is out with a new report on what he calls “algorithmic accountability reporting,” or journalism that seeks to assess the many formulas that shape our daily lives, from public school teacher evaluations to prices set by online travel sites. His audience is working reporters, but there’s insight, too, in the 37-page report for those people, local officials among them, who might come into contact with the notion of public policy produced by such fancy calculations.

Wrapping one’s mind around what “algorithmic policy” entails can be difficult. One example has to do with the speed limit: Instead of the static miles-per-hour rates we have now, we might have a flexible speed limit that readjusts itself on the fly based on conditions like weather, darkness or moose in the road. Other examples of where formulas can be tapped to figure out optimal public policy-related choices include where to roll out Google Fiber, or where to serve up taxis, or how much bail to set for accused criminals (something former New Jersey attorney Anne Milgram is working on, as detailed in this piece on Code for America).

Algorithm-based policy choices could force a rethinking of what policy is meant to achieve in the first place. By their data-based nature, they often make it possible to evaluate in real time how policies are working. Diakopoulos gives one more example he’s stumbled upon in his work:

The city of New York uses prioritization algorithms built atop reams of data to rank buildings for fire-code inspections, essentially optimizing for the limited time of inspectors and prioritizing the buildings most likely to have violations that need immediate remediation. Seventy percent of inspections now lead to eviction orders from unsafe dwellings, up from 13 percent without using the predictive algorithm — a clear improvement in helping inspectors focus on the most troubling cases.

So, there are upsides. There are also challenges, ones that become obvious when you consider the commercial applications of this approach. Diakopoulos highlights Google Autocomplete and differing emails that politicians send out to differing audiences. It’s tough to know from the outside just what human judgements — and biases — go into these formulas. That’s why Diakopoulos gets away with calling the algorithm-powered mechanisms at work “black boxes.”

Still, there’s hope. Diakopoulos has guidance for reporters who might want to shine a light inside what’s going on. Part of it consists of “reverse engineering” the formulas by studying what they spit out, whether that’s public information that can be scraped or stuff that has to be shaken loose through freedom-of-information requests.

A second bit involves the old journalistic trick of asking people questions, in this case about their decision-making process that went into formula-building: “Interviews and document investigation are especially important here in order to understand what was fed into the algorithm, in terms of data, parameters, and ways in which the algorithm is used,” Diakopoulos writes.

Interestingly, the very tactics that journalists may use to get to the bottom of algorithms are among the tools that public policymakers have available to them as a matter of course. The advantage of being a lawmaker or an elected official is that you can, in theory at least, demand that you be made to understand the formulas driving teacher rankings, bail amounts, safety inspections and what have you. You can reasonably demand data on their output, too, but it helps to first know what you’re asking for.

Read the full report here.

In case you’re interested in reading more about “algorithmic regulation,” Tim O’Reilly wrote a chapter on the topic in the recent Beyond Transparency book.

Nancy Scola is a Washington, DC-based journalist whose work tends to focus on the intersections of technology, politics, and public policy. Shortly after returning from Havana she started as a tech reporter at POLITICO.