Ionia, a small Michigan city that’s home to about 11,000 people, has a tendency to be overlooked. Its motto is a touch defensive: “More Than You Imagine!” The town was once featured in a national Associated Press story, but the New York Times misspelled its name in the headline as “Iona.”

If Ionia does ring a bell, it is probably because of the prisons. Thirty miles northwest of state capital Lansing, it’s home to five of Michigan’s 34 correctional facilities, including both the state’s oldest and newest lock-ups. The prisons sprawl over 4,000 acres The Michigan Reformatory, which was built by prisoners in the late 1870s, is bordered on the west by Ionia’s Wall Street — named not for the stock market, of course, but for the 15-foot stone wall that separates the prison from old family homes.

Ionia is a prison town. That is its primary industry. That’s why there’s a shiver that runs through town when word spreads about the possible closure of one of those facilities. Most of the thousands of inmates incarcerated in them come from 130-some miles away in southeast Michigan, where Detroit is the hub. The distance makes for a severe reduction in family visits and programming, which ultimately increases recidivism — that is, it increases crime. But building prisons here is what puts food on the table for local families, where the median income is $36,315. (Compare that to the state median, $48,273.) And Ionia is not unusual. In the 1990s, with mass incarceration accelerating, a new prison was built in rural America nearly every 15 days. Between 1980 and 2002, the majority of prisons were built in small towns; about 350 new prisons were put in rural counties. Before 1980, only 36 percent of prisons were in rural America.

But that was then. Today, in an era of increased scrutiny on mass incarceration and strained state budgets, prisoners are paroled more frequently and receive shorter sentences. Between 2011 and 2014, at least 89 correctional facilities in 25 states were shut down. Even communities that have long been home to a prison are pushing back, including northeast Philadelphia, where the House of Correction has stood for 140 years. Officials want to build a new prison to replace it, but the impassioned skepticism of local residents is postponing plans.

All this raises a pressing question: How do you detach a community from its dependency on the prison economy, without doing undue harm to local citizens? Is it even possible to wholly extract these forbidding fortresses from their intended purpose? After all, they were designed to be a place that nobody wants to be in. Puzzling out a way to find a new use, especially in rural areas and small towns, is a critical challenge for 21st-century planners.

“Our nation is heavily, heavily incarcerated,” says Raphael Sperry, president of Architects / Designers / Planners for Social Responsibility (ADPSR). “If we’re at all moving toward social justice, we will end up with a lot of these facilities doing different things.” ADPSR asks architects to pledge that they will not design new prisons. It’s also facilitated workshops in California for communities to re-imagine how to use a closed prison or jail. In Stockton, adults could see their local lock-up becoming a community college. Kids who attended the meeting want it to become a water park.

An interesting experiment in prison reuse is playing out in New York, where Governor Andrew Cuomo is recalibrating the corrections system. Early in his administration, he traded on his popular support to begin the process of closing prisons, and so far, 13 adult prisons and numerous juvenile facilities have been shuttered. This was tricky to do. Even though New York’s prison population had decreased by 22 percent, closure was viewed warily by the small prison towns and the strong unions of corrections officers. But the backlash was subdued by the government’s commitment to give a $10 million economic transformation grant to each community that lost a prison.

It’s more than the economy that needs to change. Tracy Huling is a New York resident, a Soros Justice Fellow and founder of Yes in My Backyard, a site dedicated to prison reuse. She is on a mission to help communities move beyond their identities and limitations as prison towns — and that includes digging into the emotional history of historic and intergenerational prison towns. “How do you help people change their culture, when forcing a change in industry can be very painful and disruptive and even create some war within the community?” Huling asks.

To explore the answer to that question, Huling founded the Prison Public Memory Project. It is designed to use culture to change culture. It is a vehicle for documenting and interpreting the history of prison towns, in partnership with local residents, “so that their history is not ripped away from them with no acknowledgement that there was anything good that might have come from their work.” Indeed, while this work is about a community grappling with its own dark histories, it is also about acknowledging positive outcomes, whether it was staff members who were able to pay off their mortgages with their stable jobs or people who, while in prison, learned how to read. “We acknowledge the suffering, but we also need to acknowledge that people make meaning out of that,” Huling says.

They have begun with Hudson, New York, where a very old local prison has served, variously, as a reformatory for women, a detention center for girls and a men’s prison. Because of its age, it is on the shortlist for closure, and Huling was interested in beginning the memory project before the community absorbs the shock. Local people were hired to begin with archival research and oral histories. Programming, from illustrated lectures to community photo shoots to story sharing at the library, invites residents into the conversation.

“Communities with prisons often absorb the culture from inside the walls, the code of silence,” Huling says. “There are good reasons for that and bad reasons for that.” By gently interrupting that pattern, the team is constructing a creation story of the community of Hudson.

After its third year of working in Hudson, the Prison Public Memory Project received funding to take what it’s learned and do two things: to create a permanent site of prison memory in Hudson and to create a national organization that will provide assistance to towns around the country that are transitioning from their prison culture. Huling is on the brink of putting together a national board of directors.

“When I started this project, there were a lot of skeptics,” Huling says. “They said these places can’t be repurposed. They should just be burned down. That’s easy for people to say who are outside the communities, and outside the whole field of community development. People who understand the relationship of place and industry and culture are the people most excited about the possibility of using these places to crank up something new and exciting in the community.”

When repurposing prisons, it’s worth considering how their structures are effectively little cities unto themselves, complete with their own sewer, water, and food and recreation facilities. That’s encouraged planners to think big.

In a much-celebrated urban example of prison repurposing, a nonprofit called the Osborne Association is turning a South Bronx prison into a multi-purpose assistance center tailored toward the needs of returning citizens. (More than 60 percent of the state’s prison population hails from the five boroughs.) New York State has helped fund this through the pool of money that was set aside to help prison communities transition. After years of significant planning and support-building, the seven-story facility is on its way toward reconstruction and becoming a national model for repurposing prisons in a nonprofit service model.

In North Carolina, Growing Change is repurposing small prisons into sustainable farms and recreation centers. The state once had an unusual system of county prisons, but they were closed en masse, leaving about 50 closed corrections facilities across the state. Now a coalition of public and private agencies are flipping these prisons into therapeutic centers where young people and veterans can be trained in agriculture and entrepreneurial business.

“Look at this project in North Carolina,” Sperry says. “They’re using an old guard tower as a recreational climbing tower for at-risk youth. That’s a very poetic use. It was once used to surveil them, to threaten them, and now it’s used for them to learn how to overcome challenges, and build their hope and leadership skills.”

And in the small upstate New York community of Warwick, where the Mid-Orange Correctional Facility was closed in 2011, the community’s Republican mayor decided he wanted his own community to decide how to repurpose the 730-acre site, rather than outsiders, or chance. He initiated a major community planning process to choose a new way forward. The Mid-Orange property was surveyed and environmentally remediated. Zoning regulations were examined. And, after about a year, the community came up with a plan that divided the old prison into multi-use parcels. Warwick will retain some land as green space, reconnecting parts of the community that had been divided from each other. Trans Tech, a manufacturer of school buses, is leasing a portion of the property to develop electric buses. Thirty-six acres of the facility were sold to a private company that will transform them into an indoor-outdoor sports recreation facility.

Other prisons around the country, especially those with notorious histories and significant architecture, have become prison museums, or haunted houses and sites for paranormal tours. This can be done in a way that is more or less tasteful, and Huling points out that the Eastern State Penitentiary in Philadelphia, one of the first prisons in America, raises about 65 percent of its operating budget from its “nights of terror” in Halloween season. And that budget is spent on thoughtful historic programming and education.

For prison towns looking to find their way to a new identity, Huling cautions them against trading one unwanted land use for another. Exchanging a prison for, say, a factory farm, doesn’t go far. And transitioning the prison into another lock-up, like an immigrant detention center, is also unsustainable in the long run. “We know that prisons, even though they provide good-paying jobs, are not actually good for economic development over a long period of time,” Huling says.

Most of all, a state shutting down a prison and walking away is the worst possible strategy. It leaves the community — which does not own the facility — uniquely vulnerable. Communities must work with state and federal government in developing a proactive plan for transitioning the site to new use. “Because government investment is part of what turns a prison town into a prison town, government has a responsibility to provide a pathway out of it,” says Sperry. “At the very least, they should provide a job planning process.”

Also deserving of state support: an appraisal and a proper environmental remediation. And if state governments plead that they don’t have the money for this? “It’s [cost is] so much smaller than the salaries paid to run an entire facility,” Sperry says. “Asking for this is like a drop in the bucket. It’s a very reasonable request.”

A proper transition should also include a long-term initiative to collect and acknowledge the history of the prison, as Huling is pioneering. That includes a permanent site of prison memory.

“Those places have affected thousands and thousands of lives very profoundly,” Huling says. “It’s both the right thing to do, and a very proactive way to build community.”