In the debates about what is fueling widening inequality and persistent wage stagnation among lower earners, the decline of unions usually takes a back seat to globalization and automation. But a new study from the Economic Policy Institute suggests that weakened union power and shrinking union density may play a bigger role than previously thought. That’s because its not just union members who benefit from collective bargaining.

According to the report, “unions, especially in industries and regions where they are strong, help boost the wages of all workers by establishing pay and benefit standards that many nonunion firms adopt.” Highly unionized states have driven the push for an increased minimum wage, for example. Employers of nonunion workers might raise pay to prevent workers from organizing or leaving for higher paying union jobs.

But with only 1 in 20 private-sector workers in a union today, compared to 1 in 3 in the 1950s, nonunion workers are no longer getting that union boost. Unionization has declined by two thirds among private-sector men (from 34 percent to 10 percent), by more than a half for women (from 16 to 6 percent) and by more than two-thirds for men without college degrees (from 38 to 11 percent).

The effect is lower wages for everyone. The study finds that private-sector nonunion men’s weekly wages would have been 5 percent higher in 2013 had union density remained at its 1979 level. That’s $52 a week and $2,704 a year. Multiply that by the 40.2 million nonunion private-sector men in this country, and the annual wages lost total $109 billion.

For nonunion men with a high school diploma or less, the effect is even starker. Weekly wages would be an estimated 9 percent higher, the study finds, with a union density at 1979 levels. That’s an annual wage loss of $3,172. Consider that for non-college graduates, the wage loss from increased trade with nations who pay their workers less is estimated at 5 percent.

Nonunion women’s wages have seen less of a decrease as a result of union decline, given that women were not as unionized as men in 1979. Still, in 2013 weekly wages for nonunion women with or without high school diplomas would have been 2 to 3 percent higher with 1979 union density. Looking at all 32.9 million full-time, private-sector nonunion women, that’s a loss of $24 billion in annual wages.

Citing earlier research that found union erosion can explain about a third of the growth of wage inequality among men between 1972 and 2007, and about a fifth among women, the report states, “At least for middle-wage men, the impact of the erosion of unions on the wages of both union and nonunion workers is likely the largest single factor underlying wage stagnation and wage inequality.”