The Obama rule holds that the Social Security Administration must report anyone who requires third-party assistance to manage their Social Security benefits is placed on the National Instant Criminal Background Check System, or NICS.
But Republicans, who control the Senate, argued it stigmatised the disabled and voted it down 57 to 43.
Federal law already bans the sale of guns to the mentally ill, but doesn't require states to report people with records of mental illness (being committed to a mental hospital, for instance) to the background check system, so they often don't.
The House voted earlier this month to rescind the same rule.
The measure was initially proposed as part of Obama's push to expand gun control regulations following the 2012 school massacre in Newtown, Connecticut, when Adam Lanza killed more than 25 children and teachers, as well as his mother.
"If you can't manage your own financial affairs, how can we expect that you're going to be a responsible steward of a risky, lethal firearm", he said.
"If there are problems with this rule, they can be addressed by fixing it", said Senate Minority Whip Richard J. Durbin, Illinois Democrat.
The rule, meant to strengthen gun background check laws, was created in the final months of President Barack Obama's administration.
"If a specific individual is likely to be violent due to the nature of their mental illness, then the government should have to prove it", Grassley said, according to the news service. "This regulation obviously and simply does not achieve that". The American Civil Liberties Union has joined with the NRA in fighting the regulation, as has an independent federal agency charged with advising the president and Congress on government policy. That was a loophole that the federal mandate to the Social Security Administration could have, ideally, partially closed.
"The regulation at issue here was issued by the Social Security Administration under President Obama".
"These killings must stop and this rule, as implemented past year, will help to do that", said Clarence Anthony, the CEO and executive director of the National League of Cities.