Downsizing after retirement often makes no sense — are you locking yourself in for a smaller life than you should?

So what’s keeping so many older Americans and retirees from making the big move? In short: money.

Here’s what you need to know to make the right choice for you when the time comes.

The hidden costs of downsizing

On the surface, downsizing seems like a simple math problem. Take the fair market value of a large four-bedroom detached house and subtract the market value of a smaller, two-bedroom apartment and the difference is your ticket to financial freedom.

However, this simple calculation overlooks many of the hidden costs of buying and selling homes. Agent commissions, closing costs, taxes, home repairs, mortgage repayment and other additional costs can all add up to 10% to 15% of your home’s final selling price, according to Experian.

It can also cost as much as $10,000 if you’re moving long-distance, such as from the East Coast to the West Coast, according to Rocket Mortgage.

Beyond these transaction costs, there’s also capital gains taxes to consider. The Internal Revenue Service (IRS) exempts up to $500,000 in capital gains for a married couple filing taxes together. But if you’ve owned your home for a few decades in a relatively high-cost-of-living region, there’s a chance your gains exceed this threshold.

For many millionaire homeowners, this tax is an additional barrier to downsizing.

For others, the mortgage rate is a key concern. As Redfin notes, the majority of baby boomers (58%) have already paid off their mortgages. Even if they haven’t, many benefit from ultra-low mortgage rates — and that’s not something they’re keen to give up with rates now at 6.65%.

And finally, selling your home can have implications on your Medicare costs thanks to IRMAA (income-related monthly adjusted amount).

Simply put, there are many common scenarios where downsizing doesn’t make financial sense. The costs and taxes can diminish the appeal of moving to a smaller home, let alone the lifestyle challenge of adapting to a smaller space.

Redfin adds there’s also a non-monetary rationale behind staying put.

“There are also social and lifestyle reasons to stay put: Baby boomers, in their sixties and seventies, may want to stay in the neighborhoods they’ve lived in for a long time, close to their friends, family, work and/or recreational activities.”

That being said, downsizing isn’t always a bad idea.

When downsizing makes sense

There are specific circumstances where downsizing is clearly the right financial and lifestyle decision.