My husband and I are retirees with $2.5 million in investments. Our 4% $200,000 mortgage is keeping us up at night – should we pay it off?

Yet for retirees with a growing retirement investment portfolio and a home mortgage with a low interest rate, rushing to pay off the mortgage using investment cash has its pros and cons.

Consider Leslie, 57, and her husband Rick, 59, who want to use some of their $2.5 million investment account to pay off a $200,000 mortgage with a 4% interest rate. The couple also plan to retire, each by age 62.

With their golden years almost in reach, should they raid the investment portfolio or keep paying down the mortgage and leave their stocks, bonds and funds alone?

Here’s what retirement savings experts advise for couples like Leslie and Rick, who feel they’re caught between the rock and the proverbial hard place with a nearly-paid-off mortgage.

Start by asking where that $200K is actually coming from

Knowing your exact current and estimated portfolio picture is just as important as the mortgage rate. “On paper, paying off a 4% mortgage gives you a guaranteed 4% return because you’re eliminating that interest expense,” Steve Sexton, CEO of California-based Sexton Advisory Group, told Moneywise. “If the money stays invested and earns 6-7% over time, you may come out ahead financially.”

Here, two often underrated household finance factors come into play. “The portfolio’s return on investment isn’t guaranteed, and in retirement you don’t necessarily have decades to recover from a bad market at the wrong time,” Sexton noted.

Leslie and Rick will also want to compare the mortgage rate with the after-tax return on the investments, and not just the headline return. Plus, Sexton advises the couple to examine whether the mortgage interest is actually providing a tax benefit. “Mortgage interest generally has to be claimed as an itemized deduction, and many retirees may get more benefit from taking the standard deduction instead,” he said. (The IRS explains the mortgage-interest deduction rules here.)

Ultimately, the key questions Leslie and Risk face before going forward are critical ones:

• What is the after-tax cost of this mortgage?
• What would we realistically expect this $200k to earn?
• What tax bill would we create by accessing it?
• How much liquidity would we have left afterward?

Make a decision that will help you sleep better at night

Having peace of mind is important, and it should count heavily when Lelsie and Rick make their decision.

“I would never tell someone to look at the numbers if the debt is really causing them stress,” George Dimov, CPA and founder at Dimov Associates, a New York-based tax and accounting firm, told Moneywise.

After all, a couple with two and a half million dollars is not losing sleep over the interest rate, Dimov noted. Instead, they’re losing sleep over having to make a payment every month. “Paying off the mortgage to get rid of that stress is a choice even if the math says it is slightly better to keep investing,” Dimov said.

There are also hidden costs to paying off a mortgage

Leslie and Risk are both invested mostly in their company retirement plans, and that’s a big factor, too.

“If you take two hundred thousand dollars out of an IRA or 401k, every dollar is considered regular income,” Dimov noted. “To get two hundred thousand dollars, you might have to withdraw two hundred and sixty thousand dollars or more.”

That move alone can push the couple into a higher tax bracket, make more of their Social Security taxable, and trigger a Medicare Income-Related Monthly Adjustment Amount (IRMAA) surcharge two years later.