Is there a penalty or fee for early payoff?

No, there is no fee or penalty for paying off your first mortgage early.