Hmmm... interesting question. The reference in 133 to voting rights specified that the loan would not qualify unless the expanded voting rights were applied. But in defining the voting rights, it described what the ESOP had to provide with respect to the rights of the securities acquired with the loan.
IRC Section 133(b)(7) VOTING RIGHTS OF EMPLOYER SECURITIES. --A loan shall not be treated as a securities acquisition loan for purposes of this section unless --
(A) the employee stock ownership plan meets the requirements of section 409(e)(2) with respect to all employer securities acquired by, or transferred to, the plan in connection with such loan (without regard to whether or not the employer has a registration-type class of securities), and
(B) no stock described in section 409(l)(3) is acquired by, or transferred to, the plan in connection with such loan unless --
(i) such stock has voting rights equivalent to the stock to which it may be converted, and
(ii) the requirements of subparagraph (A) are met with respect to such voting rights.
So, if the loan is paid off, it seems like you don't care about the status of the loan anymore and you may be able to amend the plan to revert to the general voting rights on the shares. But, I don't know that the government would be forestalled from arguing that the expanded voting rights applied to the shares acquired with the loan, rather than just for the term of the loan.
Fence sitting, aren't I? I am going to irritate you further by saying that I think there is sufficient ambiguity that you should check with experienced ERISA counsel and perhaps either go for a letter ruling or make the amendment and highlight in your determination letter application. Note, I did look briefly at 411(d)(6) and the associated regulations and do not see where voting pass-through is a protected benefit. But this is free advice, so you need to check into that, also.