we want to buy a home in another state for our retirement home and was told it had to be our primary residence we do not own the home were in now it will be our first home why cant we purchase it with a esop loan?

Do the terms of the ESOP allow for loans to plan participants?

If you don't plan on living in it once you buy it then it's not your primary residence. It doesn't matter if you don't own the home you are living in now, what matters is that it is your primary residence.