Industry veterans are wary of prospective borrowers who can’t pay for agents, and of compensation guidelines clashing with government mortgage lending programs.