Nonbank brokers and lenders typically scale back after the spring homebuying season and the housing market could be particularly slow for the rest of this year.