Despite data pointing to overall health, concerns about the direction of the economy are raised among consumers, businesses and investors alike, the Fed governor said.