Asset owners, from stocks to housing, were also the big winners in the past few years.