'Owning a second home worth over $5 million should come with a responsibility to the city that makes it valuable,' the city's mayor said.