Office building in London, without Goldman Sachs paying huge wages to staff operating it's business interests in the EU, is just an empty building.
And if you think EU will just sit idle while UK is rorting it with taking all the benefits of large EU market while blocking things UK doesn't want. Well, that's very hard to imagine playing out in the real world.
I also doubt China and USA (especially with Trump) won't try to force deals that suit them. Once UK leaves the safety of 2nd largest economy in the world, which isn't as easy prey as UK on it's own.
Anyway, I hope the whole thing plays out ok for all and people don't suffer from the whole process!