I must have been lucky or something but I've still yet to be confronted by a security guard - it all seems very pointlessly aggressive and confrontational. If you were taking photos to stake out the building some way, you'd do it from miles away with a zoom lens. I have been stopped by people obviously worried I was from the Social and I've told them the same thing - if I wanted to take your photo covertly, I'd do it from bleeding miles away and you'd never even know a lens had been pointed at you.

Fair play for standing your ground.