You don’t even have to be a fan of the show to enjoy this charming town in southwest London. Here’s why you should visit.