They're a bit like Marmite...........??
It's your garden so you can do whatever you like. But if you feel you can't remove them because your neighbour likes the hedge, personally I would start by giving them a harsh trim on your side. Then in 6 months you can say they looked awful, were told they weren't going to recover, and they just had to go. Then you can dig them up and put something to your taste in their place.