We like a challenge, that's the problem! I inherited a bright pink one last year in this house and it's about to burst into flower and looks great. It has no maintenance or food and the previous owner was not a gardener so I doubt it ever has. If they don't have the right conditions they won't perform at their best - most plants are the same to be honest. There are lots of plants that struggle in the cold wet sticky clay we get here so I go with that rather than fight it. I have built raised beds here so that I can grow bearded Irises (which I love) and they need sharp draining soil and sun. They'd never last if I planted them in the same bed as the rhodo!
This is what mine looked like last spring: