I have parted ways with a few Endo's over the years. My first one moved out of state to teach at the Albany Med, so I was passed on to another Pediatric Endo. he was nice, but not the same as my original endo. He broke up with me my sophomore year of college, saying that I might want a big boy doctor as if I had any serious issues and was in the hospital he might not be able to see me or something like that. It was sad when that relationship ended though, I was with him a good 5 years. I moved to a big boy endo. and had him for two years before he broke up with me since he was moving to Pittsburgh I think to teach. Which is what puts me with my current endo. I like her alot, although it is sad that I go in, she cmall talks for 2 minutes, weighs me, looks at my numbers, gives me my a1c, looks at my feet, asks if I have had my eyes checked, and smacks me on the butt and sends me out the door. I spend more time in the waiting room than I do talking with her. But I am okay with that right now.
However, sometimes switching doctor's could be a good thing. Your doctor may just be trying to help you find the motivation in your treatment that he feels you are not doing right now. There is no harm in trying and there probably is someone better out there, who can help you find what you need in your treatment to make things better, so to speak.