Jen, you may recall I posted about my search for a new endo. I think I had some of the same concerns as you did. I was particularly concerned at the time to not cause my previous endo to start a war against me. Dr's talk, you have records exchanged. If a Dr labels you as a pain, non-compliant, etc, you may have difficulty finding a new Dr. So I would encourage you to not burn bridges even if you think this guy has not given you the best care.
If you do consider switching within a practice, you may have difficulty as many practices don't allow you to switch between Drs. They may want to avoid competition and poaching of patients. They may want to discourage you from "shopping" a diagnosis. Whatever. Many practices have written policies against this, it would be good to understand what you are up against before trying to change within a practice.
ps. And what I discovered is that sometimes a Dr can be the worlds best specialist and still be a lousy Dr.