Try pivoting on the heel, before you shift your weight onto the foot, as opposed to pivoting on the ball after the weight is already on it.
I had an interesting experience regarding this question when I switched from traditional Tae Kwon Do to Shaolin Kung Fu years ago:
In TKD, there was a very intense focus on all the little details of how exactly to position which part of your body. Pivoting my foot on the ball prior to a kick, with most of the weight on it, was something I practiced over and over and over, for years, with limited success.
When I switched to Kung Fu, I immediately noticed that everybody was pivoting on the heel, and, more specifically, pivoting before they put their weight on the foot. This felt very natural, was very quick, and I quickly forgot my TKD training.
Perhaps more interestingly, from that day on, that was about it on pivoting. To TKD or Karate students, Kung Fu seems to have a great degree of anarchy in it. I asked my teachers on many occasions for detailed instructions like how far I should pivot, 45, 90, 180 degrees? I was accustomed to those things mattering a lot.
The answer was always something like "don't bother, that depends on far too many things to make this a useful thing to worry about". I asked, what does it depend on, and they said, are you wearing footwear? How important is effectiveness vs elegance? What's the ground like? What shape is your body in (like, if you're not warm, or slightly out of shape, you should pivot less)? Do you need to be very quick? Are you aiming high or low? What technique comes next? Is it important to hide your intent? Etc.