You shouldn't, it's not a press fit. Although they can be a little stubborn, if you know what I mean. Spray some WD40 on the shaft and just work it back and forth a little bit. It will slowly come off.