apraying lube on it is just masking a bigger problem. If you spray lube on this then add another medium to attract dirt, salt and grime to be introduced into the areas you see already wearing and will accelerate this by turning the lube and grime into a sanding paste and just doing more damage.
1. Can you pull the pole out by hand back at the dock easily? If so then it's more than likely not your pole which is the biggest problem.
2. I'd nearly be pulling the whole halyard/pole system apart, inspecting/replacing what's stuffed and generally focus around that. 2mm clearance is more than enough for a pole to move freely. Those triangulated pole out-kite up systems gain friction very quickly and very easily due to the amount of turns and blocks they run through. Just one bound up block can make the whole thing hard.
Just because you see the pole with a bit of wear on it doesn't mean that's the issue. It could just be light rubbing as expected from something that slides in and out all the time.