When setting up a freshly built car I have difficulties with this myself from time to time. The jam nut setting is the most sensitive aspect of linkage adjustment. It's fairly easy to adjust the shift shaft length, but the clocking of the shaft with the transmission when tightening the jam nut can be a bugger.
If not adjusted JUST RIGHT, the trouble I have is putting the trans in first gear reliably and smoothy from neutral, or down shifting from 3rd to 1st. If slightly mis-adjusted this won't be smooth. But usually when slightly misadjusted, once into 1st, upshifting to 2nd then 3rd and 4th is a slam dunk. But then getting it back into first is usually my problem, and it takes a VERY SLIGHT mis-adjustment to cause this problem.
Yes, it's 1st gear that gives me the most trouble, but with patience I finally get it sorted out to work. Once I get first working right, reverse has never failed me, nor any other gear.
Adjusting the length of the shaft is as simple as taking off the shifter (using a conventional VW style shifter) from the mount (typically two bolts) and center the ball socket (front to back) in the middle of the mount hole when the transmission is in neutral. Eyeballing this front to back adjustment for shaft length is usually easy and reliable. The side to side adjustment is done with the jam nut and that's the tricky one. To do that you have to reinstall the shifter with the two bolts. Usually there's some slop in the shifter bolt holes to provide some adjustment capabilitiy forward and back, if needed. I always install the shifter centered on those hole slots so I have it in the middle, front to back.
To adjust the side to side clocking of the shaft with the jam nut I put the transmission in first. That can be tricky with your linkage out of adjustment. It can take a few tries. My final check to confirm it's in first is....... with the clutch out and ignition off, I bump the starter switch to confirm it feels like first gear. The car will jump forward, be careful. It's safer to do this with wheels of the ground using jack stands, but never do.
After confirming the transmission is in 1st gear, AND the linkage adjusted to right length, I have a buddy help out to hold the shifter in 1st gear position. While that guy holds the shifter to the left/forward position for 1st, putting a good amount of tension on the shifter, I tighten the jam nut as careful as possible, trying to make sure no torque is put on the front portion of the shaft that goes to the shifter. For my jam nut I use two big crescent wrenches, but others are different.
I've often thought that someone needs to invent a slick shifter adjuster that make this adjustment process easier. There's got to be a better method than the typical jam nut solution. A good engineer with an imagination could do a lot better.