I suspect your tower has some worn parts in it that need attention. The 1\2" ball on the end of the shift lever may be worn square. The slots in in the shifting forks may also be worn.
The attached article may be of some help. Some of the suppliers are now carrying reproduction forks. I would steer clear of them and find some better originals or have yours welded up and milled.
Tom Endy
http://www.santaanitaas.org/wp-conte...estoration.pdf