It occurred to me that the shift fork was acting like a disc brake pad. That would explain the non-contact area (inside the horseshoe) wear. Intuitively, it seems like the damage would occur only when this gear is delivering the power, the drive dogs providing the lateral locking force necessary to hold the interference fit. That said, it's a non-spinning (splined) gear that sees full engine torque, regardless of which gear is delivering power to the C/S. Thus, the altercation could occur in 1,2,3, or 4.
As was suggested earlier, lateral misalignment may be the issue. Unfortunately, the fork-to-gear tolerances are so precise I don't know how one could measure them, in 1,2,3 & 4, with the motor assembled. Case gasket vs no gasket, thrust washer(s) vs none might point you in the right direction. However, going by the wear I measured, I dunno how anyone could see what's going on...we're talking about a lot less than the thickness of a case gasket, or a thrust washer. I can easily see how this tranny was still shifting normally. It also looks like a few thousandths of an inch could make all the difference.
Rather than yet another longwinded dissertation, how about going back to basics. With two pairs of hands, it's possible to apply rotation and a little torque to both the M/S and C/S...while shifting gears (either via the shift lever, or fixing bolt). Leave the gear and new shift fork clean & dry, so as not to diminish any braking action. With as much heat-discoloration as the old fork developed, you may be able to feel when the fork is being forced against the gear. Alternatively, or additionally, Dykem or a Sharpie, could be used to mark the fork tips. You'd then spin the tranny, in a given gear (4th seems the most likely position to have created the problem), pull the drum assembly and look for signs of contact. Plan on at least a half-dozen repetitions, on the bench. Hopefully, you get some answer(s) with which you are comfortable before you feel like Charlie Chaplin in "Modern Times".