Not knowing my way around motors like everyone helping out here. I failed to get the brushes out. I tried prying the brass cap off but that didn't seem to help although it moved a little. However while doing this, I noticed black particles on my fingers. I tried the battery again and it now seems to be working equally well in both directions. I didn't have time to put it back in the engine to test as that will have to wait until after I get back from TrainFest. Thanks all for the help so far.
That REALLY points toward brushes now.
By prying on that cap, you probably jostled the brush under it. It may have been stuck in its cavity, so it wasn't smoothly riding up and down in there under the spring pressure, against the commutator. If it was sticking, that could definitely cause
bad behavior in only one direction. A brush really has to have a very smooth, curved surface that lays exactly mated up against that round commutator, and it has to be able to maintain constant, but not to hard, pressure.
Any sticking, or a broken chip at the edge of the brush, etc, can make them act very badly.
And as this is an old motor, the brush and its spring might be getting near the end of its life.