A lot of model manufacturers play fast and loose with gear design, so you may have inadvertently inherited their sloppiness. Are the axle gears you're using straight cut or helical? For good gear design they should be helical to match the worm, something that is rarely seen in N scale as it costs more to tool it properly.
Does it run well without the axles?