Or, if you can find the right color, common mortar sand. Much of the Milwaukee main in Montana didn't have "ballast" at all. It was all river rock, whatever color the local rocks were. One source I read, if unknown reliability, claimed that the railroad didn't like sharp rocks, because they wore the ties out faster.
On the other hand, by the 1970s, good ballast would have been wasted on the Pacific Extension, because the track wasn't good enough to need it.