I just buried mine in dirt - actually mortar sand, with a layer of very fine dirt on top to smooth it out. The sand doesn't crack, at least on a permanent layout, the way clay does, and the local dirt is mostly clay.
Unless it's a new yard, there won't be much ballast visible, just dirt, oil and grease, and "crud". I'd be tempted to glue it all down with diluted white glue, as usual, after adding some craft paint to the mix.