You shouldn't have metal shavings from the track in your motor, unless you have some VERY old track. The last steel track I saw in N scale was made in the late 60s or early 70s.
On the other hand, metal from ballast is a very real possibility, especially if you use real dirt for ballast or scenery. The sand where I grew up has enough iron particles in it that they can be collected with a magnet. Run a small magnet, it doesn't have to be very strong, through the sand, and it comes out fuzzy with iron.