Time is not a reliable way to do it. These are not stepper motors and load will change the time it takes to travel. All it would take is for the mechanism to be a little stiffer one way than the other to quickly get it out of time. Limit switches are the best way for the mechanism only to be used reliably.
The best solution would be to etch a board with the limit switches built in, using the existing wipers on the servo mech as the contacts to control it. That may get into transistorized control of the motor though because I don't think the wiper contacts would handle the current load to use them as the switch, maybe they would though, it could be worth a try.