A second half brace from Filip Duricic ended Scotland's qualifying hopes in Novi Sad.
Rovers had two players in the starting eleven as Grant Hanley and Jordan Rhodes both featured, but on a sticky surface Scotland struggled to carve open any meaningful opportunities, one of the best falling to Rhodes on 58 minutes but the Rovers striker shot tamely.
It would prove to be costly as Serbia raced straight up the other end with Duricic applying the finishing touch after a goalmouth scramble.
On 65 minutes the tie was over as the Serbian striker netted another following a defensive mix-up between Gary Caldwell and Alan Hutton.
Grant Hanley had a strong display at the back for Scotland, he played the full match, whilst Rhodes, who had a frustrating night, was substituted with ten minutes to go.