Colin Hendry's young side more than held their own against highly-rated opposition, creating the better chances throughout, but fell to a late Billy Clifford goal at Cobham.
Rovers gave a debut to Academy goalkeeper David Raya, whilst fresh from returning from loan spells with Fleetwood Town and York City, Ryan Edwards and Jack O'Connell formed the defensive partnership.
It was the Lancashire club who were lively early on with Chelsea captain Danny Pappoe called upon to make a vital block to thwart Micah Evans.
From the resulting corner Rovers thought they had scored through Jamie Maclaren but Aziz Deen-Conteh cleared off the line via the underside of the crossbar.
Good defending from Peter Wylie prevented the home side registering their first effort after 25 minutes, getting back in time with an important tackle on Milan Lalkovic.
That was a rare Chelsea chance in a first half in which Rovers dominated, a mix-up between Pappoe and goalkeeper Jamal Blackman allowed Osayamen Osawe in but he fired wide.
Rovers continued to threaten by Jamie Maclaren volleyed wide when well placed.
The second half saw Chelsea grow more into the game, Partick Bamford forcing David Raya into early action, the young Spaniard saving well.
But Rovers were holding strong and aside from a defensive mix-up when Wylie had to clear off his own line, the home side rarely threatened a resolute back line.
However just as it looked like Rovers would claim a worthy point in their final game of the season they were undone by another defensive error.
Tim Payne's back pass to Ryan Edwards was underhit and that allowed Billy Clifford to nip in and the midfielder cooly finished.
The result was harsh on Rovers as they had been the dominant side, creating the better chances in the match.
"For the second time in a row we are lose because of a mistake on our part which is difficult to accept," said Colin Hendry after the game.
"We played well, we had loads of chances and couldn't score, but they capitalise on a mistake of our making which is hard to take."
CHELSEA: Blackman, Bangura (sub Nortey 40 mins), Pappoe (c), Nkumu, Deen-Conteh, Ssewankambo (sub Hunte 30 mins), Musonda, Clifford, Bamford, Starkey, Lalkovic.
Subs not used: Beeney (GK), Muleba, Seremba.
ROVERS: Raya, Wylie, Bauress, O'Sullivan, O'Connell (c) (sub Daly 70 mins), Edwards, Evans, Payne, Slew, Maclaren, Osawe.
Subs not used: Urwin (GK), R. Hanley, Brown.