David Lowe's side went into the game in fine form after wins over Middlesbrough and Bolton, but came unstuck in the second half against Tottenham.
A first half hat-trick from Dean Rittenberg put Rovers in command, but four unanswered second half goals by the visitors saw Spurs claimed all three points.
On a sunny Saturday morning Rovers were quick out of the blocks, a storming run from Sam Lavelle in the 4th minute saw the defender find Rittenberg but the Tottenham goalkeeper made a fine one-handed save.
Sixty seconds later the deadlock was broken, Sam Joel fed Ian Vicars on the overlap down the right and his centre was swept home by Rittenberg.
On 13 minutes Rovers grabbed their second of the game, Liam Langford charging from left back to the edge of the area to slip in Rittenberg and the in-form forward rounded McGee in the Spurs goal to score.
The visitors were stunned and on 39 minutes Rittenberg completed his hat-trick, when a scramble from a corner saw the striker force the ball over the line for his 7th goal of the season.
Rovers looked in total control but a goal from Tottenham right on the stroke of half-time from Harry Winks gave them renewed enthusiasm and they came out a different side in the second half.
Rittenberg could have made it 4-1 a minute after the restart but his volley from Joel's cross was just wide.
Spurs made the home side pay as they scored a minute later, Will Miller cutting in from the left to fire into the far corner of the goal to make the scoreline to 3-2.
On 69 minutes Spurs were level. Will Butterwick was adjudged to have tripped Nathan Oduwa as he raced into the area, the linesman on that side giving the penalty.
Substitute Shayon Harrison made no mistake to make it 3-3 with 20 minutes to go.
Rovers tried to wrestle the advantage away from the visitors, but the intensity to their play, so prevalent in the first half, wasn't there in the second, the midweek match against Bolton obviously taking it's toll.
With eight minutes remaining Tottenham completed the comeback by taking the lead for the first time, Harrison saw a shot hit the bar but as Rovers struggled to clear their lines, Oduwa pounced.
Rovers could have equalised on 89 minutes when Dean Rittenberg powered a header goalwards from a cross from the right, but McGee made an important save.
To rub salt into the wounds Tottenham raced away with Harrison grabbing his second with a break away goal.
It was a disappointing end to the match. Rovers had dominated the first half, working tirelessly to give Tottenham very little space. But as Rovers tired Tottenham took full advantage to claim all three points.
ROVERS: Williams, Vicars, Langford, Butterwick, Lavelle, Wassi, Harris, Torres (sub Scott 64 mins), Rittenberg, Joel (sub Tanner 70 mins), Doyle (sub Wall 46 mins).
Subs not used: Stanford, Pierce.