Rotherham midfielder Martin Woods twice came within a whisker of an opener in the first minute as the League One strugglers gave Norwich a fright.
A gap opened up and Woods smashed a low shot from just inside the box that keeper Paul Gallacher dived and parried away.
But as the home side kept up the pressure, Michael Keane controlled and rolled the ball back to Woods who produced another fierce effort with Gallacher stranded but Jason Shackell produced a last ditch goal line clearance.
And moments later when a free-kick came Rotherham's way it was Woods who hit another blistering effort from 25 yards but it was blocked.
Norwich though, fielding a weakened side, had veteran Peter Thorne to thank for taking the lead after six minutes with their first effort.
Jurgen Colin swung in a cross from the left and Thorne drifted away from his marker before heading a looping effort over Neil Cutler for only his second Norwich goal.
Rotherham could have been level after 22 minutes when Gallacher gifted them a chance when he raced off his line for a cross and dropped it for young striker Ryan Taylor.
He lashed a low shot with Gallacher still on the floor but Colin stepped in smartly to intercept and clear.
A minute later Rotherham were level when Keane notched up his first goal for the club. David Worrell fed Taylor who turned and picked out the Irish midfielder who lifted the ball over Gallacher for a cool finish.
Two minutes before the break Norwich were grateful for another timely intervention from Shackell when Lee Williamson produced a goalbound header from a Woods cross.
Four minutes into the second half Williamson smashed home a penalty to take the lead after Shackell clutched the shirt of Will Hoskins when he darted into the area.
But in the 54th minute Norwich were level when Cutler spilled a shot from substitute Paul McVeigh and young striker Jarvis blazed home the rebound.
Rotherham were still reeling a minute later when an Ian Henderson cross found Craig Fleming to head home.
Jarvis added a flattering fourth at the death.