Tessem replaced James Beattie and made an instant impact rifling a shot into the roof of the net two minutes later after goalkeeper Robert Green had failed to hold Chris Marsden's fierce left-foot drive.
Ormerod had been unlucky not to open the scoring in the third minute when his left-foot effort looked destined for the far corner only to hit the right-hand post while Beattie also tested Green on two more occasions in the first half.
But Norwich were by no means overawed as they attempted to reach the sixth round and could have taken a lead into half-time themselves with a little bit more luck.
McVeigh and David Nielsen both sent flashing drives narrowly wide while goalkeeper Antti Niemi had to save bravely at the feet of the latter to deny him a clear-cut goalscoring chance.
Norwich defender Malky Mackay wasted a good chance ten minutes into the second half whern he headed Phillip Mulryne's free-kick wide after finding space in the area before Swensson and Tessem put Southampton in control.
Mackay and Norwich's afternoon were summed up nine minutes from time when the centre-back was dismissed for a second bookable offence after fouling Ormerod.
It is very open and I don't think anyone will want to be drawn against us."