Henry's final scalp was that of Angelo Mathews for 95, with the Sri Lankan captain having broken out of a form rut to lift the tourists into contention after they were 33-3.
The in-form opener and Williamson did a fine fix job, though, embarking on a 122-run partnership, which saw both men pass 50, Guptill with a swept four in the 17th over and Williamson raising his bat five overs later.
Williamson, again leading New Zealand with Brendon McCullum still sidelined by a back injury, had looked in good touch until he was undone by a poor ball from Tillakaratne Dilshan going down the leg side.
Despite losing Taylor, cameos from Luke Ronchi (37 not out) and Mitchell Santner (21 not out) took New Zealand to 294 for five in their quota of 50 overs.
For that reason Sri Lanka left out legspinner Jeffrey Vandersay in favour of fast bowler Nuwan Pradeep, meaning they were carrying three frontline quicks and two seam-bowling allrounders in their XI.
Mathews and Chandimal produced a 93-run stand for the fourth wicket before Trent Boult, making his first appearance in the series, dismissed Chandimal for 50.
Sri Lanka lost half their side with 161 runs on the board when Thisara Perera (15) was caught in the covers. In the 34 over Sri Lanka were 161/5 with the required run rate just over 8 runs an over.
Mathews then found company in Milinda Siriwardana, who smacked three fours and two sixes in a 22-ball 39 before falling to Mitchell Santner.
Nevertheless, the match will still be a cracker given that it is poised to determine whether Sri Lanka can draw the series or if New Zealand can take it with a commanding 3-1 margin.
The match then turned in the next three overs when Siriwardana and Chamara Kapugedara (10) were both dismissed and the chase effectively ended when Mathews became Henry's fourth victim in the 46th over.
The Black Caps now lead the series 2-1.
Dilshan fell to Trent Boult for five while Henry picked up the wickets of Danushka Gunathilaka and Lahiru Thirimanne for 15 and two respectively.