The 17-time Grand Slam winner was due to play last week in Madrid but withdrew with back pain. He came to Rome early to train, drawing 6,000 fans to a weekend practice session.
He has laid low in recent days, declining pre-match media responsibilities as he waited to let his racquet speak for him and his current clay form.
Federer is playing at the Foro Italico for the 16th time as he seeks his first title here. The Swiss has finished runner-up four times, most recently last year against Novak Djokovic.
Rome marks only the fourth tournament of the season for Federer, who underwent knee surgery in early February, the first operation of his career.
Victory over Zverev sent Federer’s record in 2016 to 11-3. He next plays promising young Austrian 13th seed Dominic Thiem, who defeated Portugal’s Joao Sousa 6-3, 6-2.
Zverev, aged 19 and ranked 44th, could not have asked for a more challenging Rome debut as he faced off against his tennis idol. His loss left him 0-8 against top 10 opponents.
Federer broke for 4-2 on his way to collecting the opening set in 28 minutes.
The Swiss then earned a break in the third game of the second set but then lost serve for 2-all.
The veteran broke back on his fourth opportunity in the 11th game for a 6-5 lead and finished off the welcome win a game later, a drop shot setting up two match points with Zverev driving wide on the first.
In women’s second-round play, Spanish third seed Garbine Muguruza crushed Russian Ekaterina Makarova 6-1, 6-0 while Madrid quarter-finalist Irina-Camelia Begu of Romania upset injured fourth seed Victoria Azarenka 6-3, 6-2.
The Belarusian withdrew last week from Madrid with back pain and was unable to fight off her Rome challenger after winning all five of their previous meetings.
“I was trying but it didn’t work,” Azarenka said. “The back is not good right now. I’m very disappointed obviously with that.
“I thought I was okay and was ready to play and came back, so I’m just disappointed. We’ll see what happens. I’m not going to think about the future.”
Czech Barbora Strycova beat Britain’s Heather Watson 6-4, 6-2, while Muguruza called her winning effort “a great match”.
“Everything I did was working. For sure, it’s one of the best matches of the year because everything went well,” said the Spaniard, who will face Latvia’s Jelena Ostapenko next.