No way to tell without seeing it, and since it's purely a judgment call I might disagree with the referee right there, and who's to say I'd be right?
All I can tell you is how the rules work. I'm not going to quote it word for word but here's what it says, paraphrased.
A defensive player cannot charge into a kicker after the kick has been made. There is leeway given to a player or players if it's not reasonably certain a kick would be made. So if the player is truly trying to make a tackle on a guy he thinks is a runner, he gets a pass. HOWEVER, if the kick is made, then the defenders do not have a free ticket to nail the punter. If the referee determines that contact could have been avoided, then roughing would be the correct call.
Is that clear as mud?
I had a play once where a punter took a botched snap and started running upfield. Two guys had him lined up and at the very last second, he kicked it, then they sandwiched him. I had no flag and the coach went nuts. But I've also had a punter run then punt and a guy clearly watched the punt fly right past him, then he still knocked the heck out of the punter and I flagged him. that coach went nuts, so I guess in those situations one coach is going to go crazy no matter which way it goes..