You guys did pretty well with this one. I remember being given this many years ago and I insisted it had to be a safety to kick a ball through your own endzone. I was ready to go to the mat for that. How could it not be a safety in both cases?
Turns out, it's not a safety in either case. B foul by illegally kicking the ball in the field of play, but the original kick never ended becasue the only way a kick can end is when it is possessed or goes into the endzone or out of bounds.
As has been said, ALL kicks into B's endzone are touchbacks.
To clarify, it's the kick by A -- the place kicker or the punter -- that is the kick that went into B's endzone resulting in a touchback. Even though the B player kicked the ball, he did not possess it, so the original kick never ended and the result of the play is a touchback, regardless of whether it's a kickoff or a punt.
Of course, there is a small matter of an illegal kick by the B player.
An illegal kick is a 15 yard penalty. If the play is a free kick (kickoff), the options for A are to decline the penalty and it's B's ball at the 20 because of the touchback, or take 15 yards from the previous spot and rekick: So there will be a rekick from the B45 yard line.
If the play was a punt from the 40, it's a post scrimmage kick foul by B. That means the basic spot for PSK that ends in a touchback is the 20. Because the spot of the foul was behind the basic spot (the 10), the penaty will be half the distance to the goal line from the 10, so B will have the ball at their own 5.