That depends on how many officials are in the crew. If there are five or fewer, it can be the referee or umpire because the side officials are downfield in coverage. In these mechanics, the umpire moves up to the ine of scrimage (or the edge of the expanded neutral zone) to rule on ineligibles downfield. He can assist. If I'm trailing a QB and I'll mark the spot mentally, then run to that spot and look at the box.
If there are six or seven offivials, then there are deep wings (a technical term) can cover deep coverage and the line judge can stay at the line of scrimmage to look across the field at the box and rule on an illegal forward pass or ineligible receiver downfield