Photometric stereo has shown its merits in industrial computer-aided optical inspection, but until now has only been used for defect detection as opposed to 3D reconstruction of the inspected surfaces. The reason is that in practice, the geometry of the measurement is not modelled accurately enough to infer surface shapes quantitatively. We derive a new extended photometric stereo model from physically based radiometric relations and effects. The new model is sufficiently accurate for surface reconstruction, nominal-actual comparison and reverse engineering. Thus the measurement of surface normals, which is routinely done on specular surfaces with deflectometry and offers very high sensitivity for surface defects, is now possible on scattering surfaces as well. The formalism is very general and can easily be adapted to different hardware set-ups.