The image coordinates and
recovered depths
give complete estimates
of the 3D coordinates
of the reference points
in camera-centered coordinates.
There remains the determination of a similarity transformation
between two sets of 3D points 
.
The best rotation in the least squares sense can be found in
closed-form using quaternions [4,2].
The determination of the translation and scale immediately follow
from the rotation.