Considering coordinate errors of both control points and non-control points, and different weights between control points and non-control points, this contribution proposes an extended weighted total least squares (WTLS) iterative algorithm of 3D similarity transformation based on Gibbs vector. It treats the transformation parameters and the target coordinate of non-control points as unknowns. Thus it is able to recover the transformation parameters and compute the target coordinate of non-control points simultaneously. It is also able to assess the accuracy of the transformation parameters and the target coordinates of non-control points. Obviously it is different from the traditional algorithms that first recover the transformation parameters and then compute the target coordinate of non-control points by the estimated transformation parameters. Besides it utilizes a Gibbs vector to represent the rotation matrix. This representation does not introduce additional unknowns; neither introduces transcendental function like sine or cosine functions. As a result, the presented algorithm is not dependent to the initial value of transformation parameters. This excellent performance ensures the presented algorithm is suitable for the big rotation angles. Two numerical cases with big rotation angles including a real world case (LIDAR point cloud registration) and a simulative case are tested to validate the presented algorithm.