On the selection of a two-dimensional signal constellation in the presence of phase jitter and Gaussian noise

A long-standing communications problem is the efficient coding of a block of binary data into a pair of in-phase and quadrature components. This modulation technique may be regarded as the placing of a discrete number of signal points in two dimensions. Quadrature amplitude modulation (QAM) and combined amplitude and phase modulation (AM-PM) are two familiar examples of this signaling format. Subject to a peak or average power constraint, the selection of the signal coordinates is done so as to minimize the probability of error. In the design of high-speed data communication systems this problem becomes one of great practical significance since the dense packing of signal points reduces the margin against Gaussian noise. Phase jitter, which tends to perturb the angular location of the transmitted signal point, further degrades the error rate. Previous investigations have considered the signal evaluation and design problem in the presence of Gaussian noise alone and within the framework of a particular structure, such as conventional amplitude and phase modulation. We present techniques to evaluate and optimize the choice of a signal constellation in the presence of both Gaussian noise and carrier phase jitter. The performance of a number of currently used or proposed signal constellations are compared. The evaluation and the optimization are based upon a perturbation analysis of the probability density of the received signal given the transmitted signal. Laplace's asymptotic formula is used for the evaluation. Discretizing the signal space reduces the optimal signal design problem under a peak power constraint to a tractable mathematical programming problem. Our results indicate that in Gaussian noise alone an improvement in signal-to-noise ratio of as much as 2 dB may be realized by using quadrature amplitude modulation instead of conventional amplitude and phase modulation. New modulation formats are proposed which perform very well in Gaussian noise and additionally are quite insensitive to moderate amounts of phase jitter.