Complex modulus is one fundamental mechanical property of viscoelastic composites. In some existing methods on the prediction of complex modulus, inhomogeneities were usually assumed to be spherical and symmetric, and some complicated calculations were often performed to deal with the inverse Laplace transform. Most importantly, the effect of interaction among randomly oriented inhomogeneities on the complex modulus of composites has not been studied carefully. In this study, to address these challenges, aggregates in asphalt concrete are modeled as asymmetric ellipsoids. To account for the impact of aggregate interactions on the complex modulus, the orientation interaction model (OIM) is employed in conjunction with the elastic-viscoelastic correspondence principle. Based on OIM, the Laplace transform of the composite relaxation modulus is derived from the component characteristics. Then, according to the relationship between relaxation modulus and complex modulus for viscoelastic materials, the complex modulus of composites is obtained directly from the Laplace transform of the composite relaxation modulus, so the inverse Laplace transform is avoided. Model predictions agree well with test data, and it is found the aggregate geometry has a considerable influence on the composite property. The proposed model also captures the decrease of the composite dynamic Poisson's ratio with the increase of loading frequencies. The effects of component volume contents and the Poisson's ratio of the matrix on the composite dynamic modulus are analyzed, and model predictions are consistent with experimental observations.