To investigate the installation damage of geogrids during roller compaction under rockfill condition, a three-dimensional discrete element model for roller compaction of geogrid-reinforced rockfill was established. The rockfill was modeled by irregular rigid block elements, while the geogrids were modeled by bonding basic ball elements. The model parameters were then calibrated by triaxial consolidated-drained and tensile tests. The displacements of the geogrids in three perpendicular directions, and the strength of the geogrids was analyzed. Additionally, the effects of compaction parameters on the installation damage of the geogrids were studied. The results showed that deformation of the geogrids was relatively small in the roller-driving direction but significant in the roller-axis and settlement directions. The damage modes of the geogrids could be mainly classified into three types: rib fracture, rib end fracture, and node fracture. The installation damage of the geogrid was derived mainly from its uneven deformation and fracture, and after roller compaction the strength distributions at different locations of the geogrid layer showed a normal distribution. Furthermore, the installation damage of the geogrids increased with increasing excitation force and compaction passes but decreased with increasing overlying rockfill thickness, roller velocity, and excitation frequency.