Establishing a unified constitutive model to simulate the hot deformation behaviors, microstructure evolution and fracture behaviors under different stress states during the hot forming of titanium alloy is indispensable. The high temperature tensile tests were first carried out on different stress states of forged Ti-6Al-4 V alloy specimens to analyze the flow behaviors, microstructure evolution and fracture mechanism. The results show that the effect of temperature on fracture elongation is more significant than strain rate. High temperature and low strain rate will increase the dynamic recrystallization (DRX) volume fraction and softening effect, which inhibits the nucleation and growth of voids, thereby enhancing the plastic deformation ability of the alloy. The DRX volume fraction, grain size and stress triaxiality were introduced into the unified Gurson-Tvergaard-Needleman (GTN) damage model using the internal state variables. The parameters of GTN model were modified by the Response Surface Method (RSM) and compared with the high temperature tension. Finally, the established GTN damage model was successfully applied to finite element (FE) simulation under different stress states. The correlation coefficient R of predicted stress is 0.989, and the maximum errors of DRX volume fraction and grain size are 9.86% and 6.54%. The research results can provide a basis for the performance control in hot working of titanium alloy.