n-Hexane and tert-Butanol (TBA) are frequently utilized to refine the conditions for Grignard, Wittig, and Friedel-Crafts alkylation reactions. During the synthesis of loratadine, the formation of azeotropic mixtures n-hexane and TBA, which are challenging to separate through conventional distillation, is inevitable. This study utilizes the COSMO-RS model to identify suitable ionic liquids (ILs) for the separation of the n-hexane - TBA azeotropic system. Based on solvent capacity and selectivity, 1-ethyl-3-methylimidazolium trifluoroacetate ([EMIM][TFA]), 1-butyl-3-methylimidazolium trifluoroacetate ([BMIM][TFA]), and 1-hexyl-3-methylimidazolium trifluoroacetate ([HMIM][TFA]) were selected as extractants. The liquid–liquid equilibrium (LLE) data for the n-hexane - TBA - ILs ternary system were measured at 303.15 K and atmospheric pressure. Distribution coefficients and selectivity were calculated to evaluate the performance of the extractants, with the NRTL model used to correlate the experimental LLE data. The consistency of the NRTL model parameters was corroborated through topological analysis associated with the Gibbs tangent principle. Quantum chemical calculations, including interaction energy, ESP analysis, IGMH analysis, and QTAIM topological analysis, were performed to explore the separation mechanism at the molecular level. The results indicated that the interaction energies between the ILs and TBA were higher than those between the ILs and n-hexane, indicating a stronger attraction of ILs to TBA. Consequently, the ILs effectively separated TBA from the n-hexane - TBA azeotropic system, with extraction capacities ranked as [EMIM][TFA] > [BMIM][TFA] > [HMIM][TFA]. The quantum chemical calculations successfully explained the experimental results, aligning with COSMO-RS model predictions and confirming their reliability. IGMH and QTAIM topological analyses systematically explored the types and strengths of interactions, revealing that hydrogen bonds are predominant between the ILs and TBA, with additional contributions from van der Waals forces. Furthermore, the hydrogen bond strengths between TBA and the anions and cations of the ILs are classified as strong and moderate, respectively. This work provides valuable insights into the separation of azeotropic systems using ILs, elucidating the underlying mechanisms behind this process.