A synergistic solvent extraction system comprising trioctylamine (TOA) and ligands with hydroxyl and carboxyl groups can efficiently recover boric acid (H3BO3) and separate boron isotopes. However, the structure of ligands might impact H3BO3 extraction, boron isotope separation, and solvent loss, which has not been thoroughly investigated. This study initially evaluated the influence of ligand's type, pKa, and substituents on H3BO3 extraction efficiency, as well as the impact of the B(4)-O structure (boron is bound to four oxygen atoms) in the organic phase on isotope separation efficiency. Subsequently, by synthesizing the highly hydrophobic 2-hydroxydodecanoic acid (HYA), the extraction performance and mechanism of the TOA/HYA system were investigated. The findings highlight the superior extraction efficiency when employing di-phenolic hydroxyl, phenolic hydroxyl + carbinol hydroxyl, and alcoholic hydroxyl + carboxyl ligands compared to phenolic hydroxyl + carboxyl, phenolic hydroxyl + ethanol hydroxyl, diol hydroxyl, and dicarboxylic ligands. The organic phase anion complex, exclusively comprising the B(4)-O structure, enhances isotope separation effectiveness. The TOA/HYA system achieves an 80% single-stage extraction efficiency for H3BO3. H3BO3 and HYA are extracted into the organic phase at a ratio of 1:2, with the anion complex solely containing the B(4)-O structure. This study paves the way for the construction of novel boric acid extraction and boron isotope separation systems.