This study explores the synthesis of a novel titanium oxide-cetyltrimethylammonium bromide (TO@CTAB) nanocomposite for the effective removal of malachite green (MG) and methyl orange (MO) dyes. The optimization of the nanocomposite’s performance was carried out using response surface methodology (RSM). The adsorption characteristics were further evaluated through isotherm models, kinetic studies and thermodynamic analyses. The mesoporous nature of TO@CTAB was confirmed through BET analysis, revealing a pore diameter of 4.625 nm. The crystalline size of TO@CTAB is 54.78 nm, and its crystalline index is 70.84%. The optimal operating conditions were established based on the results obtained from the ANOVA. The Langmuir isotherm model demonstrates superior adsorption performance compared to the Freundlich isotherm model, with adsorption efficiencies of 317.46 mg/g for MO and 306.748 mg/g for MG. The pseudo-second-order model, with an R2 value of 0.998 and 0.997 for MO and MG, respectively, provides a more accurate and reliable explanation of the adsorption process compared to the pseudo-first-order model. Furthermore, the high reusability and minimal deterioration of TO@CTAB were observed for up to 5 cycles. The analysis of the adsorption mechanism indicates that the adsorption of MG and MO occurs through H-bonding, electrostatic and π-π interactions. A comprehensive cost analysis of the process was conducted to evaluate the cost-effectiveness; total expenditure incurred during the process was determined to be within acceptable limits. TO@CTAB was assessed using real wastewater samples, demonstrating a decolourization efficiency of 82%. Additionally, it resulted in a reduction of COD, BOD, TSS and TDS.