Helically coiled tubes are employed as heat transfer tubes in Once Through Steam Generator (OTSG) of High Temperature Gas-cooled Reactor (HTGR) due to their compact structure, large heat transfer area and excellent thermal expansion adaptability. However, the helical geometry induces centrifugal forces and secondary flows in the tube, resulting in notable differences in flow and heat transfer characteristics compared to that of straight tubes. This study conducted an experimental investigation on the onset of nucleate boiling (ONB) and the subcooled boiling heat transfer coefficient in helically coiled tubes with a large curvature ratio (δ = 0.109). The experimental parameters cover broad ranges. The system pressures are ranging from 3.5 to 7 MPa, mass fluxes are from 300 to 1100 kg/(m2·s) and heat fluxes are from 50 to 600 kW/m2. The experimental results indicate that the ONB can occur even when the average inner wall temperature is below the fluid’s saturation temperature. An increase in heat flux advances the ONB, while increases in mass flux and system pressure delay it. Enhancements in both heat flux and mass flux improve the subcooled boiling heat transfer coefficient. Additionally, higher system pressure also increases the heat transfer coefficient, although this effect diminishes as the quality increases. Based on the experimental data and dimensionless analysis, new correlations were proposed for predicting ONB and calculating the subcooled boiling heat transfer coefficient in helically coiled tubes. Both new correlations exhibit more accurate predictive capabilities, with mean absolute percentage error (MAPE) values of 6.20 % and 8.86 %, respectively.