In the present paper, the thermal conductivity (TC) of a hydraulic oil-based nanofluid in the presence of ternary nano-additives, graphene oxide (GO), iron oxide (Fe3O4), and titanium dioxide (TiO2), is analyzed in a wide range of volume fractions (VFs), temperatures, and mixing ratios (MRs). The stability of ternary hybrid nanofluids (THNFs) and size distribution of nanomaterial is obtained through zeta potential and dynamic light scattering (DLS) tests. Zeta potential and DLS tests indicated the remarkable stability of the samples with the GO (2): Fe3O4(1): TiO2(1) MR. Analysis of the measurements revealed that the enlargement in temperature and VFs improved the TC of THNFs for all MRs (1:1:1, 2:1:1, 1:2:1, 1:1:2). The highest TC enhancement is observed at the highest temperature (65 ◦C) and VF (1%), which for the MRs of 1:1:1, 1:1:2, 1:2:1, and 2:1:1 equal to 36.04%, 26.28%, 25.95%, and 33.86%, respectively. Furthermore, considering the average TC enhancement in the presence of nano-additives for various temperatures, MRs of GO(1): Fe3O4(1): TiO2(1) and GO(1): Fe3O4(1): TiO2(2) indicated the best and worst efficiency with 30.46% and 22.01%, respectively. The RSM method is applied to provide a simple and efficient formula-based model to describe the TC of THNFs in terms of input variables. In addition, a novel genetic algorithm-based optimization of training/structure parameters of Gaussian process regression (GPR) as a leading machine learning algorithm is developed, which provided thoroughly precise outcomes (R2test = 0.9994 and R2 train = 0.9998) for the prediction of TC of THNFs. The sensitivity analysis for the present THNFs revealed that the TC sensitivity is maximized at the highest VF and temperature.
