A numerical investigation is performed to evaluate the effect of nanoparticle volume concentration and Reynolds number on the heat transfer and entropy generation characteristics of a hybrid nanofluid containing tetramethylammonium hydroxide (TMAH) coated Fe3O4 (magnetite) nanoparticles and gum arabic (GA) coated carbon nanotubes (CNTs) flowing inside a counter-flow double-pipe heat exchanger. Variable thermophysical properties are employed such that thermal conductivity is considered dependent on temperature and concentration, while viscosity is dependent on temperature, concentration and shear rate. The results demonstrate that the overall heat transfer coefficient and the entropy generation rate augment with the increase of Reynolds number, CNT concentration and magnetite concentration. In addition to the assessment and analysis of the outcomes, models of the overall heat transfer coefficient and global total entropy generation are developed in terms of the Fe3O4 and CNT concentrations and Reynolds number using neural network. Then, genetic algorithm is utilized in combination with compromise programming in order to obtain the optimal cases with maximum heat transfer and minimum total entropy generation. To achieve the minimum total entropy generation along with the maximum heat transfer, applying the nanofluids with great nanoparticle concentrations alongside low Reynolds numbers is suggested