An efficient solution to prevent multiple access interference and reduce transmitter power in cellular communication systems is cooperative communications. In multiuser cooperative communications, the users transmit their signals to the receivers through multiple relay nodes. Simultaneous optimum detection of multiuser signals considering the constraints of transmitter power and SINR satisfaction is the main challenge of these systems. As known, the optimum detector in multiuser communications is maximum likelihood (ML) detector but it is not applicable because of computational complexity. In this paper, to detect the desired transmitted signal in receivers, the problem of optimum multiuser detection is optimized using Lagrange multipliers and due to complexity of the resulted output, modified dimension-reduction genetic algorithm is applied based on dividing the problem with large number of parameters (users) into several simpler problems with less number of parameters (users). The main idea is the selection of more critical parameters in first-round optimization. The optimization is done subjected to two constraints as SINR and minimum transmitter power to guarantee the quality of service (QoS) in the cooperative communication system. Simulation results show that our proposed algorithm’s performance is near the optimum ML detector and it is more efficient in comparison to other well-known sub-optimum algorithms in the literature with satisfying the QoS of the system.