The spectrum scarcity in communication systems has attracted many researches in the area of cognitive communication which is known as a precious solution to overcome this problem. Overlay (OL) and underlay (UL) strategies are two main techniques in the cognitive systems (CSs). On the other side, cooperative communication improves the reliability and capacity of data communications when using the multiple-relay scheme. In this paper, assuming the multiple-relay cooperative cognitive communication system, the quality of service (QoS) parameters as effective capacity (EC) and outage probability (OP) are analyzed in details. A closed-form mathematical expression is obtained for OP and EC in Rayleigh fading channel considering different constraints on the CS such as false detection of busy/idle state of primary user (PU), or interference threshold of secondary user on PU in OL and UL strategies, respectively. Moreover, the comparison of OL and UL strategies in terms of the QoS parameters is done. As will be shown in this paper, simulation and numerical results verify each other and follow the mathematical analysis of the QoS parameters.
