The radio spectrum of communication systems is largely occupied by wideband technologies, and thus, efficient usage of spectrum is a crucial requirement from regulators' and service providers' perspectives. Cognitive communication has been proposed as a potential solution to resolve the spectrum scarcity problem. Cognitive radio schemes can be jointly applied with cooperative communications to improve the performance under both overlay and underlay strategies as the two main cognitive strategies supported by multiple relays. This combination significantly improves the reliability and capacity of data communication. In this paper, two QoS parameters, namely, the effective capacity and outage probability, have been thoroughly analyzed in a Nakagami‐m fading channel that is the typical channel condition often considered in the transmission of communication systems. Closed‐form expressions are obtained for effective capacity and outage probability in both underlay and overlay strategies. Simulation results verify the effectiveness of the mathematical modelling of the proposed QoS parameters.
