Optimizing 5G Massive MIMO Systems Using DNN and DSAC-T: A Scalable Adaptive Beamforming Framework
Keywords:
5G network, Beamforming;, Deep neural networks, DSAC-T, Massive multiple-input multiple-outputAbstract
Advanced Fifth Generation (5G) technologies play a crucial role in meeting the growing demand for high-capacity wireless communication, leading to the adoption of massive multiple-input multiple-output (MIMO) systems. This paper proposes an adaptive beamforming framework for 5G networks that integrates deep neural networks with the Distributional Soft Actor-Critic with Three Refinements (DSAC-T) algorithm. The framework optimizes beamforming vectors and transmission parameters in orthogonal frequency division multiplexing (OFDM)/MIMO systems while addressing challenges such as computational complexity, scalability, and system instability in dynamic channel conditions. DSAC-T improves learning stability through twin value distribution learning and critic gradient adjustments while mitigating policy overestimation using variance-based target return clipping. Simulation results demonstrate its superiority over traditional reinforcement learning methods, showing enhanced spectral efficiency, energy efficiency, latency reduction, and bit error rate performance. The framework remains robust across diverse channel conditions, making it well-suited for real-time deployment in 5G and beyond. This work provides a scalable and efficient approach to optimizing communication systems and lays a foundation for future research on reinforcement learning-based beamforming in advanced wireless networks.References
[1] G. Eappen, J. Cosmas, R. Nilavalan, and J. Thomas, Deep learning integrated reinforcement learning for adaptive beamforming in B5G networks, IET Communications, 16(20), 2022, 2454-2466.
[2] K. Suh, S. Kim, Y. Ahn, S. Kim, H. Ju and B. Shim, Deep reinforcement learning-based network slicing for beyond 5G, IEEE Access, 10, 2022, 7384-7395.
[3] M. S. Mandloi, P. Gupta, A. Parmar, P. Malviya and L. Malviya, Beamforming MIMO array antenna for 5G-millimeter-wave application, Wireless Personal Communications, 129(1), 2023, 153-172.
[4] A. Jumaah and A. Qasim, Hybrid beamforming for massive MIMO in 5G wireless networks, AIP Conference Proceedings, 3079, 2024, 060020.
[5] A. Klawonn, M. Lanser and J. Weber, A domain decomposition–based CNN-DNN architecture for model parallel training applied to image recognition problems, SIAM Journal on Scientific Computing, 46(5), 2024, C557-C582.
[6] H. Li, T. Wei, A. Ren, Q. Zhu and Y. Wang, Deep reinforcement learning: framework, applications, and embedded implementations, Proceedings of 2017 IEEE/ACM International Conference on Computer-Aided Design (ICCAD), Irvine, USA, 2017, 847-854.
[7] A. A. A. Reji and S. Muruganantham, Building detection based on faster RCNN with distributional soft actor-critic with three refinements, Applications of Modelling and Simulation, 8, 2024, 201-212.
[8] G. Wu, Y. Li, S. Luo, G. Song, Q. Wang, J. He and H. Zhu, A joint inverse reinforcement learning and deep learning model for drivers' behavioral prediction, Proceedings of the 29th ACM International Conference on Information & Knowledge Management, Ireland, 2020, 2805-2812.
[9] Y. Arjoune and S. Faruque, Experience-driven learning-based intelligent hybrid beamforming for massive MIMO mmwave communications, Physical Communication, 51, 2022, 101534.
[10] J. Duan, Y. Guan, S. E. Li, Y. Ren, Q. Sun and B. Cheng, Distributional soft actor-critic: Off-policy reinforcement learning for addressing value estimation errors, IEEE Transactions on Neural Networks and Learning Systems, 33(11), 2021, 6584-6598.
[11] Y. Yan, B. Zhang, C. Li, J. Bai and Z. Yao, A novel model-assisted decentralized multi-agent reinforcement learning for joint optimization of hybrid beamforming in massive MIMO mmWave systems, IEEE Transactions on Vehicular Technology, 72(11), 2023, 14743-14755.
[12] F. Zhang, J. Li and Z. Li, A TD3-based multi-agent deep reinforcement learning method in mixed cooperation-competition environment, Neurocomputing, 411, 2020, 206-215.
[13] Y. Ren, J. Duan, S. E. Li, Y. Guan and Q. Sun, Improving generalization of reinforcement learning with minimax distributional soft actor-critic, Proceedings of the 2020 IEEE 23rd International Conference on Intelligent Transportation Systems (ITSC), Rhodes, Greece, 2020, 1-6.
[14] J. Duan, W. Wang, L. Xiao, J. Gao and S. E. Li, DSAC-T: Distributional soft actor-critic with three refinements, arXiv preprint, arXiv:2310.05858, 2023.
[15] J. Liu, J. Chen, Z. Liu and H. Zhou, Enabling feedback-free MIMO transmission for FD-RAN: A data-driven approach, arXiv preprint, arXiv:2311.14329, 2023.
[16] M. A. Albreem, A. Kumar, M. H. Alsharif, I. Khan and B. J. Choi, Comparative analysis of data detection techniques for 5G massive MIMO systems, Sustainability, 12(21), 2020, 9281.
[17] S. A. Khwandah, J. P. Cosmas, P. I. Lazaridis, Z. D. Zaharis and I. P. Chochliouros, Massive MIMO systems for 5G communications, Wireless Personal Communications, 120(3), 2021, 2101-2115.
[18] D. V. Ratnam and K. N. Rao, Bi-LSTM based deep learning method for 5G signal detection and channel estimation, AIMS Electronics and Electrical Engineering, 5(4), 2021, 334-341.
[19] A. Kumar, S. Chakravarty, S. Suganya, H. Sharma, R. Pareek, M. Masud and S. Aljahdali, Intelligent conventional and proposed hybrid 5G detection techniques, Alexandria Engineering Journal, 61(12), 2022, 10485-10494.
[20] A. Kumar, N. Gour, H. Sharma, M. Shorfuzzaman and M. Masud, Hybrid detection techniques for 5G and B5G M-MIMO system, Alexandria Engineering Journal, 75, 2023, 429-437.
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2025 Sandhya Bolla, Manwinder Singh, B. Ramesh, B. Swapna, M. Anand, P. M. Diaz
This work is licensed under a Creative Commons Attribution 4.0 International License.
Authors who publish with this journal agree to the following terms:Authors hold and retain copyright, and grant the journal right of first publication, with the work after publication simultaneously licensed under a Creative Commons Attribution 4.0 License CC BY that permits any use, reproduction and distribution of the work and article without further permission provided that the original work is properly cited.
Authors are permitted and encouraged to post their work online in institutional repositories, website and other social media before and after publication, as it can lead to productive exchanges, as well as earlier and greater citation of published work.
