Optimization Model of Ramp Input Rate, Variable Speed Limit and Highway Tolls based on Reinforcement Learning
Document Type : Original Article
Authors
1 Professor, School of Civil Engineering, Iran University of Science and Technology, Tehran, Iran.
2 M.Sc., Student, School of Civil Engineering, Iran University of Science and Technology, Tehran, Iran.
Abstract
jams and, as a result, long queues, increased travel time, increased chances of accidents, and environmental pollution. Several methods, such as ramp metering control, variable speed limit, lane change control, tolls, etc., have been studied and investigated to control highways. One of the approaches to improve the performance of these controls is their simultaneous and coordinated integration and implementation. In this paper, we will investigate the effect of the combination of three concurrent controls of ramp metering, variable speed limit, and highway tolls on the traffic conditions of the highway network. For this purpose, the reinforcement learning method has been used to manage the control values as variables. Traffic data is first predicted by predictive models based on neural networks and then given to the simulation of Sumo software. The controller model then selects the ramp rate, speed limit, and toll values based on the simulation outputs and applies them to the network. The results of implementing the proposed model show the significant impact of the model in improving traffic parameters . The model reduced the average travel time by 11.5% compared to the condition without controls, which is more than the effect of implementing single controls or a double combination of the mentioned methods. In addition, in the event of an accident in the network, the proposed management can significantly improve the network conditions compared to the situation without control.
Keywords
-Arango, M., )2019(, “Toll Road With Dynamic Congestion Pricing Using Reinforcement Learning”.
-Belletti, F., Haziza, D., Gomes, G. And Bayen, A.M., (2017), “Expert Level Control of Ramp Metering Based on Multi-Task Deep Reinforcement Learning”, IEEE Transactions on Intelligent Transportation Systems, 19(4), Pp.1198-1207.
-Carlson, R.C., Manolis, D., Papamichail, I. And Papageorgiou, M., (2012), “Integrated Ramp Metering and Mainstream Traffic Flow Control on Freeways Using Variable Speed Limits”, Procedia-Social And Behavioral Sciences, 48, Pp.1578-1588.
-Carlson, R.C., Papamichail, I. And Papageorgiou, M., (2014), “Integrated Feedback Ramp Metering and Mainstream Traffic Flow Control on Motorways Using Variable Speed Limits”, Transportation Research Part C: Emerging Technologies, 46, Pp.209-221.
-Carlson, R.C., Papamichail, I., Papageorgiou, M. And Messmer, A., (2010), “Optimal Motorway Traffic Flow Control Involving Variable Speed Limits and Ramp Metering”, Transportation Science, 44(2), Pp.238-253.
-Cheng, Q., Liu, Z., Liu, F. And Jia, R., (2017), “Urban Dynamic Congestion Pricing: An Overview and Emerging Research Needs”, International Journal of Urban Sciences, 21(Sup1), Pp.3-18.
-Chiou, Y.C., Huang, Y.F. And Lin, P.C., (2012.), “Optimal Variable Speed-Limit Control under Abnormal Traffic Conditions”, Journal of the Chinese Institute of Engineers, 35(3), Pp.299-308.
-Davarynejad, M., Hegyi, A., Vrancken, J. And Van Den Berg, J., (2011), “Motorway Ramp-Metering Control with Queuing Consideration Using Q-Learning, in 2011 14th International IEEE Conference on Intelligent Transportation Systems (ITSC), IEEE, pp. 1652-1658.
-Fares, A. and Gomaa, W., (2014), June, “Freeway Ramp-Metering Control Based on Reinforcement Learning”, in 11th IEEE International Conference on Control & Automation (ICCA), IEEE., pp. 1226-1231.
-François-Lavet, V., Henderson, P., Islam, R., Bellemare, M.G. and Pineau, J., (2018), “An Introduction to Deep Reinforcement Learning”, Arxiv Preprint Arxiv:1811.12560.
-Fu, R., Zhang, Z. And Li, L., (2016), “Using LSTM and GRU Neural Network Methods for Traffic Flow Prediction”, in 2016 31st Youth Academic Annual Conference of Chinese Association of Automation (YAC), IEEE, pp. 324-328.
-Ghods, A.H., Kian, A.R. And Tabibi, M., (2009), “Adaptive Freeway Ramp Metering and Variable Speed Limit Control: A Genetic-Fuzzy Approach”, IEEE, Intelligent Transportation Systems Magazine, 1(1), pp.27-36.
-Hegyi, A., De Schutter, B. And Hellendoorn, H., (2005), “Model Predictive Control for Optimal Coordination of Ramp Metering and Variable Speed Limits”, Transportation Research Part C: Emerging Technologies, 13(3), pp.185-209.
-Khazraeian, S., Koohifar, F. And Kalantari, N., (2017), “A Nonlinear Optimal Static Controller for Ramp Control (NOSCO)”, in Proceedings of the 96th Annual Meeting of the Transportation Research Board.
-Khondaker, B. And Kattan, L., (2015), “Variable Speed Limit: An Overview”, Transportation Letters, 7(5), pp.264-278.
-Kušić, K., Dusparic, I., Guériau, M., Gregurić, M. And Ivanjko, E., (2020), “Extended Variable Speed Limit Control Using Multi-Agent Reinforcement Learning, in 2020 IEEE 23rd International Conference on Intelligent Transportation Systems (ITSC), IEEE, pp. 1-8.
-Li, Z., Xu, C., Guo, Y., Liu, P. And Pu, Z., (2020), “Reinforcement Learning-Based Variable Speed Limits Control to Reduce Crash Risks Near Traffic Oscillations on Freeways”, IEEE Intelligent Transportation Systems Magazine, 13(4), pp.64-70.
-Lombardi, C., Picado-Santos, L. and Annaswamy, A.M., (2021), “Model-Based Dynamic Toll Pricing: An Overview”, Applied Sciences, 11(11), p.4778.
-Lu, X.Y., Varaiya, P., Horowitz, R., Su, D. and Shladover, S.E., (2011), “Novel Freeway Traffic Control with Variable Speed Limit and Coordinated Ramp Metering”, Transportation Research Record, 2229(1), pp.55-65.
-Ma, M.H., Yang, Q.F., Liang, S.D. And Li, Z.L., (2015), “Integrated Variable Speed Limits Control and Ramp Metering for Bottleneck Regions on Freeway”, Mathematical Problems in Engineering.
-Mahajan, N., Hegyi, A., Van De Weg, G.S. And Hoogendoorn, S.P., (2015), “Integrated Variable Speed Limit and Ramp Metering Control Against Jam Waves--A COSCAL V2 Based Approach”, in 2015 IEEE 18th International Conference on Intelligent Transportation Systems, IEEE, pp. 1156-1162.
-Mizuta, A., Roberts, K., Jacobsen, L., Thompson, N. And Colyar, J., (2014), “Ramp Metering: A Proven, Cost-Effective Operational Strategy: A Primer (No. FHWA-HOP-14-020)”, United States, Federal Highway Administration.
-Papageorgiou, M. , Hadj-Salem, H. And Blosseville, J.M., 1991,” ALINEA: A Local Feedback Control Law for On-Ramp Metering”, Transportation Research Record, No. 1320, Transportation Research Board, Washington, D.C., pp. 58 – 64
-Papageorgiou, M., Blosseville, J. M., And Hadj Salem, H., (1990), “Modeling and Real-Time Control of Traffic Flow on The Southern Part of Boulevard Peripherique in Paris: Part II: Coordinated On-Ramp Metering”, Transportation Research. Vol. 24A, No. 5, pp. 361-370.
-Papamichail, I., Kampitaki, K., Papageorgiou, M. And Messmer, A., (2008), “Integrated Ramp Metering and Variable Speed Limit Control of Motorway Traffic Flow”, IFAC Proceedings Volumes, 41(2), pp.14084-14089.
-Papamichail, I., M.Papageorgiou, V. Vong, & J. Gaffney, (2010), “Heuristic Ramp-Metering Coordination Strategy Implemented at Monash Freeway, Australia, Research Record”, Journal of the Transportation Research Board,
No. 2178, 2010, pp. 10–20.
No. 2178, 2010, pp. 10–20.
-Recht, B., (2019), “A Tour of Reinforcement Learning: The View from Continuous Control”, Annual Review of Control, Robotics, and Autonomous Systems, 2, pp.253-279.
-Rezaee, K., Abdulhai, B. And Abdelgawad, H., (2012), September, “Application of Reinforcement Learning with Continuous State Space to Ramp Metering in Real-World Conditions”, in 2012 15th International IEEE Conference on Intelligent Transportation Systems, IEEE, pp. 1590-1595.
-Schmidt-Dumont, T. And Van Vuuren, J.H., (2015), “Decentralised Reinforcement Learning for Ramp Metering and Variable Speed Limits on Highways”, IEEE Transactions on Intelligent Transportation Systems, 14(8), p.1.
-Schmidt-Dumont, T. And Van Vuuren, J.H., (2019), “A Case for the Adoption of Decentralised Reinforcement Learning for The Control of Traffic Flow on South African Highways”, Journal of the South African Institution of Civil Engineering, 61(3),
pp.7-19.
pp.7-19.
-Schulman, J., Wolski, F., Dhariwal, P., Radford, A. And Klimov, O., (2017), “Proximal Policy Optimization Algorithms”, Arxiv Preprint Arxiv:1707.06347.
-Sewak, M., (2019), “Deep Reinforcement Learning”, Singapore: Springer Singapore.
-Shaaban, K., Khan, M.A. And Hamila, R., (2016), “Literature Review of Advancements in Adaptive Ramp Metering”, Procedia Computer Science, 83, pp.203-211.
-Siami-Namini, S., Tavakoli, N. And Namin, A.S., (2018), “A Comparison of ARIMA And LSTM In Forecasting Time Series”, in 2018 17th IEEE International Conference on Machine Learning and Applications (ICMLA), IEEE, pp. 1394-1401.
-Vinitsky, E., Parvate, K., Kreidieh, A., Wu, C. and Bayen, A., (2018), “Lagrangian Control through Deep-Rl: Applications to Bottleneck Decongestion”, in 2018 21st International Conference on Intelligent Transportation Systems (ITSC), IEEE, pp. 759-765.
-Yamak, P.T., Yujian, L. and Gadosey, P.K., (2019), “A Comparison Between Arima, Lstm, And Gru for Time Series Forecasting”, In Proceedings of the 2019 2nd International Conference on Algorithms, Computing and Artificial Intelligence pp. 49-55.
-Zhang, Y., Atasoy, B., Akkinepally, A. and Ben-Akiva, M., (2019), “Dynamic Toll Pricing Using Dynamic Traffic Assignment System with Online Calibration”, Transportation Research Record, 2673(10), pp.532-546.