End-to-End Steering Controller with CNN-based Closed-loop Feedback for Autonomous Vehicles

Many significant research achievements in the past few decades have demonstrated that convolutional neural networks (CNNs) could be capable of steering wheel control which is the basic and essential maneuver of autonomous vehicles. We propose an end-to-end steering controller with CNN-based closed-loop feedback for autonomous vehicles that improves driving performance compared to traditional CNN-based approaches. This paper demonstrates that the proposed neural network, DAVE-2SKY, is able to learn to inference steering wheel angles for the lateral control of self-driving vehicles through initial supervised pre-training and subsequent reinforced closed-loop post-training with images from a camera mounted on the vehicle. We used the PreScan simulator and Caffe deep learning framework for training under diversified circumstances in a software in the loop (SIL) simulation environment. We used DRIVE ^TM PX2 computer to implement a self-driving car for an experimental validation of the proposed end-to-end controller. The performance of the proposed system has been investigated under simulations and on-road tests as well. This work shows that the CNN-based end-to-end controller performs robust steering control even under partially observable road conditions, which indicates the possibility of fully self-driving vehicles controlled by CNN-based end-to-end steering controllers.