Behavioral Cloning Project

Overview

This project uses Simulator developed by Udacity to collect data and test models.

When thinking about autonomous driving problems, there would be several very difficult problems that will blow your mind. Take, for instance, the lane keeping problem, developers usually need to break it down into path planning, control and lane detections. However, based on a paper written by NVIDIA, deep learning suprisingly performs well in solving this problem without explicitly discomposing the problem into several sub-problems. We call this End-to-End Deep Learning, or Behavioral Cloning.

This project simulates the End-to-End Learning described in NVIDIA's paper. Before the learning process, we need to collect data by driving around an environment and record images captured and our driving behaviors. By feeding our model the driver's behaviors, the model will learn and clone the desired behavior when running in a similar environment. After training the model we built, the outcome should be a mapping from Camera Images to Desired Steering Commands.

I break down this project into 4 parts:

1. Data Collection: In this part, we will manually drive our car in the simulator and record all images taken by 3 cameras on the car and the corresponding steering commands we made. The more data we collect, the better our model will perform.

2. Pre-processing: Process the images we collected by doing Gaussian Blurring and Coverting it to RGB Color Format

3. Training: In this part, we design our Neural Network Model and train it to minimize the mean square error between model predicted steering command and our recorded command for each image. Refer to "model_nvidia.py" for the CNN model developed by NVIDIA.

4. Testing: Run the trained model ("model.h5") in the simulator. The car is set to run in constant speed. Check how your model performs. This time, the input should be images captured by a single camera and output the steering commands.

Of course we cannot use this single model to solve the autonomous driving problems in the real world. This is just a simple way to let a machine learn how to drive. But from this project we can see how deep neural networks are able to learn such a complicated problem without extracting features explicitly.

Project Manual

This repository contains starting files for the Behavioral Cloning Project.

In this project, you will use what you've learned about deep neural networks and convolutional neural networks to clone driving behavior. You will train, validate and test a model using Keras. The model will output a steering angle to an autonomous vehicle.

We have provided a simulator where you can steer a car around a track for data collection. You'll use image data and steering angles to train a neural network and then use this model to drive the car autonomously around the track.

We also want you to create a detailed writeup of the project. Check out the writeup template for this project and use it as a starting point for creating your own writeup. The writeup can be either a markdown file or a pdf document.

To meet specifications, the project will require five files:

• model.py (script used to create and train the model)
• drive.py (script to drive the car - feel free to modify this file)
• model.h5 (a trained Keras model)
• a report writeup file (either markdown or pdf)
• video.mp4 (a video recording of your vehicle driving autonomously around the track for at least one full lap)

This README file describes how to output the video in the "Details About Files In This Directory" section.

Instruction

The goals / steps of this project are the following:

• Use the simulator to collect data of good driving behavior
• Design, train and validate a model that predicts a steering angle from image data
• Use the model to drive the vehicle autonomously around the first track in the simulator. The vehicle should remain on the road for an entire loop around the track.
• Summarize the results with a written report

Dependencies

Required dependencies are listed in './requirements.txt' file including:

• tensorflow 2.2
• scikit-learn

etc.

It is recommended (but optional) to use conda to create an exclusive environment for each python project. For example,

conda create -n sdc python=3.6

You can use the following command to get your environment ready,

pip install -r requrements.txt

The following resources can be found in this repository:

• drive.py
• video.py
• writeup_template.md

The simulator can be downloaded from the following links based on your system,

Version 2, 2/07/17

Linux Mac Windows

Version 1, 12/09/16

Linux Mac Windows 32 Windows 64

Instructions: Download the zip file, extract it and run the executable file.

Details About Files In This Directory

drive.py

Usage of drive.py requires you have saved the trained model as an h5 file, i.e. model.h5. See the Keras documentation for how to create this file using the following command:

model.save(filepath)

Once the model has been saved, it can be used with drive.py using this command:

python drive.py model.h5

The above command will load the trained model and use the model to make predictions on individual images in real-time and send the predicted angle back to the server via a websocket connection.

Note: There is known local system's setting issue with replacing "," with "." when using drive.py. When this happens it can make predicted steering values clipped to max/min values. If this occurs, a known fix for this is to add "export LANG=en_US.utf8" to the bashrc file.

Saving a video of the autonomous agent

python drive.py model.h5 run1

The fourth argument, run1, is the directory in which to save the images seen by the agent. If the directory already exists, it'll be overwritten.

ls run1

[2017-01-09 16:10:23 EST]  12KiB 2017_01_09_21_10_23_424.jpg
[2017-01-09 16:10:23 EST]  12KiB 2017_01_09_21_10_23_451.jpg
[2017-01-09 16:10:23 EST]  12KiB 2017_01_09_21_10_23_477.jpg
[2017-01-09 16:10:23 EST]  12KiB 2017_01_09_21_10_23_528.jpg
[2017-01-09 16:10:23 EST]  12KiB 2017_01_09_21_10_23_573.jpg
[2017-01-09 16:10:23 EST]  12KiB 2017_01_09_21_10_23_618.jpg
[2017-01-09 16:10:23 EST]  12KiB 2017_01_09_21_10_23_697.jpg
[2017-01-09 16:10:23 EST]  12KiB 2017_01_09_21_10_23_723.jpg
[2017-01-09 16:10:23 EST]  12KiB 2017_01_09_21_10_23_749.jpg
[2017-01-09 16:10:23 EST]  12KiB 2017_01_09_21_10_23_817.jpg
...

The image file name is a timestamp of when the image was seen. This information is used by video.py to create a chronological video of the agent driving.

video.py

python video.py run1

Creates a video based on images found in the run1 directory. The name of the video will be the name of the directory followed by '.mp4', so, in this case the video will be run1.mp4.

Optionally, one can specify the FPS (frames per second) of the video:

python video.py run1 --fps 48

Will run the video at 48 FPS. The default FPS is 60.

Why create a video

1. It's been noted the simulator might perform differently based on the hardware. So if your model drives succesfully on your machine it might not on another machine (your reviewer). Saving a video is a solid backup in case this happens.
2. You could slightly alter the code in drive.py and/or video.py to create a video of what your model sees after the image is processed (may be helpful for debugging).

Tips

• Please keep in mind that training images are loaded in BGR colorspace using cv2 while drive.py load images in RGB to predict the steering angles.