Extremely Large Network Outputs

[lukazikus]

After training the model with the 24GB training dataset, I loaded it into a client on CARLA, just to realize that it outputs extremely large values (i.e. in the thousands) as opposed to values limited by 1. Is there a normalization step that needs to be applied?

[markus-hinsche]

Hi Chris,

It happened to me too sometimes that the values for the outputs didn't match the distribution of the data.
Some things that might work:

• try different hyper-parameters (e.g. lambda_steer, lambda_throttle etc.). In your case, only speed might be learned correctly, but throttle, brake, and steer might be barely taken into account.
• restart the training (due to a different initialization it might not converge correctly)

I also described these points here.
https://medium.com/merantix/journey-from-academic-paper-to-industry-usage-cf57fe598f31

We also had to adjust the controller (https://github.com/carla-simulator/imitation-learning/blob/master/agents/imitation/imitation_learning.py) a bit, due to the throttle unit, but since you get also too big values for steer and brake, this is less likely.

[lukazikus]

Hi Markus,

Thanks for the response. I realized that after 90000 iterations, the model started to have these values explode. Before, the car did follow the path mostly well.

I was wondering if you know how we would be able to augment the existing training set's h5 files? We would like to go around the map and collect new training data to add to the learning process.

Thanks,
Chris

[markus-hinsche]

I realized that after 90000 iterations, the model started to have these values explode

I experienced the same behavior (this is very obvious to see in Tensorboard). Every time I train and if this happens, it happens at a different amount of steps. I didn't debug this so far, it would be good to find out why.

how we would be able to augment the existing training set's h5 files?

This is done on the fly just before training already: see https://github.com/merantix/imitation-learning/blob/master/imitation/input_fn.py#L196

We would like to go around the map and collect new training data to add to the learning process.

To collect data, refer to https://github.com/carla-simulator/imitation-learning/ and the associated paper

[lukazikus]

Hi Markus,

Sorry if I was unclear. I meant more in terms of how to record images taken from the CARLA simulator and convert them to the h5 files in the same format as the ones in Codevilla's training dataset. I don't think he shared the code for this, but I was wondering if there are any samples I can look at since I am relatively new to using the CARLA/H5 API.

[markus-hinsche]

We didn't add additional training images/sequences so I would have to research to find out myself