Scenario creation

Overview

To create driving scenarios, including both ones that we record as videos, as well as being able to pick out "metric" scenes that can be used to evaluate lenses, sensors, and object detectors, we rely on Matlab's Automated Driving Toolbox (ADT) to help us with the basics.

First, we get a road. We have pre-built roads (e.g., road_020). At this time, it is best to use one of these pre-built roads.

The ADT has an app called the Driving Scenario Designer (DSD). This is an interface that allows us to place assets on a road visually. It also has panels that allow us to set the orientation and trajectory of moving objects.

To work correctly with ISETAuto you need to:

• Add a camera to the subject vehicle (that DSD calls the Ego Vehicle).
• Set the Scenario name to match the name of the Road you are using, so ISETAuto knows which base scene to use.
• Similarly, set the names of Vehicles and Actors to match those of existing assets in ISETAuto (for example, the scenario below uses "car_058" (F150), "pedestrian_001", "deer_001", etc. NOTE: This requires that you have access to the 'remoteResources' directory that contains assets for ISETAuto.
• Set the "Step" time to something reasonable for ISETAUto -- as we use it as the rate we compute scenes, which is expensive.

Dave has implemented a large set of functions that convert the DSD output into a format that we can render with PBRT. Essentially the create a simulation in ISETAuto that runs in parallel to the simulation engine running in Matlab's Driving Scenario.

After creating a Scenario in the DSD, we export it to a Matlab function (using the appropriate Menu item). This creates a Matlab file that we will very slightly modify to use our subclass of the driving scenario object. Our subclass provides the interface to ISET3d methods. Typically this means changing the call to drivingScenario (Matlab's class) to ia_drivingScenario (our sub-class). That call is typically around the area of line 79-81.

An example of a file that has been output from the DSD is initialPAEBTest.m in 'isetauto/data/scenes/road/road_020'. That file has been (slightly) modified by replacing one call that looks roughly like this:

 scenario = drivingScenario('StopTime', 5, 'SampleTime', 0.25);

To

scenario = ia_drivingScenario('StopTime', 5, 'SampleTime', 0.25);

The ia_drivingScenario is the subclass with the needed hooks.

As coded, the scenario uses Matlab's YOLOv4 object detector to look for pedestrians. That requires the Computer Vision Toolbox, and the YOLOv4 detector model auxiliary download.

Advanced Options

• dataOnly flag runs the scenario with the car at full speed and no rendering, to provide an idea of time versus distance

• debug flag floats the camera above the car, looking down, to provide a bird's eye view of the scene (but of course isn't realistic)

Side notes

We would like to build simple roads that come from OpenDrive format files. but we are not quite done with that. Zhenyi can do it manually, but we don't have a purely automated way of going through the export and conversion. Zhenyi does something with Blender.