Tesla has a lot of footage collected by vehicles included in the Full Self-Driving program, but it's a lot of work to go through all that manually. Right now, the work of identifying and labeling cars, people, other road users, traffic signs and pretty much anything else that a car might come across on the road is being done by hand.
But Tesla is working on a tool to automate this gargantuan task, get AI to do it and it already looks like it’s doing what it is supposed to. Videos showing an early version of the auto-labeling tool working was shown by the company’s AI boss, Andrej Karpathy, on Twitter and if you understand what’s happening, it’s pretty fascinating to watch.
The software is not yet running in vehicles, but you can see it very easily identifies primarily pedestrians and other cars - even cars parked in lots quite far from the road, which are only visible to the vehicle’s side-facing cameras, are clearly marked with blue boxes.
Road signs are marked in green, traffic lights are orange and you can see the AI is able to clearly define where the sidewalk is (marked in red) and the actual road (which gets a slight purple hue). The system also seems to use teal to mark the edge of what the car can drive on and it also appears able to detect if a surface is paved or not - it shows the unpaved area around trees on the sidewalk as green, as well as other unpaved areas.
Karpathy’s description for the two videos that he posted makes it seem harder to understand what’s happening, but this tool does what its name says - it automatically labels. The lingo used in the description seems aimed at people who are familiar with neural networks and artificial intelligence, so perhaps Tesla is looking to attract new talent to help get this tool implemented and working.
More self-driving cars
Source: Andrej Karpathy / Twitter