Tele-operation

What exactly is tele-operation?

Tele-operation - or also called Remote Control - means the steering of an unmanned vehicle or a machine by a human operator that is not in the vehicle. Typically, the remote operator or driver is in a control room that could be 1000s km's away from the vehicle. Tele-operation has been around for a while: from toys to big machines. But more recently advanced data reduction techniques this feature is finding its place in as a stem of autonomous driving. New AI-models enable vehicles to operate autonomously need a lot of training. For that training, many hours of safe operations are required to capture sufficient sensor data (video, radar, lidar, IMU). During this process the vehicle is not yet autonomous and needs a human guide. Initially for 100% of the time. Later, and as autonomous AI-capabilities grow, the human driver is required for part-time operation of the vehicle. For instance, to manoeuvre the vehicle along complex or new situations where the AI does not yet know what to do. In that case a human operator takes over control and steers the vehicle until a ‘low complex situation’ where AI can take over from the human again.

This brings first efficiency to a business: namely that 1 operator can steer more vehicles. So, evolving from a 1:1 driver-to-vehicle ratio to a 1:N ratio.

The second efficiency is about location-independency of tele-operators. This works as follows. Drivers need to be in a physical location to steer a car, forklift, machine, etc. They can't be in more locations at the same time. Therefore, business must plan and hire for sufficient drivers to cover the peak demand per physical location. Leading to ‘buffers’ or idle time of staff. When these drivers become remote operators, they can switch from one physical location to another physical location in seconds and hence can be deployed more efficiently as there is the ability to match within seconds the demand and supply over several physical locations.

The above effects help companies that struggle to find sufficient, well-trained drivers. It is often a challenge in many markets (e.g. in countries with an aging population or with a shortage of certified drivers) to find enough staff. In addition, it opens job opportunities for disabled people that previously were not accessible to them.

Teleoperation can be broken down into a few implementation options and categories, including how the vehicle is controlled, the extent of control, the level of control, the use case, and more.

‍

The purpose of tele-operation

For autonomous driving, teleoperation serves the following purposes.

Remote driving: Driving the vehicle with a remote operator at the early stages of autonomy is the bread and butter for the AI-model training schemes. The use of tele-operation has penetrated through every company that aims for autonomous driving like forklifts, delivery robots, trucks, cars etc.

Vehicle fallback mechanism: Even the best trained AI-models will fall short in its decision-making when faced with anomalies and edge cases. Covering every possible real-world scenario is practically impossible. The ideal solution here is to assist the vehicle through tele-operation, this indeed serves as a safety requirement demanded by law or self-elected.

‍

The driver seat

‍

While it's easy to think that the remote operator is just like driving a vehicle in a video game. Just with real video images of surrounding of the vehicle instead of computer animations. However, the off-line reality is more challenging.

When a vehicle gives full control to the operator holding only the safety based anti-crash mechanisms at the most; is termed as direct control. The tele-operator has full control of the vehicle and performs all aspects of driving including steering, acceleration and braking.

The growth to next level from direct control is with levels of commands (which are more relevant to the AV fallback use case). The teleoperators gives direct orders to the vehicle that regulate its manoeuvre without the tele-operator governing every single step.

Low-level commands: These are direct and precise orders to the vehicle given by the tele-operators like steer 50 degrees left etc. for it to perform as accurately as possible.

High level commands: These commands are more of directions such as changing lanes, continue driving straight or slowing down etc. In this case, the vehicle decides for itself how it will perform the actual operations that facilitate the commands.

Providing the vehicle with additional information is also considered a type of command, that is often included in the vehicle decision making plan and execution. The AI also would pro-actively ask the human tele-operator how it should react to specific situations that it does not know how to deal with — for example, whether it’s okay to overtake a stalled moving truck that is blocking a lane.

‍

Key technologies for tele-operation

A stable and “delay-less” connectivity between the vehicle and the operator is most critical to tele-operation. Although cellular bandwidth availability has increased over the last decades it's still a challenge to stream multiple 4k videos and sensor data at a safe level of transmission delay (called latency) over current 4G networks. 5G networks are being rolled out but will not have national coverage any time soon. Uninterrupted, low latency connectivity is mandatory for safe tele-operations to ensure safe and uninterrupted operations. Secondly, the perceived video quality is essential for teleoperators as main actors. When the stability of connectivity is challenged by e.g. low-bandwidth conditions, teleoperators would have to wait for the video stream or rely on choppy images with block artefacts. That makes the steering of the vehicle unsafe. Even more problematic are disconnections because of the network's inability to stream the (buffered) video in real-time. This leads to unsafe situations and disruptions of work.

For this there is a solution. Teraki’s “ROI” video pre-processing is a technique that reduces latency by factor 5x by reducing data without the loss of visual quality. Teraki does this through AI-powered, state-of-the-art selection of relevant areas of the video.

This intelligent and configurable technology combines best of both world: a 4x – 5x lowering of data throughput whilst delivering the identical visual quality for the human (measured in e.g. VMAF). Below a graph where one can see the data reduction performance when delivering a high visual quality for the driver – in this case: 99.8% VMAF.

‍

A second technology used to circumvent the threats of transmission delays and disconnections is so-called “network bonding”. With this hardware-based technology the vehicle transmits the video to the wireless network(s) (cellular and/or wifi) that has – at any given time – the best available bandwidth. By doing this the data load is dynamically and intelligently spread over the available ‘wireless pipes’. As the bandwidth availability of networks changes over time and over locations, this must be a continuous seeking of available bandwidth and allocating transmission accordingly in a dynamic fashion.

‍

The hardware and software selection for the vehicle determines the functional success for tele-operation. Power consumption and hardware costs play an important role here. Teraki's ROI software elevates low-powered hardware to do highly performant processing at the edge.

‍

Reaching autonomy

During the training phase of the AI-models, human oversight of the vehicle is required. As self-driving AI-models get better, the tele-operator becomes able to steer multiple vehicles, as human intervention is only needed for the ‘edge cases’, the difficult or uncommon situations. It may take quite some tim for AI to become fully autonomous. In the years in between, we will witness a growth of remote tele-operators steering cars, robots, forklifts, mining machines, etc. However, they will be steering not one but multiple vehicles as during the “easy parts” (straight roads, moving from A to B in a warehouse, etc.) will be done independent by the vehicle. It's about taking over in these more complex events that humans will still be required. In the meanwhile, AI learns from the tele-operator driver how to tackle such more complex situation.

‍

Conclusion

Tele-operation is a field that is expected to rapidly grow rather sooner than later. In time, there will be many vehicles and robots on their way to autonomy. In the meanwhile, they will be operated remotely. Teraki is the best equipped partner to achieve this road to full autonomy, starting with state-of-the-art tele-operation. The company offers a clear and manageable roadmap from remote control to full autonomy.

‍

Get in touch

Feel free to contact us at info@teraki.com for a conversation what we could mean for improving the efficiency, flexibility and staffing challenges of your operations.