"Car-man unity, man-machine driving together" -- Exploration and breakthrough of intelligent cockpit

1

Smart Cockpit Overview

Since the advent of the car, the car has experienced two eras: the mechanical age and the electrical age. The cars of these two eras are vehicles with driving as the main task, and the internal space area of the car is mainly divided into fixed driver's seat and fixed passenger seat. Due to the rise of cutting-edge technologies such as autonomous driving, artificial intelligence, and 5G in recent years, the automotive industry has gradually integrated functions such as assisted autonomous driving and intelligent assistants in automobiles, making driving easier and faster. However, people's understanding of the value of automobiles has also changed from travel tools to "third space", and the cockpit is the core carrier to realize space shaping. According to IHS Market's prediction, the market size of the global automobile intelligent cockpit will reach 68.1 billion US dollars by 2030, when the domestic cockpit market size will also exceed 160 billion yuan, and China's share of the global market will rise from 23% to about 37%, becoming the world's most important intelligent cockpit market.

 

Source: IHS Market

So what exactly is a smart cockpit? The industry has not yet formed a clear and unified definition.

IHS Market believes that "intelligent cockpit refers to a silicon-based life complex that integrates intelligent and networked technologies, software and hardware, and can realize intelligent perception and intelligent decision-making of cockpit space through continuous learning and iteration. Different from mechanized cockpit and electronic cockpit, intelligent cockpit can perceive the cockpit environment more efficiently and intelligently, and it is endowed with more personality characteristics and concrete existence-intelligent technology and software shape the foundation of the soul, while hardware equipment constructs the intelligent functional sensory system".

According to a market analysis of Minsheng Securities, "the intelligent cockpit is developed through the integration and integration of hardware, human-computer interaction system and software, with driving information display system and infotainment system as the core carriers, and under the support of the underlying software architecture, complete the functions related to human-computer interaction, and finally realize the intelligent cockpit".

Roland Berger's intelligent cockpit white paper believes that "mainly covers the innovation and linkage in the cockpit interior and electronic fields, and is a human-computer interaction (HMI) system built from the perspective of consumer application scenarios to embrace the emerging technology trends in the automotive industry"

According to the journal Software and Integrated Circuits, "intelligent cockpit should be able to cover both in-vehicle infotainment and driving information, provide users with intuitive and efficient human-computer interaction, and support the transmission and processing of massive data to achieve a complete automated control experience".

It can be seen that the intelligent cockpit is a complex of multiple interrelated functional systems such as driving system, entertainment information system, safety protection system, seat system, audio system, air conditioning system, in-car monitoring system, etc., generally including cockpit domain controller, (AR)HUD, central control touch screen, central instrument panel, streaming media rearview mirror, various sensors, voice controller and other main intelligent components. Among them,) human-computer interaction (HMI,Human Machine Interaction) is the most essential feature of intelligent cockpit. It is no exaggeration to say that the core requirement of intelligent cockpit is more humanized human-computer interaction, which is a core goal throughout the design of various subsystems in cockpit. Every innovation of human-computer interaction technology has become the key to the improvement of the intelligent level of automobiles, directly affecting (driving) the development direction of the intelligent cockpit and even the entire intelligent automobile industry.

The following section will analyze in detail the development trend of human-computer interaction system for intelligent cockpit, and systematically analyze a number of key technologies, and finally discuss in detail the basis and cut-in recommendations of the Zhijia Center in this field.

 

2

Development trend of cockpit human-computer interaction system

Throughout the development of car cockpit interaction technology, we can divide the stage and predict the trend from three different angles: 1) according to the interaction medium, 2) according to the direction of interaction initiation, and 3) according to the interactive presentation mode.

2.1 from touch interaction to voice interaction to multi-modal interaction

According to the interactive media, the interactive system design of the intelligent cockpit has experienced the traditional touch interactive era, entered the current voice-based intelligent interactive stage, and will move towards the ultimate form of multi-mode interaction in the future.

In the era of touch interaction, people and the cockpit mainly interact through mechanical buttons. The driver triggers a certain function of the cockpit through buttons, such as radio broadcasting, window lifting, air conditioning control, etc. At this stage, all kinds of equipment/systems in the cockpit, such as on-board infotainment system, LCD instrument, seat, etc., are based on independent ECU's independent decision-making control without any related influence. Each system is like an "island, in addition, due to limited space, the number of buttons cannot be increased, and the functions of each system/device in the cabin are relatively simple and direct. In the later period, the mechanical instrument panel (digital display screen) gradually evolved into a color LCD screen, and the mechanical buttons evolved into touch buttons, which greatly improved the interactive experience of the cockpit. Many innovative applications such as navigation, car-machine Bluetooth interconnection, multi-style instrument panel, and central control large screen appeared.

After entering the era of voice interaction, the driver's (human) hands have been further released, the cockpit more through dialogue to obtain control intentions or control reminders, more artificial intelligence is injected into the overall design of the cockpit. The voice interaction system is divided into four links: awakening, listening, understanding and broadcasting, and further improves the recognition rate with lip recognition, eye gaze tracking, sound source positioning and other auxiliary means. It has the characteristics of personification and emotion, and conveys the concept of automobile brand to users with language and service. With the introduction and rapid development of voice interaction systems, the intelligent cockpit has gradually changed from human adaptation to machines to machines that constantly adapt to people, and in some scenarios can even pass the Turing test.

Multi-mode interaction is the ideal model of the next generation of human-computer interaction. What is multi-mode interaction? Simply put, it is to use gestures, eye tracking, voice and other ways to interact. The modal here is similar to the human "senses", multi-modal is the fusion of a variety of senses, corresponding to the human vision, hearing, touch, smell, taste five senses. With the further development of sensor technology and artificial intelligence technology, a single perception method cannot fully understand the true intention of the driver (person), and the recognition rate is further improved. The intelligent cockpit must integrate multiple perception methods, from a voice-based interaction mode to a multi-mode interaction mode with visual perception as the core and voice and other perception methods as the assistance, forming an independent three-dimensional perception layer. At present, it has become an industry trend that a variety of sensing technologies are applied to intelligent cockpit at the same time. The average number of sensors installed in a single cockpit is expected to reach more than 11 in 2025, of which more than 50% will be visual sensors (cameras). New products recently introduced by many car companies such as Mercedes-Benz, BMW, Chang 'an and GAC have realized multi-modal human-computer interaction to varying degrees.

2.2 from passive to active interaction

According to the direction of interaction, the intelligent cockpit has changed from passive interaction mode to active interaction mode.

In the early cockpit, all interactions were passive (from the perspective of the cockpit). The pilot (person) issues an instruction through a certain mode, and the cockpit passively receives and executes it. The cockpit never, and cannot, autonomously perform a function or initiate some kind of communication with the pilot (person). In this era, drivers need full control of the car, continuous attention to all kinds of environmental information, and rapid decision-making. The design goals of the cockpit interaction system are shorter control paths, faster response times, and simpler control methods to achieve an upgrade from "operation" to "control.

With the improvement of the level of autonomous driving, the driver will have more time and choice to do various tasks besides driving, which opens up new ways for interaction, and active interaction is introduced into the concept of intelligent cockpit. More and more sensors are integrated into the cockpit to understand the behavior, emotions and selection trends of the driver and/or passengers, so as to provide precise functions and services. From passive reception, the cockpit begins to shift to active perception, problem detection, Communication scheme. In this stage, the cockpit already has a relatively complete perception ability, and has a certain ability to analyze and understand. It can issue reminders to form interaction when necessary according to the current state of the car/body, the state of the car, the environment, and the mental state of the occupants., And propose relevant solutions for the driver (person) to make decisions. Drivers no longer need to pay attention to the internal and external environment of the vehicle all the time, and can release more time and energy to engage in other interesting activities, realizing the leap from "control" to "enjoyment.

2.3 from explicit interaction to implicit interaction

Display interaction is the most basic way of interaction in HMI design, language-based interaction, display-based interaction, haptic-based interaction, and multi-modal hybrid interface interaction are all explicit interaction modes, and it is actually an active interaction mode initiated by the driver (human). Its basic characteristics are triggered by the driver and have a perceptible activation timing. For example, voice interaction is initiated by wake-up words, button/touch interaction is initiated by the driver's button, and even more complex visual interactions are mostly triggered by eyeball gaze or specific gestures.

Implicit interaction, on the other hand, is carried out in a state where the driver (person) is completely insensitive. The human-computer interaction system can correctly identify the driving state and potential intention by continuously detecting the voice, posture, facial expression, and physiological characteristics of the occupants in the cabin, and can adjust the car's various systems to the best state at every moment. For example, if the car knows the passenger's full body posture (sitting, lying, etc.), its safety functions (such as airbags, steering, braking, and anti-collision modes) can adjust the deployment in time. Upon detecting driver distraction or fatigue, the intelligent vehicle assistant may choose to provide a visual or vibratory alert, or engage in a dialogue with the driver to keep them alert. If a non-optimal body position is detected, the seat system can push the driver to the correct body position for optimal driving attention. Similarly, if the driver's mood is found to be anger or sadness, the intelligent vehicle assistant can recommend soothing music, control the temperature in the car, and create a relaxed environment. In short, audio-visual interaction is the way to let the machine know you. Smart cars can accurately identify the behavior and activities of drivers and passengers, which has a profound impact on in-vehicle interaction systems and safety functions.

To sum up, when the cockpit interaction technology realizes multi-mode interaction, active interaction and implicit interaction, the driver will no longer feel that he is communicating with the machine, but will feel that the cockpit has become his own "body". In that era, we will truly realize"Car-man unity, man-machine driving"The ultimate goal.

 

3

Key Technology of Human-Computer Interaction in Cockpit

The key technologies commonly used or rapidly developed in the current cockpit human-computer interaction system include but are not limited to the following:

3.1 AR-HUD

HUD is currently a popular market, projecting driving information onto the windshield to keep the driver's head up. The HUD is usually complementary to the traditional dashboard display and provides supplementary information related to driving. For example, the information they project can be any information from which radio station plays to your speed limit, or they can help drivers navigate effectively in accident-prone and low visibility situations. HUD technology is rapidly being perfected as it is seen as the next major technological advance in automotive interior technology. For example, the use of AR holographic technology for HU has recently attracted the interest of the automotive industry. AR-HUD technology is not new, it was used on Boeing 727 aircraft in the 1990 s. The AR-HUD uses optical projection to present virtual information augmentation in the driver's line of sight without requiring the driver to make frequent adjustments between the real world and the dashboard/navigation data.

3.2 DMS/OMS

DMS is the abbreviation of English Driver Monitor System, that is, driver monitoring system. At present, the mainstream is to use 2D or 3D camera scheme (usually with infrared function) to realize the identification of the driver, the detection function of driver fatigue driving and dangerous behavior, which is an important part of the popular ADAS (Advanced Driver Assistance System) system. The DMS can also be upgraded to OMS-the in-vehicle occupant monitoring system. The camera can not only track the driver's state, including some reminders of the driver's driving fatigue, but also track the rear passengers to meet the relevant riding needs.

DMS/OMS system provides original video data for all kinds of visual intelligent perception algorithms, which is the physical basis of all visual intelligent perception algorithms.

3.3 gesture recognition

Through the in-vehicle vision camera (DMS/OMS), the driver's hand movements are accurately recognized, and a set of predefined gesture standards are formed to control various functions in the cockpit. Different gestures can express rich semantics, and enable the driver to get rid of the position constraints of the central control touch screen, and initiate interaction more freely and conveniently. Gesture recognition is one of the various types of vision-based intelligent recognition algorithms, and is being promoted and applied in the intelligent cockpit industry.

3.4 physiological state perception

The physiological state sensing system can realize the physiological state monitoring of the occupant through the fusion of visual, infrared and tactile sensing means. The system functions include heart rate detection, body temperature monitoring, breathing detection, distraction/fatigue detection, dangerous driving reminder, emotion recognition and comfort, and gesture recognition. It should have all-weather sensing capability to provide physiological status monitoring and reminding of cabin personnel anytime and anywhere.

3.5 cloud edge-end fusion computing, more offline tilt

Users have higher requirements for the delay of human-computer interaction systems. The biggest problem caused by online interaction is the system delay, which depends on the offline local interaction of the network. In the case of sufficient end-side computing power, the delay is generally twice as fast as that of the cloud.

The cloud has stronger computing power and is an important supplement to the computing power of the end side. After integration, it can better provide users with refined and gradually modified services. Based on the federal learning algorithm, a small model can be trained on the end side to send the model to the cloud for joint training, which may further improve the accuracy of the algorithm model and make the habits reach thousands of people for different users. Personalized models are downloaded over-the-air to the end-side for use after full training in the cloud, and support offline computing for better computing power allocation.

Data privacy aspects. People prefer to leave the privacy data (including human physiological characteristics and occupant conversation voice) in the cockpit for processing on the end side rather than uploading it directly to the cloud, thus avoiding accidental disclosure of sensitive information.

3.6 multi-tone zone isolation

The purpose of multi-tone zone isolation is to allow people in the cabin to enjoy their own audio without the constraints of headphones. Personal audio isolation can divide the cabin of a car into several independent listening areas, each of which can hear independent audio. Through this technology, the driver can choose whether he wants to hear the sound of other sound zones, allowing the driver to concentrate on driving even in a noisy environment. It is hoped that in the future cockpit, audio zoning technology will recognize the position of the occupant and dynamically generate independent audio zones based on the position.

3.7 sound source positioning

There are two main purposes of using sound source positioning. The first purpose is to know who is talking and provide personalized services. For example, when the co-driver says "open the window", only the co-driver's window will be opened; the second purpose It is to realize directional pickup. Due to the large noise in the environment inside and outside the car, the discussion and echo between passengers will affect the pickup quality of voice interaction, therefore, sound source localization allows the microphone array to lock in a certain direction of sound. Sound source localization has developed from the double-tone zone to the four-tone zone. The double-tone zone means that the cockpit is divided into left and right sides. It is impossible to identify whether the driver or the passenger on the left side of the rear row is speaking, so it has greater limitations. The four-tone zone can divide the cockpit into four areas: the driver, the co-driver, the left side of the rear row and the right side of the rear row, which can better solve the problem of the two-tone zone.

 

4

Technical planning of the Zhijia Center in the field of technology

4.1 intelligent cockpit is an important application scenario of video Internet of Things

First of all,The essence of intelligent cockpit is human-computer interaction, and the most important dimension of human-computer interaction is visual perception. The smart car is actually an external extension of the family space. It is a special manned robot that serves family travel. The smart cockpit is the first interface for human-vehicle communication and is the most intuitive feeling of people's intelligence of the car. Some psychological studies have shown that, according to the experiments of Chiratella (Treicher), 83% of the information that humans come into contact with comes from vision, and vision is still the most efficient way for people to obtain information. Therefore, human-computer interaction technology, especially visual perception, is an important research content of intelligent cockpit.

Video Internet of Things is an important basic technology of visual perception system.The visual perception system takes video data as the processing object, and its core is inseparable from various video processing technologies. At present, there are many challenges in intelligent perception, semantic coding, narrow-band transmission, heterogeneous storage, analysis and understanding of ultra-large-scale video data, which seriously hinder the development of various applications centered on video. The new generation of video Internet of Things integrates multi-disciplinary key technologies such as multimedia, Internet of Things, big data, artificial intelligence, etc., and provides stable and reliable basic capabilities and service facilities for the "Vision of Everything" scene.

with visual perception as the core interaction formSmart cockpit will be an important product application in the future of video IoT, is a new business market that major video technology service enterprises are competing.

Technology Accumulation of 4.2 Zhijia Center in Video and IOT

Relying on the main construction businesses such as mobile housekeeping, home wisdom and digital village, etc., in recent years, the smart home center has precipitated a number of key technical achievements in video connection, including but not limited:

1) Successfully developed fully independent intellectual property rights to support ultra-large-scale access to the intelligent video Internet of Things system.

In the face of the increasing demand of the video IOT industry, Zhijia Center, relying on AIoTel's chief expert studio and technology accumulation, has successfully developed a video IOT capability system capable of carrying ultra-large-scale connection terminal access after 8 years of polishing, focusing on nebula architecture, semantic coding, media flash transmission, video direct storage, illusion encryption, eye rainbow communication, north-south cloud bridge and visual computing scheduling, the relevant performance indicators have reached the advanced level of the industry, which has laid a solid foundation for supporting the rapid development of various upper-level video services, ensuring the technical safety of core areas, and building a technical benchmark for related industries.

2) Successful construction of dozens of central computing nodes distributed nationwide, has been connected to tens of millions of video terminals of the visual network and infrastructure platform.

The Zhijia Center has cooperated deeply with the mobile cloud to complete the construction of dozens of exclusive IDC computer rooms across the country, realizing the nearby access of users and terminals in 31 provinces. Currently, the activation volume of the whole network equipment exceeds 50 million, the usage of cloud servers exceeds 50000, the cloud storage scale exceeds 800PB, and an ultra-large-scale video storage cluster has been built.

3) Successfully build a visual network ecosystem covering mainstream equipment/chip manufacturers in the industrial chain.

Through the equipment SDK enabling mode, we have reached a deep partnership with domestic and foreign first-line video networking terminal manufacturers. We have cooperated with more than 200 manufacturers/brands, docking more than 680 terminal models in 12 final categories such as smart speakers, cameras, cat's eye doorbells, smart door locks, smart desk lamps, smart rearview mirrors, etc., and Top50 ecological brands are fully enabled. With a number of video SOC chip manufacturers to establish strategic partnerships, docking its main chip 18 models, listed commercial 11 models. The Vision Union Terminal Ecological Alliance has been established, playing a leading role in the industry ecosystem.

Assumption of the next research work in 4.3

Promote the key technologies related to the video networking, accelerate the scale construction of the video networking platform, and build a national team and main force in the field of video technology;

Continue to upgrade the ecology of the Vision Union, and gradually cultivate a large number of domestic high-end video chip manufacturers;

Strengthen cooperation with smart cockpit tie 1 suppliers to promote the company's core video technology in the smart cockpit application scenario as soon as possible.

The title of the original text has been revised, and the original title is: "Car-man Unity, Man-machine Co-driving"-Prospective Research on Intelligent Cockpit Perception Interaction Technology