Broadcast speech recognition and control system based on Internet of Things sensors for smart cities

1 Introduction

As we talk much more quickly than we write, voice-first technology not only increases safety but also improves customer experience by making communication more convenient. Communication has improved by allowing spoken commands to be sent to touchless control devices. A transition to voiceless is supported by a number of variables in addition to the users’ low-effort, high-comfort experience. Devices with speech recognition are portable and available anywhere. Widespread adoption of smart speakers in contemporary homes and Natural language processing (NLP) advancements enable sentiment analysis and broad context understanding using machine learning, and artificial intelligence development to provide tailored experiences and voice-controlled IoT devices’ widespread use.

The IoT is growing to a large extent. One of the most cutting-edge technologies, like IoT, may be utilized for managing, connecting, and monitoring devices linked to the Internet. Obtaining an adequate range of IoT devices, security, and related IoT device services for the average way of life, including a novel use case value proposition proposal, are some of the major challenges [1]. A hyper-connected environment where items are connected to one another, mobile devices, and the Internet has emerged as a result of information technology’s (IT) fast expansion. The core of this highly linked network is the IoT, also known as machine-to-machine communication [2].

The IoT has been used for data and voice integration into IoT applications, such as voice over long-term evolution, offering a flexible way to provide human interaction and communication that can provide a more cost-effective user interface than traditional methods like touch screens or data input with proper implementation and consideration of costs and human factors. The Android framework and Firebase as the cloud are used in this article as a paradigm for smart home computerization. The Arduino Node MCu Wi-Fi Module’s presence placed restrictions on each device [3].

As an emerging strategic industry, the Industrial Internet of Things (IIoT) is widely used as an important foundation and core technology in industrial and agricultural production, transportation, and other aspects. Sensor technology plays an important role in the development of the IIoT [4]. Based on this premise, it is necessary to conduct a comprehensive and in-depth study on the application of intelligent sensors in the IoT, to achieve continuous optimization and upgrading of the IIoT [5].

Secondly, the user end of the Internet of Things can be effectively expanded and extended to all objects, and its ultimate goal is to effectively realize things and people, and things and things. Through the effective connection between the network and all objects, information exchange can be effectively carried out, and then effective identification, management, and control can be achieved [6]. With the continuous development and progress of society, the unique advantages of the IoT are gradually highlighted [7].

The IoT, the enabling technology that has made widespread digitization possible and produced the idea of smart cities, is at the center of smart city projects. For devices to transfer data to the cloud and receive instructions for carrying out tasks, they must be constantly connected to the Internet. This phenomenon is known as the “Internet of Things.” In the IoT, data are gathered and data analytics operations are carried out to extract information to aid in decision- and policy-making. According to the estimates, there will be more than 75 billion Internet-connected gadgets by 2025 [8].

Speech is the most consistent with human natural habits of a way of interaction, and with the development of speech interaction technology, its application in human–machine interface and multimedia is also increasingly common. Language is the most natural and effective means of human communication and is also the signal with the largest amount of information among many carriers, with the highest level of intelligence. This requires that the robot must have the capability of natural language understanding (NLU). The lexer can be used to realize various methods of lexical analysis, such as right linear grammar, and regular expression automata, which are equivalent in description ability but different in complexity [9]. Considering the simple and graceful character of regular expressions, the system uses regular expressions for lexical analysis to realize NLU [10].

2 Literature review

The essence of a sensor is a kind of detection device, which can accurately perceive the information to be measured and effectively transform the relevant information into electrical signals or other forms of information for effective dissemination. Under the effective support of sensors, automatic detection, and control to obtain comprehensive enhancement with the further development of IT, Internet technology is also in the time development pace to get effective development, to be able to fully meet the demand of the development of the IoT, by a high-performance microprocessor with the intelligence technology in the use of traditional sensors, to derive a new intelligent sensor [11]. With the current rapid development of high and new technologies, the realization of intelligent sensors needs to be completed in the following ways: the first is to rely on the non-integrated way, the traditional sensor signal processing circuit, and the microprocessor for scientific and reasonable integration, to achieve the creation of an intelligent sensing system [12].

Since the new century and the rapid development of artificial intelligence in 2017, artificial intelligence ushered in a real new era, and intelligent voice as an important branch of artificial intelligence will become the next outlet and key direction. With the rapid development of robot technology, the traditional control methods such as joystick, keyboard, and handle gradually eliminate voice control as a new control method because of the complicated operation and low efficiency. Because of its advantages such as high control efficiency and simple operation, it has been widely used.

For 8 years running, China has dominated the global market for industrial robots. The action is a component of a larger national effort to deal with a graying population and use cutting-edge technologies to drive industrial upgrading. Major productivity issues and the quickening of robot labor’s replacement have started to be resolved thanks to rapid advancements in artificial intelligence and robotics. The human workforce presents enormous difficulties. As a result of intelligent robots’ influence on labor supply and demand, wage equilibrium may alter as a result of changes in labor supply and demand. Intelligent robots have the potential to soon replace humans in several occupations, lowering wages. Over time, the contribution of labor to the national revenue will decline as intelligent robots operate. Increased minimum wages have a substantial influence on the deployment of robots by more productive, private, skilled labor-intensive, coastal businesses [13].

Complex autonomous and semi-autonomous robotic system control and monitoring is a difficult task. In order to operate as a robotic middleware system operating on Ubuntu Linux, the Robot operating system (ROS) was created. It permits, among other things, hardware abstraction, message-passing between individual processes, and package management. However, there is a lack of active support for ROS apps on smartphones and tablets. Use the ROS Visualization (rviz), which includes a full desktop installation, to see robot data, establish marker locations, or define the boundary perimeters. To make it easier to interact with various robotic systems, an Android application that offers a variety of control and visualization options similar to rviz and is also adaptable and simple to use is needed. Unfortunately, rviz is not available for mobile devices running Android, which accounts for 75% of all mobile devices [14].

In this article, an intelligent voice control system based on the ROS platform is designed, and the speech recognition software is built by the Baidu voice recognition software development kit (SDK). The control system can recognize voice commands and convert them to text information, control the system according to the different words of receiving information from the corresponding instruction, control the robot’s movement of the intelligent robot based on voice control overcomes the traditional robot operation trivial low efficiency and so on shortcomings, for the intelligent robot with flexible control provides the basis.

3 Method

3.1 Current status of relevant technology development

3.1.1 ROS introduction

As a complex research object of multinode and multitask, the robot needs to establish a set of matching systems, including a physical peripheral human–machine interface, communication interface, and simulation tool, to complete the corresponding design [15]. Since entering the new century, the development of the field of robot control is changing with each passing day, and the traditional robot design method is no longer suitable for the requirements of the development of time [16]. In 2010, the open-source ROS was born, which provides a new solution for the development of software and hardware for robot systems.

With the development of science and technology, the social division of labor is gradually detailed, which reflects the characteristics of multilevel robot-related hardware, such as DC motor, sensor, driver, controller, and other components from different manufacturers. ROS, as an operating system suitable for robot programming, can integrate various components of a robot in a loosely coupled way, providing a communication architecture for the communication between robot systems.

As shown in Figure 1, ROS, as a post-operating system, differs from those operating systems in the conventional sense. As an open-source operating system, ROS organizes various environmental control decisions. The publish-subscribe messaging architecture offered by ROS is meant to facilitate the rapid and simple creation of distributed computing systems. It offers a wide range of tools for setting up, launching, introspecting, troubleshooting, visualizing, logging, testing, and terminating distributed computing systems. With a focus on mobility, manipulation, and perception, it also offers a large selection of libraries that implement practical robot functions. With a strong emphasis on integration and documentation, a sizable community supports and advances ROS. The hundreds of ROS packages that are offered by developers worldwide may be found and learned about at ros.org, which serves as a one-stop shop for this purpose.

Figure 1

Comparison of operating systems.

As shown in Figure 2, ROS uses communication modules to realize the network connection among various modules in the system, mainly through topic-based asynchronous data communication. The ROS is composed of a variety of processes and communicates with each other through a point-to-point network structure.

Figure 2

Communication structure of ROS.

The special endpoint-to-endpoint communication mode of ROS can greatly reduce the computational pressure caused by multiple complex functions, especially in the scenario of multiple robots, the ROS system has the characteristics of multilanguage support, and language neutrality framework structure can meet the preferences of various developers [17].

ROS integrates existing open-source project code to integrate a wide variety of functional projects into the development architecture [18]. Compared with the traditional complex construction method of robot systems, it is more convenient and faster.

3.1.2 Introduction to intelligent speech technology

Since the beginning of the 21st century, advances in machine learning algorithms and computer performance have brought about more efficient methods for training neural networks deep neural networks, which exhibit better accuracy and performance than traditional models. On desktop computers, mobile devices, and human-robot interfaces, voice interaction with NLU has been thoroughly investigated. However, a limited study has been done on voice interaction with NLU in argumented reality (AR). In AR, adopting voice interaction has advantages such as high naturalness and being hands-free. Because NLU has a broader grammar, it may be implemented with a speech interface that is more flexible. We look at how much this flexibility may increase interaction accuracy when utilizing voice in augmented reality. Users’ voice command intent is determined by the NLU component. Without prior constraints, users can use different verbalizations of the same command. Intents, entities, keywords, and semantic interpretation are all automatically extracted from texts using NLU. NLU is frequently used in combination with voice recognition to examine the text that has been extracted from spoken language. Deep learning techniques like the attention-based encoder–decoder and recursive autoencoder are the foundation of the majority of NLU algorithms.

With the application of deep learning and the continuous progress of natural speech algorithms, speech recognition technology has been greatly developed. The speech recognition rate of some companies has reached 97% in the case of quiet rooms, and the stability and recognition rate have met the requirements of wide application. On the other hand, the rapid development of NLP technology has greatly reduced the difficulty for machines to understand the semantics of human voices and improved their ability to communicate with people. The principle of speech recognition is shown in Figure 3.

Figure 3

Principles of voice control flow in speech recognition.

3.2 Overall design of the system

Based on the summary of existing scientific research achievements, combined with the application of real scenes and the development status of intelligent speech recognition technology, this article uses the advantages and characteristics of ROS to design an intelligent speech control system. The system is composed of software and hardware, and its structure is shown in Figure 4.

Figure 4

Software and hardware of the control system.

The hardware part is mainly composed of the motor drive module of the main controller, power module, WiFi module, Bluetooth module, laser ranging module, etc. [19]. The main controller on the robot platform receives the control signal sent by the PC and directly or indirectly controls the movement or operation of a certain function of the robot platform [20].

3.4 Software module

Based on the characteristics of the detailed and multilevel design of the robot industry, the overall architecture of the robot platform is designed as shown in Figure 5. In the PC, the system should complete the functions of voice interaction and monitoring, control the movement of the robot with real-time voice, and monitor the movement status of the robot.

Figure 5

Architecture of the Robot platform.

Monitoring module: Multiple sensors can be configured on the robot platform. Sensors such as the LiDAR camera and gyroscope constantly generate data during operation, and the control system should timely process the data transmitted by each sensor. To better detect whether the running status of each node is normal, it is necessary to detect the running data of each node in real time and give an alarm when the data is abnormal.

Voice control: The Baidu voice recognition node is used to launch the file to obtain voice commands from the speech recognition node, convert them into words, and publish them to the corresponding topic. The robot control node receives text messages by subscribing to the topic to control the main controller on the robot platform and control the movement of the robot platform.

Navigation module: According to the shape and size of the robot, the parameters of the navigation function package are configured. When the robot platform is running ROS, the transformation tree and sensor data of ROS message type are released to realize the autonomous navigation function of the robot platform.

3.5 Automatic speech recognition (ASR)

At present, most intelligent voice control systems adopt hardware modules as speech recognition units. The commonly used ones are the LD3320 speech recognition chip from ICROUTE and the YQ5969 series intelligent voice control module from Shenzhen Renmai Information. These are non-specific speech recognition chips that only recognize the same language and have nothing to do with differences in age and gender. These chips increase the hardware cost for the design of the control system, and the recognition rate is not high, which leads to the unstable performance of the language recognition of the control system.

ROS has some available speech tools, such as Sphinx and Festival, but it is not widely used because of its low recognition accuracy and inadaptability to dialects.

Baidu intelligent speech development platform provides an intelligent speech development kit SDK, which makes the secondary development more convenient and quicker to use Baidu voice application programming interface to realize speech recognition and TTS function package, which can easily realize speech recognition, synthesis, and other functions. The robot control node receives written instructions by subscribing to this topic and sends them to the main controller of the robot platform, which controls the movement of the DC motor through the drive module, thus completing the voice control. The voice control process is shown in Figure 6.

Figure 6

Voice control flow.

4 Results and discussion

WER measures how many mistakes there are per thousand words in a transcript. Better voice recognition accuracy results from a reduced WER in speech-to-text. When transcribing sentences and paragraphs of words with significance (such as from book pages or newspaper articles), the WER may be more useful. It shows how many words must be added, removed, or substituted in order to change one statement into another.

WER is a standard statistic for assessing how accurately speech recognition APIs create transcripts. Along with powering complex IVRs and voice-activated devices like the Amazon Echo, speech recognition APIs are used to extract useful information from vast amounts of audio data. By developing a test, you may check the precision of your unique model. The test requires a number of audio recordings and the transcriptions that go with them. You can assess the precision of a custom model against a speech-to-text-based model or another custom model. Analyze the WER in relation to the outcomes of the voice recognition test after receiving the test results. WERs of 5–10% are regarded as being of high quality and suitable for usage. Although a WER of 20% is respectable, you may want to think about getting further training. When the WER is 30% or more, it indicates poor quality and necessitates customization and training.

WER and accuracy here were calculated under different input commands, as shown in Table 1.

According to the experimental results in Figure 7, the WER of speech recognition for six groups of different voice commands is all within 5%. Compared with the speech recognition chip as the speech recognition unit, the speech recognition software, including the speech recognition SDK, has higher efficiency, a reduced hardware cost, and a lower WER [27,28]. You must construct a SpeechConfig object in order to use the Speech SDK to contact the speech service. This class contains details about your subscription, such as your key and any related endpoint, host location, or authorization tokens. Language and acoustic modeling techniques are used in speech recognition. Acoustic modeling is used to illustrate the link between audio signals and linguistic units of speech. Language modeling, on the other hand, connects sounds to word sequences to assist in discerning between similar sounding words or phrases [29,30]. Furthermore, hidden Makarov models are frequently utilized to distinguish certain temporal speech patterns and hence increase system accuracy.

Figure 7

Experimental accuracy analysis diagram.

5 Conclusion

This article proposes a speech recognition and control system for broadcasting based on IIoT sensors. Speech recognition and robot control nodes are programmed on the robot platform using ROS to collect human voice signals, and then the speech recognition nodes are used to process the speech signals, and the robot control nodes receive commands and then control the robot movement. The system based on voice control can free human hands, solve the problems of traditional robot operation cumbersome system delay, large interpersonal interaction is not needed, greatly improve the efficiency of production regulation, more suitable for the needs of market development. The experimental results show that the control system can recognize the acquired sound signals, convert them into control instructions, and then tell the robot platform to move in the direction of the control instructions. Speech recognition rates exceed 95%. The system’s high identification rate and ease of use are requirements for most industrial systems. It potentially has a substantial impact on production and life efficiency, as well as the development of control technologies. This module responds comparatively quickly. The major limitation of this research lies in the losses that may arise due to the use of various devices and the interaction between machines and humans. Furthermore, a large set of data is required to train the speech module in order to recognize the sound signals more accurately. In order to tackle these limitations, work may be done by including more appliances to reduce the losses and noise, training the speech recognition module to adapt to different user voice conditions, and adding a voice recognition module to assure security for home automation. Furthermore, additional authentication methods can be added.