Skip to content

UI Software

Jump to bottom
fllay edited this page Feb 1, 2022 · 17 revisions

There are two parts of software. One is backend software running on a embedded linux board and another one is frontend (Web application) also running on the embedded linux board. They are connected by Rosbridge as shown below.

roscore is a program to manage all messages in ROS while the frontend is serve from server programs. There are two API servers running on the embedded linux board.

pi@raspberrypi:~/linorobot_ws/src/tutorial/scripts $ pm2 list
┌────┬────────────────────┬──────────┬──────┬───────────┬──────────┬──────────┐
│ id │ name               │ mode     │ ↺    │ status    │ cpu      │ memory   │
├────┼────────────────────┼──────────┼──────┼───────────┼──────────┼──────────┤
│ 1  │ app3               │ fork     │ 0    │ online    │ 0%       │ 210.5mb  │
│ 0  │ index              │ fork     │ 0    │ online    │ 0%       │ 53.7mb   │
└────┴────────────────────┴──────────┴──────┴───────────┴──────────┴──────────┘

app3 is a python code to server APIs for AI and index is a Nodejs code to perform house keeping.

ROS sensor nodes

ROS nodes for sensor interface is contains in kidbright_tpu_nanopi package. There are two main sensors, camera and microphone. audio_node.py is a node to interface with microphone and cam_stream.py is a node to control camera.

pi@raspberrypi:~/linorobot_ws/src/kidbright_tpu_nanopi/scripts $ ls -l
total 1112
-rw-r--r-- 1 pi pi      36 Apr 26  2021 array_hf.csv
-rwxr-xr-x 1 pi pi    5814 May 13  2021 audio_node.py
-rwxr-xr-x 1 pi pi    1088 May 17  2021 cam_stream.py
-rw-r--r-- 1 pi pi    5434 Apr 26  2021 foo.csv
-rwxr-xr-x 1 pi pi     635 Apr 26  2021 get_object_center.py
-rwxr-xr-x 1 pi pi    2016 Apr 26  2021 mpu9250ros.py
-rwxr-xr-x 1 pi pi    6414 May  7  2021 save_wave.py
-rw-r--r-- 1 pi pi    5075 Apr 26  2021 save_wave_3.py
-rwxr-xr-x 1 pi pi    6449 Aug  1 18:03 save_wave_action.py
-rw-r--r-- 1 pi pi       0 Apr 26  2021 sound.wav
-rw-r--r-- 1 pi pi 1000044 Apr 29  2021 test.wav
-rwxr-xr-x 1 pi pi    5403 May 13  2021 tpu_classify.py
-rwxr-xr-x 1 pi pi    7399 May 13  2021 tpu_detect.py
-rwxr-xr-x 1 pi pi    5743 May 10  2021 wakeword_classify.py
-rwxr-xr-x 1 pi pi    7225 Apr 26  2021 wakeword_classify.py.back
-rw-r--r-- 1 pi pi   44509 Apr 26  2021 wav.csv

We can also see from rosnode list for these two nodes.

/audio_stream_746_1643555002873
/cam_stream
/rosbridge_websocket
/rosout
/rosserial_lino
/save_wave_file
/server_stdio
/talker_27721_1643596630977
/web_video_server

Web application directory

Web application is in /home/pi/kbclientNew/nectec-client directory.

pi@raspberrypi:~/kbclientNew/nectec-client $ ls -l
total 676
drwxr-xr-x    3 pi pi   4096 Jul 28  2021 Example_ObjectDetection
-rw-r--r--    1 pi pi    399 Jul 28  2021 README.md
drwxr-xr-x    4 pi pi   4096 Jul 28  2021 aoba
-rw-r--r--    1 pi pi     53 Jul 28  2021 babel.config.js
drwxr-xr-x   10 pi pi   4096 Jan 19 06:52 dist
drwxr-xr-x    3 pi pi   4096 Jul 28  2021 eru
drwxr-xr-x    3 pi pi   4096 Jul 28  2021 hifumi
drwxr-xr-x    3 pi pi   4096 Jul 28  2021 inko
drwxr-xr-x    3 pi pi   4096 Aug  1  2021 mio
drwxr-xr-x 1038 pi pi  36864 Jul 29  2021 node_modules
-rw-r--r--    1 pi pi 597552 Jul 29  2021 package-lock.json
-rw-r--r--    1 pi pi   2527 Jul 28  2021 package.json
drwxr-xr-x    3 pi pi   4096 Jul 28  2021 public
drwxr-xr-x    8 pi pi   4096 Jul 28  2021 src
-rw-r--r--    1 pi pi    152 Jul 28  2021 vue.config.js

and the source code is in src. The web app is written using Vue framework. The projects are directories in this folder /home/pi/kbclientNew/nectec-client/dist. For example

pi@raspberrypi:~/kbclientNew/nectec-client/dist $ ls -l
total 36
drwxr-xr-x 3 pi pi 4096 Aug  2 04:48 aoba
drwxr-xr-x 4 pi pi 4096 Jan 31 12:11 cake
drwxr-xr-x 2 pi pi 4096 Aug  2 04:46 img
-rw-r--r-- 1 pi pi  484 Aug  2 04:46 index.html
drwxr-xr-x 3 pi pi 4096 Jan 31 08:16 inko
drwxr-xr-x 2 pi pi 4096 Aug  2 04:46 js
drwxr-xr-x 3 pi pi 4096 Aug  2 05:04 koguma
drwxr-xr-x 3 pi pi 4096 Aug  2 04:47 shii
drwxr-xr-x 3 pi pi 4096 Aug  2 04:46 soundApp

It can be observed that there are few projects here. aoba, cake, inko, koguma and shii. Let take a look at inko project.

Project directory

pi@raspberrypi:~/kbclientNew/nectec-client/dist/inko $ ls -l
total 3384
drwxr-xr-x 2 pi pi    4096 Aug  2  2021 images
-rw-r--r-- 1 pi pi 3434344 Jan 31 08:19 images.zip
-rw-r--r-- 1 pi pi    1637 Aug  2  2021 imclass.json
-rw-r--r-- 1 pi pi      53 Aug  2  2021 label.json
-rw-r--r-- 1 pi pi      22 Jan 31 08:19 label_map.txt
-rw-r--r-- 1 pi pi     153 Aug  2  2021 project.xml
-rw-r--r-- 1 pi pi     812 Feb  1 02:15 test1.py
-rw-r--r-- 1 pi pi    1225 Feb  1 02:15 test1.xml

Blockly code will be save into test1.xml and when user press Run, Blockly code will be converted into test1.py file. Let compare test.py content with our Blockly code.

and test1.py

class2 = None


from geometry_msgs.msg import Twist
import rospy
import time
rospy.init_node('get_center', anonymous=True)
velocity_publisher = rospy.Publisher('/cmd_vel', Twist, queue_size=1)
vel_msg = Twist()
import roslib
import rospy
from kidbright_tpu.msg import tpu_object
from kidbright_tpu.msg import tpu_objects
from std_msgs.msg import String,Int16MultiArray
import rosnode
import subprocess
import time
import os
ros_nodes = rosnode.get_node_names()
if not '/image_class' in ros_nodes:
	command='rosrun kidbright_tpu tpu_classify.py /home/pi/kbclientNew/nectec-client/dist/inko'
	process = subprocess.Popen(command, shell=True, stdout=subprocess.PIPE)
	time.sleep(10)
while not rospy.is_shutdown():
  class2 = rospy.wait_for_message('/tpu_objects', tpu_objects, timeout=4).tpu_objects
  print(class2)

Model file

Now let train our model and download it.

Before downloading the mode

pi@raspberrypi:~/kbclientNew/nectec-client/dist/inko $ ls -l
total 3384
drwxr-xr-x 2 pi pi    4096 Aug  2  2021 images
-rw-r--r-- 1 pi pi 3434344 Jan 31 08:19 images.zip
-rw-r--r-- 1 pi pi    1637 Aug  2  2021 imclass.json
-rw-r--r-- 1 pi pi      53 Aug  2  2021 label.json
-rw-r--r-- 1 pi pi      22 Jan 31 08:19 label_map.txt
-rw-r--r-- 1 pi pi     153 Aug  2  2021 project.xml
-rw-r--r-- 1 pi pi     812 Feb  1 02:15 test1.py
-rw-r--r-- 1 pi pi    1225 Feb  1 02:15 test1.xml

after the model is downloaded

pi@raspberrypi:~/kbclientNew/nectec-client/dist/inko $ ls -l
total 6772
drwxr-xr-x 2 pi pi    4096 Aug  2  2021 images
-rw-r--r-- 1 pi pi 3434344 Feb  1 02:56 images.zip
-rw-r--r-- 1 pi pi    1637 Aug  2  2021 imclass.json
-rw-r--r-- 1 pi pi      53 Aug  2  2021 label.json
-rw-r--r-- 1 pi pi      22 Feb  1 02:56 label_map.txt
-rw-r--r-- 1 pi pi     153 Aug  2  2021 project.xml
-rw-r--r-- 1 pi pi 3468864 Feb  1 02:57 retrained_model_edgetpu.tflite
-rw-r--r-- 1 pi pi     812 Feb  1 02:15 test1.py
-rw-r--r-- 1 pi pi    1225 Feb  1 02:15 test1.xml

It can be seen that there is a model file in the project directory.

retrained_model_edgetpu.tflite

Manually run python script

test1.py can be execute in the project directory using

pi@raspberrypi:~/kbclientNew/nectec-client/dist/inko $ python3 test1.py 

[cx: 0.98828125
cy: 0.98828125
width: 0.0
height: 0.0
label: "totoro"]
[cx: 0.98828125
cy: 0.98828125
width: 0.0
height: 0.0
label: "totoro"]
[cx: 0.9921875
cy: 0.9921875
width: 0.0

By running python code using command line, it will be more responsive than web UI. Moreover, we can see any error in the code on the console.

Content

Clone this wiki locally