complete all

.gitignore

@@ -4,6 +4,7 @@
 *.odt
 *.pdf 
 *.gv
+*.wav
 
 # vscode 
 *.vscode

README.md

@@ -1,74 +1,120 @@
 
-# Simon Says: Say a target, and the robot moves there.
-Spring project at Northwestern University  
-2019 April ~ June  
-(This repo will be completed by 6/19)
+# Simon Says
+Goal: I say a target object, then the robot detects it and moves there.
 
+--------------------
+
+Test detection:  
+![](doc/test_detection.gif)
+
+Robot moves to the target number:  
+![](doc/car_moves_to_target.gif)
+
+--------------------
+
+Demo video: [doc/video_demo.mp4](doc/video_demo.mp4)
+
+Content of the video demo:
+``` 
+0:00-1:10: Test object detection.
+1:10-1:20: Object detection.
+1:27-1:35: Use speech command to specify the target.
+1:36-2:03: Robot moves to the target object.
+```
 
 # 1. Intro to the project
 
-Hardware:  
+This is my spring project at Northwestern University, MS Robotics program.  
+Time: 2019/04-06.
+
+
+**Hardware:**  
 * Turtlebot3-Waffle-Pi (a 2-wheel robot)  
 * RGBD camera (Intel Realsense)
 * A white board with numbers on it, including ①, ②, ③, ④, ⑤.
 * My laptop as the main processor.
 
-Procedures and goals: 
+**Procedures and goals:** 
 1. I put the Turtlebot robot on the ground.
 2. I put the white board in front of the robot with some arbitrary pose.
 3. I say a target number in English, such as "one" or "two". 
-4. The robot detects the target number, and locates its position.
+4. The robot detects the target number, and locates its position and orientation.
 5. The robot moves to the white board and hits the target.
 
-Key techniques:  
+**Key techniques:**    
 * Speech recognition (classification).
 * Object detection to find targets.
-* Point cloud processing for locating the target's pose.
-* Control algorithm for moving robot to desired pose.
+* Point cloud processing for locating the target's orientation.
+* Use control algorithm to move robot to the desired pose.
 
-Environments:
-* ROS Melodic on turtlebot with Python 2.
+**Environments:**
+* ROS Melodic on Turtlebot with Python 2.
 * ROS Melodic on my laptop with Python 2.
 * Python 3 on my laptop for speech recognition and object detection. Communicate with ROS through file.
 
-# 2. Done
+# 2. Main components
 
 ## 2.0 Hardware setup
-including: Turtlebot, realsense camera, communication with my laptop, etc.
+Turtlebot, Realsense camera, my laptop.
 
-## 2.1 Natural language proprocessing
-I've trained an LSTM model to classify 10 types of audio segments, namely: "back", "five", "four", "front", "left", "one", "right", "stop", "three", "two".
+## 2.1 Speech commands recognition
+I've trained an LSTM model to classify 11 types of audio clips, namely: 
+> one, two, three, four, five, front, back, left, right, stop, none
 
+Code: [src/speech_recognition/](src/speech_recognition/)  
+Details: Please see this project:
+https://github.com/felixchenfy/Speech-Commands-Classification-by-LSTM-PyTorch 
 
-## 2.2 Simple navigation algorithm
+## 2.2 Object detection by Yolo
+Detect numbers 1~5 in RGB image.
 
-Given robot's current pose (x, y, theta) and a goal pose (x*, y*, theta*), I've implemented an algorithm that could drive the robot to the goal position based on a feedback control method.
+It's trained on 90 real images and 4000 thousand synthesized images, with a learning rate of 1e-3 or 1e-4, and trained for about 400 epoches in total. 
 
-## 2.3 Object detection by Yolo
-Detect numbers 1~5 in RGB image.
+I've lost the configuration file, so this training is not reproducible. But the main code, all data, and trained weights are still there, and are listed below:  
+Code: [src/detection/yolo/](src/detection/yolo/)  
+Data: [src/detection/yolo/data/README.md](src/detection/yolo/data/README.md)    
+Weights: [src/detection/yolo/weights/README.md](src/detection/yolo/weights/README.md)   
+Details of how the code works: Please see this project: https://github.com/felixchenfy/Yolo_as_Template_Matching
+
+## 2.3 Plane detection by RANSAC
+Since the target numbers are on a white board, their orientation is the normal directioni of the board's plane, which I use RANSAC to detect.
+
+Code: [src/detection/detect_plane.py](src/detection/detect_plane.py)
+
+## 2.4 Control algorithm for wheeled robot
+
+Given a robot's:  
+```
+current pose: (x, y, theta)  
+goal pose:    (x*, y*, theta*)
+```
+I've implemented a **"drive to pose"** algorithm that could drive the robot to the goal position based on feedback control.
+
+Code: [src/turtlebot_control/move_turtle_to_pose.py](src/turtlebot_control/move_turtle_to_pose.py)  
+Reference: [Robotics, Vision and Control](https://link.springer.com/book/10.1007/978-3-642-20144-8), Chapter 4.1.1.4--Moving to a Pose.
+
+# 3. How to run
 
-## 2.4 Object pose estimation
-First, detect plane in point cloud data.  
-Then, from Yolo's result, we can locate the target's position and orientation.
+For connecting to Turtlebot and setting things up,  
+please see:  [doc/readme1_start_turtlebot.md](doc/readme1_start_turtlebot.md)
 
-# 3. TODO
-1. Integrate the whole thing and take a demo video.  
-    (Complete by June 14)
-2. Update this README.   
-    (Complete by June 18)
+For testing whether the Turtlebot is working OK, as well as some other tests,  
+please see:
+[doc/readme2_tests.md](doc/readme2_tests.md)
 
-# 4. Main commands
+Then, the main commands are illustrated below:
 
-see [doc/main_commands.md](doc/main_commands.md)
+* Start script for **Object detection**:  
+    > $ roscd simon_says; open_anaconda; cd src/detection; python yolo_response.py
 
-# 5. Other commands for testing
+* Run main program [src/main.py](src/main.py):
+    > rosrun simon_says main.py 
 
-* Test robot controller in simulation
+* Start script for inputting **Speech Command**:   
+    > $ roscd simon_says; open_anaconda; cd src/speech_recognition; python voice_speaker.py  
 
-    In [lib_turtlebot.py](src/turtlebot_control/lib_turtlebot.py), set **IF_IN_SIMULATION** to True to test the robot in simulation.  
-    In [move_turtle_to_pose.py](src/turtlebot_control/move_turtle_to_pose.py), in the **def main(** function, select a test case to test.
+    (After starting this, the keyboard key "R" should be pressed only when you want to record the audio)
 
-    Then:  
-    > $ roslaunch turtlebot3_gazebo turtlebot3_empty_world.launch  
-    > $ rosrun simon_says move_turtle_to_pose.py   
+    Press key "R" to record audio and send your command to the laptop.
 
+    After receiving a good command, the robot will move to the target.

data/raw_rgbd_images/README.md

@@ -0,0 +1,7 @@
+
+The output of this script:  
+src/realsense_io/sub_and_save_rgb_and_depth_images.py
+
+are the recorded color/depth images of the Realsense camera.
+
+These images will be saved to this folder.

doc/readme1_start_turtlebot.md

@@ -0,0 +1,45 @@
+
+# 1. Commands to start Turtlebot 
+
+
+## 1.1 On my laptop
+
+Connect to Turtlebot:  
+```
+Connect to the hidden wifi: RoboLab  
+User name on Turtlebot's Ubuntu: pi-cvlab
+```
+
+Set my laptop's IP in ~/.bashrc:   
+```
+export ROS_IP=192.168.1.204
+```
+
+
+Start ROS master:  
+> $ roscore
+
+Open another terminal and ssh into Turtlebot:  
+> $ ssh pi@192.168.1.218   
+
+You may or may not open vncserver:
+> $ sudo service vncserver@1 start
+
+## 1.2 On Turtlebot  
+
+Add following to ~/.bashrc:   
+```
+export ROS_MASTER_URI=http://192.168.1.204:11311
+export ROS_HOSTNAME=192.168.1.218
+export ROS_IP=192.168.1.218
+```
+
+Launch sensors and bringups:  
+> pi@pi-cvlab:~$ roslaunch turtlebot3_bringup turtlebot3_robot.launch
+
+Launch realsense camera (if needed):   
+> pi@pi-cvlab:~$ roslaunch realsense2_camera rs_camera.launch align_depth:=true enable_infra1:=false enable_infra2:=false color_fps:=30 depth_fps:=30
+
+Notes: Raspberry PI is slow. For Realsense, please see this for solving error:    
+https://github.com/IntelRealSense/realsense-ros/issues/669
+https://github.com/IntelRealSense/realsense-ros  

doc/readme2_tests.md

@@ -0,0 +1,33 @@
+
+# 2. Tests to run on my laptop  
+> $ roscd simon_says
+
+## 2.1 Test if connection is OK:  
+> $ rostopic list  
+> $ rostopic echo /odom  
+> $ roslaunch turtlebot3_teleop turtlebot3_teleop_key.launch
+
+## 2.2 Test realsense camera
+> $ roslaunch simon_says rviz_realsense.launch
+
+## 2.3 Test control algorithm
+In [src/turtlebot_control/lib_turtlebot.py](src/turtlebot_control/lib_turtlebot.py), set "IF_IN_SIMULATION = False".  
+Then:
+> $ rosrun simon_says move_turtle_to_pose.py
+
+## 2.4 Use recorded images to replace RealSense Camera
+
+rosrun this script: [src/realsense_io/fake_rgbd_image_publisher.py](src/realsense_io/fake_rgbd_image_publisher.py)  
+It reads in color/depth images from the disk and publishes them with the same format as RealSense Camera.
+
+
+# 3. Other commands for testing
+
+* Test robot controller in simulation
+
+    In [lib_turtlebot.py](src/turtlebot_control/lib_turtlebot.py), set "**IF_IN_SIMULATION** = True" to test the robot in simulation.  
+    In [move_turtle_to_pose.py](src/turtlebot_control/move_turtle_to_pose.py), in the **"def main("** function, select a test case to test.
+
+    Then:  
+    > $ roslaunch turtlebot3_gazebo turtlebot3_empty_world.launch  
+    > $ rosrun simon_says move_turtle_to_pose.py   

src/detection/yolo/data/README.md

@@ -0,0 +1,5 @@
+Please download data from:  
+> https://drive.google.com/file/d/1u_AfKcUOVgfUBVS6nl4aHkO1MDNLI2u1/view?usp=sharing
+
+and unzip it to this README's folder, which is:  
+> src/detection/yolo/data/

src/detection/yolo/weights/README.md

@@ -0,0 +1,5 @@
+Please download weight file from:  
+> https://drive.google.com/file/d/1fBtAwaFmPLNaG8jvCsxA9lUXfPeT-Fpq/view?usp=sharing
+
+and unzip it to this README's folder, which is:  
+> src/detection/yolo/weights/

src/lib_for_main.py

@@ -56,7 +56,7 @@ def realsense_camera_link_to_color_image():
     @staticmethod
     def turtlebot_to_camera():
         x_dis = 0.082 # 0.055 ~ 0.1. TODO: Determine this value
-        y_dis = 0 # camera is placed right in front of the turtlebot
+        y_dis = 0 # camera is placed right in front of the Turtlebot
         z_dis = 0.01 # 0.01 ~ 0.02. This value doesn't matter at all. In my case, I don't care the height
         pos = [x_dis, y_dis, z_dis]
         R = np.identity(3)

src/speech_recognition/src/s0_record_audio.sh

@@ -0,0 +1,2 @@
+#!/bin/bash
+python utils/lib_record_audio.py --device 0