{"id":78,"date":"2021-11-22T20:20:23","date_gmt":"2021-11-22T20:20:23","guid":{"rendered":"https:\/\/petersen-projects.com\/?p=78"},"modified":"2021-11-23T11:20:49","modified_gmt":"2021-11-23T11:20:49","slug":"moving-in-vr-with-machine-learning","status":"publish","type":"post","link":"https:\/\/petersen-projects.com\/?p=78","title":{"rendered":"Moving in VR with Machine Learning"},"content":{"rendered":"\n<p>Last time we created an ESP32 controller to move in VR.<\/p>\n\n\n\n<p>But I wanted to try something different&#8230; This time, with machine learning, which I&#8217;ve never used before.<\/p>\n\n\n\n<p>My goal was to gather data from myself in an idle state and also moving. I&#8217;ve changed some of the previous code and created a new repo to contain the changes that I&#8217;ve made (if you&#8217;re interested).<\/p>\n\n\n\n<p>Previous post: <a href=\"https:\/\/petersen-projects.com\/?p=30\" target=\"_blank\" rel=\"noreferrer noopener\">https:\/\/petersen-projects.com\/?p=30<\/a><\/p>\n\n\n\n<p>Repo to the code: <a rel=\"noreferrer noopener\" href=\"https:\/\/github.com\/petersen-projects\/openvr-walker-machine-learning\" target=\"_blank\">https:\/\/github.com\/petersen-projects\/openvr-walker-machine-learning<\/a><\/p>\n\n\n\n<p>My video explaining the project (in portuguese):<\/p>\n\n\n\n<figure class=\"wp-block-embed aligncenter is-type-video is-provider-youtube wp-block-embed-youtube wp-embed-aspect-16-9 wp-has-aspect-ratio\"><div class=\"wp-block-embed__wrapper\">\n<iframe loading=\"lazy\" title=\"Movimentando-se em VR com Machine Learning\" width=\"696\" height=\"392\" src=\"https:\/\/www.youtube.com\/embed\/q8f-V9Wff8w?feature=oembed\" frameborder=\"0\" allow=\"accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture\" allowfullscreen><\/iframe>\n<\/div><\/figure>\n\n\n\n<p>So let&#8217;s get to it!<\/p>\n\n\n\n<p class=\"has-large-font-size\">Architecture<\/p>\n\n\n\n<p>For this example, the achitecture is as follows (same as before):<\/p>\n\n\n\n<div class=\"wp-block-image\"><figure class=\"aligncenter size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"503\" height=\"361\" src=\"https:\/\/petersen-projects.com\/wp-content\/uploads\/2021\/11\/image-3.png\" alt=\"\" class=\"wp-image-90\" srcset=\"https:\/\/petersen-projects.com\/wp-content\/uploads\/2021\/11\/image-3.png 503w, https:\/\/petersen-projects.com\/wp-content\/uploads\/2021\/11\/image-3-300x215.png 300w\" sizes=\"auto, (max-width: 503px) 100vw, 503px\" \/><\/figure><\/div>\n\n\n\n<p>So our program works as follows:<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li>ESP32 reads data from the MPU6050 and sends it via socket to our python server;<\/li><li>Python Server checks to see if I&#8217;m moving using Machine Learning, and based on that, presses keyboard buttons accordingly;<\/li><li>SteamVR Driver checks to see if any keyboard presses were made and sends the actions to the game.<\/li><\/ul>\n\n\n\n<p class=\"has-large-font-size\">Step One &#8211; ESP32 Sending Data<\/p>\n\n\n\n<p>I&#8217;ve changed my arduino code, now it only sends the data from the board to the server, via socket. I&#8217;m reading all of the outputs &#8211; Accelerometer X, Y and Z and Gyro X, Y and Z.<\/p>\n\n\n\n<p>All of this gathered data is sent to the server for persisting in a csv type file. I&#8217;ve gathered this data in two ways &#8211; idle and moving.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>void loop() {\n  if (client.connected()) {\n    sensors_event_t a, g, temp;\n    mpu.getEvent(&amp;a, &amp;g, &amp;temp);\n\n    HTTPClient http;\n    String currAccelX = String(a.acceleration.x);\n    String currAccelY = String(a.acceleration.y);\n    String currAccelZ = String(a.acceleration.z);\n    String currGyroX= String(g.gyro.x);\n    String currGyroY= String(g.gyro.y);\n    String currGyroZ= String(g.gyro.z);\n    String result = String(currAccelX + \",\" + currAccelY + \",\" + currAccelZ + \",\" + currGyroX + \",\" + currGyroY + \",\" + currGyroZ);\n    Serial.println(result);\n    client.print(result);\n  } else {\n    client.stop();\n    connectSocket();\n  }\n  Serial.println(\"----\");\n  delay(read_delay);\n}<\/code><\/pre>\n\n\n\n<p>As you can see, the code above is super simple and only gathers all of the info obtained from the MPU-6050 board and sends it directly to our python server.<\/p>\n\n\n\n<p class=\"has-large-font-size\">Step two &#8211; Python for Gathering Data<\/p>\n\n\n\n<p>First of all we have to gather some data. How can we do that?<\/p>\n\n\n\n<p>On the esp32 side no code changes need to be made. The main goal of this device now is only to send data, nothing more.<\/p>\n\n\n\n<p>Python now has 3 scripts which we can call (gather data, create our model and the esp32 controller).<\/p>\n\n\n\n<p>If you check the code that it&#8217;s in my repo, you can see that there&#8217;s a file called <code>gather_data.py<\/code>. This file can be run directly from python3 and will persist all of the data that the board sends to us.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code># For gathering data - set current state.\ncurrent_state = IDLE<\/code><\/pre>\n\n\n\n<p>On the code above, you must set the type of input that we are recording in this session &#8211; IDLE or MOVING. I&#8217;ve put also 5 seconds of wait time just to get things ready and start moving, so the data will be as correct as possible.<\/p>\n\n\n\n<p>The gather_data script will generate a csv file with the chosen name for the session, I&#8217;ve chosen <code>gathering_idle.csv<\/code> and <code>gathering_moving.csv<\/code>, respectively. Both files merged into one, called <code>consolidated_data.csv<\/code>, which contains the data to train our model.<\/p>\n\n\n\n<p class=\"has-large-font-size\">Step Three &#8211; Training our Model<\/p>\n\n\n\n<p>Now that we&#8217;ve gathered all of our data, it&#8217;s time to create our model and persist it.<\/p>\n\n\n\n<p>For this goal, I&#8217;ve created a script called create_model.py, which reads our <code>consolidated_data.csv<\/code> and fits it into a model. I&#8217;m using <code>ExtraTreesClassifier<\/code> (<a href=\"https:\/\/scikit-learn.org\/stable\/modules\/generated\/sklearn.ensemble.ExtraTreesClassifier.html\" target=\"_blank\" rel=\"noreferrer noopener\">https:\/\/scikit-learn.org\/stable\/modules\/generated\/sklearn.ensemble.ExtraTreesClassifier.html<\/a>) algorithm.<\/p>\n\n\n\n<p>After running this script, a new model will be persisted inside the models folder, with the name of your choice. Mine is <code>train-data<\/code>.<\/p>\n\n\n\n<p>The accuracy that I got was of <code>0.99<\/code>, which is way more than I&#8217;ve hoped for.<\/p>\n\n\n\n<p class=\"has-large-font-size\">Step Four &#8211; Changes to the OpenVR Driver<\/p>\n\n\n\n<p>I&#8217;ve refactored a little bit of the previous code, just to keep it a little bit cleaner. The way it works is the same as before, but I&#8217;ll show you what I&#8217;ve done:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>void ControllerDriver::RunFrame()\r\n{\r\n\tfloat currMovSpeed = GetCurrentSpeed();\r\n\t\/\/ Update Y Axis according to the keyboard pressed buttons.\r\n\tUpdateYAxis(joystickYHandle, trackpadYHandle, currMovSpeed);\r\n\t\/\/ Update X Axis according to the keyboard pressed buttons.\r\n\tUpdateXAxis(joystickXHandle, trackpadXHandle, currMovSpeed);\r\n}<\/code><\/pre>\n\n\n\n<p>I&#8217;ve simplified the RunFrame function and broke it into different functions. I think there&#8217;s a lots of room to improve the code, but I&#8217;m happy the way it is for now (it works :B).<\/p>\n\n\n\n<p class=\"has-large-font-size\">Step Four &#8211; Using the Controller<\/p>\n\n\n\n<p>After gathering the data and creating our model, it&#8217;s time to use our controller. But does it work?<\/p>\n\n\n\n<p>Yes! In my video you can see a little bit of the device working. The results were even better than the last time, when I&#8217;ve used only a threshold. Almost no false positives and the gaming section went smoothly. Overall, I&#8217;m very pleased with the results. I&#8217;ll definitely use this device to move in my VR games from now on.<\/p>\n\n\n\n<p class=\"has-large-font-size\">Super Ultra Advanced Gear<\/p>\n\n\n\n<p>How do I supply energy to the device? How do I strap it in my body?<\/p>\n\n\n\n<div class=\"wp-block-image\"><figure class=\"aligncenter size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"680\" src=\"https:\/\/petersen-projects.com\/wp-content\/uploads\/2021\/11\/55ce6550-00ca-44d9-986d-39767a7f257b-1024x680.jpg\" alt=\"\" class=\"wp-image-111\" srcset=\"https:\/\/petersen-projects.com\/wp-content\/uploads\/2021\/11\/55ce6550-00ca-44d9-986d-39767a7f257b-1024x680.jpg 1024w, https:\/\/petersen-projects.com\/wp-content\/uploads\/2021\/11\/55ce6550-00ca-44d9-986d-39767a7f257b-300x199.jpg 300w, https:\/\/petersen-projects.com\/wp-content\/uploads\/2021\/11\/55ce6550-00ca-44d9-986d-39767a7f257b-768x510.jpg 768w, https:\/\/petersen-projects.com\/wp-content\/uploads\/2021\/11\/55ce6550-00ca-44d9-986d-39767a7f257b.jpg 1280w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><figcaption> Advanced gear for advanced users <\/figcaption><\/figure><\/div>\n\n\n\n<p>I use the super fancy equipment shown in the photo above. A super simple running belt to put the device in, an old Raspberry Pi 2 box to keep the hardware from moving while I use it and a power bank. Hey, don&#8217;t judge, it works fine :B<\/p>\n\n\n\n<p class=\"has-large-font-size\">Softwares that track movement<\/p>\n\n\n\n<p>I recommend doing this project for fun, but you can also find some options that achieve the same without any work:<\/p>\n\n\n\n<p><a href=\"https:\/\/store.steampowered.com\/app\/798810\/Natural_Locomotion\/\">https:\/\/store.steampowered.com\/app\/798810\/Natural_Locomotion\/<\/a><\/p>\n\n\n\n<p><a href=\"https:\/\/store.steampowered.com\/app\/1143750\/VRocker\/\">https:\/\/store.steampowered.com\/app\/1143750\/VRocker\/<\/a><\/p>\n\n\n\n<p><a href=\"https:\/\/store.steampowered.com\/app\/1366950\/Driver4VR\/\">https:\/\/store.steampowered.com\/app\/1366950\/Driver4VR\/<\/a><\/p>\n\n\n\n<p>Super interesting projects, I recommend checking them out!<\/p>\n\n\n\n<p class=\"has-large-font-size\">Whats In for the Future<\/p>\n\n\n\n<p>I&#8217;ve seen body recognition done with a webcam, which is something that I&#8217;m interested in thinkering with.<\/p>\n\n\n\n<p>Probably in the future I&#8217;ll try to do something with that as well. As for now, I&#8217;m gonna play a lot and enjoy the rest of my vacations (yeah, I can&#8217;t stop coding even when I&#8217;m supposed to rest :O)<\/p>\n\n\n\n<p>That&#8217;s it! See ya next time!!<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Last time we created an ESP32 controller to move in VR. But I wanted to try something different&#8230; This time, with machine learning, which I&#8217;ve never used before. My goal was to gather data from myself in an idle state and also moving. I&#8217;ve changed some of the previous code and created a new repo [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-78","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"_links":{"self":[{"href":"https:\/\/petersen-projects.com\/index.php?rest_route=\/wp\/v2\/posts\/78","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/petersen-projects.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/petersen-projects.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/petersen-projects.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/petersen-projects.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=78"}],"version-history":[{"count":39,"href":"https:\/\/petersen-projects.com\/index.php?rest_route=\/wp\/v2\/posts\/78\/revisions"}],"predecessor-version":[{"id":122,"href":"https:\/\/petersen-projects.com\/index.php?rest_route=\/wp\/v2\/posts\/78\/revisions\/122"}],"wp:attachment":[{"href":"https:\/\/petersen-projects.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=78"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/petersen-projects.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=78"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/petersen-projects.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=78"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}