{"id":30,"date":"2021-11-16T14:13:53","date_gmt":"2021-11-16T14:13:53","guid":{"rendered":"https:\/\/petersen-projects.com\/?p=30"},"modified":"2021-11-18T12:05:29","modified_gmt":"2021-11-18T12:05:29","slug":"walking-in-vr-with-esp32","status":"publish","type":"post","link":"https:\/\/petersen-projects.com\/?p=30","title":{"rendered":"Walking in VR With ESP32"},"content":{"rendered":"\n<p>Hey, hey, hey!<\/p>\n\n\n\n<p>So I&#8217;ve recently entered in the VR World.<\/p>\n\n\n\n<p>But as I was playing, I&#8217;ve just couldn&#8217;t feel the immersive experience using the analogic stick from the controller.<\/p>\n\n\n\n<p>I&#8217;ve seen some expensive treadmills that maps your walking directly to the games and I had an idea.<\/p>\n\n\n\n<p>I surely can&#8217;t create a safe enough treadmill for me to walk on, but I&#8217;m sure I can think of something&#8230;<\/p>\n\n\n\n<p>Spoiler alert: I thought of something!<\/p>\n\n\n\n<p><\/p>\n\n\n\n<p>At the end of this post I&#8217;ll share all of my ideas and why I&#8217;ve decided to stick with this main solution.<\/p>\n\n\n\n<p>All of my code is in my github repo, here: <a href=\"https:\/\/github.com\/petersen-projects\/openvr-walker-with-python\" target=\"_blank\" rel=\"noreferrer noopener\">https:\/\/github.com\/petersen-projects\/openvr-walker-with-python<\/a><\/p>\n\n\n\n<p class=\"has-large-font-size\">Step One: Writing my OpenVR Driver<\/p>\n\n\n\n<p>So in order to make all of this work, firstly I must add my own OpenVR Driver to SteamVR.<\/p>\n\n\n\n<p>But how can we do that? <\/p>\n\n\n\n<p>I strongly suggest you take a look at this video from FinallyFunctional, as I created my own driver following his steps:<\/p>\n\n\n\n<figure class=\"wp-block-embed is-type-video is-provider-youtube wp-block-embed-youtube wp-embed-aspect-16-9 wp-has-aspect-ratio\"><div class=\"wp-block-embed__wrapper\">\n<iframe loading=\"lazy\" title=\"Open VR Driver Tutorial\" width=\"696\" height=\"392\" src=\"https:\/\/www.youtube.com\/embed\/LzEIOBnbC8k?feature=oembed\" frameborder=\"0\" allow=\"accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture\" allowfullscreen><\/iframe>\n<\/div><\/figure>\n\n\n\n<p>Also, his github repo is here: <a href=\"https:\/\/github.com\/finallyfunctional\/openvr-driver-example\" target=\"_blank\" rel=\"noreferrer noopener\">https:\/\/github.com\/finallyfunctional\/openvr-driver-example<\/a><\/p>\n\n\n\n<p>I&#8217;ve changed the RunFrame method to the following:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>const float STOPPED = 0.0;\nconst float WALK_SPEED = 0.5;\nconst float RUN_SPEED = 0.95;\n\nvoid ControllerDriver::RunFrame()\n{\n\t\/\/ Set movement speed.\n\tfloat currMovSpeed = WALK_SPEED;\n\tif (GetKeyState('R') &amp; 0x8000) {\n\t\tcurrMovSpeed = RUN_SPEED;\n\t}\n\n\t\/\/ Direction of movement.\n\tif (GetKeyState('W') &amp; 0x8000) {\n\t\tVRDriverInput()-&gt;UpdateScalarComponent(joystickYHandle, currMovSpeed, 0);\n\t\tVRDriverInput()-&gt;UpdateScalarComponent(trackpadYHandle, currMovSpeed, 0);\n\t}\n\telse if (GetKeyState('S') &amp; 0x8000) {\n\t\tVRDriverInput()-&gt;UpdateScalarComponent(joystickYHandle, -currMovSpeed, 0);\n\t\tVRDriverInput()-&gt;UpdateScalarComponent(trackpadYHandle, -currMovSpeed, 0);\n\t}\n\telse if (!(\n\t\t(GetKeyState('W') &amp; 0x8000) &amp;&amp;\n\t\t(GetKeyState('S') &amp; 0x8000)\n\t\t)) {\n\t\tVRDriverInput()-&gt;UpdateScalarComponent(joystickYHandle, 0.0f, 0); \/\/move forward\n\t\tVRDriverInput()-&gt;UpdateScalarComponent(trackpadYHandle, 0.0f, 0); \/\/move foward\n\t}\n\n\t\/\/ We can also move sideways.\n\tif (GetKeyState('A') &amp; 0x8000) {\n\t\tVRDriverInput()-&gt;UpdateScalarComponent(joystickXHandle, -currMovSpeed, 0);\n\t\tVRDriverInput()-&gt;UpdateScalarComponent(trackpadXHandle, -currMovSpeed, 0);\n\t}\n\telse if (GetKeyState('D') &amp; 0x8000) {\n\t\tVRDriverInput()-&gt;UpdateScalarComponent(joystickXHandle, currMovSpeed, 0);\n\t\tVRDriverInput()-&gt;UpdateScalarComponent(trackpadXHandle, currMovSpeed, 0);\n\t}\n\telse if (!(\n\t\t(GetKeyState('A') &amp; 0x8000) &amp;&amp;\n\t\t(GetKeyState('D') &amp; 0x8000)\n\t\t)) {\n\t\tVRDriverInput()-&gt;UpdateScalarComponent(joystickXHandle, 0.0f, 0);\n\t\tVRDriverInput()-&gt;UpdateScalarComponent(trackpadXHandle, 0.0f, 0);\n\t}\n\n}<\/code><\/pre>\n\n\n\n<p>The way the above code works is, when we receive an input from the keyboard, we do something with the joystick.<\/p>\n\n\n\n<figure class=\"wp-block-table is-style-stripes\"><table><tbody><tr><td><strong>Key<\/strong><\/td><td><strong>Action<\/strong><\/td><\/tr><tr><td><strong>W<\/strong><\/td><td>Move Forward<\/td><\/tr><tr><td><strong>R<\/strong><\/td><td>While moving, use Full Speed (RUN)<\/td><\/tr><tr><td><strong>A<\/strong><\/td><td>Strafe Left<\/td><\/tr><tr><td><strong>D<\/strong><\/td><td>Strafe Right<\/td><\/tr><tr><td><strong>S<\/strong><\/td><td>Move backwards<\/td><\/tr><\/tbody><\/table><figcaption>Keys and respective actions<\/figcaption><\/figure>\n\n\n\n<p>I&#8217;ve kept the support for multiple types of movement, but when I was testing those let me kinda dizzy, so when I&#8217;m playing I only move forward.<\/p>\n\n\n\n<p>Now that our driver is created and installed, it should be shown on Steam as follows:<\/p>\n\n\n\n<div class=\"wp-block-image\"><figure class=\"aligncenter size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"309\" height=\"155\" src=\"https:\/\/petersen-projects.com\/wp-content\/uploads\/2021\/11\/image.png\" alt=\"\" class=\"wp-image-41\" srcset=\"https:\/\/petersen-projects.com\/wp-content\/uploads\/2021\/11\/image.png 309w, https:\/\/petersen-projects.com\/wp-content\/uploads\/2021\/11\/image-300x150.png 300w\" sizes=\"auto, (max-width: 309px) 100vw, 309px\" \/><figcaption>New controller added as a &#8220;C&#8221; in SteamVR<\/figcaption><\/figure><\/div>\n\n\n\n<p>You can also test your new controller by going to SteamVR Settings &gt; Controllers &gt; Test Controller<\/p>\n\n\n\n<p>Choose your newly added controller and press any supported key on your keyboard, it should show that is moving forward \/ backwards \/ strafing left or right.<\/p>\n\n\n\n<p class=\"has-large-font-size\">Step Two &#8211; Writing a simple Python Socket<\/p>\n\n\n\n<p>After we created and installed our OpenVR Driver, we now have to receive data from our ESP32.<\/p>\n\n\n\n<p>I&#8217;ve decided to go to a socket approach, but feel free to do in any way you like.<\/p>\n\n\n\n<p>Our Python server receives data from our ESP32, and if we&#8217;re moving, Python will then press the according key in our keyboard.<\/p>\n\n\n\n<p>Simple as that.<\/p>\n\n\n\n<p>The moving code consists of 4 numbers:<\/p>\n\n\n\n<figure class=\"wp-block-table is-style-stripes\"><table class=\"has-black-color has-white-background-color has-text-color has-background\"><tbody><tr><td><strong>Number<\/strong><\/td><td><strong>Action<\/strong><\/td><\/tr><tr><td>0\/1<\/td><td> <em>Stopped \/ Moving<\/em> <\/td><\/tr><tr><td>0\/1<\/td><td> <em>Walking \/ Running<\/em> <\/td><\/tr><tr><td>0\/1\/2<\/td><td> <em>Center \/ Forward \/ Backwards <\/em><\/td><\/tr><tr><td>0\/1\/2<\/td><td> Center \/ Strafe Left \/ Strafe Right <\/td><\/tr><\/tbody><\/table><figcaption>Numbers and respective actions<\/figcaption><\/figure>\n\n\n\n<p>A lot of these variables are unused since I got dizzy \/ sick when I&#8217;ve tried to use them. So I&#8217;ve decided to only run forward.<\/p>\n\n\n\n<p>Using the table above, an example would be:<\/p>\n\n\n\n<p>1110 = Moving, Running, Forward, Center<\/p>\n\n\n\n<p class=\"has-large-font-size\">Step Three &#8211; Assembling and programming our ESP32<\/p>\n\n\n\n<p>I&#8217;m using an MPU-6050 to detect when I&#8217;m walking. I&#8217;ve added some checks to see if I&#8217;m moving, a threshold on my code serves that purpose with a simple check logic (if gyro and accelerometer surpasses X, the controller should input forward):<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>void mpu_read() {\n  \/* Get new sensor events with the readings *\/\n  sensors_event_t a, g, temp;\n  mpu.getEvent(&amp;a, &amp;g, &amp;temp);\n\n  accelXCurrValue = abs(a.acceleration.x);\n  float accelXDiff = accelXCurrValue - accelXOldValue;\n  float accelXDiffAbs = abs(accelXDiff);\n\n  gyroXCurrValue = abs(g.gyro.x);\n  float gyroXDiff = gyroXCurrValue - gyroXOldValue;\n  float gyroXDiffAbs = abs(gyroXDiff);\n  \n  if (gyroXDiffAbs > gyroXMoveThreshold &amp;&amp; accelXDiffAbs > accelXMoveThreshold) {\n    currentMovement = MOVING;\n  } else {\n    currentMovement = STOPPED;\n  }\n\n\n  gyroXOldValue = gyroXCurrValue;\n  accelXOldValue = accelXCurrValue;\n}<\/code><\/pre>\n\n\n\n<p>The code above uses the gyro and the accelerometer, both in X axis. Both are important to map the way I walk \/ jog. The thresholds are absolute values, to make the checking process easier.<\/p>\n\n\n\n<p><em>* If you want to move sideways or backwards, use Y axis and Z axis, respectively &#8211; but be careful because it can cause sickness \/ dizziness.<\/em><\/p>\n\n\n\n<p>The plot below shows when I&#8217;m walking \/ jogging, so you can have an idea of how the logic of the thresholds works:<\/p>\n\n\n\n<div class=\"wp-block-image\"><figure class=\"aligncenter size-large is-resized\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/petersen-projects.com\/wp-content\/uploads\/2021\/11\/gyroaccabsvals-1024x507.png\" alt=\"\" class=\"wp-image-58\" width=\"840\" height=\"415\" srcset=\"https:\/\/petersen-projects.com\/wp-content\/uploads\/2021\/11\/gyroaccabsvals-1024x507.png 1024w, https:\/\/petersen-projects.com\/wp-content\/uploads\/2021\/11\/gyroaccabsvals-300x149.png 300w, https:\/\/petersen-projects.com\/wp-content\/uploads\/2021\/11\/gyroaccabsvals-768x380.png 768w, https:\/\/petersen-projects.com\/wp-content\/uploads\/2021\/11\/gyroaccabsvals.png 1505w\" sizes=\"auto, (max-width: 840px) 100vw, 840px\" \/><figcaption><em>Red line = Accelerometer \/ Blue line = Gyro<\/em><\/figcaption><\/figure><\/div>\n\n\n\n<p>I&#8217;ve assembled as follows:<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"767\" src=\"https:\/\/petersen-projects.com\/wp-content\/uploads\/2021\/11\/image-1-1024x767.png\" alt=\"\" class=\"wp-image-51\" srcset=\"https:\/\/petersen-projects.com\/wp-content\/uploads\/2021\/11\/image-1-1024x767.png 1024w, https:\/\/petersen-projects.com\/wp-content\/uploads\/2021\/11\/image-1-300x225.png 300w, https:\/\/petersen-projects.com\/wp-content\/uploads\/2021\/11\/image-1-768x575.png 768w, https:\/\/petersen-projects.com\/wp-content\/uploads\/2021\/11\/image-1.png 1280w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n\n\n\n<p>You can find out more at <a href=\"https:\/\/learn.adafruit.com\/mpu6050-6-dof-accelerometer-and-gyro\/arduino\" target=\"_blank\" rel=\"noreferrer noopener\">https:\/\/learn.adafruit.com\/mpu6050-6-dof-accelerometer-and-gyro\/arduino<\/a><\/p>\n\n\n\n<p class=\"has-large-font-size\">Step four &#8211; Does it work??<\/p>\n\n\n\n<p>I&#8217;m using the game &#8220;GORN&#8221; to test my walk detection with ESP32.<\/p>\n\n\n\n<p>So now that we have all ready, let&#8217;s test it!<\/p>\n\n\n\n<p>I&#8217;ve thought of putting this device in my ankles, with two MPU-6050. But I&#8217;ve discarded the idea for simplicity. Perhaps in the future I&#8217;ll go back to this, but for now, having it in my chest works just fine.<\/p>\n\n\n\n<figure class=\"wp-block-embed is-type-video is-provider-youtube wp-block-embed-youtube wp-embed-aspect-16-9 wp-has-aspect-ratio\"><div class=\"wp-block-embed__wrapper\">\n<iframe loading=\"lazy\" title=\"Walking \/ Jogging with SteamVR + ESP32 + OpenVR\" width=\"696\" height=\"392\" src=\"https:\/\/www.youtube.com\/embed\/ly5nex0q1JM?feature=oembed\" frameborder=\"0\" allow=\"accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture\" allowfullscreen><\/iframe>\n<\/div><\/figure>\n\n\n\n<p>It works well, some false positives but I think thats just a calibration issue. I&#8217;m sure that I&#8217;ll have a lot of fun using this new controller that I&#8217;ve made!<\/p>\n\n\n\n<p class=\"has-large-font-size\">What were the other options that I&#8217;ve tried<\/p>\n\n\n\n<p>I mainly tried DLL Injection in a process, but discarded the idea because I think I&#8217;ll always have to program new code to different games. I wanted a global solution that would work out of the box with any VR game that uses OpenVR.<\/p>\n\n\n\n<p>I also went to try some reverse engineering in unity dll&#8217;s, tried to debug processes and so on. But I gave up after three days getting nowhere.<\/p>\n\n\n\n<p class=\"has-large-font-size\">Whats in for the future and final thoughts<\/p>\n\n\n\n<p>I&#8217;m thinking of moving the MPU-6050 to my ankles and add a new one, to track both legs. As for now, I really don&#8217;t know if I&#8217;m able to connect more than one module to the ESP32 as I didn&#8217;t find anything mentioning it.<\/p>\n\n\n\n<p>As it stands it&#8217;s a pretty good solution, more immersive and helped a lot with dizziness \/ move sickness when using the analogic from the controller to move. I really liked this new controller that I&#8217;ve made.<\/p>\n\n\n\n<p>Thats it, I hope you&#8217;ve liked this post and I hope to have helped you!<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Hey, hey, hey! So I&#8217;ve recently entered in the VR World. But as I was playing, I&#8217;ve just couldn&#8217;t feel the immersive experience using the analogic stick from the controller. I&#8217;ve seen some expensive treadmills that maps your walking directly to the games and I had an idea. I surely can&#8217;t create a safe enough [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-30","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"_links":{"self":[{"href":"https:\/\/petersen-projects.com\/index.php?rest_route=\/wp\/v2\/posts\/30","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/petersen-projects.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/petersen-projects.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/petersen-projects.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/petersen-projects.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=30"}],"version-history":[{"count":30,"href":"https:\/\/petersen-projects.com\/index.php?rest_route=\/wp\/v2\/posts\/30\/revisions"}],"predecessor-version":[{"id":76,"href":"https:\/\/petersen-projects.com\/index.php?rest_route=\/wp\/v2\/posts\/30\/revisions\/76"}],"wp:attachment":[{"href":"https:\/\/petersen-projects.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=30"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/petersen-projects.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=30"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/petersen-projects.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=30"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}