For my testing I wanted to test three things: How fast a player is able to perform each feature in the game without being told how to do it, How players react to the interaction task, and How long it takes them to complete the course. I would sit each player down and instruct them on how to open the application, and told them that while they are playing they should speak their thoughts out loud. I then would explain that you can move around the course with your hand, and that their goal would be to collect coins, complete the interaction task before every banner, and reach the finish line.

Caden

Caden was the first tester, and he managed to complete the course in 10 minutes and 37 seconds. After he started the application, it took him only 1 second to learn how to rotate the hand, as he looked left and right once the game started. He was a little surprised to be rotating at first, but caught on and begun moving the hand forward after about 0:10. Caden made sure to grab every coin he could for the first part of the course, and looked to be moving fairly rapidly throgh it also. When he came to the first interaction task, he almost ran right through it, but stopped and turned around once he saw the buttons appear. He tried to run into them at first, but when that didn’t work he tried using his left hand. Seeing that his left hand didn’t work, he went back to trying his right, and then flipped it, triggering the mode change at 1:52. He started the task, pointed towards the object, and grabbed it at 2:10. He had some trouble with the first object as he was getting used to grabbing and moving, and accidentally skipped the third object by pressing done, but in the end he completed the first interaction task by 4:03. The rest of the course was what I expected, but when he reached the floating coins, he did an almost perfect jump at 7:51. I was very surprised as he hadn’t done a single jump the entire time, and he managed to get almost every coin in the arc of the jump. After he finished, his biggest complaint was how shakey the object grabbing and moving was, and that it you could accidentally have your hand over the done button and skip an object, but otherwise he said he liked moving around and how you could jump.

Thomas

Thomas was able to complete the course in 9 minutes and 41 seconds. Like Caden, he rotated the hand within the first 5 seconds, and then begun moving forward at 0:13. As he was moving to collect coins, he accidentally triggered a jump at 0:39, and was very surprised to suddenly be in the air. He then begun to try and trigger another jump, and spent some time jumping in the air. He continued to collect the final bit of coins and reached the interaction task, and changed to the interaction mode at 1:40. Thomas grabbed the first object at 2:20, but didn’t try to perfectly allign the objects like Caden attempted, and finished the interaction task at 3:28. After he passed through the banner, he continued collecting coins and jumping around, but accidentlly gained too much speed and zoomed past the second interaction task and into the banner, which got rid of the task. He asked if he needed to restart but I told him it was fine, and he continued but went noticeably slower through the final leg of the course. When he reached the last interaction task, he tried to align the objects to the best of his ability, but was having difficulties moving it to the correct angle. His final thoughts were again that moving objects was a hassle with how much you have to move if you want to get it in the correct position, and that rotating is a little too fast for him.

Sally

Our final tester, Sally completed the course after 11 minutes and 23 seconds. Once the game started, she rotated the hand after 0:03, looking around the environment like Caden did. She found that rotating was making her a little dizzy, but claimed that because she was sitting down it wasn’t so bad. She began moving at the 0:16, following the coins, but said that moving the hand was tiring. She grabbed her wrist, and triggered the interaction mode at 0:50. She was suprised at first, but quickly changed back to moving mode, and then spent some time switching between the two modes. Once she reached the first interaction task, she changed modes and begun working on the first object, but was a bit too far away from the object. She then accidentally swiped over the done button, moving her to the next object. She switched modes to get a bit closer to the task, and then grabbed the second object at 2:51. She carefully aligned the objects and finished the interaction task at 4:29. She continued through the course, going for every coin she could, and when she came to an interaction task she would align the objects to the best of her ability. She would move around, both in-game and in the real world, and approach the objects from different angles to get them in the best position possible. After completing the second interaction task and going down the hill, she noticed the floating coins and stopped right before the beginning of the arc. She then attempted to jump by lifting her hand rapidly, then tried again by fanning her fingers out, but her fingers were not in the jump collider before so it didn’t work. She tried again with her hand beginning in a fist, and this time it triggered a jump at 8:54. She spent some time perfecting her technique, and then begun collecting the floating coins. She didn’t get them all in one jump, as she had not built up enough speed going into the jump, but she managed to grab them all after 4 jumps. After completing the final task and then the course, she said that it was a bit strenuous to keep moving your fingers to get around, but thought the movement system was fun. Like the other playtesters, her biggest complaint was the challenge of aligning the objects for the task. She felt that it was too much effort while also being too shakey.

Overall, I was surprised on how fast the playtesters were able to get through the course with no prior knowledge of the game and its systems. As I thought, the interaction method would be the biggest complaint of the testers, but they all managed to change modes and grab objects without me guiding them, which was what I cared the most about. It was also interesting, but i guess also expected, that every playtester started rotating within the first 5 seconds of beginning the game. Its only natural to want to get your surroundings before beginning something. It was very fun to see them attempting the course, learning how they can move the fastest, interact, and grab things, but it was great seeing how once they learned something, they instantly adapted to it and begun utilizing it. I think that because they aren’t having to use a controller, and are purely interacting with the game through their hands, it somehow makes the system click faster within the players, but I would need more playtesters and a version with controllers to actually test that.

Here you can see me running through the course to get a better idea of what the playtesters went through.

I first started on creating this mode switching functionality by creating another two colliders above and below where the players tracked hand is. From the hand’s palm I added a raycast, that works almost exactly like the raycast used for rotation. If the ray hits the upper trigger, the hand enters its interaction mode. And when the player points their palm back down and hits the lower trigger, they go back to the moving mode:

 if (Physics.Raycast(ray, out hit, 100f, triggerLayer))
 {
     if (hit.collider.CompareTag("SelectionTrigger"))
     {
         if (currentMode != LocomotionMode.Selection)
         {
             hand.SetActive(false);
             ovrRigTransform.position = hand.transform.position + new Vector3(0f,-0.6f, 0f);
             currentMode = LocomotionMode.Selection;
             if (selectionObject != null)
                 selectionObject.SetActive(true);
             Debug.Log("Selection Mode Activated");
         }
     }
     else if (hit.collider.CompareTag("MovingTrigger"))
     {
         if (currentMode != LocomotionMode.Moving && myGrab.isSelected != true)
         {
             hand.SetActive(true);
             currentMode = LocomotionMode.Moving;
             hand.transform.rotation = Quaternion.Euler(0, hand.transform.rotation.eulerAngles.y, hand.transform.rotation.eulerAngles.z);
             if (selectionObject != null)
                 selectionObject.SetActive(false);
             Debug.Log("Moving Mode Activated");
         }
     }
 }

Here you can get a better idea of the collision triggers and how the ray interacts with them.

For the first version of this interaction mode, I had the moving hand be what you would use to interact with the task, but I had trouble refining its movement and ability to actualy grab and select things. Here you can see that the hand would just touch the buttons instantly and skip past the tasks.

I then decided that I would deactivate the large hand when you enter selection mode, and teleport the camera rig to where the hand was. While in this state, the player would be able to see a long pole coming out of their right hand, and that pole allows the player to press the buttons and grab objects:

Because of these new modes that the hand could be in, I had to go back to some of the previous methods and have them only work while in moving mode, as I didn’t want the player to be rotating while trying to do the interaction tasks. Or have the large hand somehow walk away while they were doing a task.

In order to actually grab and interact with the tasks, I had to modify the MyGrab script that was already included in the parcour to work with the interaction mode. The code that checked when an object entered a trigger collider was moved to a new script called triggerHandler, which would call to myGrab and feed it the object that the player is interacting with. I had to do this because I was having trouble with selected objects not actually following the correct parent or even detecting that something was interacting with them. The problem was that the game object that handles a players hand is layered in a way that the top most parent object doesn’t move exactly with the players real hand, so I had to go down a few layers and attach the script to the correct child. With the game now actually detecting that the hand is trying to interact with something, I needed to be able to move the objects. The OVRHand can automatically detect when a player is pinching their fingers, so I had it that when a player pinched their fingers on an object, it sets that object to be a child of the hand, and thus moves it with the hand.

if (locomotionTechnique != null && locomotionTechnique.currentMode == LocomotionTechnique.LocomotionMode.Selection)
{
    if (isInCollider && selectedObj != null)
    {
        if (!isSelected && hand.GetFingerIsPinching(OVRHand.HandFinger.Index))
        {
            isSelected = true;
            selectedObj.transform.SetParent(anchor.transform, true);
        }
        else if (isSelected && !hand.GetFingerIsPinching(OVRHand.HandFinger.Index))
        {
            isSelected = false;
            selectedObj.transform.SetParent(null, true);
        }
    }
}

Here you can see me trying to orient an object in the correct direction. My hands are fairly shakey so I have a hard time getting it correct, but that is also a flaw with my implementation. The raw hand tracking isn’t fine enough to use, and I needed to smooth it out more.

You can enter move mode and get closer then go back to interaction mode, which helps a bit in getting it correct. But it is still possible to reorient an object in any way, you just need to grab the object from different angles and distances.

What I had to do instead was just create a box collider that is roughly the size of the hand, and we would just move that around the track. I still wanted the player to move their fingers in order to move forward, so I used the same idea that I used for detecting the head rotating: Make a trigger collider where when a finger enters it, the hand moves forward. I added new tags to the capsule colliders on each fingertip, and then begun working on the scripting for the movement. This would be another new method in the locomotion technique script.

First we would keep track of what fingers have entered the collider, and what time that they entered:

 HashSet<Collider> activeFingers = new HashSet<Collider>();
 foreach (Collider collider in colliders)
 {
     if (collider.CompareTag("FingerTip"))
     {
         activeFingers.Add(collider);

         // Record the entry time if the finger is new
         if (!fingerEntryTimes.ContainsKey(collider))
         {
             fingerEntryTimes[collider] = Time.time;
         }
     }
 }

This would be so that we can compare the time the finger entered with the set time that the finger is allowed to keep accelerating the hand:

float targetSpeed = 0;
foreach (var finger in fingerEntryTimes.Keys)
{
    if (Time.time - fingerEntryTimes[finger] <= fingerAccelerationTime)
    {
        targetSpeed += maxSpeed / fingerDetectionThreshold;
    }
}
targetSpeed = Mathf.Clamp(targetSpeed, 0, maxSpeed);

This time limit that each finger has to accelerate the hand was added because the player could just hold their hand in a fist and move around the map, which was not what I intended.

After the target speed is determined, we pass it into another method called MoveHandSmoothly. This would actually transform the position of the hand based on the given target speed:

 float speedDifference = targetSpeed - currentSpeed;

 if (fingerDetected) 
 {
     currentSpeed += Mathf.Sign(speedDifference) * acceleration * Time.deltaTime;
     currentSpeed = Mathf.Clamp(currentSpeed, 0, maxSpeed); 
 }
 else 
 {
     currentSpeed += Mathf.Sign(speedDifference) * deceleration * Time.deltaTime;
     currentSpeed = Mathf.Clamp(currentSpeed, 0, maxSpeed); 
 }

 transform.position += moveDirection * currentSpeed * Time.deltaTime;

But it would also slow the hand down when no fingers are detected:

 else if (!fingerDetected && _currentSpeed > 0)
{
    float speedDifference = targetSpeed - _currentSpeed;
    currentSpeed += Mathf.Sign(speedDifference) * deceleration * Time.deltaTime;
    currentSpeed = Mathf.Clamp(currentSpeed, 0, maxSpeed); 
    transform.position += moveDirection * currentSpeed * Time.deltaTime;
}
else if (!fingerDetected && currentSpeed == 0)
{
    moveDirection = Vector3.zero;
}

Here you can see the fixed movement, where the player must keep moving their fingers if they want to move fast around the course.

And here is a better look at the colliders that are used to move the hand.

Now that the hand is able to run around the course, it also needs to be able to jump, as some of the coins at the end of the course are in the air. What I decided to do was check when all the fingers are within the collision box, and then when the player fans out their hand, it adds upwards force to the hand. I needed to track the time when all fingers are inside the box, and then check if they left in a certain amount of time to trigger the jump:

if (activeFingers.Count == fingerDetectionThreshold) 
{
    allFingersInTime = Time.time; 
}
else if (activeFingers.Count == 0 && allFingersInTime != -1f && Time.time - allFingersInTime <= jumpTimeThreshold)
{
    hand.GetComponent<Rigidbody>().AddForce(Vector3.up * jumpForce, ForceMode.Impulse);
    allFingersInTime = -1f; // Reset the timer
}
else if (activeFingers.Count == 0)
{
    allFingersInTime = -1f; 
}

Here you can see that when all the fingers leave the collider, the hand jumps up! This can be chained one after the other, so you can actually fly around if you want!

Since I want the camera to be following this hand as it is moving around the course, I first tried making the camera rig a child of the hand, but as I tested manually moving the hand around in the editor, the camera movement was way too stiff. It was almost making me sick, so I disconnected the camera from the hand, and decided on moving the camera with a script, so its more smooth and adjustable.

First I would create a offset position behind the hand that the camera would be moving to:

Vector3 offset = -correctedForward * distance + correctedUp * height;
targetPosition = hand.position + offset;

transform.position = Vector3.Lerp(transform.position, targetPosition, followSpeed * Time.deltaTime);

Then I would get the rotation needed to look at the hand, and smoothly nudge the camera back to looking at it:

Vector3 directionToWrist = wrist.position - transform.position;
targetRotation = Quaternion.LookRotation(directionToWrist, correctedUp);

transform.rotation = Quaternion.Slerp(transform.rotation, targetRotation, rotationSpeed * Time.deltaTime);

This now locked the camera to always be a set distance behind and above the hand, and always looking at it, but I still needed a way for the player to actually rotate the hand while they play. I settled on making two colliders on either side of the hand, and raycasting from the center of the players head, so when they turn their head and hit one of the triggers, the hand begins rotating. These two box colliders were made children of the hand game object, and given a tag to designate what side of the hand they were. I needed to script the actual rotation, so I attached the locomotion technique script to the hand, and created a method that is called on update to handle rotation. It would draw the ray, then check if it hit the correct trigger layer, and then act accordingly depending on if it was right or left. Below is how it handles hitting the left rotation trigger, but it is the same idea for the right as well.

if (Physics.Raycast(ray, out hit, 100f, triggerLayer))
{
    if (hit.collider.CompareTag("LeftRotationTrigger"))
    {
        _isRotatingLeft = true;
        _isRotatingRight = false;
    }
}   

if (_isRotatingLeft)
{
    float rotationAmount = -rotationSpeed * Time.deltaTime;
    transform.Rotate(Vector3.up * rotationAmount);
    hand.transform.rotation = _handRotationOffset * Quaternion.Euler(0, transform.eulerAngles.y, 0);
}

And because of the code we made to have the camera always follow the hand, it rotates with it as expected. Here you can see it in action!

Coins are scattered about, and the track isn’t just a straight line, so the player must have a way to alter their direction of movement. The track isn’t completely flat, so we must make sure the locomotion won’t break if we are at an incline. Some coins are in the air, so we need a way to reach those. All of these must be taken into account while we develop the implementation.

There is also the selection task portion of the parkour, where a T-shaped block spawns and it must be oriented in the correct spot. If we are having only one hand to work with, that is our main way to move around the game, how can we also have it be our main mode of object interaction?

You can fly, teleport, pilot giant machines, transform into different forms, etc etc etc.

Of course, many of these things have been done over and over, and with the IARVR project, we need to come up with something unique that isn’t just the first thing that comes to mind.

So the first thing I started doing was thinking, ‘what is something I don’t see in VR games that I would want to experiment with?’ What I thought of first was recording movement and gestures, and then assigning them to inputs that would replay those actions. I imagined a small control room where to your left you would record yourself, lets say grabbing an item, and then you would take a floppy disk of that action and slide in into a main control pannel. This pannel would have multiple buttons, switches, levers, and slots so the player could choose what goes where. Outside of this control room, we would see the robot that all these recorded movements are affecting, and that would be the game. Piloting this robot through replays and tank controls. You would move the legs with two levers like Plankton in Spongebob, you could rotate the body with a big handwheel, but anything with its arms would need to be recorded.

I Liked the idea, but then I started going through everything I would need to implement it. Recording the player, saving those movements, playing BACK those recordings on a different model. I tried messing around with the Unity Animation Recorder, but I was having trouble translating the movements of my hands from the animation file to a kinematic humanoid. If I spent more time playing around with it, i’m sure I could get something to work, but I wanted to experiment more.

My next idea came to me while I was messing around with the no-controller mode on the Quest 3. What if you could pilot your hand like Thing from the Addams Family? Have it skitter and crawl around an environment?

I would have a seperate ridgidbody hand that mimics exactly what your hand is doing, and what I wanted was for it to be moved through pure physics. It would be a challenge to get it moving and steady, and take a good amount of work to master. The player would also need to be able to interact with the environment, and since we would already be using the right hand for movement, what if it could switch between moving around and grabbing things?

This would become my core principle of the project. Purely with the hand tracking data of a player’s right hand, they should be able to do anything needed by the game with only that one hand.

But how can we design a VR locomotion and interaction technique that utilizes only one hand, and is intuitive to the player?

This is what I would be asking myself throughout the project and while I implement the way the hand would move and interact with the game.

Using my available github page, I’ve forked the Reverie, which is a Jekyll-powered theme that helps give the blog some formatting and has a bunch of build in things.

Hopefully, this should make it to where formatting on mobile and on the computer isn’t a pain, and that adding new blog posts is smooth.

In order to upload the gifs and videos for my posts I ended up having to download Github Desktop and set up Github Large File Storage for the repository, which ended up being harder than I thought because this repo was a fork. For some reason, large file storage doesn’t work on forks of public repositories, so I had to disconnect this fork from the original repo, and only then could I upload the large files I needed to upload. Other than that hiccup, this has exposed me to more of the github ecosystem, something that I desperately need to familiarize myself with if I am to continue developing VR applications in the future.