Lode: Vicon

This was a project I did for a client of 8D Games (December 2019). The company called Lode asked to combine VR with mo-cap. Lode is specialized in the complete spectrum of medical ergometry. And is renowned as a manufacturer of high quality ergometers. The Lode product range varies from bicycle ergometers and treadmills to recumbent, arm and supine ergometers and ergometry software.

The purpose of the software was to demonstrate the potential use of combining the HTC Vive (VR device) with a Vicon system (mo-cap camera's and software) to be beneficial for rehabilitation. A requirement was that the software was able to record and help with analyzing patient movements. As it was to be used to demonstrate the use of the systems, it was required that it was quick to setup a user and should only last for a couple of minutes to show the full potential.

Before anything was being programmed, we [collegues from 8D Games and I] brainstormed for game ideas. There were ideas of a dance game, or a Crash Bash type game. But, at this state, it was important to put as little effort as possible with the highest amount of impact. Thus game the idea of a boxing game; few mechanics, direct play and engaging. This would be the first target as a demo.

The day came that we got the HTC Vive and Vicon system from Lode. They explained to me how the system worked, the initial setup and how to calibrate the mo-cap system. After they left, I said my colleagues goodbye and transferred all my stuff to the VR and mo-cap setup. Now I had a bit over 30 hours to create a boxing game demo in Unity.

First and foremost, the tactic I always use: Where am I now? Where do I need to be? And how do I get there? To answer the first question -Where am I now?- I have to know what data I've got in my hands. I started to read the documentation and logged the data of the HTC Vive and Vicon system. For both systems I was able to read the position and rotation of each component.

Where do I need to be? I need a boxing game, but before anything I have to make sure the data from the VR device and mo-cap system can be merged. This was the first goal: getting the transformation matrix of both system to convert to Unity game engine space. I glued some passive markers to the VR headset. This way I could track the position and rotation of the head in both systems. There is one problem remaining though: What is up?

I tried to solve this using the data of the HTC Vive system to figure out the up vector for the Vicon system. But, under these tight time constrains I had to give up this approach. Quickly I did plan B: adding a calibration system. If the person is in a T-pose, I could figure out what left and right is. Most humans can't rotate their head 180 degrees, so I also knew the forward axis. This gives enough information to solve the remaining information and create the transformation matrix. After that I simplified the solution, which can be found below.

Now I had the capability to convert both systems to one transformation space. With the remaining time, it was just building as much as I could. I added some rigidbodies, sound and particle effects. You now could hit the rigidbodies and get feedback.

To improve immersion, I imported the Unity robot character into the project. Did some test with the built-in IK systems to resolve the missing joints that were not recorded by the mo-cap system. This immediately made the demo feel so much more than just VR+. You could now see your body, but as a robot.

To get more serious and create something useful for the client I added some commands to record users. The recording could then be saved. When the recording was played back a character was created and performed frame by frame the movements that were recorded. This made it possible to study the recorded movements.

At this time my colleagues were missing me. And the deadline was due. The client was happy and really interested in the software that has been made.

I would like to have gotten more time on this hardware and to improve what I have made. Like removing the need of callibration and figuring out the transformation matrices based on just the two systems; using multiple frames with rotation, to match these. With this system a lot of new games can be explored; as the feeling of seeing your body and feet in VR is undescribable.

More info is available here: 8D Games

public error Calibrate()
{
	double[] lh;
	{
		var output = Client.GetSegmentGlobalTranslation("L_Hand", "L_Hand");
		if (output.Result == Result.Success && !output.Occluded)
			lh = output.Translation;
		else return new error("Calibration failed: Could not get data from left hand. Make sure L_Hand is defined in the Vicon software.");
	}

	double[] rh;
	{
		var output = Client.GetSegmentGlobalTranslation("R_Hand", "R_Hand");
		if (output.Result == Result.Success && !output.Occluded)
			rh = output.Translation;
		else return new error("Calibration failed: Could not get data from right hand. Make sure R_Hand is defined in the Vicon software.");
	}

	double[] vp;
	{
		var output = Client.GetSegmentGlobalTranslation("Vive", "Vive");
		if (output.Result == Result.Success && !output.Occluded)
			vp = output.Translation;
		else return new error("Calibration failed: Could not get data from Vive");
	}
	
	var relativeToLocal = default(Matrix4x4);
	var positionOffset = Vector3.zero;
	{
		var right = new Vector3((float)(rh[0] - lh[0]), 0, (float)(rh[2] - lh[2]));
		right.Normalize();
		
		var up = Vector3.up;
		var forward = Vector3.Cross(up, right);
		
		relativeToLocal.SetRow(0, right);
		relativeToLocal.SetRow(1, up);
		relativeToLocal.SetRow(2, forward);
		relativeToLocal.SetRow(3, new Vector4(0, 0, 0, 1));

		positionOffset.x = (float)(vp[0] * MilliMeter);
		positionOffset.y = (float)(vp[1] * MilliMeter);
		positionOffset.z = (float)(vp[2] * MilliMeter);
		positionOffset -= forward * 0.1f;
	}
	
	var localToWorld = Camera.worldToLocalMatrix;
	var localOffset = Vector3.zero;
	{
		var right = localToWorld.GetRow(0);
		right.y = 0;
		right.Normalize();

		var forward = localToWorld.GetRow(2);
		forward.y = 0;
		forward.Normalize();

		localToWorld.SetRow(0, right);
		localToWorld.SetRow(1, new Vector4(0, 1, 0, 0));
		localToWorld.SetRow(2, forward);

		var cp = localToWorld.GetRow(3);
		cp.x = -cp.x;
		cp.y = -cp.y;
		cp.z = -cp.z;
		LocalOffset = cp;

		localToWorld.SetRow(3, new Vector4(0, 0, 0, 1));
		localToWorld = localToWorld.inverse;
	}

	RelativeToLocal = relativeToLocal;
	LocalToWorld = localToWorld;
	PositionOffset = positionOffset;
	LocalOffset = localOffset;

	var json = new Json.Object
	{
		{"relative_to_local", Json.ParseValue(RelativeToLocal) },
		{"local_to_world", Json.ParseValue(LocalToWorld) },
		{"position_offset", Json.ParseValue(PositionOffset) },
		{"local_offset", Json.ParseValue(LocalOffset) },
	}.ToString();
	PlayerPrefs.SetString(PrefsKey, json);
	PlayerPrefs.Save();
	return error.ok;
}

void LateUpdate()
{
	// ...
	
	for (var i = 0; i < Nodes.Length; i++)
	{
		if (!ApplyOrientation(ref Nodes[i], Client, 1.0f, time))
		{
			Nodes[i].position += Nodes[i].velocity * delta;
			Nodes[i].velocity *= 0.9f;
		}
		
		var node = Nodes[i];
		var position = node.position;
		var relativePosition = position - PositionOffset;
		var localPosition = RelativeToLocal * new Vector4(relativePosition.x, relativePosition.y, relativePosition.z, 1);
		localPosition.x += LocalOffset.x;
		localPosition.y += LocalOffset.y;
		localPosition.z += LocalOffset.z;
		
		var worldPosition = (Vector3)(LocalToWorld * localPosition);
		node.transform.position = worldPosition;
	}
}

public static bool ApplyOrientation(ref Orientation orientation, Client client, float scale, float time)
{
	var name = orientation.name;
	var output = client.GetSegmentGlobalTranslation(name, name);
	if (output.Result == Result.Success && !output.Occluded)
	{
		var translation = output.Translation;
		var position = new Vector3(
			scale * (float)(translation[0] * MilliMeter),
			scale * (float)(translation[1] * MilliMeter),
			scale * (float)(translation[2] * MilliMeter));
		var velocity = (position - orientation.position) / (time - orientation.time);
		orientation.position = position;
		orientation.velocity = velocity;
		orientation.time = time;
		return true;
	}
	return false;
}