7

I'm trying to get the bone rotations related to their parents, but I end up getting pretty weird angles.

I've tried everything, matrix multiplications, offsets, axis swapping, and no luck.

guard let bodyAnchor = anchor as? ARBodyAnchor else { continue }

let skeleton = bodyAnchor.skeleton
let jointTransforms = skeleton.jointLocalTransforms

for (i, jointTransform) in jointTransforms.enumerated() {

    //RETRIEVE ANGLES HERE
}

In //RETRIEVE ANGLES HERE I've tried different approaches:

let n = SCNNode()
n.transform = SCNMatrix4(jointTransform)
print(n.eulerAngles)

In this try, I set the jointTransformation to a SCNNode.transform so I can retrieve the eulerAngles to make them human readable and try to understand what's happening.

I get to work some joints, but I think it's pure coincidence or luck, because the rest of the bones rotate very weird.

In other try I get them using jointModelTransforms (Model, instead of Local) so all transforms are relative to the Root bone of the Skeleton.

With this approach I do matrix multiplications like this:

LocalMatrix = Inverse(JointModelMatrix) * (ParentJointModelMatrix)

To get the rotations relative to its parent, but same situation, some bones rotate okay other rotate weird. Pure coincidence I bet.

Why do I want to get the bone rotations?

I'm trying build a MoCap app with my phone that passes to Blender the rotations, trying to build .BVH files from this, so I can use them on Blender.

This is my own rig:

I've done this before with Kinect, but I've been trying for days to do it on ARKit 3 with no luck :(

2 Answers 2

1

Using simd_quatf(from:to:) with the right input should do it. I had trouble with weird angles until i started normalising the vectors:

guard let bodyAnchor = anchor as? ARBodyAnchor else { continue }

let skeleton = bodyAnchor.skeleton
let jointTransforms = skeleton.jointLocalTransforms

for (i, jointTransform) in jointTransforms.enumerated() {
    // First i filter out the root (Hip) joint because it doesn't have a parent
    let parentIndex = skeleton.definition.parentIndices[i]
    guard parentIndex >= 0 else { continue } // root joint has parent index of -1

    //RETRIEVE ANGLES HERE
    let jointVectorFromParent = simd_make_float3(jointTransform.columns.3)
    let referenceVector: SIMD3<Float>
    if skeleton.definition.parentIndices[parentIndex] >= 0 {
         referenceVector = simd_make_float3(jointTransforms[parentIndex].columns.3)
    } else {
         // The parent joint is the Hip joint which should have
         // a vector of 0 going to itself
         // It's impossible to calculate an angle from a vector of length 0,
         // So we're using a vector that's just pointing up
         referenceVector = SIMD3<Float>(x: 0, y: 1, z: 0)
    }
    // Normalizing is important because simd_quatf gives weird results otherwise
    let jointNormalized = normalize(jointVectorFromParent)
    let referenceNormalized = normalize(referenceVector)
    let orientation = simd_quatf(from: referenceNormalized, to: jointNormalized)
    print("angle of joint \(i) = \(orientation.angle)")
}

One important thing to keep in mind though: ARKit3 tracks only some joints (AFAIK the named joints in ARSkeleton.JointName). The other joints are extrapolated from that using a standardized skeleton. Which means, that the angle you get for the elbow for example won't be the exact angle the tracked persons elbow has there.

4
  • Going to try this :-) Will let you know
    – c4b4d4
    Nov 10, 2019 at 20:02
  • Tried it, but the angles didn't seem to change. I'm always getting a T-Pose in the orientation variable, no matter if person is moving. Am I doing it wrong? I just took the orientation variable for each bone and applied it to my rigged character. Should I do something with this orientation before applying it to my character instead?
    – c4b4d4
    Nov 11, 2019 at 0:08
  • Ok, TBH what I'm actually using in my own project is the jointModelTransforms and then the difference between the joint position and the parent joint position to get the "bone vector". And this definitely works for me.
    – jaetzold
    Nov 12, 2019 at 10:44
  • Can you elaborate more your answer with code? So I understand what you are doing. I don’t see the jointModelTransform in your posted answer for example.
    – c4b4d4
    Nov 12, 2019 at 15:35
0

Just a guess… does this do the job?

  let skeleton = bodyAnchor.skeleton
  let jointTransforms = skeleton.jointLocalTransforms

  for (i, jointTransform) in jointTransforms.enumerated() {
    print(Transform(matrix: jointTransform).rotation)
  }
1
  • This is what I already tried, and I can tell ArKit 3 is horrible for the moment for MoCap. The T-Pose is not even zero-ed. :( If anyone has achieved anything, leave a comment
    – c4b4d4
    Oct 28, 2019 at 1:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.