0

I'm kind of stuck right now when it comes to applying a specific texture on my 3d obj model.

Easiest solution of all would be to do let test = SCNScene(named: "models.scnassets/modelFolder/ModelName.obj"), but this requires that the mtl file maps the texture file directly inside of it which is not something that's possible with my current workflow.

With my current understanding, this leaves me with the option of using a scattering function to apply textures to a specific semantic, something like such :

if let url = URL(string: obj) {
    let asset = MDLAsset(url: url)
    guard let object = asset.object(at: 0) as? MDLMesh else {
        print("Failed to get mesh from asset.")
        self.presentAlert(title: "Warning", message: "Could not fetch the model.", firstBtn: "Ok")
        return
    }

    // Create a material from the various textures with a scatteringFunction
    let scatteringFunction = MDLScatteringFunction()
    let material = MDLMaterial(name: "material", scatteringFunction: scatteringFunction)
    let property = MDLMaterialProperty(name: "texture", semantic: .baseColor, url: URL(string: self.textureURL))
    material.setProperty(property)
    
    // Apply the texture to every submesh of the asset
    object.submeshes?.forEach {
        if let submesh = $0 as? MDLSubmesh {
            submesh.material = material
        }
    }

    // Wrap the ModelIO object in a SceneKit object
    let node = SCNNode(mdlObject: object)
    let scene = SCNScene()
    scene.rootNode.addChildNode(node)

    // Set up the SceneView
    sceneView.scene = scene
    ...
}

The actual problem is the semantics. The 3d models are made on Unreal and for many models there's a png texture which has 3 semantics inside of it, namely Ambient Occlusion, Roughness and Metallic. Ambient Occlusion would need to be applied on the red channel, Roughness on the greed channel and Metallic on the blue channel.

How could I achieve this? An MdlMaterialSemantic has all of these possible semantics, but metallic, ambient occlusion and roughness are all separate. I tried simply applying the texture on each, but obviously this did not work very well.

Considering that my .png texture has all of those 3 "packaged" in it under a different channel, how can I work with this? I was thinking that maybe I could somehow use a small script to add mapping to the texture in the mtl file on my end in the app directly, but this seems sketchy lol..

What are my other options if there's no way of doing this? I've also been trying to use fbx files with assimpKit, but I couldn't manage to load any textures, just the model in black...

I am open to any suggestion, if more info is needed, please let me know! Thank you very much!

1 Answer 1

1

Sorry, I don't have enough rep to comment, but this might be more of a comment than an answer!

Have you tried loading the texture png image separately (as a NS/UI/CGImage) and then splitting it into three channels manually, then applying these channels separately? (Splitting into three separate channels is not as simple as it could be... but you could use this grayscale conversion for guidance, and just do one channel at a time.)

Once you have your objects in SceneKit, it is possibly slightly easier to modify these materials. Once you have a SCNNode with a SCNGeometry with a SCNMaterial you can access any of these materials and set the .contents property to almost anything (including a XXImage).

Edit:
Here's an extension you can try to extract the individual channels from a CGImage using Accelerate. You can get a CGImage from an NSImage/UIImage depending on whether you're on Mac or iOS (and you can load the file directly into one of those image formats).

I've just adapted the code from the link above, I am not very experienced with the Accelerate framework, so use at your own risk! But hopefully this puts you on the right path.

extension CGImage {
    enum Channel {
        case red, green, blue
    }
    
    func getChannel(channel: Channel) -> CGImage? {
        // code adapted from https://developer.apple.com/documentation/accelerate/converting_color_images_to_grayscale
        guard let format = vImage_CGImageFormat(cgImage: cgImage) else {return nil}
        guard var sourceImageBuffer = try? vImage_Buffer(cgImage: cgImage, format: format) else {return nil}

        guard var destinationBuffer = try? vImage_Buffer(width: Int(sourceImageBuffer.width), height: Int(sourceImageBuffer.height), bitsPerPixel: 8) else {return nil}
        
        defer {
            sourceImageBuffer.free()
            destinationBuffer.free()
        }
    
        let redCoefficient: Float = channel == .red ? 1 : 0
        let greenCoefficient: Float = channel == .green ? 1 : 0
        let blueCoefficient: Float = channel == .blue ? 1 : 0
        
        let divisor: Int32 = 0x1000
        let fDivisor = Float(divisor)
        
        var coefficientsMatrix = [
            Int16(redCoefficient * fDivisor),
            Int16(greenCoefficient * fDivisor),
            Int16(blueCoefficient * fDivisor)
        ]
        
        let preBias: [Int16] = [0, 0, 0, 0]
        let postBias: Int32 = 0
        
        vImageMatrixMultiply_ARGB8888ToPlanar8(&sourceImageBuffer,
                                               &destinationBuffer,
                                               &coefficientsMatrix,
                                               divisor,
                                               preBias,
                                               postBias,
                                               vImage_Flags(kvImageNoFlags))
        
        guard let monoFormat = vImage_CGImageFormat(
            bitsPerComponent: 8,
            bitsPerPixel: 8,
            colorSpace: CGColorSpaceCreateDeviceGray(),
            bitmapInfo: CGBitmapInfo(rawValue: CGImageAlphaInfo.none.rawValue),
            renderingIntent: .defaultIntent) else {return nil}
        
        guard let result = try? destinationBuffer.createCGImage(format: monoFormat) else {return nil}
        
        return result
    }
}
6
  • Ah, so it is possible to split the image into three channels? I had a lot of trouble this information. Would you perhaps know of an easy way of doing this? Looks a tad bit advanced for my knowledge, maybe a pod or extension could help me I guess. I'll try to look more into it in the meantime!
    – David
    Oct 29, 2021 at 13:06
  • @David, it's definitely possible. I've added an extension above you can try for a solution using vImage/Accelerate that I adapted from the previous link. That should be fast enough (though I am not an expert with this framework). Nov 1, 2021 at 17:50
  • Depending on the rest of your code, there's various other ways you could do this. If you already have your own Metal code, then it would be easy with a compute shader into a metal texture, that's probably what I'd do. Or, if you use CoreImage, you could use a CIColorPolynomial filter. Or you could just read the raw bytes of the CGImage on the CPU. Nov 1, 2021 at 17:54
  • Alright, thanks! I've managed to make it work, but it's way too slow to process this on demand. I'll have to look into another solution, maybe doing the conversion on the api side when the models & textures are saved on the database, not sure. My other goal in the meantime would be to load a fbx model with textures :) Thanks for the help!
    – David
    Nov 2, 2021 at 13:25
  • Re-tested with your code and it actually works and is fast enough! Thanks a lot for your help :)
    – David
    Nov 4, 2021 at 15:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.