2

I want to use ExoPlayer to progressively stream videos from urls, but display the video in Unity3D. I first tested it with a surfaceview in a native android app and it works brilliantly. However, I need this functionality to be contained into an aar library and use it as a plugin for Unity. I am VERY close to make it work (or so it seems), but I know my lack of OpenGL knowledge is not letting me advance. Should the texture be created in Unity-side or in the plugin-side? When should it be re-created and how? When should it be destroyed and how?

I tried several combinations and only a black texture is shown. Audio works in each case. The code is a mess, as now I'm still in the "discovering" stage and I'm trying everything I can think of. I didn't delete all the comments in the code, so you can see what other approaches I used. I also put some questions in the comments. I know the solution needs unity+exoplayer+opengl+android knowledge at the same time, so I also posted this on Unity Android forum and linked the posts to each other. Hopefully, I can combine the answers I get here with those ones and come up with the solution.

Android side:

public class ExoPlayerForUnity {
    static SurfaceTexture surfaceTexture;
    static Surface surface;

    static Context getUnityContext() {
        try {
            Class<?> unityPlayerClass = Class.forName("com.unity3d.player.UnityPlayer");
            Field currentActivity = unityPlayerClass.getField("currentActivity");
            Object o = currentActivity.get(null);
            return (Context)o;
        } catch (ClassNotFoundException e) {
            e.printStackTrace();
        } catch (NoSuchFieldException e) {
            e.printStackTrace();
        } catch (IllegalAccessException e) {
            e.printStackTrace();
        }

        return null;
    }


    public static int init( String uriString) {

        Uri videoURI = Uri.parse(uriString);

        final Activity unityContext = (Activity)getUnityContext();

        final Handler handler = new Handler();
//        BandwidthMeter bandwidthMeter = new DefaultBandwidthMeter();
//        TrackSelection.Factory videoTrackSelectionFactory = new AdaptiveVideoTrackSelection.Factory(bandwidthMeter);
//        TrackSelection.Factory videoTrackSelectionFactory = new FixedTrackSelection.Factory();
//        TrackSelector trackSelector = new DefaultTrackSelector(mainHandler, videoTrackSelectionFactory);
        TrackSelector trackSelector = new DefaultTrackSelector(handler, null);
        LoadControl loadControl = new DefaultLoadControl();
        final SimpleExoPlayer player = ExoPlayerFactory.newSimpleInstance(unityContext, trackSelector, loadControl, null, false);

        DefaultBandwidthMeter defBandwidthMeter = new DefaultBandwidthMeter();
        DataSource.Factory dataSourceFactory = new DefaultDataSourceFactory(unityContext,
                Util.getUserAgent(unityContext, "yourApplicationName"), defBandwidthMeter);
        ExtractorsFactory extractorsFactory = new DefaultExtractorsFactory();
        MediaSource videoSource = new ExtractorMediaSource(videoURI,
                dataSourceFactory, extractorsFactory, new Handler(), new ExtractorMediaSource.EventListener() {
            @Override
            public void onLoadError(IOException error) {
                error.printStackTrace();
            }
        });


        // Maybe useful?: http://stackoverflow.com/questions/33324753/how-to-use-unity-createexternaltexture-on-android
        int[] textures = new int[1];
        GLES20.glGenTextures(1, textures, 0);
        final int texture = textures[0];

//        //GLES20.glActiveTexture(GLES20.GL_TEXTURE0);
//        GLES20.glBindTexture(GLES20.GL_TEXTURE_2D, texture);
//        //GLES20.glTexImage2D(GLES20.GL_TEXTURE_2D, 0, GLES20.GL_RGB, 1024, 512, 0, GLES20.GL_RGB, GLES20.GL_UNSIGNED_BYTE, null);
//        GLES20.glTexParameteri( GLES20.GL_TEXTURE_2D, GLES20.GL_TEXTURE_MIN_FILTER, GLES20.GL_LINEAR );
//        GLES20.glTexParameteri( GLES20.GL_TEXTURE_2D, GLES20.GL_TEXTURE_MAG_FILTER, GLES20.GL_LINEAR );
//        GLES20.glTexParameteri( GLES20.GL_TEXTURE_2D, GLES20.GL_TEXTURE_WRAP_S, GLES20.GL_CLAMP_TO_EDGE );
//        GLES20.glTexParameteri( GLES20.GL_TEXTURE_2D, GLES20.GL_TEXTURE_WRAP_T, GLES20.GL_CLAMP_TO_EDGE );
//        GLES20.glBindTexture( GLES20.GL_TEXTURE_2D, 0 );

        GLES20.glBindTexture(GLES11Ext.GL_TEXTURE_EXTERNAL_OES, texture);
        GLES20.glTexParameteri( GLES11Ext.GL_TEXTURE_EXTERNAL_OES, GLES20.GL_TEXTURE_MIN_FILTER, GLES20.GL_LINEAR );
        GLES20.glTexParameteri( GLES11Ext.GL_TEXTURE_EXTERNAL_OES, GLES20.GL_TEXTURE_MAG_FILTER, GLES20.GL_LINEAR );
        GLES20.glTexParameteri( GLES11Ext.GL_TEXTURE_EXTERNAL_OES, GLES20.GL_TEXTURE_WRAP_S, GLES20.GL_CLAMP_TO_EDGE );
        GLES20.glTexParameteri( GLES11Ext.GL_TEXTURE_EXTERNAL_OES, GLES20.GL_TEXTURE_WRAP_T, GLES20.GL_CLAMP_TO_EDGE );
        GLES20.glBindTexture( GLES11Ext.GL_TEXTURE_EXTERNAL_OES, 0 );



        // TODO work with attachToGLContext maybe?
        surfaceTexture = new SurfaceTexture(texture);

        surfaceTexture.setOnFrameAvailableListener(new SurfaceTexture.OnFrameAvailableListener() {
            @Override
            public void onFrameAvailable(final SurfaceTexture sf) {
                handler.post(new Runnable() {
                    @Override
                    public void run() {
                        // Should I use this or update it in Unity-side with tex.UpdateTextureExternal(tex.GetNativeTexturePtr()) ?
                        //GLES20.glBindTexture(GLES11Ext.GL_TEXTURE_EXTERNAL_OES, texture);
                        //sf.updateTexImage();
                        //GLES20.glBindTexture( GLES11Ext.GL_TEXTURE_EXTERNAL_OES, 0 );
                    }
                });
            }
        });

        surface = new Surface(surfaceTexture);

        Timer t = new Timer();
        t.schedule(
                new TimerTask() {
                    @Override
                    public void run() {
                        Log.v("ExoPlayerForUnity", "buff: " + player.getBufferedPercentage());
                    }
                }, 3000, 1000
        );

        player.setVideoSurface(surface);
        player.prepare(videoSource);
        player.setPlayWhenReady(true);

        return texture;
    }
}

Unity side:

public class ExoTestPlugin : MonoBehaviour
{
    new Renderer renderer;
    Texture2D rt;
    //RenderTexture rt; // maybe this?
    AndroidJavaClass jc;
    int texName;
    float curScale = 0f;

    void init()
    {
        renderer = GetComponent<Renderer>();
        Camera.onPreRender += OnPreRender2;

        jc = new AndroidJavaClass("com.thefallengames.exoplayerforunity.ExoPlayerForUnity");

        texName = jc.CallStatic<int>("init", 0, "http://clips.vorwaerts-gmbh.de/big_buck_bunny.mp4");


        //rt = new Texture2D(640, 360, TextureFormat.RGBA32, true, true);
        //rt = new Texture2D(640, 360, TextureFormat.ETC2_RGBA8, true, true);
        //rt = new RenderTexture(640, 360, 24, RenderTextureFormat.Default, RenderTextureReadWrite.Default);
        //rt.Create();

        rt = Texture2D.CreateExternalTexture(1024, 512, TextureFormat.ETC_RGB4, true, true, (IntPtr)texName);

        renderer.material.mainTexture = rt;
    }

    void Update () {

        // Here I'm just starting the playback on first tap and then toggling between 2x and 1x scale on any subsequent taps
        if (Input.touchCount > 0 && Input.GetTouch(0).phase == TouchPhase.Ended)
        {
            if (curScale == 0f)
            {
                curScale = 1f;
                init();

                return;
            }

            if (curScale == 2f)
                curScale = .5f;
            else
                curScale = 2f;
            transform.localScale *= curScale;
        }
    }

    void OnPreRender2(Camera cam)
    {
        if (jc == null) // not inited
            return;

        if (UnityEngine.Random.Range(0, 60) != 0) // updating ~once per second only, just for debugging
            return;

        rt.UpdateExternalTexture((IntPtr)texName);
    }
}

EDIT: Also checked this post, but it just says it's possible, it doesn't give any useful details.
Note that I also tried to create the texture in unity and pass the pointer to the java-side and that doesn't seem to work either

7
  • But init() is never called.... Aren't you suppose to call that from the Start function function?
    – Programmer
    Nov 7, 2016 at 18:38
  • Hi! Look for this comment "// Here I'm just starting the playback on first tap and then toggling between 2x and 1x scale on any subsequent taps"
    – lutian
    Nov 7, 2016 at 18:45
  • Ok, I see that.
    – Programmer
    Nov 7, 2016 at 18:57
  • No one?..... :(
    – lutian
    Nov 23, 2016 at 12:06
  • Did you figure this out? I'd like to render from ExoPlayer to a native OpenGLES texture too
    – ephemer
    Jul 26, 2017 at 16:52

0

Your Answer

Reminder: Answers generated by Artificial Intelligence tools are not allowed on Stack Overflow. Learn more

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.