I want to use ExoPlayer to progressively stream videos from urls, but display the video in Unity3D. I first tested it with a surfaceview in a native android app and it works brilliantly. However, I need this functionality to be contained into an aar library and use it as a plugin for Unity. I am VERY close to make it work (or so it seems), but I know my lack of OpenGL knowledge is not letting me advance. Should the texture be created in Unity-side or in the plugin-side? When should it be re-created and how? When should it be destroyed and how?
I tried several combinations and only a black texture is shown. Audio works in each case. The code is a mess, as now I'm still in the "discovering" stage and I'm trying everything I can think of. I didn't delete all the comments in the code, so you can see what other approaches I used. I also put some questions in the comments. I know the solution needs unity+exoplayer+opengl+android knowledge at the same time, so I also posted this on Unity Android forum and linked the posts to each other. Hopefully, I can combine the answers I get here with those ones and come up with the solution.
Android side:
public class ExoPlayerForUnity {
static SurfaceTexture surfaceTexture;
static Surface surface;
static Context getUnityContext() {
try {
Class<?> unityPlayerClass = Class.forName("com.unity3d.player.UnityPlayer");
Field currentActivity = unityPlayerClass.getField("currentActivity");
Object o = currentActivity.get(null);
return (Context)o;
} catch (ClassNotFoundException e) {
e.printStackTrace();
} catch (NoSuchFieldException e) {
e.printStackTrace();
} catch (IllegalAccessException e) {
e.printStackTrace();
}
return null;
}
public static int init( String uriString) {
Uri videoURI = Uri.parse(uriString);
final Activity unityContext = (Activity)getUnityContext();
final Handler handler = new Handler();
// BandwidthMeter bandwidthMeter = new DefaultBandwidthMeter();
// TrackSelection.Factory videoTrackSelectionFactory = new AdaptiveVideoTrackSelection.Factory(bandwidthMeter);
// TrackSelection.Factory videoTrackSelectionFactory = new FixedTrackSelection.Factory();
// TrackSelector trackSelector = new DefaultTrackSelector(mainHandler, videoTrackSelectionFactory);
TrackSelector trackSelector = new DefaultTrackSelector(handler, null);
LoadControl loadControl = new DefaultLoadControl();
final SimpleExoPlayer player = ExoPlayerFactory.newSimpleInstance(unityContext, trackSelector, loadControl, null, false);
DefaultBandwidthMeter defBandwidthMeter = new DefaultBandwidthMeter();
DataSource.Factory dataSourceFactory = new DefaultDataSourceFactory(unityContext,
Util.getUserAgent(unityContext, "yourApplicationName"), defBandwidthMeter);
ExtractorsFactory extractorsFactory = new DefaultExtractorsFactory();
MediaSource videoSource = new ExtractorMediaSource(videoURI,
dataSourceFactory, extractorsFactory, new Handler(), new ExtractorMediaSource.EventListener() {
@Override
public void onLoadError(IOException error) {
error.printStackTrace();
}
});
// Maybe useful?: http://stackoverflow.com/questions/33324753/how-to-use-unity-createexternaltexture-on-android
int[] textures = new int[1];
GLES20.glGenTextures(1, textures, 0);
final int texture = textures[0];
// //GLES20.glActiveTexture(GLES20.GL_TEXTURE0);
// GLES20.glBindTexture(GLES20.GL_TEXTURE_2D, texture);
// //GLES20.glTexImage2D(GLES20.GL_TEXTURE_2D, 0, GLES20.GL_RGB, 1024, 512, 0, GLES20.GL_RGB, GLES20.GL_UNSIGNED_BYTE, null);
// GLES20.glTexParameteri( GLES20.GL_TEXTURE_2D, GLES20.GL_TEXTURE_MIN_FILTER, GLES20.GL_LINEAR );
// GLES20.glTexParameteri( GLES20.GL_TEXTURE_2D, GLES20.GL_TEXTURE_MAG_FILTER, GLES20.GL_LINEAR );
// GLES20.glTexParameteri( GLES20.GL_TEXTURE_2D, GLES20.GL_TEXTURE_WRAP_S, GLES20.GL_CLAMP_TO_EDGE );
// GLES20.glTexParameteri( GLES20.GL_TEXTURE_2D, GLES20.GL_TEXTURE_WRAP_T, GLES20.GL_CLAMP_TO_EDGE );
// GLES20.glBindTexture( GLES20.GL_TEXTURE_2D, 0 );
GLES20.glBindTexture(GLES11Ext.GL_TEXTURE_EXTERNAL_OES, texture);
GLES20.glTexParameteri( GLES11Ext.GL_TEXTURE_EXTERNAL_OES, GLES20.GL_TEXTURE_MIN_FILTER, GLES20.GL_LINEAR );
GLES20.glTexParameteri( GLES11Ext.GL_TEXTURE_EXTERNAL_OES, GLES20.GL_TEXTURE_MAG_FILTER, GLES20.GL_LINEAR );
GLES20.glTexParameteri( GLES11Ext.GL_TEXTURE_EXTERNAL_OES, GLES20.GL_TEXTURE_WRAP_S, GLES20.GL_CLAMP_TO_EDGE );
GLES20.glTexParameteri( GLES11Ext.GL_TEXTURE_EXTERNAL_OES, GLES20.GL_TEXTURE_WRAP_T, GLES20.GL_CLAMP_TO_EDGE );
GLES20.glBindTexture( GLES11Ext.GL_TEXTURE_EXTERNAL_OES, 0 );
// TODO work with attachToGLContext maybe?
surfaceTexture = new SurfaceTexture(texture);
surfaceTexture.setOnFrameAvailableListener(new SurfaceTexture.OnFrameAvailableListener() {
@Override
public void onFrameAvailable(final SurfaceTexture sf) {
handler.post(new Runnable() {
@Override
public void run() {
// Should I use this or update it in Unity-side with tex.UpdateTextureExternal(tex.GetNativeTexturePtr()) ?
//GLES20.glBindTexture(GLES11Ext.GL_TEXTURE_EXTERNAL_OES, texture);
//sf.updateTexImage();
//GLES20.glBindTexture( GLES11Ext.GL_TEXTURE_EXTERNAL_OES, 0 );
}
});
}
});
surface = new Surface(surfaceTexture);
Timer t = new Timer();
t.schedule(
new TimerTask() {
@Override
public void run() {
Log.v("ExoPlayerForUnity", "buff: " + player.getBufferedPercentage());
}
}, 3000, 1000
);
player.setVideoSurface(surface);
player.prepare(videoSource);
player.setPlayWhenReady(true);
return texture;
}
}
Unity side:
public class ExoTestPlugin : MonoBehaviour
{
new Renderer renderer;
Texture2D rt;
//RenderTexture rt; // maybe this?
AndroidJavaClass jc;
int texName;
float curScale = 0f;
void init()
{
renderer = GetComponent<Renderer>();
Camera.onPreRender += OnPreRender2;
jc = new AndroidJavaClass("com.thefallengames.exoplayerforunity.ExoPlayerForUnity");
texName = jc.CallStatic<int>("init", 0, "http://clips.vorwaerts-gmbh.de/big_buck_bunny.mp4");
//rt = new Texture2D(640, 360, TextureFormat.RGBA32, true, true);
//rt = new Texture2D(640, 360, TextureFormat.ETC2_RGBA8, true, true);
//rt = new RenderTexture(640, 360, 24, RenderTextureFormat.Default, RenderTextureReadWrite.Default);
//rt.Create();
rt = Texture2D.CreateExternalTexture(1024, 512, TextureFormat.ETC_RGB4, true, true, (IntPtr)texName);
renderer.material.mainTexture = rt;
}
void Update () {
// Here I'm just starting the playback on first tap and then toggling between 2x and 1x scale on any subsequent taps
if (Input.touchCount > 0 && Input.GetTouch(0).phase == TouchPhase.Ended)
{
if (curScale == 0f)
{
curScale = 1f;
init();
return;
}
if (curScale == 2f)
curScale = .5f;
else
curScale = 2f;
transform.localScale *= curScale;
}
}
void OnPreRender2(Camera cam)
{
if (jc == null) // not inited
return;
if (UnityEngine.Random.Range(0, 60) != 0) // updating ~once per second only, just for debugging
return;
rt.UpdateExternalTexture((IntPtr)texName);
}
}
EDIT:
Also checked this post, but it just says it's possible, it doesn't give any useful details.
Note that I also tried to create the texture in unity and pass the pointer to the java-side and that doesn't seem to work either
init()is never called.... Aren't you suppose to call that from theStartfunction function?