Android/AndroidTV Placeshifter client (Discussion)

[tmiranda]

Nice work Sean. I'm going to try this later today on my Samsung. I'll also start looking at the STV to see if I can come up with something more phone friendly.

[stuckless]

Originally Posted by dinki

I'm on 4.4.4 so I guess this opengl es 2 is the problem. This is a phone dependent thing?

Does it crash, or simply won't install? If it's a crash, then it could be something in the code... if it won't install, then it's definately an opengles 2 issue.

Originally Posted by tmiranda

Nice work Sean. I'm going to try this later today on my Samsung. I'll also start looking at the STV to see if I can come up with something more phone friendly.

That would be awesome!

[dinki]

Originally Posted by stuckless

Does it crash, or simply won't install? If it's a crash, then it could be something in the code... if it won't install, then it's definately an opengles 2 issue.

The error I get is "Can't open file" when trying to install apk on my phone. Downloaded again just to see but same result. It's somehow dependent on opengl es 2 even for the install? I know nothing of how these things work.

*EDIT*

According to this site: http://tabshowdown.blogspot.com/2014...-best-mid.html it appears my phone supports opengl es 3:

As benchmarks will show, the Adreno 305 GPU is no match for current high-end mobile GPUs, but at the same time it's one of the most modern GPUs, architecturally speaking, thanks to its full OpenGL ES 3.0 compatibility. It may not have the power to run demanding games or benchmarks super-smoothly, but at least it has excellent API compatibility, unlike Nvidia's Tegra 4, which may have more processing power, but can't run OpenGL ES 3.0 applications.

Unless I'm reading that wrong, I'd guess it's backward compatible with 2.0 right?

[stuckless]

Originally Posted by dinki

The error I get is "Can't open file" when trying to install apk on my phone. Downloaded again just to see but same result. It's somehow dependent on opengl es 2 even for the install? I know nothing of how these things work.

There is a manifest entry that requires es2... so Android may be preventing you from installing it, because of that. Setting that was require to force Android to use hardware acceleration on the Canvas. Otherwise it would be REAL slow.

[Fuzzy]

Originally Posted by stuckless

The android Canvas, much like the Java2d canvas is pretty snappy for drawing, but when you start to scale it, it gets laggy. Currently each rendered frame takes anywhere between 100-200ms to render, which is far too long, especially when you consider that sagetv typically has an animation duration of 200-300ms. So, if an animation in sagetv creates 5 frames and each of those frames takes 200ms each to render, then our 250ms animation just took a full second to render, and that's the sluggishness you are seeing in the UI today.

So my next challenge for this is to try and create a pure opengl ES2 renderer. My problem is that I haven't looked at opengl in years (probably since college)... and ES2 uses shader functions, and vertex buffer objects, and I've never looked at either of these. So combining my lack of knowledge in opengl with the fact that the opengl code that sagetv is using currently has no direct mapping to opengl ES... this next step is going to be slow But moving to opengl is the only way we are going to get buttery smooth transition, like we have on the HD300.

If anyone that has experience with VBOs and Android OpenGL ES and they want to help or work with me, then it would be appreciated. To be honest, to get this working in opengl based on where the code is now, is minimal effort for anyone that understands opengl. I've written the load bitmap into a texture function (Android provided a nice utility for that), but I have no idea how I'm supposed to map a loaded texture to a region of the screen using VBOs.

Android has a GLSurfaceView that manages the offscreen buffering from what I can tell, so getting this work, should be fairly straight forward. In the mean time, I'll be reading article after article on opengl, vbos, and android opengl es

Out of curiosity, how quickly would it render at full screen resolution (whatever resolution the device runs at), thus skipping the scaling?

[dinki]

Sean, did you see my edit above? I may have been editing while you were replying. I _think_ my phone has es3, so should I be able to use es2 code? Sorry if that's a dumb question.

[stuckless]

Originally Posted by Fuzzy

Out of curiosity, how quickly would it render at full screen resolution (whatever resolution the device runs at), thus skipping the scaling?

When I tell sagetv that my screen resolution is 1920x1080... it stil sends me 720x480 bitmaps... so I don't know It's like that I need to set some other GFX options, but I don't know what they are... Maybe Jeff can help.

Originally Posted by dinki

Sean, did you see my edit above? I may have been editing while you were replying. I _think_ my phone has es3, so should I be able to use es2 code? Sorry if that's a dumb question.

So not an es2 issue, which is good, so not sure why it would not install.

[Fuzzy]

Originally Posted by stuckless

When I tell sagetv that my screen resolution is 1920x1080... it stil sends me 720x480 bitmaps... so I don't know It's like that I need to set some other GFX options, but I don't know what they are... Maybe Jeff can help.

In MiniClientRenderer, there appears to be a UI_CLIENT_RESIZE_EVENT_REPLY handler that will set new values for osd_rendering_width, osd_rendering_height_ntsc & osd_rendering_height_pal, which are otherwise defaulted at 720x480 if this event is never triggered.

You'll likely need to send this event immediately after connecting to get it to send a higher res ui. On the client end, it looks like this is done via:

MiniClientConnection.postResizeEvent(java.awt.Dimension size)

[stuckless]

Originally Posted by Fuzzy

In MiniClientRenderer, there appears to be a UI_CLIENT_RESIZE_EVENT_REPLY handler that will set new values for osd_rendering_width, osd_rendering_height_ntsc & osd_rendering_height_pal, which are otherwise defaulted at 720x480 if this event is never triggered.

You'll likely need to send this event immediately after connecting to get it to send a higher res ui. On the client end, it looks like this is done via:

MiniClientConnection.postResizeEvent(java.awt.Dimension size)

I can try posting a resize event... I just thought that setting the GFX_RESOLUTION=1920x1080 would tell sagetv that my resolution is 1920x1080. If I get time tonight, I'll try the posting of the resive event and see what happens.

[Fuzzy]

Originally Posted by stuckless

I can try posting a resize event... I just thought that setting the GFX_RESOLUTION=1920x1080 would tell sagetv that my resolution is 1920x1080. If I get time tonight, I'll try the posting of the resive event and see what happens.

Yeah, I haven't figured out the difference between those two, but they appear to be completely unrelated to each other on the server end.

[Narflex]

Fuzzy is right...that's the event you need to let the server know what the actual UI size is.

GFX_RESOLUTION is actually used for output resolutions from the device...not for the size of the actual UI plane itself. In the Android TV case though, it probably doesn't really matter.

[Taddeusz]

Trying to see if I can use j2objc to convert the MiniClient to work with iOS. Doesn't work with what is in the repository. Do you have the source for your cut down MiniClient available?

[stuckless]

Originally Posted by Taddeusz

Trying to see if I can use j2objc to convert the MiniClient to work with iOS. Doesn't work with what is in the repository. Do you have the source for your cut down MiniClient available?

I don't have it published, yet, because to be honest, the code is quite scary I've been hacking and slashing to get running I'll see about pushing out the code that I have in the next day or so, and you can try that as a base. I've removed all the AWT dependencies, etc, so it's a good base to start from.

[stuckless]

Originally Posted by Narflex

Fuzzy is right...that's the event you need to let the server know what the actual UI size is.

GFX_RESOLUTION is actually used for output resolutions from the device...not for the size of the actual UI plane itself. In the Android TV case though, it probably doesn't really matter.

FYI... I posted the resize event to match the native resolution, and I disabled scaling. The UI is incredibly crisp, but sadly, the Canvas cannot keep up with the rendering. Rendering frame times are almost the same as before, between 100-200ms. I guess I'll be learning opengl

From my logging, I can see it doing about 120 frame operations (mostly texture rendering) and those frames take 100-200ms... On some frames, like going into Plugins, it's doing 2300 frame operations... and those take about 500-600ms per frame

[tmiranda]

Just downloaded and installed the .apk. Sometimes it connects to the server and sometimes it doesn't (go from the blue connecting to .... screen back to the black screen with the ip address on it. Here is some of the log when it fails:

Mon 9/21 17:14:23.737 [VideoFrame-90b68673f8cb@181849a] Did not find a player socket connection; notifying the client to force a reconnect on the other channel
Mon 9/21 17:14:23.741 [VideoFrame-90b68673f8cb@181849a] MiniUI client error24-1:java.io.IOException: An established connection was aborted by the software in your host machine
Mon 9/21 17:14:23.745 [VideoFrame-90b68673f8cb@181849a] java.io.IOException: An established connection was aborted by the software in your host machine
Mon 9/21 17:14:23.745 [VideoFrame-90b68673f8cb@181849a] sun.nio.ch.SocketDispatcher.write0(Native Method)
Mon 9/21 17:14:23.748 [VideoFrame-90b68673f8cb@181849a] sun.nio.ch.SocketDispatcher.write(Unknown Source)
Mon 9/21 17:14:23.751 [VideoFrame-90b68673f8cb@181849a] sun.nio.ch.IOUtil.writeFromNativeBuffer(Unknown Source)
Mon 9/21 17:14:23.754 [VideoFrame-90b68673f8cb@181849a] sun.nio.ch.IOUtil.write(Unknown Source)
Mon 9/21 17:14:23.757 [VideoFrame-90b68673f8cb@181849a] sun.nio.ch.SocketChannelImpl.write(Unknown Source)
Mon 9/21 17:14:23.760 [VideoFrame-90b68673f8cb@181849a] java.nio.channels.Channels.write(Unknown Source)
Mon 9/21 17:14:23.763 [VideoFrame-90b68673f8cb@181849a] java.nio.channels.Channels.access$000(Unknown Source)
Mon 9/21 17:14:23.766 [VideoFrame-90b68673f8cb@181849a] java.nio.channels.Channels$1.write(Unknown Source)
Mon 9/21 17:14:23.768 [VideoFrame-90b68673f8cb@181849a] com.jcraft.jzlib.ZOutputStream.write(ZOutputStream.java:88)
Mon 9/21 17:14:23.770 [VideoFrame-90b68673f8cb@181849a] sage.MiniClientSageRenderer.sendBufferNow(MiniClientSageRenderer.java:6489)
Mon 9/21 17:14:23.772 [VideoFrame-90b68673f8cb@181849a] sage.MiniClientSageRenderer.forceMediaReconnect(MiniClientSageRenderer.java:6467)
Mon 9/21 17:14:23.774 [VideoFrame-90b68673f8cb@181849a] sage.MiniClientSageRenderer.getSocketChannelInfo(MiniClientSageRenderer.java:811)
Mon 9/21 17:14:23.777 [VideoFrame-90b68673f8cb@181849a] sage.MiniClientSageRenderer.getSocketChannel(MiniClientSageRenderer.java:766)
Mon 9/21 17:14:23.779 [VideoFrame-90b68673f8cb@181849a] sage.MiniClientSageRenderer.getPlayerSocketChannel(MiniClientSageRenderer.java:740)
Mon 9/21 17:14:23.781 [VideoFrame-90b68673f8cb@181849a] sage.MiniClientSageRenderer.getPlayerSocketChannel(MiniClientSageRenderer.java:735)
Mon 9/21 17:14:23.783 [VideoFrame-90b68673f8cb@181849a] sage.MiniPlayer.initDriver0(MiniPlayer.java:2979)
Mon 9/21 17:14:23.785 [VideoFrame-90b68673f8cb@181849a] sage.MiniPlayer.load(MiniPlayer.java:645)
Mon 9/21 17:14:23.787 [VideoFrame-90b68673f8cb@181849a] sage.VideoFrame.timeSelected(VideoFrame.java:3403)
Mon 9/21 17:14:23.789 [VideoFrame-90b68673f8cb@181849a] sage.VideoFrame.processJob(VideoFrame.java:2450)
Mon 9/21 17:14:23.791 [VideoFrame-90b68673f8cb@181849a] sage.VideoFrame.run(VideoFrame.java:1614)
Mon 9/21 17:14:23.793 [VideoFrame-90b68673f8cb@181849a] sage.Pooler$PooledThread.run(Pooler.java:252)

The UI is not as slow as I would have thought given your comments. I'm running it on a Samsung Galaxy 6.

I've never done any real work on the phone. Where in the filesystem do I find the Sage tv "home" directory? I'm specifically looking for the STV file.

Edit: I tried turning the animatios off to see if that would make things faster. It put the device ino some kind of loop where the UI would flash on and off. I had to delete the .properties file on the server to correct the problem.

Edit2: The swipe gesture seem to be backwards. Swiping from right to left goes back. I would have guessed that swiping left to right would be back and right to left would be forward?

[stuckless]

Originally Posted by tmiranda

Just downloaded and installed the .apk. Sometimes it connects to the server and sometimes it doesn't (go from the blue connecting to .... screen back to the black screen with the ip address on it. Here is some of the log when it fails:

Mon 9/21 17:14:23.737 [VideoFrame-90b68673f8cb@181849a] Did not find a player socket connection; notifying the client to force a reconnect on the other channel
Mon 9/21 17:14:23.741 [VideoFrame-90b68673f8cb@181849a] MiniUI client error24-1:java.io.IOException: An established connection was aborted by the software in your host machine
Mon 9/21 17:14:23.745 [VideoFrame-90b68673f8cb@181849a] java.io.IOException: An established connection was aborted by the software in your host machine
Mon 9/21 17:14:23.745 [VideoFrame-90b68673f8cb@181849a] sun.nio.ch.SocketDispatcher.write0(Native Method)
Mon 9/21 17:14:23.748 [VideoFrame-90b68673f8cb@181849a] sun.nio.ch.SocketDispatcher.write(Unknown Source)
Mon 9/21 17:14:23.751 [VideoFrame-90b68673f8cb@181849a] sun.nio.ch.IOUtil.writeFromNativeBuffer(Unknown Source)
Mon 9/21 17:14:23.754 [VideoFrame-90b68673f8cb@181849a] sun.nio.ch.IOUtil.write(Unknown Source)
Mon 9/21 17:14:23.757 [VideoFrame-90b68673f8cb@181849a] sun.nio.ch.SocketChannelImpl.write(Unknown Source)
Mon 9/21 17:14:23.760 [VideoFrame-90b68673f8cb@181849a] java.nio.channels.Channels.write(Unknown Source)
Mon 9/21 17:14:23.763 [VideoFrame-90b68673f8cb@181849a] java.nio.channels.Channels.access$000(Unknown Source)
Mon 9/21 17:14:23.766 [VideoFrame-90b68673f8cb@181849a] java.nio.channels.Channels$1.write(Unknown Source)
Mon 9/21 17:14:23.768 [VideoFrame-90b68673f8cb@181849a] com.jcraft.jzlib.ZOutputStream.write(ZOutputStream.java:88)
Mon 9/21 17:14:23.770 [VideoFrame-90b68673f8cb@181849a] sage.MiniClientSageRenderer.sendBufferNow(MiniClientSageRenderer.java:6489)
Mon 9/21 17:14:23.772 [VideoFrame-90b68673f8cb@181849a] sage.MiniClientSageRenderer.forceMediaReconnect(MiniClientSageRenderer.java:6467)
Mon 9/21 17:14:23.774 [VideoFrame-90b68673f8cb@181849a] sage.MiniClientSageRenderer.getSocketChannelInfo(MiniClientSageRenderer.java:811)
Mon 9/21 17:14:23.777 [VideoFrame-90b68673f8cb@181849a] sage.MiniClientSageRenderer.getSocketChannel(MiniClientSageRenderer.java:766)
Mon 9/21 17:14:23.779 [VideoFrame-90b68673f8cb@181849a] sage.MiniClientSageRenderer.getPlayerSocketChannel(MiniClientSageRenderer.java:740)
Mon 9/21 17:14:23.781 [VideoFrame-90b68673f8cb@181849a] sage.MiniClientSageRenderer.getPlayerSocketChannel(MiniClientSageRenderer.java:735)
Mon 9/21 17:14:23.783 [VideoFrame-90b68673f8cb@181849a] sage.MiniPlayer.initDriver0(MiniPlayer.java:2979)
Mon 9/21 17:14:23.785 [VideoFrame-90b68673f8cb@181849a] sage.MiniPlayer.load(MiniPlayer.java:645)
Mon 9/21 17:14:23.787 [VideoFrame-90b68673f8cb@181849a] sage.VideoFrame.timeSelected(VideoFrame.java:3403)
Mon 9/21 17:14:23.789 [VideoFrame-90b68673f8cb@181849a] sage.VideoFrame.processJob(VideoFrame.java:2450)
Mon 9/21 17:14:23.791 [VideoFrame-90b68673f8cb@181849a] sage.VideoFrame.run(VideoFrame.java:1614)
Mon 9/21 17:14:23.793 [VideoFrame-90b68673f8cb@181849a] sage.Pooler$PooledThread.run(Pooler.java:252)

The UI is not as slow as I would have thought given your comments. I'm running it on a Samsung Galaxy 6.

I've never done any real work on the phone. Where in the filesystem do I find the Sage tv "home" directory? I'm specifically looking for the STV file.

Edit: I tried turning the animatios off to see if that would make things faster. It put the device ino some kind of loop where the UI would flash on and off. I had to delete the .properties file on the server to correct the problem.

Edit2: The swipe gesture seem to be backwards. Swiping from right to left goes back. I would have guessed that swiping left to right would be back and right to left would be forward?

Swiping should move in the direction you swipe, like using arrow keys. So, if you swipe from left to right, that's like hitting the right arrow key. We can customize this behaviour, since what I have now, is very rough to get some interaction with the screen.

As for turning off animations, yeah, I noticed lots of issues as well... so I left them on Obviously there is much more work that I need to do there.

[tmiranda]

And I realized that as a miniclient it uses the STV from the server. I'll start messing about with it to see if I can come up with something a bit more small-screen friendly.

[wnjj]

Originally Posted by stuckless

Swiping should move in the direction you swipe, like using arrow keys. So, if you swipe from left to right, that's like hitting the right arrow key. We can customize this behaviour, since what I have now, is very rough to get some interaction with the screen.

Not sure I agree with that. Pressing arrow keys usually means you want to move a "cursor" in that direction, so the media moves the opposite direction.

Swiping, on the other hand, is swiping the media itself. Swiping left to right on my phone shows my the screen that was previously to the left, which is what left-arrow does on my Windows machine.

Think of arrow keys as moving the pencil and swiping as moving the paper.

[tmiranda]

Originally Posted by wnjj

Not sure I agree with that. Pressing arrow keys usually means you want to move a "cursor" in that direction, so the media moves the opposite direction.

Swiping, on the other hand, is swiping the media itself. Swiping left to right on my phone shows my the screen that was previously to the left, which is what left-arrow does on my Windows machine.

Think of arrow keys as moving the pencil and swiping as moving the paper.

I agree with this.

[stuckless]

Originally Posted by wnjj

Not sure I agree with that. Pressing arrow keys usually means you want to move a "cursor" in that direction, so the media moves the opposite direction.

Swiping, on the other hand, is swiping the media itself. Swiping left to right on my phone shows my the screen that was previously to the left, which is what left-arrow does on my Windows machine.

Think of arrow keys as moving the pencil and swiping as moving the paper.

Originally Posted by tmiranda

I agree with this.

I'm not disagreeing... So far I've spent about 1 hr working on gestures, and about 2 weeks working on the UI. All i can say, is that it's not done yet When it is done, people can probably just remap it. But to to be honest, when you start using the app more, you'll see how the swiping as implemented, makes sense in some cases, not in others. for example if you take the sagetv menus. when you tap a menu, it will focus it. When you swipe left to right, it opens that menu (simulates right key press). You are saying that you want to swipe right to left, to open the menus Touch for SageTV has to consistently map to a mouse event and/or keystroke, and I'm not convinced that swiping right to left, should open a menu, that will fly out, right to left. But as I said, once all the big things are done, I'll look more into the gestures, and if time permits, I'll give an option to reverse the swiping actions.