HOWTO: Using Sage Remote APIs for Java Development

[stuckless]

This is a post on setting up the sagex-apis (Sage Remote APIs) so that you can develop the java portion of your customization, and fully test it, without having to build and deploy a jar to SageTV. Building and deploying a jar, and then restarting SageTV to pick up the changes is time consuming, and your time is better spent writing code

Overview
First, just a quick overview of the sagex-apis. This is not a new plugin. This is the first plugin that I wrote when I started developing for sagetv, because I needed a way to develop for sagetv like any other desktop/web application, ie, write and test on my desktop.

The sagex-apis is a thin api wrapper on top of Sage's Native API. The api can function like the normal SageAPI when deployed to a Sage Server, or it can communicate remotely with the server. You don't have to do anything special in your code to allow for this, since the API knows when it is running inside the sagetv server or remotely. The is very little overhead when running inside the sage server, since sage objects are passed around, as is. The is some overhead when running remotely, since objects have be serialized and transmitted over the network.

Installation
To get started, you'll need to install sagex-services - SageTV Remote API Services on the server to which you want to communicate. This plugin includes all the remote api capabilities (json/xml over http + rmi), which is why it will also depends on Jetty as well.

Here's a screen shot of the Plugin Manager with sagex-services highlighted.

Next, you should make sure that the RMI services is enabled, since that is how your remote Java code will communicate with the sagetv server. Go to the plugin manager, installed plugins, and select sagex-services, and then configure plugin. You should see the following screen, and just make sure that the first option, Enable RMI Remote API is True.

Testing and Using
So, you are now all set on the server, next go to your workstation and open up your IDE. And create short piece of test code, something like,

package test;

import sagex.api.Global;

public class TestMisc {
	public static void main(String args[]) throws Exception {
		System.out.println("Sage OS: " + Global.GetOS());
	}
}

And run it from within your IDE. If all goes well you'll see something like this in the output console window.

INFO - Configured Root Logger
LOG4J: Configured Root Logger
INFO - Configured Logging for: sagex-api using file: sagex-api.log4j.properties
LOG4J: Configured Logging for: sagex-api using file: sagex-api.log4j.properties
Embedded SageAPI is not functional.  We are most likely running remotely.
Adding Remote Server: sean-desktop
Adding Remote Server: mediaserver
Sage OS: Linux

The first few lines about the logging is normal, since the sagex-apis will write to the stdout until the log4j logger is configured.

The line about "Embedded SageAPI is NOT functional" is letting your know that you are not running this code inside the sage server, but remotely, which is what you want.

Following that, the sagex apis will begin discovering remote servers to which it can communicate. On my network, I have main server (mediaserver) and my devlopment server (sean-desktop). In the event that there is more than one server returned, then sagex will use the first one, in this case, sean-deskop.

And finally the last line is proof that it is working. It is the result of your System.out command, which shows the OS that the Sage Server is running on, in my case Linux.

I have more than one SageTV server
Sometimes, if you have more than one server, OR if the discovery code fails (which it may), then you can force a server in your test code.

package test;

import sagex.SageAPI;
import sagex.api.Global;
import sagex.remote.rmi.RMISageAPI;

public class TestMisc {
	public static void main(String args[]) throws Exception {
		SageAPI.setProvider(new RMISageAPI("mediaserver"));
		System.out.println("Sage OS: " + Global.GetOS());
	}
}

Using SageAPI.setProvider(), you force the sagex api to explicitly use an implementation. In this case, I'm use the RMI provider and giving it the host that I want to use (which could be an ip address as well).

The output now, looks like this.

INFO - Configured Root Logger
LOG4J: Configured Root Logger
INFO - Configured Logging for: sagex-api using file: sagex-api.log4j.properties
LOG4J: Configured Logging for: sagex-api using file: sagex-api.log4j.properties
Sage OS: Linux

Note, well still have all the initial logging, but the lines about remote servers is gone, since we've told the API to use an explicit implementation with a connection.

I'm Old School
You might have an old library that uses the SageTV.api() and SageTV.apiUi() calls, and you don't really want to convert all that code into wrapped sagex-apis. You can still take advantage of the sagex-apis, with some minimal effort.

Simply replace SageTV.api() and SageTV.apiUI() with SageAPI.call() in the old library. Recompile your library, and now your library can be used remotely. I've successfully done this will all the nielm jars, and even Greg's api wrapper. In fact, initially I wasn't going to build a set of wrappers, and all I had was the SageAPI.call() method, that proxied the SageTV.api() method. But, since autogenerating a set of wrappers is fairly easily, I extended it to make a full blown set of wrappers that also work remotely.


What about Web Applications
I develope and test BMT web ui, which is a fairly complex web application that depends on phoenix and bunch of other libraries, and I do it entirely outside of the sagetv server. This makes for a nice environment for developing web applications, since you fully test and develop on your local jetty/tomcat server and when you are ready, you can build a war and publish to sagetv.

To do this, simply, include the sagex-api.jar in your web classpath. That's it. If you need force the sagex apis to a particular server, you can do that using the servlet's init() method, and do something like...

if (SageAPI.isRemote()) {
   SageAPI.setProvider(new RMISageAPI("mediaserver"));
}

The SageAPI knows when it is remote, so using the SageAPI.isRemote() method is a nice way to isolate code that you only want executed when you are remote.

JUnit Testing
The other reason I created the sagex-apis is that I needed a better way to unit test code from my eclipse development environment. I won't get into JUnit testing, only to say, that by dynamically setting a new API provider, you can unit test code, without using a real sagetv server.

I provide a simple stub provider, sagex.stub.StubSageAPI, that can be be used without fear that you are affecting a live server.

In your unit test code, you can add the following line..

SageAPI.setProvider(new StubSageAPI());

A stub api, is just that, it is a stub. It satifies a dependency on the api, but it doesn't add any value. You can interact with it, but don't expect things like GetMediaFiles() to return data (unless you call AddMediaFile first). This is bare API that can be used for unit testing.

Another option, which I use as well, is I dynamically create a SageAPIProvider implementation using easymock. This enables me to fill the api provider with mock data so that I can do a unit test.

Having options to unit test your code, is a great way easily add quality to your plugin.

There is much more that I could talk about with the sagex-services/sagex-api, but this is a good start, for now

Good luck.

[jreichen]

Thanks for the good article Sean. Once I get caught up on everything I'll write unit tests After doing that all day it's hard to come home and go through it all again.

I suppose there's a certain procedure that needs to be followed to set up Jetty? If anybody should know it would be me but I haven't tried it

If you have the time would you mind sharing more about testing with mock objects? It seems some things like GetProperty would be easier than others such as GetAiringsForShow.

Jason

[stuckless]

I'm not much of a fan of writing unit tests myself, but with projects the size of bmt and phoenix, they need them. Without unit testing (especially phoenix), I'd be scared about making any changes to the core

Also, I'm fairly relaxed with the unit tests. ie, I create tests and will group a bunch of operations together, but some people are real sticklers about having every little test separated out into their own test case, etc. (I'm not like that )

(note, easymock can be very complicated to get your head around, but once you do, it can be a useful unit testing tool)

As for easymock, here's a simple junit test that I wrote a awhile ago to test the Year value coming back from phoenix IMetadata proxy, that proxies the sagetv GetShowYear and/or Get/SetMediaFileMetadata.

		final Map<String, String> simpleMetadataMap = new HashMap<String, String>();
        ISageAPIProvider prov = createNiceMock(ISageAPIProvider.class);
        expect(prov.callService(eq("GetShowYear"), (Object[]) anyObject())).andAnswer(new IAnswer<Object>() {
            public Object answer() throws Throwable {
                return simpleMetadataMap.get(((Object[])getCurrentArguments()[1])[0]);
            }
        }).anyTimes();

        expect(prov.callService(eq("GetMediaFileMetadata"), (Object[]) anyObject())).andAnswer(new IAnswer<Object>() {
            public Object answer() throws Throwable {
                return simpleMetadataMap.get(((Object[])getCurrentArguments()[1])[0]);
            }
        }).anyTimes();
        
        expect(prov.callService(eq("IsMediaFileObject"), (Object[]) anyObject())).andAnswer(new IAnswer<Object>() {
            public Object answer() throws Throwable {
                return true;
            }
        }).anyTimes();
        
        replay(prov);
        SageAPI.setProvider(prov);

        simpleMetadataMap.put("mediafile1", "2010");
        simpleMetadataMap.put("mediafile2", "ten");
        simpleMetadataMap.put("mediafile3", "");
        simpleMetadataMap.put("mediafile4", null);
        
        IMetadata md = AiringMetadataProxy.newInstance("mediafile1");
        assertEquals(phoenix.metadata.GetYear("mediafile1"),2010);

        md = AiringMetadataProxy.newInstance("mediafile2");
        assertEquals(phoenix.metadata.GetYear("mediafile2"),0);

        md = AiringMetadataProxy.newInstance("mediafile3");
        assertEquals(phoenix.metadata.GetYear("mediafile3"),0);

        md = AiringMetadataProxy.newInstance("mediafile4");
        assertEquals(phoenix.metadata.GetYear("mediafile4"),0);

        md = SageMediaFileMetadataProxy.newInstance("mediafile1");
        assertEquals(phoenix.metadata.GetYear("mediafile1"),2010);
        
        md = SageMediaFileMetadataProxy.newInstance("mediafile2");
        assertEquals(phoenix.metadata.GetYear("mediafile2"),0);
        
        md = SageMediaFileMetadataProxy.newInstance("mediafile3");
        assertEquals(phoenix.metadata.GetYear("mediafile3"),0);
        
        md = SageMediaFileMetadataProxy.newInstance("mediafile4");
        assertEquals(phoenix.metadata.GetYear("mediafile4"),0);
	}

The first thing that I did was create a Map to hold a mediafile as the key, and a single piece of Metadata, which is year. This will be used by easymock to locate data.

		final Map<String, String> simpleMetadataMap = new HashMap<String, String>();

I mentioned in the original post that the sagex-api can use a "provider" implementation, and so the next line of code, uses the easymock framework to create "Mock" implementation of the ISageAPIProvder interface. (easymock works with interfaces quick nicely because it uses the underlying java Proxy classes, but it can deal with classes as well, but requires more work).

        ISageAPIProvider prov = createNiceMock(ISageAPIProvider.class);

The next part of the code, sets up the playback of what to return when certains parameters are passed to the ISageAPIProvider.callService() method. Since everything filters down to ISageAPIProvider.callService(), this is where "mock" the sagetv communications. The nice thing about the sage api from a mocking perpective is that all sage objects are just "Object" types. So, in my mock environment, I'm using a "String" to represent a native SageTV MediaFile. In my easymock implementation, I'm relying on the fact that the first parameter to a SageAPI is the Object (ie, MediaFile) that is being worked on, so in the easymock area, I know that I'll get back the "String" representing my MediaFile, and I use that to lookup that value that I want from the map.

        expect(prov.callService(eq("GetShowYear"), (Object[]) anyObject())).andAnswer(new IAnswer<Object>() {
            public Object answer() throws Throwable {
                return simpleMetadataMap.get(((Object[])getCurrentArguments()[1])[0]);
            }
        }).anyTimes();

Next, I'm setting the Mocked ISageAPIProvider as the main API Provider for the sagex apis. This means that my new mock API Provider (which only handles 3 commands) will handle ALL communication that goes throw the sagex apis.

SageAPI.setProvider(prov);

And the final part of setting up the test case is that I'm adding 4 mediafiles to my map, that simply map the mediafile to single piece of metadata (which is the year). This is an extremely simple mapping, the Year, is the only thing that I'm concerned about.

        simpleMetadataMap.put("mediafile1", "2010");
        simpleMetadataMap.put("mediafile2", "ten");
        simpleMetadataMap.put("mediafile3", "");
        simpleMetadataMap.put("mediafile4", null);

Now that the setup is done, the rest is your standard jUnit testing, where I run throw 4 tests, one for each mediafile.

        IMetadata md = AiringMetadataProxy.newInstance("mediafile1");
        assertEquals(phoenix.metadata.GetYear("mediafile1"),2010);

AiringMetadataProxy is a Java Proxy class that Proxies the IMetadata interface against the native Airing object. (ie, it calls Apis on the Airing, like GetShowYear)

Next I run through 4 tests, using the SageMediaFileMetadataProxy, which is Proxy that sends all Metadata requests to the GetMediaFileMetadata api for the given MediaFile.

        md = SageMediaFileMetadataProxy.newInstance("mediafile1");
        assertEquals(phoenix.metadata.GetYear("mediafile1"),2010);

This might seem like a lot of work, just to test how the 2 proxy classes handle the fact that a Year can be a Number, String, Empty or Null, but this actually tests a lot more. It tests whether or not the underlying Proxy classes with handle the conversion of String to Number, and since this logic is used by numeric fields in IMetadata, then I know, if this test passes then the others should as well. (I know, this is where I get fairly relaxed on my junit assumptions).

This test illustrates that the entire process, while ultimately depending on SageTV, can work quite well without SageTV being present, as long as we use the sagex apis and we provider a provider to handle the communication.

I use a combination of easymock and Stub Provider in my unit testing, but you could have just as easily created your own stub provider and not used easymock at all. (I used it here, because I wanted to better understand how easymock worked).

        ISageAPIProvider prov = new ISageAPIProvider() {
			@Override
			public Object callService(String context, String name, Object[] args) throws Exception {
                                // don't worry about context
				return callService(name, args);
			}
			
			@Override
			public Object callService(String name, Object[] args) throws Exception {
				if ("GetShowYear".equals(name)) {
					return simpleMetadataMap.get(args[0]);
				} else if ("GetMediaFileMetadata".equals(name)) {
					return simpleMetadataMap.get(args[0]);
				} else if ("IsMediaFileObject".equals(name)) {
					return simpleMetadataMap.containsKey(args[0]);
				} else {
					System.out.println("Unhandled: " + name);
					return null;
				}
			}
		};
        SageAPI.setProvider(prov);

Ultimately the flexibility is there for you to handle the API calls however you want.

In some of my tests, I simply need to return some media files, based on whether the API is recievng "T" (Recordings) or "TL" (Archived Recordings). So, it looks like this.

        ISageAPIProvider prov = createNiceMock(ISageAPIProvider.class);
        expect(prov.callService(eq("GetMediaFiles"), aryEq(new Object[] {"T"}))).andAnswer(new IAnswer<Object[]>() {
            public Object[] answer() throws Throwable {
                return new Object[] {"1","2"};
            }
        });

        expect(prov.callService(eq("GetMediaFiles"), aryEq(new Object[] {"TL"}))).andAnswer(new IAnswer<Object[]>() {
            public Object[] answer() throws Throwable {
            	System.out.println("GETMEDIAFILES called");
                return new Object[] {"3","4"};
            }
        });

And again, because every sagetv object is an "Object" I can return whatever I want to represent a mediafile, as long as I'm consistent in using it that way.

Hope this gives better clarification on how to use easymock and sagex apis (or simply create your own provider to emulate sagetv.

And one final note about unit tests. While I tend to run them from the IDE, I do have an ant task that runs through the entire set of cases as well. I'm not using a TestSuite, since I need each test in a separate JVM to ensure propper setup and tear down of the Junit tests. (I'm lazy, and I don't want to do my own setup and teardown in each test case )

Here's my ant task...

	<target name="unittests" depends="build">
		<mkdir dir="${target}/testclasses"/>
		<mkdir dir="${target}/unittests"/>
		<javac debug="true" classpathref="project.class.path" source="1.5" target="1.5" srcdir="src/test/java" destdir="${target}/testclasses"/>
		<junit printsummary="yes" haltonfailure="yes" fork="true">

		  <classpath>
		  		<pathelement path="${target}/testclasses"/>
				<pathelement location="target/build/classes" />
				<fileset dir="lib" includes="*.jar"/>
				<pathelement path="${sage.home}/Sage.jar"/>
		  </classpath>

		  <formatter type="xml"/>
			<test name="test.junit.TestMenus" haltonfailure="no" todir="${target}/unittests"/>
			<test name="test.junit.TestOnlineVideos" haltonfailure="no" todir="${target}/unittests"/>
			<test name="test.junit.TestMiscStuff" haltonfailure="no" todir="${target}/unittests"/>
			<test name="test.junit.TestTaskManager" haltonfailure="no" todir="${target}/unittests"/>
			<test name="test.junit.TestFanartLocations" haltonfailure="no" todir="${target}/unittests"/>
			<test name="test.junit.TestUtilAPI" haltonfailure="no" todir="${target}/unittests"/>
			<test name="test.junit.TestDateParsers" haltonfailure="no" todir="${target}/unittests"/>
			<test name="test.junit.TestConfigurationMetadata" haltonfailure="no" todir="${target}/unittests"/>
			<test name="test.junit.TestFormattedTitles" haltonfailure="no" todir="${target}/unittests"/>
			<test name="test.junit.TestImageAPI" haltonfailure="no" todir="${target}/unittests"/>
			<test name="test.junit.TestMetadata" haltonfailure="no" todir="${target}/unittests"/>
			<test name="test.junit.TestKeypadRegexSearch" haltonfailure="no" todir="${target}/unittests"/>
			<test name="test.junit.TestEDLCommercials" haltonfailure="no" todir="${target}/unittests"/>
			<test name="test.junit.TestMediaResources" haltonfailure="no" todir="${target}/unittests"/>
			<test name="test.junit.TestSkinAPI" haltonfailure="no" todir="${target}/unittests"/>
			<test name="test.junit.TestTrailersVFS" haltonfailure="no" todir="${target}/unittests"/>
			<test name="test.junit.TestTransformFactory" haltonfailure="no" todir="${target}/unittests"/>
			<test name="test.junit.TestGetYearError" haltonfailure="no" todir="${target}/unittests"/>
			<test name="test.junit.TestDynamicVariables" haltonfailure="no" todir="${target}/unittests"/>
			<test name="test.junit.TestReplacements" haltonfailure="no" todir="${target}/unittests"/>
			<test name="test.junit.TestRemoveAll" haltonfailure="no" todir="${target}/unittests"/>
			<test name="test.junit.TestProgressMonitor" haltonfailure="no" todir="${target}/unittests"/>
			<test name="test.junit.TestEvents" haltonfailure="no" todir="${target}/unittests"/>
			<test name="test.junit.TestDownloadManager" haltonfailure="no" todir="${target}/unittests"/>
			<test name="test.junit.TestUserProfiles" haltonfailure="no" todir="${target}/unittests"/>
			<test name="test.junit.TestFanartDownloading" haltonfailure="no" todir="${target}/unittests"/>
			<test name="test.junit.TestVFSBuilder" haltonfailure="no" todir="${target}/unittests"/>
			<test name="test.junit.TestFilePatterns" haltonfailure="no" todir="${target}/unittests"/>
			<test name="test.junit.TestSearchQuery" haltonfailure="no" todir="${target}/unittests"/>
			<test name="test.junit.TestMetadataProviderFactory" haltonfailure="no" todir="${target}/unittests"/>
			<test name="test.junit.TestMediaBrowser" haltonfailure="no" todir="${target}/unittests"/>
		</junit>		
	</target>

[stuckless]

Originally Posted by jreichen

I suppose there's a certain procedure that needs to be followed to set up Jetty? If anybody should know it would be me but I haven't tried it

I'm sure what you mean? Do you mean getting Jetty running inside of eclipse? Or getting the Sage Webserver running inside eclipse (which you can do, if you build sagex friendly versions of the nielm jars)

For bmt, I use GWT, so it has it's own Jetty webserver. There's nothing special that I have to do to get it working.

[jreichen]

Originally Posted by stuckless

I'm sure what you mean? Do you mean getting Jetty running inside of eclipse? Or getting the Sage Webserver running inside eclipse (which you can do, if you build sagex friendly versions of the nielm jars)

I mean getting my build of Jetty running - either inside or outside of Eclipse. I customized Jetty to work with Sage (tried to keep that to a minimum as much as possible) so I can't be sure of what it will take to set it up on a machine without Sage until I try it.

Originally Posted by stuckless

For bmt, I use GWT, so it has it's own Jetty webserver. There's nothing special that I have to do to get it working.

That's probably done with an Eclipse plugin, and I'm not going to add something like that to my todo list.

Thanks for putting together the mock example. One thing I was hung up on was how to get a Sage Airing or MediaFile object and then pass it around to other APIs. It looks like you've covered that in your example.