Removing the JAR package type from plugins

[Slugger]

Why do we (I) want to drop the JAR package type in plugin manifests? In the current system, jars must (or really should) be packaged as separate, independent plugins, which are then depended on by any plugins that need the plugin. This is ok (well not really), but leads to a system where one user ends up owning the jar file and others are at the mercy of that owner if it ever needs to be
upgraded.

Under the current system, there's nothing preventing people from packaging conflicting jars with different plugin ids and if a user installs the magic combination of plugins, they end up with jar conflicts on their classpath.

A plugin dev can accidentally include jar files as part of their plugin which are already available as a plugin by someone else, again leading to classpath collisions.

The need to package and maintain jar files as separate plugins is very tedious (especially when there are better tools to do this).

My solution (and fall & probably winter project) is to eliminate the JAR package type and instead require manifests to declare their jars from known repositories (specifically maven central, jcenter & a custom repository specific for SageTV). Plugin manifests declare all jars needed for their plugin and the Sage core will then dynamically "build" the JARs folder at SageTV launch based on the installed & enabled plugins.

The biggest advantage is that the core will then manage and determine all the jars you need and resolve conflicts between plugins. For example, Plugin A declares it's using commons-codec 1.0 and Plugin B declares it's using commons-codec 1.2. The core will notice this and do one of two things (I haven't decided which is best yet):

1. Install commons-codec 1.2 jar and ignore the 1.0 jar. This is the default behaviour and probably how I'd go. This is usually safe, but not always as you can break the code relying on the older jar.

2. The other option is to detect the conflict between plugins and reject the 2nd dev who submits requiring that second dev to either upgrade/downgrade to whichever version was submitted by the first plugin to register.

Assuming #1, when a user installs plugins, the declared jars for all those plugins are "merged" together at startup and then the resulting list of jars that satisfies all installed plugins are then automatically installed to the SageTV/JARs folder.

Another part of this plan calls for hosting of all plugins in one of three locations: maven central, jcenter, or a SageTV specific repository. The latter being hosted on bintray.

And the final piece of this will be single command build and delivery of your plugins to bintray via a gradle plugin. With this gradle plugin, you can literally run a single command that will create a new SageTV plugin project that can immediately be loaded in your favourite IDE. Another command will provide instant publishing of your plugin to bintray and automatic submission to the Sage plugin repository, including automatic creation of your XML manifest file so that Sage users see your new builds.

So as to not rock the boat too much, I will continue to use an XML manifest format for the repository within Sage server/client, however the changes I'm proposing will make the Sage v7 repository invalid. This will result in a new plugin repository for v9.

I've been playing around with some prototypes of the new manifest format here and there for the last few weeks and I actually have a build of the Sage server that consumes the new format and installs plugins using the new format, including dynamically building the SageTV/JARs folder at startup. I've declared summer is over and after playing around for an hour or two here and there for the last few weeks, I'm ready to dive into this and take it to the next level.

I know this breaks the use of the Sage v7 repo, but I think it solves a lot of problems and tedious things about the v7 plugin repository and will eventually make the lives of developers that much easier.

So is this something that piques the interest of the main/official repository? I am definitely implementing this in my fork and it will be the fork I end up moving to when I move my prod box to v9. And it will be the only way I write plugins going forward. If this is something that the majority has zero interest in, that's cool and I'll just work away on it on my own. If it's something that may be of interest to main/official, but not exactly as I've described then let's talk about how people see this working and see if we can come up with a strategy that we can get into main.

So here's what a manifest will look like under this system. Pretty much the same except you'll see a new <Jars> section and packages of type JAR are no longer acceptable. Again, I have a prototype build of the Sage server that already processes this manifest and successfully installs the plugin, including the jar file management.

<SageTVPlugin>
  <Name>SageTV OSS Sample Plugin</Name>
  <Identifier>sage-oss-sample</Identifier>
  <Version beta='true'>0.1.0</Version>
  <Author>Slugger</Author>
  <CreationDate>2015.09.19</CreationDate>
  <ModificationDate>2015.09.19</ModificationDate>
  <ServerOnly>true</ServerOnly>
  <PluginType>Standard</PluginType>
  <ImplementationClass>sagex.oss.plugins.samples.SamplePlugin</ImplementationClass>
  <ResourcePath>plugins/sage-oss-sample</ResourcePath>
  <Dependency>
    <Plugin>sagex-api-remote</Plugin> <!-- this would pull in jetty, etc, etc and setup remote api access; you should still list sagex-api jar in <Jars> via dependencies{} if using it -->
    <MinVersion>7.1.9</MinVersion>
  </Dependency>
  <Dependency>
    <Core />
    <MinVersion>9.0.0.0</MinVersion>
  </Dependency>
  <Dependency>
    <JVM />
    <MinVersion>1.7</MinVersion>
  </Dependency>
  <Jars>
    <Jar>sagex.oss.plugins.samples:sage-oss-sample:0.1.0</Jar> <!-- Always include yourself, obviously! -->
    <Jar>sagex:sagex-api:7.1.9</Jar>
  </Jars>
  <Package>
    <PackageType>System</PackageType>
    <Location>https://dl.bintray.com/slugger/sage-oss-plugins/sagex/oss/plugins/samples/sage-oss-sample/0.1.0/sage-oss-sample-0.1.0-conf.zip</Location>
    <MD5>dd5eaa824b6fd638727a9eecf9b6576a</MD5>
  </Package>
  <Description><![CDATA[
A sample SageTV plugin that really doesn't do anything useful, but
	is a complete working example that can be installed on your SageTV server.
  ]]></Description>
  <Webpage>https://github.com/Slugger/sage-oss-sample/</Webpage>
  <ReleaseNotes><![CDATA[
Initial release.
  ]]></ReleaseNotes>
</SageTVPlugin>

The above manifest was automatically built by my gradle plugin, which needs some work, but for a PoC does quite well.

build.gradle

plugins {
	id 'sagex.sagetv-oss' version '1.0.0'
	id 'java'				// change to 'groovy' if you want to write in groovy
	id 'eclipse'			// change to 'idea' if you're an IntelliJ user
	id 'maven-publish'		// needed if you're publishing your plugin to a maven repo
	id 'com.jfrog.bintray' version '1.1' // if that maven repo is at bintray, use this
}

group = 'sagex.oss.plugins.samples' // when publishing to maven repo, this is the group
version = '0.1.0'
sourceCompatibility = '1.7' // JVM dependency is set based on this value; if not set, it is set based on the JVM version used to compile

project.ext.resourcePath = 'plugins/sage-oss-sample' // needed here to avoid circular reference

dependencies { // Anything from group 'sage' will not be included in generated manifest
	compile 'sage:sagetv-core:9.0.0.0'
        compile 'sagex:sagex-api:7.1.9'

	/*
	 * By default, dependencies will include their transitive dependencies when
	 * processed by SageTV.  You can suppress inclusion of these transitive deps
	 * by setting the transitive property to false
	 * 
	 * compile('some.package:foo:1.0') { transitive = false }
	 */
}

/* Package up my Sage plugin packages -- NEVER jars */

/*
 *  So this is a fake zip of useless stuff, but a real plugin would zip up
 *  config files, graphics, STVs, or whatever else your plugin needs to operate.
 *  The only thing you don't ever put in these packages are jar files to be included
 *  in the SageTV classpath
 *
 *  Don't forget to add this to your artifacts for upload to bintray!
 */

task mkZipSrc {
	def src = new File(project.buildDir, project.resourcePath)
	outputs.dir src.parentFile
	doLast {
		src.mkdirs()
		new File(src, 'foo.txt') << 'foobar'
	}
}
 
task mkPkgs(type: Zip) {
	classifier = 'conf'
	from mkZipSrc
	baseName = project.name
	destinationDir = new File(project.buildDir, 'zips')
}

/*
	Your Sage plugin XML manifest will be built based on the contents below.
	Some contents are auto generated based on the other parts of this build file.
*/	
sageManifest {
	name = 'SageTV OSS Sample Plugin'
	//identifier = 'sage-oss-sample' // optional else project.name
	author = 'Slugger'
	created = '2015.09.19'
	isBeta = true
	resourcePath = project.resourcePath
	webPage 'https://github.com/Slugger/sage-oss-sample/'
	isServerOnly = true
	pluginType = 'Standard'
	implementationClass = 'sagex.oss.plugins.samples.SamplePlugin'
	description = '''A sample SageTV plugin that really doesn't do anything useful, but
	is a complete working example that can be installed on your SageTV server.'''
	core '9.0.0.0' // You must specify a core dependency, if you require it
        dependency 'sagex-api-remote' , '7.1.9'
	/*
		The files referred to below must exist when this block executes in order to
		compute the md5 values.  If the files don't exist, the build will fail.
		Setup your task dependencies accordingly.
	*/
	project.files(mkPkgs).files.each {
		pkg type: 'System', location: new URL("https://dl.bintray.com/slugger/sage-oss-plugins/${project.group.replaceAll('\\.', '/')}/$project.name/$project.version/$it.name"), file: it
	}
	releaseNotes = '''Initial release.'''
}

mkSageManifest {
	dependsOn mkPkgs // Make sure your packages are built before trying to generate manifest
}

/* Maven Publishing Tasks Below */

task sourcesJar(type: Jar, dependsOn: classes) {
	classifier = 'sources'
	from sourceSets.main.allSource
}

task javadocJar(type: Jar, dependsOn: javadoc) {
	classifier = 'javadoc'
	from javadoc.destinationDir
}

artifacts {
	archives sourcesJar, javadocJar, mkPkgs
}

I removed the maven publish and bintray upload tasks in this gradle file for brevity, but it works. The only thing that isn't part of this yet is the auto submission to the plugin repo because, well, I still need to write a new plugin submission server, which I'll be starting on when I dive into this project within the next week or so.

With the above gradle file, a plugin dev simply needs to run the following commands:

gradle sageInit

In an empty directory, this command creates a new sample SageTV plugin with a sample java or groovy class (depending on which plugin you include). If you also include the eclipse or idea plugin then this init task will also create the files necessary to allow you to immediately import the new project within your IDE and begin working immediately. If you don't include the java, groovy nor IDE plugins then you get a basic skeleton that would be useful for plugins that don't include code (STVs, logos, themes, etc.). Even these types of plugins can benefit from this gradle plugin by providing automatic build and submission facilities (though admittedly, I haven't tested these types of plugin uses yet).

So you run sageInit to create your plugin project, you write your code, you build your zip packages of additional resources, test it, and when it's all done and working you'd run:

gradle mkSageManifest

If you don't want to host on bintray and don't want automatic submission of your manifest to the plugin repository then you'd run the above, which will build the xml manifest file complete with references to all the jar files you used to build code plus any other packages you created with the md5 hashes computed and included, etc. After this command completes all you'd have to do is manually upload the generated xml file to the plugin repository.

However, you will instead be able to do this:

gradle sageSubmit

This command does the mkSageManifest command but then takes that result and automatically submits it to the plugin repository. I haven't begun to implement this feature yet, but it's coming.

The gradle plugin will provide all of this functionality, but it won't be required to add plugins to my repository, it'll just be a lot more manual work to get there.

Feedback, thoughts?

[PLUCKYHD]

Seems like allot of work but I do like the Jar change personally. I agree that having the users handle the jars was less then ideal and the multiple commitments of the same class libraries wasn't necessary.

I do have one comment on the different version (ie one plugin uses 1.0 and another uses 1.2 of this.jar) you option to deny installation seems the best but also can suck to force a developer to have to update if anything does break. Again this would probably be rare that it would actually break something therefor a simple change in their manifest would kick that problem.

Overall I like the majority of what I see.

[skiingwiz]

Some thoughts about moving to new v9 plugin repository.

• I have no problem with this in general.
• Who will host the plugin submission/validation service?
• Does your plan include importing all/some v7 plugins into the new repository?
• Will your gradle plugin support publishing to the v7 repo as well as the v9?*

* No is not a bad answer to this. Someone else could always add this if it's desirable. I'm not even sure if anyone is planning to continue developing for v7.

[Slugger]

Originally Posted by PLUCKYHD

Seems like allot of work but I do like the Jar change personally. I agree that having the users handle the jars was less then ideal and the multiple commitments of the same class libraries wasn't necessary.

I do have one comment on the different version (ie one plugin uses 1.0 and another uses 1.2 of this.jar) you option to deny installation seems the best but also can suck to force a developer to have to update if anything does break. Again this would probably be rare that it would actually break something therefor a simple change in their manifest would kick that problem.

Overall I like the majority of what I see.

So in my current env, I use the policy that conflicts are resolved automatically by simply using the latest version of the conflicting jars. This is the default policy in gradle (which is what I'm using under the hood to manage the jar files within the JARs folder at runtime). It's the default because it's usually safe, but that's also because it's in the context of a build, not in the context of an application runtime (i.e. in a build, problems are likely to be caught before you release, whereas in the context of the Sage core, we're computing these jars to be used at runtime so if there is a problem, users are going to hit it). I would like to at least start with this policy and see how it goes because with this policy people are free to use whatever jars they want. For the most part this should be ok -- Plugin A uses x-1.0.jar and Plugin B uses x-1.1.jar or even x-1.25.jar. We end up with x-1.25.jar and assuming x is backwards compatible in the 1.n release then all is good (and it should be). We are more likely to hit problems if A uses x-1.0.jar and B uses x-2.0.jar. Depending on the jar, this is still likely to be ok, but we're starting to get into a worrisome situation.

The alternative is to deny Plugin B when its jars conflict with already registered plugins and force B to use whatever version is already registered by someone else. I don't like this because we're back to the "someone owns the jar" problem. But if people keep causing problems then this would be an option.

Originally Posted by skiingwiz

Some thoughts about moving to new v9 plugin repository.
I have no problem with this in general.
Who will host the plugin submission/validation service?

It will be created as a docker image. Anywhere we can host a docker image is a viable candidate.

(Big picture, anyone can run their own container of this image and point their Sage server at it as an additional source of plugins or we run multiple "official" containers where one is "released" plugins and another is "beta testing" plugins, etc. Users who want to install beta plugins can enable the additional containers, etc. But this is way down the road.)

Does your plan include importing all/some v7 plugins into the new repository?

Not really. My hope would be people would port their v7 plugins to this repo. I won't be doing it except possibly for plugins I own (and actively use).

Will your gradle plugin support publishing to the v7 repo as well as the v9?*

* No is not a bad answer to this. Someone else could always add this if it's desirable. I'm not even sure if anyone is planning to continue developing for v7.

No because in order to do the things this plugin does (auto generate the entire manifest, etc.), I need to control the entire environment (all jars must come from known maven repos, plugins are stored on bintray, etc.). That controlled environment is only available in v9, with my proposed changes here.

With that said, I've written another gradle plugin already (and it's currently published) for Sage v7, which does some of the things my v9 plugin does (or will do). I can only do a subset of things in the v7 plugin because the plugin format/repository/etc. is basically a free for all so I can't deduce many of the things I'm able to with my v9 changes.

I've ported my sdepg plugin to use gradle and my Sage 7 gradle plugin for building and automatic submission (you just have to wait for the email and click the link). Have a look there if interested. You'll notice a lot of the manifest can't be deduced and requires a lot more manual configuration. Again, that's because you have to provide jar depdencies as sagetv plugins instead of just pulling them from the gradle dependencies config, etc. It works, but it's just a stop gap allowing me to quickly publish updates to the OSS sdepg plugin, should that become necessary before I finish my work on this.

[GKusnick]

I have a few questions and issues with this proposal.

When you talk about merging the JAR requirements and dynamically installing JARs at SageTV startup, where do you propose to install them from? Will there be a local cache, or do you plan to pull them down from the net every time? It would be a shame if it's no longer possible to run SageTVClient without an internet connection.

Breaking compatibility with the V7 plugin repository seems like it will create a big disincentive to upgrade to V9, especially for plugin devs. I'll use myself as an example. I have not yet set up a V9 dev system; I will at some point, but right now I have other priorities. When I get around to doing so, there will apparently be a whole new set of tools to learn, and that's OK, but it's one reason I'm not in a rush to do it.

Here's how I imagined it would go: I set up a V9 test box, replicate my existing V7 configuration on it, validate it for functionality and stability, and then move my production server over to V9. Having done that, I'd then be in a position to get my feet wet with V9 development at my leisure.

Now you're telling me that if this proposal is adopted, that plan is out the window. No matter how stable V9 is, I won't be able to migrate my production server without first diving headlong into V9 development in order to port all my plugins over and find replacements for any orphaned V7 plugins on which I depend. There's no leisurely learning curve in this scenario, no incremental migration of one plugin at a time. It's all or nothing; I can't switch production versions until the whole job is done. And until it's done, I have to keep a V7 dev system alive to address any emergencies that may arise.

So it may help if we can spell out exactly what "the whole job" entails. Does it require coding to a new plugin API? Recompiling JARs? Or just repackaging existing JARs with new manifests? Could there be some sort of automated tool that does 90% of the work of porting a V7 plugin to V9, with just a few blanks to be filled in by the plugin dev?

Basically, the closer we can stay to scenario #1, the sooner plugin devs are likely to come on board with V9. A radical discontinuity this early in the game is likely to leave a lot of people and plugins behind.

[Monedeath]

....And Jetty is a known quantity as being the first thing that would break painfully as things stand currently in Sage with this proposed change.

[tmiranda]

There was a long discussion about this in another thread. The conclusion there was that each plugin should have it's own classpath where it's jars would go.

[stuckless]

I like the idea of using gradle (or any set of tools really) to jumpstart new plugin development and to provide the facilities to quickly publish updates, handling everything from packaging to notification.

I've given this a lot of thought as well, and I think there is concern when we start cutting off old plugins, entirely, and start making devs port plugins to V9. Echoing, Greg's concerns, I can see a lot of devs not going to V9 until there is very stable build, and if the devs don't bring the plugins to v9, then many people might not go v9. ie, SageTV is great, but the plugins make it awesome.

I think that we should still support the V7 repo format in V9. I'd go further to say that I think we should support multiple repos, where the repo could be a v9 or v7 format.

Internally, there is no reason why we can't support both v7 and v9 formats, and we could even have a "strict mode" whereby if enabled, you can ONLY load v9 plugins. If it's disabled, then you get access to v7+v9, but doing that could lead to some instability.

In my mind v9 plugins, when installed, would get thier own dedicated area for resources and configuration settings (I think Greg proposed the idea of storing plugin configuration settings with the plugin, to make it easier to pick it up and isntall it somewhere else).

In supporting v7 and v9 plugins, it could lead to some instability, which is by I think having a strict mode is important. But I think we can mitigate some of that by doing jar inspections during isntallation (and/or on startup). ie, if strict mode is enabled, we can catalog the jar files (ie, read the MANIFEST.MF) and try to detect possible duplication of jars in that way. In some ways we need something like this anyways, since we can't prevent people from simply copying jars to the JARs area. Something like this would have caught the whole lucene jar issue that we have now.

Over time, I think as more and more people get on board with v9 then v7 plugins will eventually get migrated to v9 and people can run entirely in strict mode.

I'm less concerned about publishing my plugins to v7 and v9 and more concerned that I can still get access for v7 plugins while running v9. I'l probably never update a v7 plugin once I start v9 plugin development. If people want to get the features of v9 then upgrade.

Originally Posted by tmiranda

There was a long discussion about this in another thread. The conclusion there was that each plugin should have it's own classpath where it's jars would go.

This would be good, but to do this right you need a solid module system like osgi. Java 9 is bringing in project JigSaw, a new modular system, and maybe we that can help solve the class loading issues.... Until then, I say we wait on that, and see what Java 9 brings.

@GKusnick - All jars are downloaded locally. We go to a repo to find it, much like we do today with SageTV repos, and then they get downloaded and stored locally. ie, there would never be a need to be online, once you you've installed a plugin.

[Slugger]

Originally Posted by GKusnick

I have a few questions and issues with this proposal.

When you talk about merging the JAR requirements and dynamically installing JARs at SageTV startup, where do you propose to install them from? Will there be a local cache, or do you plan to pull them down from the net every time? It would be a shame if it's no longer possible to run SageTVClient without an internet connection.

First pull is from the net (predefined maven repos: maven central, jcenter & a Sage specific repo) and the jars cached locally. Subsequent uses of the same jar are pulled from the local cache always. References to new/previously unused jars would be pulled from the net on first use then stored in local cache for subsequent uses.

Haven't investigated the SageClient side yet, but I feel like it would be possible to farm the jar "build" back to the server always and ship the jars to the client, if a no net access SageClient is truly a concern.

EDIT: SageClient directly downloads plugin packages when you install a plugin on a client, right? If so, then net access is a requirement to install plugins in v7 on a client. If not, then I'd have to look into that.

Breaking compatibility with the V7 plugin repository seems like it will create a big disincentive to upgrade to V9, especially for plugin devs. I'll use myself as an example. I have not yet set up a V9 dev system; I will at some point, but right now I have other priorities. When I get around to doing so, there will apparently be a whole new set of tools to learn, and that's OK, but it's one reason I'm not in a rush to do it.

Agreed, existing devs with an existing v7 system that works are less likely to upgrade. But I counter that new potential devs are more likely to develop on a system with more modern, dev friendly tools. This solution definitely doesn't work for everyone.

Here's how I imagined it would go: I set up a V9 test box, replicate my existing V7 configuration on it, validate it for functionality and stability, and then move my production server over to V9. Having done that, I'd then be in a position to get my feet wet with V9 development at my leisure.

Now you're telling me that if this proposal is adopted, that plan is out the window. No matter how stable V9 is, I won't be able to migrate my production server without first diving headlong into V9 development in order to port all my plugins over and find replacements for any orphaned V7 plugins on which I depend. There's no leisurely learning curve in this scenario, no incremental migration of one plugin at a time. It's all or nothing; I can't switch production versions until the whole job is done. And until it's done, I have to keep a V7 dev system alive to address any emergencies that may arise.

So it may help if we can spell out exactly what "the whole job" entails. Does it require coding to a new plugin API? Recompiling JARs? Or just repackaging existing JARs with new manifests? Could there be some sort of automated tool that does 90% of the work of porting a V7 plugin to V9, with just a few blanks to be filled in by the plugin dev?

Basically, the closer we can stay to scenario #1, the sooner plugin devs are likely to come on board with V9. A radical discontinuity this early in the game is likely to leave a lot of people and plugins behind.

Require coding to a new plugin API? No.

Recompile JARs? No. (Assuming the Sage.jar in OSS remains backwards compatible with v7; nothing I'm doing here would break any existing compatibility.)

Repackage JARs with new manifests? Yes. This is basically what this is all about. Remove the JAR package type from manifests and replace it with pointers to jars in maven repos then let the core (with help from gradle) manage those jars at runtime.

Existing v7 plugins can be migrated simply by generating a new manifest, dropping any JAR package types in their old manifest and instead moving those jar files to a maven repo known to Sage. Though untested, all v7 plugins today should be able to be migrated without any code recompile, etc. Just generate a new manifest using the new format, move jars to a maven repo and that should be it. I PoC'd this idea with one of my plugins with success, but that doesn't mean there won't be bumps in the road with other plugins that I haven't anticipated. However, I'm fairly confident all (or certainly most) plugins could be migrated with nothing more than a new manifest and relocation of jar files.

Furthermore, existing v7 plugins that do not contain any JAR package types in their manifest and do not depend on any plugins with JAR packages would "just work" without modification. Simply submit the unmodified manifest to this new repo and that should be it.

Could this migration be automated? Maybe. As you say, like 90% of it probably can be, which leaves a fill in the blank for a dev.

Originally Posted by Monedeath

....And Jetty is a known quantity as being the first thing that would break painfully as things stand currently in Sage with this proposed change.

Break how? The jetty plugin is already a bit of a mess, which is why things blow up if you try to run it alongside Java 8, for example. Not to mention, the jetty plugin is 3 versions behind stable. Jetty plugin uses jetty 6.x. If you go to the jetty web site, in big bold letters it reads that jetty 7.x & 8.x have long since been eol'd.

Is someone going to have to update the jetty plugin? Yes. But whether or not main goes this route, someone desperately needs to update the jetty plugin anyway.

Originally Posted by tmiranda

There was a long discussion about this in another thread. The conclusion there was that each plugin should have it's own classpath where it's jars would go.

Yeah, I think I raised this issue in that thread at some point: If all plugins run in their own isolated classloader then plugins can't talk amongst each other (via calling into another plugin's jar). This isn't necessarily bad, but it would break a lot of existing v7 plugins. For example, SJQ calls into SageAlert, but if they each run in their own classloader then SJQ can't see SageAlert's classes at runtime -- without including its own copy of the SageAlert jar, which then causes a whole new set of problems. If you let plugins see the other's classloader to access those classes then you're not really gaining anything by using separate classloaders -- and actually you're probably just creating a bigger mess.

This strategy would most likely break any plugin in the v7 repo that depends on another plugin that contains a JAR package type (because, presumably, you're depending on that plugin because you're calling into its jar file).

[Fuzzy]

I think the first step is to simply create the v9 repository, which would still be functionally similar to the current one, but implement <include> tags to bring in additional repository files, including the v7 repository xml, and a local 'AdditionalPlugins.xml' file that, could be nothing more than a series of further <includes> to additional plugin sources. This, combined with the other improvements I had talked about in the previous discussion - the ability for multiple versions of a plugin to exist in the repository, and only show the highest version that is compatible with your system (currently, you can't really have more than one plugin with a given name).

[Slugger]

Originally Posted by stuckless

I think that we should still support the V7 repo format in V9. I'd go further to say that I think we should support multiple repos, where the repo could be a v9 or v7 format.

Internally, there is no reason why we can't support both v7 and v9 formats, and we could even have a "strict mode" whereby if enabled, you can ONLY load v9 plugins. If it's disabled, then you get access to v7+v9, but doing that could lead to some instability.

The reason I don't think you can support both v7 and a new gradle based v9 jar management system is because the v7 repo is a free for all w.r.t. jar file management. The jars that are zipped and referenced in the v7 repo are a mixed bag. If every jar zipped up in the repo were properly tagged with a version number (and that version number were correct) then perhaps something could be done to allow a v7 and v9 repo to coexist. But as it stands, many jars don't have a version and so gradle won't be able to detect nor resolve conflicts and so if a v9 plugin refers to a jar that a v7 plugin also refers to via a Sage 7 JAR package reference then you end up with classpath conflicts.

Even if something were possible to allow the coexistence, coding that logic would be a nightmare that is likely to lead to more instability and issues.

Edit: Yes, the plugin itself has a version for the jar file, but gradle can't work off of that. Gradle works off of the version encoded in the file name. And many jars repackaged in the sage repo don't follow that naming convention. Furthermore, even if they did, gradle would need to also know the group id otherwise it wouldn't recognized two jars with the same name as being the same anyway. And since the jars in the sage repo aren't in a maven repo format, gradle isn't going to be able to sort it out automatically and you'll end up with classpath conflicts.

In my mind v9 plugins, when installed, would get thier own dedicated area for resources and configuration settings (I think Greg proposed the idea of storing plugin configuration settings with the plugin, to make it easier to pick it up and isntall it somewhere else).

Yeah, one of my validation points for plugin submissions is to enforce installation only to SageTV/plugins/<id>/ (or something like that). I wouldn't be able to stop code from writing elsewhere at runtime, but I can scan the zip files and make sure installation only occurs in the plugin resource location.

In supporting v7 and v9 plugins, it could lead to some instability, which is by I think having a strict mode is important. But I think we can mitigate some of that by doing jar inspections during isntallation (and/or on startup). ie, if strict mode is enabled, we can catalog the jar files (ie, read the MANIFEST.MF) and try to detect possible duplication of jars in that way. In some ways we need something like this anyways, since we can't prevent people from simply copying jars to the JARs area. Something like this would have caught the whole lucene jar issue that we have now.

Or we have just a v9 repo.

In my env, I do prevent people from copying random jars to the JARs area because right now the way it works on my dev box is when you start Sage it deletes everything in JARs, scans the list of activated plugins, computes the jars needed for those enabled plugins, resolves conflicts and copies the resulting list of jars to JARs folder and starts the Sage JVM. I suppose there's a small window available for someone (malicious or otherwise) to inject random jars between the time I copy the computed jars into the JARs folder and start the JVM, but otherwise, random jars can't be loaded.

[stuckless]

@Sugger - I don't disagree with anything you are saying, but I still think you should try to solve the v7 problem

I bet there are plugins in SageTV right now that are being used, but the developer have since moved on.

Could we look at using a legacy mapping feature whereby when a library is referenced in V7 we perform a lookup and provide the groupid that way, etc... There is a finite list of v7 jars that used.

The clean state approach is good, but, my fear is that we'll end up with a Blackberry problem. Great architecture, but people are not "porting" the plugins.

This isn't easy stuff, but we should really think hard before we kill an entire ecosystem.

BTW... not sure deleting/copying the jars is necessary... if you "know" all the jars that used by the plugin manifest, you can just build the classpath url ist based on that information. that way even if there are duplicate jars with different versions, etc, then they don't get added to the v9 classloader, and even if people manually copy jars into the v9 area, they don't get picked up.

Anyways, great work on moving this along... I'm sure it'll be great for plugin devs once all the kinks are worked out.

[Slugger]

I'd be more inclined to just repackage and port v7 plugins to the new repo format than try to sort out all the logic required to maintain support between the two.

[stuckless]

Originally Posted by Slugger

I'd be more inclined to just repackage and port v7 plugins to the new repo format than try to sort out all the logic required to maintain support between the two.

I think that would be a big task for you as well.

[Fuzzy]

Well, for starters, if we pointed the v9 sage.jar to look at a new, gitable location for it's plugins xml file, and started with a copy of that file in the git, than it could be altered as needed to affect compatibility with whatever we do, and would make no immediate change to how things operate. Plugins that are found to not function properly could then easily have:

     <Dependency>
          <Core/>
          <MaxVersion>7.1.9.256</MaxVersion>
     </Dependency>

tags added to them, and we'd end up with a much more accurate inventory of what works and what doesn't. And be able to do so for plugins whose owners might have disappeared.