Plugin Dependencies: A blessing and a curse!

[Slugger]

A friendly warning for all you plugin devs out there...

Been tracking down a problem with SRE for the last few days with another user. Finally got my hands on a stack trace, which easily pointed me to the problem:

ClassNotFoundException: org.apache.commons.codec.DecoderException

So I was puzzled at first, but didn't take long to realize the issue...

I use the Apache xmlrpc client in SRE, xmlrpc depends on HttpClient3, and HttpClient3 depends on commons-codec (this is the dependency I failed to realize when packaging httpclient3 plugin jar). I never directly use commons-codec in my code so never had a compile error, etc. When I tested the plugin I already had commons-codec installed in my Sage\JARs dir because some other plugin I installed depended on it. So all my testing is fine locally, but this user didn't have commons-codec installed so SRE died almost immediately on startup everytime. Once I got the stack trace from this user it was easy to go back and figure out where the dependency was missing. Of course I also had to trace back up all the plugins that depended on httpclient3 and update their dependency to a new version of httpclient3, etc. Basically a big mess!

So my lesson here, be very, very careful with plugin dependencies. Ideally, plugins are developed on a vanilla, fresh install of Sage so as to force yourself to properly identify all dependencies and package them properly. Realistically, I don't have an extra server/license to practice what I preach so now I'll just be more mindful when packaging my jars up for the repository.

In case anyone else was using the HttpClient3 plugin for something, I have updated it in the repository with the correct dependency on commons-codec (and commons-logging).

[PLUCKYHD]

Ouch those are the problems that make you want to throw your computer out the window..

all in all though even with the extra effort it takes us devs. It one of my favorite additions to v7.

[GKusnick]

A quick Google search shows that there are a number of tools out there for automatically analyzing dependencies between JARs and classes and calculating their transitive closure. I haven't used any of them so I can't make specific recommendations, but it looks like this problem is solvable with existing tools.

The problem that keeps biting me is when I add a function to a library to support a higher-level plugin, and then forget to update the MinVersion dependency in the plugin to pull in the library upgrade automatically.

[stuckless]

Originally Posted by GKusnick

A quick Google search shows that there are a number of tools out there for automatically analyzing dependencies between JARs and classes and calculating their transitive closure. I haven't used any of them so I can't make specific recommendations, but it looks like this problem is solvable with existing tools.

The problem that keeps biting me is when I add a function to a library to support a higher-level plugin, and then forget to update the MinVersion dependency in the plugin to pull in the library upgrade automatically.

The problem with some of those tools is that they rely on a proper jar manifest to work. It became quite clear to me when I wrote the jar duplicate code in bmt, that many jars simply do not have a proper jar manifest Actually, even my jars didn't have valid manifest with version number and vender originally That being said, it's probably worth looking into. Btw.. I've been bit by the min-version as well

Another issue that I recently ran into was a case where a user had done a clean Sage7 install and bmt was failing. It had failed because there was a commons-lang-2.2 and 2-4 in the JARs directory. This should not happen using the plugin manager, but i was wondering if it's possible for a plugin to package JARS as system items and bypass the dependenices. That's the only way I can a see a clean install having duplicate jars in the JARs folder. I hope that's not the case.

[PLUCKYHD]

Originally Posted by stuckless

l

Another issue that I recently ran into was a case where a user had done a clean Sage7 install and bmt was failing. It had failed because there was a commons-lang-2.2 and 2-4 in the JARs directory. This should not happen using the plugin manager, but i was wondering if it's possible for a plugin to package JARS as system items and bypass the dependenices. That's the only way I can a see a clean install having duplicate jars in the JARs folder. I hope that's not the case.

I think that might be possible hopefully sage can add a check for that if they don't already. I saw that post and was wondering the same thing. I know originally you could also include more than one jar in the zip if it was a library plugin I am hoping that is not the case either as I haven't tried that again to know if it is still possible that was early on.

[Slugger]

Originally Posted by stuckless

The problem with some of those tools is that they rely on a proper jar manifest to work. It became quite clear to me when I wrote the jar duplicate code in bmt, that many jars simply do not have a proper jar manifest Actually, even my jars didn't have valid manifest with version number and vender originally That being said, it's probably worth looking into. Btw.. I've been bit by the min-version as well

Another issue that I recently ran into was a case where a user had done a clean Sage7 install and bmt was failing. It had failed because there was a commons-lang-2.2 and 2-4 in the JARs directory. This should not happen using the plugin manager, but i was wondering if it's possible for a plugin to package JARS as system items and bypass the dependenices. That's the only way I can a see a clean install having duplicate jars in the JARs folder. I hope that's not the case.

AFAIK, there's no stopping the situation where stuckless has already added commons-io to the repository, which provide commons-io-1.3.jar (or whatever version it is) and if I'm not careful and paying attention to what's in the repository already, I could just easily add the plugin apache-commons-io with commons-io-1.4.jar. Once a user installs a plugin of mine that depends on apache-commons-io and installs another that depends on stuckless' commons-io then you've got duplicate jars in the jars folder. By duplicate, of course, I mean different jars providing the same classes in the same package.

I believe the onus is on the devs to ensure that conflicting jars aren't introduced into the repository. It'd be ideal to have the plugin manager mange this, but it just seems like there's too many ways to get around it for an auto checker to cover every conceivable scenario.

[tmiranda]

I'd like to see an "anti-dependency" that specifies which plugins are incompatible. I'd also like to be able to update the plugin manifest to include the anti-dependency so that if we discover that two plugings conflict we can update the manifest XML WITHOUT actually updating the entire plugin. AFAIK as it stands now you can't update the manifest unless the version number increases.

[PLUCKYHD]

Originally Posted by tmiranda

I'd like to see an "anti-dependency" that specifies which plugins are incompatible. I'd also like to be able to update the plugin manifest to include the anti-dependency so that if we discover that two plugings conflict we can update the manifest XML WITHOUT actually updating the entire plugin. AFAIK as it stands now you can't update the manifest unless the version number increases.

This was discussed during the private beta and Sage is completely against it and their reasons IIRC where because they didn't want plugins to start excluding other plugins and I agree.

Really the only time you would have anti dependency that I can think of it with STVi's trying to change the same area of the stv.

Basically we have to track and let users know about those on our own.

[GKusnick]

Originally Posted by Slugger

I believe the onus is on the devs to ensure that conflicting jars aren't introduced into the repository. It'd be ideal to have the plugin manager mange this, but it just seems like there's too many ways to get around it for an auto checker to cover every conceivable scenario.

I agree. It's not reasonable to expect the submission validator to check the contents of JARs for classes that already exist in some other JAR in some other plugin. So it's up to us to be careful about packaging any common libraries as separate plugins, labeling them clearly, and checking to see if the one we want already exists in the repository before submitting it.

[stuckless]

Originally Posted by GKusnick

I agree. It's not reasonable to expect the submission validator to check the contents of JARs for classes that already exist in some other JAR in some other plugin. So it's up to us to be careful about packaging any common libraries as separate plugins, labeling them clearly, and checking to see if the one we want already exists in the repository before submitting it.

I certainly don't think the plugin manager should be checking the contents of jar files... i hope that wasn't what you got out of my original post But I don't think it's unreasonable for the plugin manager to NOT allow jar files to go into the JARs directory, unless it's defined a JAR plugin or dependency.

Originally Posted by Slugger

AFAIK, there's no stopping the situation where stuckless has already added commons-io to the repository, which provide commons-io-1.3.jar (or whatever version it is) and if I'm not careful and paying attention to what's in the repository already, I could just easily add the plugin apache-commons-io with commons-io-1.4.jar. Once a user installs a plugin of mine that depends on apache-commons-io and installs another that depends on stuckless' commons-io then you've got duplicate jars in the jars folder. By duplicate, of course, I mean different jars providing the same classes in the same package.

I believe the onus is on the devs to ensure that conflicting jars aren't introduced into the repository. It'd be ideal to have the plugin manager mange this, but it just seems like there's too many ways to get around it for an auto checker to cover every conceivable scenario.

I don't think that the plugin manager would actually let you create a "commons-io" plugin... because I "own" that name, in the sense that I created the original dependency. You could, however, create a "commonsio" or "slugger-commons-io", etc, that would allow 2 versions of the same jar file to be installed.

[tmiranda]

Originally Posted by PLUCKYHD

This was discussed during the private beta and Sage is completely against it and their reasons IIRC where because they didn't want plugins to start excluding other plugins and I agree.

I hadn't thought of that, but then again I try to see the good side of people and not the bad.

How about being able to at least add a warning along the lines of "This plugin may not work properly ..."?

[jreichen]

Originally Posted by tmiranda

How about being able to at least add a warning along the lines of "This plugin may not work properly ..."?

You could put that in the description or release notes in your plugin.xml.