Which plugins should we include as core plugins, if we were to auto-install plugins?

[nyplayer]

Originally Posted by Skirge01

Ahh! Gotcha. My mistake. In that case, I agree.

That OK I have a habit of mumbling

[Narflex]

I fully support having the SDEPG integrated into the SageTV core.

[stuckless]

Originally Posted by Narflex

I fully support having the SDEPG integrated into the SageTV core.

Core vs plugin has limited value given how easy it is to install plugins, and if the guided install gave the option to install SD plugin as part of the setup it would be a "quick" thing to do and a user wouldn't know the difference really, that it's not in core... and it could maintained and updated on a schedule outside of the core releases.

The challenge that I see with the current SD plugin is that it's orphaned and it's based on groovy, and those of us that could adopt it, already have EPG (from legacy SageTV), so finding an owner is hard. Unless you hinting that SageTV epg is going away, then I'm not sure moving the SD plugin to core is going to get priority.

I have looked at the SD plugin.. and I could move it into core, but I'd rather do it without a dependency on Groovy (I like groovy)... but at the end of day... I have EPG so it's never really getting the attention it deserves

[trk2]

Originally Posted by nyplayer

For example I do not use these.... As most android devices have Youtube which is much better.

- youtube plugin

I agree but something should be done about the youtube menu item and the Online menu item in general. The stock install provides a broken youtube which is only going to cause confusion. The "Online" menu should have the broken items removed.

[stuckless]

Originally Posted by Tiki

Since these "layer" packages consist of plugins written by different authors (who may not use or test with all the other plugins), how do you ensure compatibility with revision changes?

For example, maybe the YouTube plugin was written by an author who only uses and tests with the default Sage7 UI, but it happens to work OK with Gemstone too. So, some other person comes along and creates this "layer" package that includes both the YouTube plugin and Gemstone.

But, then later, imagine that the YouTube plugin gets updated in a way that breaks compatibility with Gemstone.

So, how do you make sure that the "layer" package uses the latest versions of all the underlying plugins, but only if they are compatible?

There is always the possibility of that happening, not matter what we do. ie, the original idea of having a "list" of plugins that auto-installed on startup would face the same issues. I think this is a reality, and if XXX was causing issues, and no-one was fixing it, then it would get removed from the "layer". Keep in mind, my "list" was a tenative list of what I thought might be of a value to a newb.. not me. And we might decide that Gemstone Layer doesn't depend on a previous layer, etc. These are plugins, they can depend, or not depend on whatever we want.

Originally Posted by nyplayer

If you go to the layer package will that mean that I have to install every plugin that is in the package even those I do not use? I lke to keep my SageTV install as light a possible.

Then I'd suggest not installing a layer, and just install the parts your want. I mean this is not mean to be a "power user" feature.. This is meant to help someone that knows very little about sagetv and the available plugins, to get a "decent" setup up and running fairly quickly after the first install. They might later decide to uninstall parts that they don't want, etc, or add new features after they browse the other plugins.

When I built the list of things to install, I did not choose the things that "I" wanted, I chose things that I felt a newb might be interested in. For example, I don't use the SageTV web server or the mobile web server or youtube or upnp browser... but I do think these are things that many people might be interested in.

Let's face it... if we have to build a list of items that we agreed that we all needed... it would be a very short list, and we'd probably agree that we don't really need layers, since it's just installing 2 plugins... and you just as easily go find those and install them. The point of a layer if really just a collection of useful plugins that make it easy to install.

It's worth noting that since a "layer" is just a plugin, anyone can create their own layer, since it's just another plugin with dependencies. And Jusjoken's idea of enabling your to export your installed plugins as a layer is great example of this... it's just a simple means that creates a fake plugin (layer) that depends on all the plugins that you have installed... you can then use that to install it on another machine, etc.

[Fuzzy]

Originally Posted by stuckless

Core vs plugin has limited value given how easy it is to install plugins, and if the guided install gave the option to install SD plugin as part of the setup it would be a "quick" thing to do and a user wouldn't know the difference really, that it's not in core... and it could maintained and updated on a schedule outside of the core releases.

The challenge that I see with the current SD plugin is that it's orphaned and it's based on groovy, and those of us that could adopt it, already have EPG (from legacy SageTV), so finding an owner is hard. Unless you hinting that SageTV epg is going away, then I'm not sure moving the SD plugin to core is going to get priority.

I have looked at the SD plugin.. and I could move it into core, but I'd rather do it without a dependency on Groovy (I like groovy)... but at the end of day... I have EPG so it's never really getting the attention it deserves

Well, in my view, the BEST solution wouldn't be to really roll the plugin into the core, but to simply recreate a simplified version of it in the core. Slugger put a lot of things in there that are not entirely necessary (like groovy scripting to modify the epg as the data comes in, and some workarounds to get it to work prior to the sagetv source being opened up). I've started reading through the Schedules Direct docs, and for basic use, the actual SD API should not be all that difficult to integrate, as far as I can tell. It seems the most extensive work would be translating the JSON objects to the types sagetv expects.

[Fuzzy]

Originally Posted by trk2

I agree but something should be done about the youtube menu item and the Online menu item in general. The stock install provides a broken youtube which is only going to cause confusion. The "Online" menu should have the broken items removed.

That needs to be written as an issue on github, so it get's tracked to be fixed.

[Fuzzy]

So, I was thinking more on this, and I can see some complications if we were to move the SDEPG away from plugin and into a core feature. I still feel it needs to be a core feature - but we will run into a lucene style issue, in that in order to properly parse out the JSON data to/from objects, we should use a readily available library (natural choice would be the github neighbor google/gson). The problem is, however, that slugger already has a gson plugin in the repository, so including gson as part of the core might screw with all those plugins with the gson dependency - now, we can probably 'fix' this by simply massaging the dependencies in the repository, but it's back to one of those v7 vs. v9 issues I think.

I still think we need to bite the bullet and get this into the core somehow - it really needs to be considered the main EPG source for SageTV going forward - I'm just not sure the best approach for that though. There are only a few plugins currently dependent on GSON, so it might be able to be done 'smoothly' with some coordinated efforts between a few devs.

Any ideas?

[stuckless]

Originally Posted by Fuzzy

So, I was thinking more on this, and I can see some complications if we were to move the SDEPG away from plugin and into a core feature. I still feel it needs to be a core feature - but we will run into a lucene style issue, in that in order to properly parse out the JSON data to/from objects, we should use a readily available library (natural choice would be the github neighbor google/gson). The problem is, however, that slugger already has a gson plugin in the repository, so including gson as part of the core might screw with all those plugins with the gson dependency - now, we can probably 'fix' this by simply massaging the dependencies in the repository, but it's back to one of those v7 vs. v9 issues I think.

I still think we need to bite the bullet and get this into the core somehow - it really needs to be considered the main EPG source for SageTV going forward - I'm just not sure the best approach for that though. There are only a few plugins currently dependent on GSON, so it might be able to be done 'smoothly' with some coordinated efforts between a few devs.

Any ideas?

This is very different issue and the whole SDEPG stuff probably deserves it's own thread (probably are few of them already )... But the "lucene" and the possible "gson" issues underscore a different issue and that is... Jars should not be plugins. Plugins should depend on Jars... but a jar itself should not be a plugin. The core should auto resolve jars using the maven repo (like every other java application). This was a concept that Slugger was looking at before he moved on. The problem is that changing it breaks all the existing plugins so the general consensus is to leave it alone and hack around it... but in reality, it's a flaw that should be fixed while the Open Source version is still immature, and break ties with V7 to ensure that you can fix it right and have a fresh start.

The other option that doesn't break V7 is to put EVERYTHING including sagetv core jars into plugins... then "gson" is owned by the sagetv core team and as other jars need to be included into core, then they can also be owned and updated by the core team.

I guess the 3rd option is do the "lucene" thing and just include whatever jars we need/want and then hack through the plugin updates/messes that arise.

[Fuzzy]

If I recall, what Slugger was proposing was not to have JAR's auto-download as needed, but to have each plugin load the JARs it specifically needs into it's own CLASSPATH.

in any case, however, I don't see how moving JARs outside of the plugin system and creating some other process to ensure they are installed and updated is an improvement. The issue I was bringing up was about JARs needed BY the sagetv core, which really need to exist before plugins are (currently) processed/handled. This also would not aid the many plugins that are using JAR packaging for the actual plugin's primary work, and not just as a shared library.

Perhaps the better solution is the 2nd option you alluded to:
- Wrap every core-depending library into their own plugins, tied to a central (maven for instance) source
- Implement a way for plugins to be pre-loaded on install (really, this could be as simple as prepackaging a version of the JAR with the install - as it already is - but include the plugin in the default sage.properties, sageclient.properties, and filetracker.properties files, so it can be recognized and managed as a plugin going forward.

I don't think the previously discussed on-start plugin loader is really a good solution for these types, since the plugin manager doesn't really run BEFORE any other process, and everything starts up simultaneously - core-depending plugins might not be there before core functions that require them.

transitioning to this might not be that difficult, since the SageTVPluginsV9.xml already overrides same-named plugins in SageTVPlugins.xml, we don't have to break backwards compatibility, as long as the team 'takes ownership' of the library plugins in the V9 repository.

Of course, this brings up the question of whether SageTV7.xml and it's graphics resources should be handled in the same way as a pre-loaded 'baked-in' UI plugin, instead of the current self-contained updating method.

[stuckless]

Originally Posted by Fuzzy

If I recall, what Slugger was proposing was not to have JAR's auto-download as needed, but to have each plugin load the JARs it specifically needs into it's own CLASSPATH.

in any case, however, I don't see how moving JARs outside of the plugin system and creating some other process to ensure they are installed and updated is an improvement. The issue I was bringing up was about JARs needed BY the sagetv core, which really need to exist before plugins are (currently) processed/handled. This also would not aid the many plugins that are using JAR packaging for the actual plugin's primary work, and not just as a shared library.

To be honest, I'm not sure people would get it, unless they lived and breathed the java ecosystem like myself, and Slugger do (along with many others). His solution, in his POC, if I recall, used gradle and deletes and re-adds the jars in the JARs folder on startup. This may sound inefficient, but, generally, it isn't since in the gradle system the jars are cached elsewhere after the first download. But what this does do is solve the problems that I've seen and fought with in SageTV where if you upgrade gson but the old version doesn't deleted, then you have possibly things like gson-1.1.jar and gson-1.2.jar both in the JARs folder Or if someone just stuffs jars in their zip file and dumps them in the JARs folder, etc. The separate classpath stuff is a whole other plugin isoloation model (loosly like osgi) which has some benefits as well (ie, different plugins can use different versions of the same jar file, etc), but also comes with a huge complexity.

Originally Posted by Fuzzy

Perhaps the better solution is the 2nd option you alluded to:
- Wrap every core-depending library into their own plugins, tied to a central (maven for instance) source
- Implement a way for plugins to be pre-loaded on install (really, this could be as simple as prepackaging a version of the JAR with the install - as it already is - but include the plugin in the default sage.properties, sageclient.properties, and filetracker.properties files, so it can be recognized and managed as a plugin going forward.

I've given that some thought as well... the challenge here is really about how to automate this during the build/publish phase so that you can include all those changes in the distribution. It's easy to track changes in a running system, but in a build time, it would be much harder to programatically figure it out.

[Fuzzy]

Originally Posted by stuckless

I've given that some thought as well... the challenge here is really about how to automate this during the build/publish phase so that you can include all those changes in the distribution. It's easy to track changes in a running system, but in a build time, it would be much harder to programatically figure it out.

There's always the option of a pre-launch environment that runs a very minimal part of sage.jar, bringing up just the properties and plugin manager, and does the plugin installs, then restarts into the full version. I honestly haven't dug too deeply into how sage.jar is launched (really not a c guy at all), but there are already multiple ways the same jar is launched, so one more might not be that crazy. This would allow even sage.jar to be 'plugin-ized' - and have this 'Sage Core' plugin dependent on the various other libraries we are talking about. This would allow future versions of 'sage core' that might use more libraries to bring them in when the core is upgraded. I have a hunch that if you setup Sage Core to be dependent on itself, it would prevent it from being able to be uninstalled (the plugin manager first checks for plugins that depend on a given plugin before uninstalling it. It'll end up posting a message that Sage Core can't be uninstalled because Sage Core depends on it. Sage Core would always be installed, and be plugin 0, so the issue of the missing circular dependency wouldn't really affect it. It'd still pass the checks to upgrade.

[Slugger]

Originally Posted by stuckless

To be honest, I'm not sure people would get it, unless they lived and breathed the java ecosystem like myself, and Slugger do (along with many others). His solution, in his POC, if I recall, used gradle and deletes and re-adds the jars in the JARs folder on startup. This may sound inefficient, but, generally, it isn't since in the gradle system the jars are cached elsewhere after the first download. But what this does do is solve the problems that I've seen and fought with in SageTV where if you upgrade gson but the old version doesn't deleted, then you have possibly things like gson-1.1.jar and gson-1.2.jar both in the JARs folder Or if someone just stuffs jars in their zip file and dumps them in the JARs folder, etc. The separate classpath stuff is a whole other plugin isoloation model (loosly like osgi) which has some benefits as well (ie, different plugins can use different versions of the same jar file, etc), but also comes with a huge complexity.

Yeah, Stuckless' description is accurate for what I was working on. But the gotcha was that it totally broke backwards compatibility with the v7 repo and required plugin devs to start delivering to a maven repo. The discussions on the forums were heavily slanted towards people not wanting that to happen (mainly to keep compatibility with the v7 plugin ecosystem) so I backed off the effort a bit, then a couple months later I ended up cutting the cord and shutting down Sage.

The details are fuzzy, but I had a working PoC that used gradle to manage all jar dependencies of the Sage installation and rebuilt the JARs folder on startup. All deps were pulled from maven, cached locally as needed, and then auto managed on each sage startup.

The branch still exists at my github, but the details are a bit fuzzy now other than I know I ran it and it was working. I had also started working on an alternate plugin manager that dealt with generating the gradle build files for submitted plugins, etc. I think I was just getting started on this before I left and so there's not much here. Finally, I put together a sample plugin that a plugin dev would submit under this new ecosystem.

Again, all the details are fuzzy now, but I didn't delete any of the repos so if the community decides to start heading down this type of direction then there's at least some reference material there for you.

As for sdepg into the core, I think Fuzzy is probably right in that just writing a new SD JSON client might be easier than trying to migrate my plugin into the core. The SD JSON API itself is really straightforward. Dropping the groovy scripting functions (and really all the extras) makes sense, but I will say that even though I was the only one that I think ever actually used them, they were very, very powerful. You can deal with all kinds of upstream problems/workarounds when you can quickly and easily massage the data as it's coming in.

For example, I saw someone struggling with Tour de France airings all having the same id even though morning/night were actually distinct. If you can process & analyze the data as it's coming, it becomes rather trivial to reassign a unique id to ensure both are recorded. It seems like these kinds of things are one offs/rare, but I had a half dozen or so different scripts preprocessing my epg data to handle various things.

[Narflex]

And for the problem with JAR compatibility relating to JSON parsing...you could probably find some Apache licensed parser which you could just change the package name on and add into the core directly for JSON parsing so it wouldn't conflict with any plugins doing it....I'd probably do it that way if I was writing the SD support myself.

[Fuzzy]

Originally Posted by Narflex

And for the problem with JAR compatibility relating to JSON parsing...you could probably find some Apache licensed parser which you could just change the package name on and add into the core directly for JSON parsing so it wouldn't conflict with any plugins doing it....I'd probably do it that way if I was writing the SD support myself.

Yeah, I had considered that - I also considered just writing a single-purpose parser into the SD class - just figured that since just about every install out there HAS GSON installed on it already, it would make sense to try to go that route.

[stuckless]

Originally Posted by Narflex

And for the problem with JAR compatibility relating to JSON parsing...you could probably find some Apache licensed parser which you could just change the package name on and add into the core directly for JSON parsing so it wouldn't conflict with any plugins doing it....I'd probably do it that way if I was writing the SD support myself.

That's what I did for sagex-api... but for the most part using the org json java files directly is just making the process harder... ie, libraries like gson make working json relatively easy... the org json apis requires you write your own micro library on top of the org json, since working with that API directly is a little tedious But yeah, I agree, it would be the best solution to avoid not pulling in additional libraries.