Moving component projects to a "feature pack" model?

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Moving component projects to a "feature pack" model?

Sanne Grinovero
# WHAT WE DID SO FAR (and worked quite ok)

Since a couple of years, when releasing Hibernate Search, Hibernate
OGM, Infinispan and other projects, besides merely uploading our jar
and pom.xml files to Nexus we also upload a large tarball containing
modules for WildFly: an "add on layer", not only useful for end users
but also some other projects rely on this (e.g. Infinispan consumes
the Hibernate Search one to keep an aggressive improvement pace).

With each build of the projects these modules are not just built but
also tested: integration tests download the latest stable WildFly
version which we target, untar it, untar our own modules as a layer,
run some Arquillian tests. Invaluable to get all dependency
definitions and classloading right!

# GOALS

 -- to make sure people can download our latest release, and it will
work on their favourite app server
 -- to be ready knowing how a stable module structure should be
shaped, when a new version of WildFly integrates our latest
 -- to benefit from the modules system by allowing different versions
to co-exist on the same appserver (some people/project/product require
this)

# NOT PERFECT

This second goal had some issues recently, namely that while we know
how we should structure the modules within WildFly, it happened that
this wasn't re-creating exactly the model of what had been carefully
tested within our project. The problem is that we have to repeat the
build scripting - result might not be identical for obvious reasons -
while most integration tests reside within the project.

# FEATURE PACKS

Tomaz was so nice to show me how to define feature packs, so that the
WildFly build can - at build time - include the structure we define.
It looks very promising, but on some areas we'll need some suggestions
or perhaps a further evolution?

# NEEDED IMPROVEMENTS / SUGGESTIONS

? How do we test the feature pack ?

The goal of having WildFly use the module structure that we build
"verbatim" is only interesting if we make sure these packs work
correctly.
Ideally we'd like to deploy them in Arquillian tests like we did with
modules, and run integration tests on each commit.

? How can end users benefit from these ?

We'd really want to enable our pool of users and contributors to use
(and test!) any of our releases from "day zero".
Typically people will want to run it on the latest stable version of
WildFly, which implies something releases before "day zero".

? Versions and slot identifiers ?

We release approximately every three weeks on average, and our slot
conventions expect that "our" module releases use the
slot={releaseVersion}, while the module included in WildFly has the
slot="main".
This seems to imply some need for slot transformations when the
feature pack is consumed.

? Aliases ?

We also include module aliases for each Major.Minor combination for
the public-api module, which resolves to the latest major.minor.micro
of the matching version.
This is important to make sure we can include references to module
dependencies maintained in other projects but limiting the scope to a
compatible range, for example we have an optional dependency on
Infinispan, not all Infinispan versions are compatible.

My preferred solution would be to have WildFly actually not use "main"
slot internally, but use our same alias pattern and in addition have
an alias module - for the public API only - which has slot "main"
resolve to the "major.minor" of the Hibernate Search version which was
included.

There are many benefits to this model, I'll avoid repeating them in
this email; I shared some thoughts before:
 - http://lists.jboss.org/pipermail/wildfly-dev/2015-April/003787.html

I would love it if a similar pattern was to be applied to other
modules, for example Hibernate Search doesn't work with any version of
Hibernate ORM so depending on "slot main" has some dangers. Even more
important when we deal with other modules released by other projects
in similar packaging.

? "System" modules ?

I noticed the feature pack structure includes the path to the module layer, i.e.
 modules/system/layers/base/..

But I take it we don't want to classify our "independently" released
modules as belonging to the system/base layer?

? Conflicting versions ?

I noticed the versions of all dependencies are listed once, in the
root metadata of the feature pack. In some projects, like Hibernate
OGM, we take strong advantage of modularity and actually bundle some
libraries multiple times - of different versions - and use the modules
magic to keep them strictly isolated.
Is this going to be possible?

Thanks in advance for any suggestions!
Sanne
_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev
Reply | Threaded
Open this post in threaded view
|

Re: Moving component projects to a "feature pack" model?

Panagiotis Sotiropoulos
# SOME ADDITIONAL CONCERNS

1. Addition of optional subsystems on Wildfly

Is there some standardized way to add optional subsystems and addtional modules (using the feature pack model) on wildfly without building the server again?
While trying to create such an optional subsystem (https://developer.jboss.org/wiki/JBoss-Automated-Metrics) , I came across
wildfly-extension-plugin (http://lzoubek.github.io/wildfly-extension-maven-plugin/) created by Libor Zoubek, which seems to do
the trick. Is this the standardized way that is used in general?


2 How about creating a unified testsuite for all the servers

Would it make sense to create a unified testsuite where the tests would be tested against all the servers (eap, wildfly releases)?
So, decoupling the testsuite from the servers, writing the tests once (one repo) and testing against all servers.
 

----- Original Message -----
From: "Sanne Grinovero" <[hidden email]>
To: [hidden email]
Sent: Thursday, May 28, 2015 11:51:13 PM
Subject: [wildfly-dev] Moving component projects to a "feature pack" model?

# WHAT WE DID SO FAR (and worked quite ok)

Since a couple of years, when releasing Hibernate Search, Hibernate
OGM, Infinispan and other projects, besides merely uploading our jar
and pom.xml files to Nexus we also upload a large tarball containing
modules for WildFly: an "add on layer", not only useful for end users
but also some other projects rely on this (e.g. Infinispan consumes
the Hibernate Search one to keep an aggressive improvement pace).

With each build of the projects these modules are not just built but
also tested: integration tests download the latest stable WildFly
version which we target, untar it, untar our own modules as a layer,
run some Arquillian tests. Invaluable to get all dependency
definitions and classloading right!

# GOALS

 -- to make sure people can download our latest release, and it will
work on their favourite app server
 -- to be ready knowing how a stable module structure should be
shaped, when a new version of WildFly integrates our latest
 -- to benefit from the modules system by allowing different versions
to co-exist on the same appserver (some people/project/product require
this)

# NOT PERFECT

This second goal had some issues recently, namely that while we know
how we should structure the modules within WildFly, it happened that
this wasn't re-creating exactly the model of what had been carefully
tested within our project. The problem is that we have to repeat the
build scripting - result might not be identical for obvious reasons -
while most integration tests reside within the project.

# FEATURE PACKS

Tomaz was so nice to show me how to define feature packs, so that the
WildFly build can - at build time - include the structure we define.
It looks very promising, but on some areas we'll need some suggestions
or perhaps a further evolution?

# NEEDED IMPROVEMENTS / SUGGESTIONS

? How do we test the feature pack ?

The goal of having WildFly use the module structure that we build
"verbatim" is only interesting if we make sure these packs work
correctly.
Ideally we'd like to deploy them in Arquillian tests like we did with
modules, and run integration tests on each commit.

? How can end users benefit from these ?

We'd really want to enable our pool of users and contributors to use
(and test!) any of our releases from "day zero".
Typically people will want to run it on the latest stable version of
WildFly, which implies something releases before "day zero".

? Versions and slot identifiers ?

We release approximately every three weeks on average, and our slot
conventions expect that "our" module releases use the
slot={releaseVersion}, while the module included in WildFly has the
slot="main".
This seems to imply some need for slot transformations when the
feature pack is consumed.

? Aliases ?

We also include module aliases for each Major.Minor combination for
the public-api module, which resolves to the latest major.minor.micro
of the matching version.
This is important to make sure we can include references to module
dependencies maintained in other projects but limiting the scope to a
compatible range, for example we have an optional dependency on
Infinispan, not all Infinispan versions are compatible.

My preferred solution would be to have WildFly actually not use "main"
slot internally, but use our same alias pattern and in addition have
an alias module - for the public API only - which has slot "main"
resolve to the "major.minor" of the Hibernate Search version which was
included.

There are many benefits to this model, I'll avoid repeating them in
this email; I shared some thoughts before:
 - http://lists.jboss.org/pipermail/wildfly-dev/2015-April/003787.html

I would love it if a similar pattern was to be applied to other
modules, for example Hibernate Search doesn't work with any version of
Hibernate ORM so depending on "slot main" has some dangers. Even more
important when we deal with other modules released by other projects
in similar packaging.

? "System" modules ?

I noticed the feature pack structure includes the path to the module layer, i.e.
 modules/system/layers/base/..

But I take it we don't want to classify our "independently" released
modules as belonging to the system/base layer?

? Conflicting versions ?

I noticed the versions of all dependencies are listed once, in the
root metadata of the feature pack. In some projects, like Hibernate
OGM, we take strong advantage of modularity and actually bundle some
libraries multiple times - of different versions - and use the modules
magic to keep them strictly isolated.
Is this going to be possible?

Thanks in advance for any suggestions!
Sanne
_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev
_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev
Reply | Threaded
Open this post in threaded view
|

Re: Moving component projects to a "feature pack" model?

Tristan Tarrant
In reply to this post by Sanne Grinovero
I just posted a partially related e-mail to this ML referencing an issue
with the transaction subsystem, but I'd like to use that example in the
context of feature packs.
Sanne is mentioning the use of FPs as a way to layer components on top
of an existing distribution, but I would also like to be able to use
components from the main distribution in a more granular fashion. Are
there any plans to further breakdown WF into component feature packs so
that they can be imported individually without having to base on a
larger FP which contains unwanted components ?

Tristan

On 28/05/2015 23:51, Sanne Grinovero wrote:

> # WHAT WE DID SO FAR (and worked quite ok)
>
> Since a couple of years, when releasing Hibernate Search, Hibernate
> OGM, Infinispan and other projects, besides merely uploading our jar
> and pom.xml files to Nexus we also upload a large tarball containing
> modules for WildFly: an "add on layer", not only useful for end users
> but also some other projects rely on this (e.g. Infinispan consumes
> the Hibernate Search one to keep an aggressive improvement pace).
>
> With each build of the projects these modules are not just built but
> also tested: integration tests download the latest stable WildFly
> version which we target, untar it, untar our own modules as a layer,
> run some Arquillian tests. Invaluable to get all dependency
> definitions and classloading right!
>
> # GOALS
>
>   -- to make sure people can download our latest release, and it will
> work on their favourite app server
>   -- to be ready knowing how a stable module structure should be
> shaped, when a new version of WildFly integrates our latest
>   -- to benefit from the modules system by allowing different versions
> to co-exist on the same appserver (some people/project/product require
> this)
>
> # NOT PERFECT
>
> This second goal had some issues recently, namely that while we know
> how we should structure the modules within WildFly, it happened that
> this wasn't re-creating exactly the model of what had been carefully
> tested within our project. The problem is that we have to repeat the
> build scripting - result might not be identical for obvious reasons -
> while most integration tests reside within the project.
>
> # FEATURE PACKS
>
> Tomaz was so nice to show me how to define feature packs, so that the
> WildFly build can - at build time - include the structure we define.
> It looks very promising, but on some areas we'll need some suggestions
> or perhaps a further evolution?
>
> # NEEDED IMPROVEMENTS / SUGGESTIONS
>
> ? How do we test the feature pack ?
>
> The goal of having WildFly use the module structure that we build
> "verbatim" is only interesting if we make sure these packs work
> correctly.
> Ideally we'd like to deploy them in Arquillian tests like we did with
> modules, and run integration tests on each commit.
>
> ? How can end users benefit from these ?
>
> We'd really want to enable our pool of users and contributors to use
> (and test!) any of our releases from "day zero".
> Typically people will want to run it on the latest stable version of
> WildFly, which implies something releases before "day zero".
>
> ? Versions and slot identifiers ?
>
> We release approximately every three weeks on average, and our slot
> conventions expect that "our" module releases use the
> slot={releaseVersion}, while the module included in WildFly has the
> slot="main".
> This seems to imply some need for slot transformations when the
> feature pack is consumed.
>
> ? Aliases ?
>
> We also include module aliases for each Major.Minor combination for
> the public-api module, which resolves to the latest major.minor.micro
> of the matching version.
> This is important to make sure we can include references to module
> dependencies maintained in other projects but limiting the scope to a
> compatible range, for example we have an optional dependency on
> Infinispan, not all Infinispan versions are compatible.
>
> My preferred solution would be to have WildFly actually not use "main"
> slot internally, but use our same alias pattern and in addition have
> an alias module - for the public API only - which has slot "main"
> resolve to the "major.minor" of the Hibernate Search version which was
> included.
>
> There are many benefits to this model, I'll avoid repeating them in
> this email; I shared some thoughts before:
>   - http://lists.jboss.org/pipermail/wildfly-dev/2015-April/003787.html
>
> I would love it if a similar pattern was to be applied to other
> modules, for example Hibernate Search doesn't work with any version of
> Hibernate ORM so depending on "slot main" has some dangers. Even more
> important when we deal with other modules released by other projects
> in similar packaging.
>
> ? "System" modules ?
>
> I noticed the feature pack structure includes the path to the module layer, i.e.
>   modules/system/layers/base/..
>
> But I take it we don't want to classify our "independently" released
> modules as belonging to the system/base layer?
>
> ? Conflicting versions ?
>
> I noticed the versions of all dependencies are listed once, in the
> root metadata of the feature pack. In some projects, like Hibernate
> OGM, we take strong advantage of modularity and actually bundle some
> libraries multiple times - of different versions - and use the modules
> magic to keep them strictly isolated.
> Is this going to be possible?
>
> Thanks in advance for any suggestions!
> Sanne
> _______________________________________________
> wildfly-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/wildfly-dev
>

--
Tristan Tarrant
Infinispan Lead
JBoss, a division of Red Hat
_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev
Reply | Threaded
Open this post in threaded view
|

Re: Moving component projects to a "feature pack" model?

Tomaž Cerar-2
In reply to this post by Sanne Grinovero

On Thu, May 28, 2015 at 11:51 PM, Sanne Grinovero <[hidden email]> wrote:
? How do we test the feature pack ?

The goal of having WildFly use the module structure that we build
"verbatim" is only interesting if we make sure these packs work
correctly.
Ideally we'd like to deploy them in Arquillian tests like we did with
modules, and run integration tests on each commit.


This is quite simple. As part of test run you provision thin WildFly with your feature pack
and run run arquillian tests. As provisionied server is thin (doesn't contain any jars) it is also lightweight.

an example would be how we test arquillian integration itself
or by provisioning wildfly-servlet(aka wildfly-web)
https://github.com/wildfly/wildfly-arquillian/tree/master/container-embedded

which can be configured to include as many feature packs as you want.
for example, take wildfly-web(-servlet) distro and add hibernate-search next to it.

then just run arquillian tests as you would normally do.

We use this in wildfly-core testsuite already (but without arquillain ) and have plans to move our wildfly full testsuite to use it as well.

--
tomaz



_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev