Subsystems to advertise implicit and optional dependencies to galleon

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

Subsystems to advertise implicit and optional dependencies to galleon

Jean-Francois Denise
Hi,

in the galleon project, in a context where we would not provision a
complete server but just a subset required to run a given configuration,
we have identified a need for subsystems to advertise to the galleon
tooling some modules in addition to the modules galleon discover by
traversing the module dependency tree.

The first case is DeploymentProcessor injecting modules into the
deployment units (implicit modules).

The deployment injected ones are not required to be a dependency of the
subsystem module, so galleon has the risk to miss some of them.

As an example, in logging subsystem we have the following non optional
injected modules :
"org.jboss.logging",
"org.apache.commons.logging",
"org.apache.log4j",
"org.slf4j",
"org.jboss.logging.jul-to-slf4j-stub"

Some of these modules are direct dependencies of logging subsystem, some
others are indirect dependencies, others could be not present at all in
the module dependency tree (eg: org.jboss.logging.jul-to-slf4j-stub).

So we are thinking at solving this issue in 2 possible ways:

1) Mandate that all injected modules become dependencies of the
subsystem module at the cost to load some useless modules at runtime.

2) Possibly better, make the subsystem to call
RuntimeCapability.addAdditionalRequiredPackages (package name being
module name) for each injected module. An existing capability or a new
one  would have to be created.

There is also the case of optional injected module dependencies (eg: ee
subsystem org.glassfish.javax.el, org.eclipse.yasson, ...). We need to
treat them differently. When one is provisioning a server using galleon
he can choose to exclude some packages from the provisioned server.
Optional packages can be excluded without making the provisioned state
invalid (as opposed to required package that can't be excluded). These
optional implicit dependencies are typical usage of this, they are not
required by the deployment unit to properly operate.

For these, we plan to use
RuntimeCapability.addAdditionalOptionalPackages. We can't make them
"optional" in JBoss module. When galleon provision a subset of the
server we don't include all optional dependencies.

This brings the case of provisioning of optional dependencies present in
JBOSS module.xml.

We have identified multiple kind of optional dependencies.

1) The optional dependencies that are referencing modules only in use
when a feature is present in the configuration of the subsystem (eg: jmx
subsystem optional dep on org.jboss.remoting-jmx due to
<remoting-connector/>, remoting subsystem optional dep on
io.undertow.core for <http-connector/>, elytron subsystem optional dep
on org.picketbox for <jacc-policy>)

In order to have these dependencies to be provisioned with the
subsystem, we can attach thanks to
RuntimeCapability.addAdditionalRequiredPackages the modules to the
feature. When the feature is present in the configuration, the module is
no more optional but required.

2) The optional dependencies that reference modules that are part of
another subsystems and only use if this other subsystem is present (eg:
org.jboss.as.jpa optional dep on org.jboss.as.ejb3,
org.jboss.as.transactions optional dep on org.jboss.remoting). These
ones are simply not taken into account

3)  The optional dependencies that reference modules that are not part
of another subsystem (so are not provisioned by another source) but are
only meaningful if the other subsystem is present. We call these ones
"passive" (eg: org.jboss.as.weld optional dependency on
org.jboss.as.weld.ejb|jpa|beanvalidation|webservices|transactions).
"passive" are analyzed. If all their dependencies are present, then they
are included. Some implicit optional dependencies can fall into this
category (eg: org.jboss.jaxrs optional dep on
org.jboss.resteasy.resteasy-validator-provider-11).

The passive optional dependencies would be advertised with
RuntimeCapability.addAdditionalPassiveOptionalPackages(optional
dependency package name).

So to summarize:

- RuntimeCapability.addAdditionalRequiredPackages for all required
implicit modules

- RuntimeCapability.addAdditionalRequiredPackages to associate optional
dependencies to a feature

- RuntimeCapability.addAdditionalOptionalPackages for all optional
implicit dependencies that are not passive

- RuntimeCapability.addAdditionalPassiveOptionalPackages for all
optional dependencies (implicit or not) that are passive

Is it something that subsystems owner would be ready to put in place to
help galleon in this task? It would require a bit of analysis of the
dependencies of your modules but the gain could be quite important. Some
early experimentation of this has shown a big reduction of the server
filesystem footprint (web server + cdi has been reduced from 156MB to
46MB). The runtime memory usage reduction is not that big, but less
modules being loaded we have a gain.

Thanks for reading.

JF

_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev
Reply | Threaded
Open this post in threaded view
|

Re: Subsystems to advertise implicit and optional dependencies to galleon

Brian Stansberry


On Fri, Nov 9, 2018 at 2:30 AM, Jean-Francois Denise <[hidden email]> wrote:
Hi,

in the galleon project, in a context where we would not provision a
complete server but just a subset required to run a given configuration,
we have identified a need for subsystems to advertise to the galleon
tooling some modules in addition to the modules galleon discover by
traversing the module dependency tree.

The first case is DeploymentProcessor injecting modules into the
deployment units (implicit modules).

The deployment injected ones are not required to be a dependency of the
subsystem module, so galleon has the risk to miss some of them.

As an example, in logging subsystem we have the following non optional
injected modules :
"org.jboss.logging",
"org.apache.commons.logging",
"org.apache.log4j",
"org.slf4j",
"org.jboss.logging.jul-to-slf4j-stub"

Some of these modules are direct dependencies of logging subsystem, some
others are indirect dependencies, others could be not present at all in
the module dependency tree (eg: org.jboss.logging.jul-to-slf4j-stub).

So we are thinking at solving this issue in 2 possible ways:

1) Mandate that all injected modules become dependencies of the
subsystem module at the cost to load some useless modules at runtime.

2) Possibly better, make the subsystem to call
RuntimeCapability.addAdditionalRequiredPackages (package name being
module name) for each injected module. An existing capability or a new
one  would have to be created.

There is also the case of optional injected module dependencies (eg: ee
subsystem org.glassfish.javax.el, org.eclipse.yasson, ...). We need to
treat them differently. When one is provisioning a server using galleon
he can choose to exclude some packages from the provisioned server.
Optional packages can be excluded without making the provisioned state
invalid (as opposed to required package that can't be excluded). These
optional implicit dependencies are typical usage of this, they are not
required by the deployment unit to properly operate.

For these, we plan to use
RuntimeCapability.addAdditionalOptionalPackages. We can't make them
"optional" in JBoss module. When galleon provision a subset of the
server we don't include all optional dependencies.

This brings the case of provisioning of optional dependencies present in
JBOSS module.xml.

We have identified multiple kind of optional dependencies.

1) The optional dependencies that are referencing modules only in use
when a feature is present in the configuration of the subsystem (eg: jmx
subsystem optional dep on org.jboss.remoting-jmx due to
<remoting-connector/>, remoting subsystem optional dep on
io.undertow.core for <http-connector/>, elytron subsystem optional dep
on org.picketbox for <jacc-policy>)

In order to have these dependencies to be provisioned with the
subsystem, we can attach thanks to
RuntimeCapability.addAdditionalRequiredPackages the modules to the
feature. When the feature is present in the configuration, the module is
no more optional but required.

2) The optional dependencies that reference modules that are part of
another subsystems and only use if this other subsystem is present (eg:
org.jboss.as.jpa optional dep on org.jboss.as.ejb3,
org.jboss.as.transactions optional dep on org.jboss.remoting). These
ones are simply not taken into account

3)  The optional dependencies that reference modules that are not part
of another subsystem (so are not provisioned by another source) but are
only meaningful if the other subsystem is present. We call these ones
"passive" (eg: org.jboss.as.weld optional dependency on
org.jboss.as.weld.ejb|jpa|beanvalidation|webservices|transactions).
"passive" are analyzed. If all their dependencies are present, then they
are included. Some implicit optional dependencies can fall into this
category (eg: org.jboss.jaxrs optional dep on
org.jboss.resteasy.resteasy-validator-provider-11).

The passive optional dependencies would be advertised with
RuntimeCapability.addAdditionalPassiveOptionalPackages(optional
dependency package name).

So to summarize:

- RuntimeCapability.addAdditionalRequiredPackages for all required
implicit modules

- RuntimeCapability.addAdditionalRequiredPackages to associate optional
dependencies to a feature

- RuntimeCapability.addAdditionalOptionalPackages for all optional
implicit dependencies that are not passive

- RuntimeCapability.addAdditionalPassiveOptionalPackages for all
optional dependencies (implicit or not) that are passive

A good question Alexey raised is whether the capability is the right concept for encapsulating this information.

In the end the key point is that the information ends up in the galleon feature-spec. That's the thing that encapsulates the configuration elements and the packages (i.e. fs content) that are needed to provide a feature. It's the feature that needs the package. We generate the galleon feature specs from our management model information. When in WF 13 we hit the first must-implement use case for getting additional package info into a feature spec I kind of arbitrarily picked 'RuntimeCapability' as the place to record that. It was a reasonable enough choice for that one use case but isn't great otherwise.

The alternative is to record this data in the ResourceDefinition, which is the primary source of the data that's used in the feature spec generation.

The conceptual argument is a capability is all about a named provider of a contract that other elements of the server can rely on; e.g. some MSC Service with a particular value type that will be exposed.  But this additional package stuff isn't really about that contract, it's about other things that will be done that likely are not relevant to other elements of the system.

A hint of a design smell would be if we end up creating RuntimeCapability instances for no other reason than to record this additional package information. Something I could see happening with this logging case you mentioned at the start.

I don't think ResourceDefinition has this problem.


Is it something that subsystems owner would be ready to put in place to
help galleon in this task? It would require a bit of analysis of the
dependencies of your modules but the gain could be quite important.

Please don't let my point above derail consideration of this question.  Use of ResourceDefinition vs RuntimeCapability is a relatively small detail. 

Some
early experimentation of this has shown a big reduction of the server
filesystem footprint (web server + cdi has been reduced from 156MB to
46MB). The runtime memory usage reduction is not that big, but less
modules being loaded we have a gain.

Thanks for reading.

JF

_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev



--
Brian Stansberry
Manager, Senior Principal Software Engineer
Red Hat

_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev
Reply | Threaded
Open this post in threaded view
|

Re: Subsystems to advertise implicit and optional dependencies to galleon

James Perkins
In reply to this post by Jean-Francois Denise
The main reason those modules are dependencies of the logging subsystem is because they are not really required for the subsystem itself with the exception of org.jboss.logging. Those module dependencies are added to deployments for convenience. So really they're not required at all unless the user is using one of those logging frameworks and they want the logging subsystem to control the logging output.

This may be somewhat of an edge case because if the user doesn't use, say slf4j, then there is no real reason to include the dependency unless of course another module uses it. I do wonder if we should remove the org.jboss.logging.jul-to-slf4j-stub module and just put the library as a resource root to org.slf4j, but that's slightly OT :)

On Fri, Nov 9, 2018 at 12:34 AM Jean-Francois Denise <[hidden email]> wrote:
Hi,

in the galleon project, in a context where we would not provision a
complete server but just a subset required to run a given configuration,
we have identified a need for subsystems to advertise to the galleon
tooling some modules in addition to the modules galleon discover by
traversing the module dependency tree.

The first case is DeploymentProcessor injecting modules into the
deployment units (implicit modules).

The deployment injected ones are not required to be a dependency of the
subsystem module, so galleon has the risk to miss some of them.

As an example, in logging subsystem we have the following non optional
injected modules :
"org.jboss.logging",
"org.apache.commons.logging",
"org.apache.log4j",
"org.slf4j",
"org.jboss.logging.jul-to-slf4j-stub"

Some of these modules are direct dependencies of logging subsystem, some
others are indirect dependencies, others could be not present at all in
the module dependency tree (eg: org.jboss.logging.jul-to-slf4j-stub).

So we are thinking at solving this issue in 2 possible ways:

1) Mandate that all injected modules become dependencies of the
subsystem module at the cost to load some useless modules at runtime.

2) Possibly better, make the subsystem to call
RuntimeCapability.addAdditionalRequiredPackages (package name being
module name) for each injected module. An existing capability or a new
one  would have to be created.

There is also the case of optional injected module dependencies (eg: ee
subsystem org.glassfish.javax.el, org.eclipse.yasson, ...). We need to
treat them differently. When one is provisioning a server using galleon
he can choose to exclude some packages from the provisioned server.
Optional packages can be excluded without making the provisioned state
invalid (as opposed to required package that can't be excluded). These
optional implicit dependencies are typical usage of this, they are not
required by the deployment unit to properly operate.

For these, we plan to use
RuntimeCapability.addAdditionalOptionalPackages. We can't make them
"optional" in JBoss module. When galleon provision a subset of the
server we don't include all optional dependencies.

This brings the case of provisioning of optional dependencies present in
JBOSS module.xml.

We have identified multiple kind of optional dependencies.

1) The optional dependencies that are referencing modules only in use
when a feature is present in the configuration of the subsystem (eg: jmx
subsystem optional dep on org.jboss.remoting-jmx due to
<remoting-connector/>, remoting subsystem optional dep on
io.undertow.core for <http-connector/>, elytron subsystem optional dep
on org.picketbox for <jacc-policy>)

In order to have these dependencies to be provisioned with the
subsystem, we can attach thanks to
RuntimeCapability.addAdditionalRequiredPackages the modules to the
feature. When the feature is present in the configuration, the module is
no more optional but required.

2) The optional dependencies that reference modules that are part of
another subsystems and only use if this other subsystem is present (eg:
org.jboss.as.jpa optional dep on org.jboss.as.ejb3,
org.jboss.as.transactions optional dep on org.jboss.remoting). These
ones are simply not taken into account

3)  The optional dependencies that reference modules that are not part
of another subsystem (so are not provisioned by another source) but are
only meaningful if the other subsystem is present. We call these ones
"passive" (eg: org.jboss.as.weld optional dependency on
org.jboss.as.weld.ejb|jpa|beanvalidation|webservices|transactions).
"passive" are analyzed. If all their dependencies are present, then they
are included. Some implicit optional dependencies can fall into this
category (eg: org.jboss.jaxrs optional dep on
org.jboss.resteasy.resteasy-validator-provider-11).

The passive optional dependencies would be advertised with
RuntimeCapability.addAdditionalPassiveOptionalPackages(optional
dependency package name).

So to summarize:

- RuntimeCapability.addAdditionalRequiredPackages for all required
implicit modules

- RuntimeCapability.addAdditionalRequiredPackages to associate optional
dependencies to a feature

- RuntimeCapability.addAdditionalOptionalPackages for all optional
implicit dependencies that are not passive

- RuntimeCapability.addAdditionalPassiveOptionalPackages for all
optional dependencies (implicit or not) that are passive

Is it something that subsystems owner would be ready to put in place to
help galleon in this task? It would require a bit of analysis of the
dependencies of your modules but the gain could be quite important. Some
early experimentation of this has shown a big reduction of the server
filesystem footprint (web server + cdi has been reduced from 156MB to
46MB). The runtime memory usage reduction is not that big, but less
modules being loaded we have a gain.

Thanks for reading.

JF

_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev


--
James R. Perkins
JBoss by Red Hat

_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev
Reply | Threaded
Open this post in threaded view
|

Re: Subsystems to advertise implicit and optional dependencies to galleon

Brian Stansberry


On Fri, Nov 9, 2018 at 12:45 PM, James Perkins <[hidden email]> wrote:
The main reason those modules are dependencies of the logging subsystem is because they are not really required for the subsystem itself with the exception of org.jboss.logging. Those module dependencies are added to deployments for convenience. So really they're not required at all unless the user is using one of those logging frameworks and they want the logging subsystem to control the logging output.

So this is this case from Jean-Francois' initial post:

"- RuntimeCapability.addAdditionalOptionalPackages for all optional 
implicit dependencies that are not passive"

(Note that it's only truly optional if the DUP registers it that way.)

If the logging subsystem doesn't actual use the module itself then it's module.xml would not record it as a dependency. This is the right thing to do regardless if the logging subsystem isn't accessing any resources from them.


This may be somewhat of an edge case because if the user doesn't use, say slf4j, then there is no real reason to include the dependency unless of course another module uses it.

AIUI the idea is that by not including these in the logging subsystem module.xml but instead recording them in the management model definition via addAdditionalOptionalPackages, this information will end up in the galleon feature spec.  Based on that galleon will provision the modules, but would allow the user to exclude them if they wanted to further optimize.

I assume that for compatibility reasons we must provision them by default.

I do wonder if we should remove the org.jboss.logging.jul-to-slf4j-stub module and just put the library as a resource root to org.slf4j, but that's slightly OT :)

On Fri, Nov 9, 2018 at 12:34 AM Jean-Francois Denise <[hidden email]> wrote:
Hi,

in the galleon project, in a context where we would not provision a
complete server but just a subset required to run a given configuration,
we have identified a need for subsystems to advertise to the galleon
tooling some modules in addition to the modules galleon discover by
traversing the module dependency tree.

The first case is DeploymentProcessor injecting modules into the
deployment units (implicit modules).

The deployment injected ones are not required to be a dependency of the
subsystem module, so galleon has the risk to miss some of them.

As an example, in logging subsystem we have the following non optional
injected modules :
"org.jboss.logging",
"org.apache.commons.logging",
"org.apache.log4j",
"org.slf4j",
"org.jboss.logging.jul-to-slf4j-stub"

Some of these modules are direct dependencies of logging subsystem, some
others are indirect dependencies, others could be not present at all in
the module dependency tree (eg: org.jboss.logging.jul-to-slf4j-stub).

So we are thinking at solving this issue in 2 possible ways:

1) Mandate that all injected modules become dependencies of the
subsystem module at the cost to load some useless modules at runtime.

2) Possibly better, make the subsystem to call
RuntimeCapability.addAdditionalRequiredPackages (package name being
module name) for each injected module. An existing capability or a new
one  would have to be created.

There is also the case of optional injected module dependencies (eg: ee
subsystem org.glassfish.javax.el, org.eclipse.yasson, ...). We need to
treat them differently. When one is provisioning a server using galleon
he can choose to exclude some packages from the provisioned server.
Optional packages can be excluded without making the provisioned state
invalid (as opposed to required package that can't be excluded). These
optional implicit dependencies are typical usage of this, they are not
required by the deployment unit to properly operate.

For these, we plan to use
RuntimeCapability.addAdditionalOptionalPackages. We can't make them
"optional" in JBoss module. When galleon provision a subset of the
server we don't include all optional dependencies.

This brings the case of provisioning of optional dependencies present in
JBOSS module.xml.

We have identified multiple kind of optional dependencies.

1) The optional dependencies that are referencing modules only in use
when a feature is present in the configuration of the subsystem (eg: jmx
subsystem optional dep on org.jboss.remoting-jmx due to
<remoting-connector/>, remoting subsystem optional dep on
io.undertow.core for <http-connector/>, elytron subsystem optional dep
on org.picketbox for <jacc-policy>)

In order to have these dependencies to be provisioned with the
subsystem, we can attach thanks to
RuntimeCapability.addAdditionalRequiredPackages the modules to the
feature. When the feature is present in the configuration, the module is
no more optional but required.

2) The optional dependencies that reference modules that are part of
another subsystems and only use if this other subsystem is present (eg:
org.jboss.as.jpa optional dep on org.jboss.as.ejb3,
org.jboss.as.transactions optional dep on org.jboss.remoting). These
ones are simply not taken into account

3)  The optional dependencies that reference modules that are not part
of another subsystem (so are not provisioned by another source) but are
only meaningful if the other subsystem is present. We call these ones
"passive" (eg: org.jboss.as.weld optional dependency on
org.jboss.as.weld.ejb|jpa|beanvalidation|webservices|transactions).
"passive" are analyzed. If all their dependencies are present, then they
are included. Some implicit optional dependencies can fall into this
category (eg: org.jboss.jaxrs optional dep on
org.jboss.resteasy.resteasy-validator-provider-11).

The passive optional dependencies would be advertised with
RuntimeCapability.addAdditionalPassiveOptionalPackages(optional
dependency package name).

So to summarize:

- RuntimeCapability.addAdditionalRequiredPackages for all required
implicit modules

- RuntimeCapability.addAdditionalRequiredPackages to associate optional
dependencies to a feature

- RuntimeCapability.addAdditionalOptionalPackages for all optional
implicit dependencies that are not passive

- RuntimeCapability.addAdditionalPassiveOptionalPackages for all
optional dependencies (implicit or not) that are passive

Is it something that subsystems owner would be ready to put in place to
help galleon in this task? It would require a bit of analysis of the
dependencies of your modules but the gain could be quite important. Some
early experimentation of this has shown a big reduction of the server
filesystem footprint (web server + cdi has been reduced from 156MB to
46MB). The runtime memory usage reduction is not that big, but less
modules being loaded we have a gain.

Thanks for reading.

JF

_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev


--
James R. Perkins
JBoss by Red Hat

_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev



--
Brian Stansberry
Manager, Senior Principal Software Engineer
Red Hat

_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev
Reply | Threaded
Open this post in threaded view
|

Re: Subsystems to advertise implicit and optional dependencies to galleon

Jean-Francois Denise
In reply to this post by James Perkins

Thank-you.


On 09/11/2018 19:45, James Perkins wrote:
The main reason those modules are dependencies of the logging subsystem is because they are not really required for the subsystem itself with the exception of org.jboss.logging. Those module dependencies are added to deployments for convenience. So really they're not required at all unless the user is using one of those logging frameworks and they want the logging subsystem to control the logging output.
There is some logic to attempt to load the module, if it succeeds, then the dependency is added as a required one. Could this be replaced by an optional dependency and no module loading? These modules would have to to be added thanks to addAdditionalOptionalPackages.

This may be somewhat of an edge case because if the user doesn't use, say slf4j, then there is no real reason to include the dependency unless of course another module uses it. I do wonder if we should remove the org.jboss.logging.jul-to-slf4j-stub module and just put the library as a resource root to org.slf4j, but that's slightly OT :)

On Fri, Nov 9, 2018 at 12:34 AM Jean-Francois Denise <[hidden email]> wrote:
Hi,

in the galleon project, in a context where we would not provision a
complete server but just a subset required to run a given configuration,
we have identified a need for subsystems to advertise to the galleon
tooling some modules in addition to the modules galleon discover by
traversing the module dependency tree.

The first case is DeploymentProcessor injecting modules into the
deployment units (implicit modules).

The deployment injected ones are not required to be a dependency of the
subsystem module, so galleon has the risk to miss some of them.

As an example, in logging subsystem we have the following non optional
injected modules :
"org.jboss.logging",
"org.apache.commons.logging",
"org.apache.log4j",
"org.slf4j",
"org.jboss.logging.jul-to-slf4j-stub"

Some of these modules are direct dependencies of logging subsystem, some
others are indirect dependencies, others could be not present at all in
the module dependency tree (eg: org.jboss.logging.jul-to-slf4j-stub).

So we are thinking at solving this issue in 2 possible ways:

1) Mandate that all injected modules become dependencies of the
subsystem module at the cost to load some useless modules at runtime.

2) Possibly better, make the subsystem to call
RuntimeCapability.addAdditionalRequiredPackages (package name being
module name) for each injected module. An existing capability or a new
one  would have to be created.

There is also the case of optional injected module dependencies (eg: ee
subsystem org.glassfish.javax.el, org.eclipse.yasson, ...). We need to
treat them differently. When one is provisioning a server using galleon
he can choose to exclude some packages from the provisioned server.
Optional packages can be excluded without making the provisioned state
invalid (as opposed to required package that can't be excluded). These
optional implicit dependencies are typical usage of this, they are not
required by the deployment unit to properly operate.

For these, we plan to use
RuntimeCapability.addAdditionalOptionalPackages. We can't make them
"optional" in JBoss module. When galleon provision a subset of the
server we don't include all optional dependencies.

This brings the case of provisioning of optional dependencies present in
JBOSS module.xml.

We have identified multiple kind of optional dependencies.

1) The optional dependencies that are referencing modules only in use
when a feature is present in the configuration of the subsystem (eg: jmx
subsystem optional dep on org.jboss.remoting-jmx due to
<remoting-connector/>, remoting subsystem optional dep on
io.undertow.core for <http-connector/>, elytron subsystem optional dep
on org.picketbox for <jacc-policy>)

In order to have these dependencies to be provisioned with the
subsystem, we can attach thanks to
RuntimeCapability.addAdditionalRequiredPackages the modules to the
feature. When the feature is present in the configuration, the module is
no more optional but required.

2) The optional dependencies that reference modules that are part of
another subsystems and only use if this other subsystem is present (eg:
org.jboss.as.jpa optional dep on org.jboss.as.ejb3,
org.jboss.as.transactions optional dep on org.jboss.remoting). These
ones are simply not taken into account

3)  The optional dependencies that reference modules that are not part
of another subsystem (so are not provisioned by another source) but are
only meaningful if the other subsystem is present. We call these ones
"passive" (eg: org.jboss.as.weld optional dependency on
org.jboss.as.weld.ejb|jpa|beanvalidation|webservices|transactions).
"passive" are analyzed. If all their dependencies are present, then they
are included. Some implicit optional dependencies can fall into this
category (eg: org.jboss.jaxrs optional dep on
org.jboss.resteasy.resteasy-validator-provider-11).

The passive optional dependencies would be advertised with
RuntimeCapability.addAdditionalPassiveOptionalPackages(optional
dependency package name).

So to summarize:

- RuntimeCapability.addAdditionalRequiredPackages for all required
implicit modules

- RuntimeCapability.addAdditionalRequiredPackages to associate optional
dependencies to a feature

- RuntimeCapability.addAdditionalOptionalPackages for all optional
implicit dependencies that are not passive

- RuntimeCapability.addAdditionalPassiveOptionalPackages for all
optional dependencies (implicit or not) that are passive

Is it something that subsystems owner would be ready to put in place to
help galleon in this task? It would require a bit of analysis of the
dependencies of your modules but the gain could be quite important. Some
early experimentation of this has shown a big reduction of the server
filesystem footprint (web server + cdi has been reduced from 156MB to
46MB). The runtime memory usage reduction is not that big, but less
modules being loaded we have a gain.

Thanks for reading.

JF

_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev


--
James R. Perkins
JBoss by Red Hat


_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev


_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev
Reply | Threaded
Open this post in threaded view
|

Re: Subsystems to advertise implicit and optional dependencies to galleon

Jean-Francois Denise
In reply to this post by Brian Stansberry



On 09/11/2018 18:32, Brian Stansberry wrote:


On Fri, Nov 9, 2018 at 2:30 AM, Jean-Francois Denise <[hidden email]> wrote:
Hi,

in the galleon project, in a context where we would not provision a
complete server but just a subset required to run a given configuration,
we have identified a need for subsystems to advertise to the galleon
tooling some modules in addition to the modules galleon discover by
traversing the module dependency tree.

The first case is DeploymentProcessor injecting modules into the
deployment units (implicit modules).

The deployment injected ones are not required to be a dependency of the
subsystem module, so galleon has the risk to miss some of them.

As an example, in logging subsystem we have the following non optional
injected modules :
"org.jboss.logging",
"org.apache.commons.logging",
"org.apache.log4j",
"org.slf4j",
"org.jboss.logging.jul-to-slf4j-stub"

Some of these modules are direct dependencies of logging subsystem, some
others are indirect dependencies, others could be not present at all in
the module dependency tree (eg: org.jboss.logging.jul-to-slf4j-stub).

So we are thinking at solving this issue in 2 possible ways:

1) Mandate that all injected modules become dependencies of the
subsystem module at the cost to load some useless modules at runtime.

2) Possibly better, make the subsystem to call
RuntimeCapability.addAdditionalRequiredPackages (package name being
module name) for each injected module. An existing capability or a new
one  would have to be created.

There is also the case of optional injected module dependencies (eg: ee
subsystem org.glassfish.javax.el, org.eclipse.yasson, ...). We need to
treat them differently. When one is provisioning a server using galleon
he can choose to exclude some packages from the provisioned server.
Optional packages can be excluded without making the provisioned state
invalid (as opposed to required package that can't be excluded). These
optional implicit dependencies are typical usage of this, they are not
required by the deployment unit to properly operate.

For these, we plan to use
RuntimeCapability.addAdditionalOptionalPackages. We can't make them
"optional" in JBoss module. When galleon provision a subset of the
server we don't include all optional dependencies.

This brings the case of provisioning of optional dependencies present in
JBOSS module.xml.

We have identified multiple kind of optional dependencies.

1) The optional dependencies that are referencing modules only in use
when a feature is present in the configuration of the subsystem (eg: jmx
subsystem optional dep on org.jboss.remoting-jmx due to
<remoting-connector/>, remoting subsystem optional dep on
io.undertow.core for <http-connector/>, elytron subsystem optional dep
on org.picketbox for <jacc-policy>)

In order to have these dependencies to be provisioned with the
subsystem, we can attach thanks to
RuntimeCapability.addAdditionalRequiredPackages the modules to the
feature. When the feature is present in the configuration, the module is
no more optional but required.

2) The optional dependencies that reference modules that are part of
another subsystems and only use if this other subsystem is present (eg:
org.jboss.as.jpa optional dep on org.jboss.as.ejb3,
org.jboss.as.transactions optional dep on org.jboss.remoting). These
ones are simply not taken into account

3)  The optional dependencies that reference modules that are not part
of another subsystem (so are not provisioned by another source) but are
only meaningful if the other subsystem is present. We call these ones
"passive" (eg: org.jboss.as.weld optional dependency on
org.jboss.as.weld.ejb|jpa|beanvalidation|webservices|transactions).
"passive" are analyzed. If all their dependencies are present, then they
are included. Some implicit optional dependencies can fall into this
category (eg: org.jboss.jaxrs optional dep on
org.jboss.resteasy.resteasy-validator-provider-11).

The passive optional dependencies would be advertised with
RuntimeCapability.addAdditionalPassiveOptionalPackages(optional
dependency package name).

So to summarize:

- RuntimeCapability.addAdditionalRequiredPackages for all required
implicit modules

- RuntimeCapability.addAdditionalRequiredPackages to associate optional
dependencies to a feature

- RuntimeCapability.addAdditionalOptionalPackages for all optional
implicit dependencies that are not passive

- RuntimeCapability.addAdditionalPassiveOptionalPackages for all
optional dependencies (implicit or not) that are passive

A good question Alexey raised is whether the capability is the right concept for encapsulating this information.

In the end the key point is that the information ends up in the galleon feature-spec. That's the thing that encapsulates the configuration elements and the packages (i.e. fs content) that are needed to provide a feature. It's the feature that needs the package. We generate the galleon feature specs from our management model information. When in WF 13 we hit the first must-implement use case for getting additional package info into a feature spec I kind of arbitrarily picked 'RuntimeCapability' as the place to record that. It was a reasonable enough choice for that one use case but isn't great otherwise.

The alternative is to record this data in the ResourceDefinition, which is the primary source of the data that's used in the feature spec generation.

The conceptual argument is a capability is all about a named provider of a contract that other elements of the server can rely on; e.g. some MSC Service with a particular value type that will be exposed.  But this additional package stuff isn't really about that contract, it's about other things that will be done that likely are not relevant to other elements of the system.

A hint of a design smell would be if we end up creating RuntimeCapability instances for no other reason than to record this additional package information. Something I could see happening with this logging case you mentioned at the start.

I don't think ResourceDefinition has this problem.
I have introduced artificial capabilities in ee, jaxrs and weld (subsystems that are targeted by the demo) to add additional packages. It seems that it would be common case. So it is actually smelly.



Is it something that subsystems owner would be ready to put in place to
help galleon in this task? It would require a bit of analysis of the
dependencies of your modules but the gain could be quite important.

Please don't let my point above derail consideration of this question.  Use of ResourceDefinition vs RuntimeCapability is a relatively small detail. 

Some
early experimentation of this has shown a big reduction of the server
filesystem footprint (web server + cdi has been reduced from 156MB to
46MB). The runtime memory usage reduction is not that big, but less
modules being loaded we have a gain.

Thanks for reading.

JF

_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev



--
Brian Stansberry
Manager, Senior Principal Software Engineer
Red Hat


_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev
Reply | Threaded
Open this post in threaded view
|

Re: Subsystems to advertise implicit and optional dependencies to galleon

James Perkins
In reply to this post by Jean-Francois Denise


On Mon, Nov 12, 2018 at 3:54 AM Jean-Francois Denise <[hidden email]> wrote:

Thank-you.


On 09/11/2018 19:45, James Perkins wrote:
The main reason those modules are dependencies of the logging subsystem is because they are not really required for the subsystem itself with the exception of org.jboss.logging. Those module dependencies are added to deployments for convenience. So really they're not required at all unless the user is using one of those logging frameworks and they want the logging subsystem to control the logging output.
There is some logic to attempt to load the module, if it succeeds, then the dependency is added as a required one. Could this be replaced by an optional dependency and no module loading? These modules would have to to be added thanks to addAdditionalOptionalPackages.

I guess it really does make sense to have them as optional dependencies in the module.xml. Using the addAddtionalOptionalPackages would work too though. I'd be good with doing it either way so which ever makes the most sense for Galleon. 

This may be somewhat of an edge case because if the user doesn't use, say slf4j, then there is no real reason to include the dependency unless of course another module uses it. I do wonder if we should remove the org.jboss.logging.jul-to-slf4j-stub module and just put the library as a resource root to org.slf4j, but that's slightly OT :)


--
James R. Perkins
JBoss by Red Hat


_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev

_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev


--
James R. Perkins
JBoss by Red Hat

_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev
Reply | Threaded
Open this post in threaded view
|

Re: Subsystems to advertise implicit and optional dependencies to galleon

Brian Stansberry
In reply to this post by Jean-Francois Denise


On Mon, Nov 12, 2018 at 7:28 AM, Jean-Francois Denise <[hidden email]> wrote:



On 09/11/2018 18:32, Brian Stansberry wrote:


On Fri, Nov 9, 2018 at 2:30 AM, Jean-Francois Denise <[hidden email]> wrote:
Hi,

in the galleon project, in a context where we would not provision a
complete server but just a subset required to run a given configuration,
we have identified a need for subsystems to advertise to the galleon
tooling some modules in addition to the modules galleon discover by
traversing the module dependency tree.

The first case is DeploymentProcessor injecting modules into the
deployment units (implicit modules).

The deployment injected ones are not required to be a dependency of the
subsystem module, so galleon has the risk to miss some of them.

As an example, in logging subsystem we have the following non optional
injected modules :
"org.jboss.logging",
"org.apache.commons.logging",
"org.apache.log4j",
"org.slf4j",
"org.jboss.logging.jul-to-slf4j-stub"

Some of these modules are direct dependencies of logging subsystem, some
others are indirect dependencies, others could be not present at all in
the module dependency tree (eg: org.jboss.logging.jul-to-slf4j-stub).

So we are thinking at solving this issue in 2 possible ways:

1) Mandate that all injected modules become dependencies of the
subsystem module at the cost to load some useless modules at runtime.

2) Possibly better, make the subsystem to call
RuntimeCapability.addAdditionalRequiredPackages (package name being
module name) for each injected module. An existing capability or a new
one  would have to be created.

There is also the case of optional injected module dependencies (eg: ee
subsystem org.glassfish.javax.el, org.eclipse.yasson, ...). We need to
treat them differently. When one is provisioning a server using galleon
he can choose to exclude some packages from the provisioned server.
Optional packages can be excluded without making the provisioned state
invalid (as opposed to required package that can't be excluded). These
optional implicit dependencies are typical usage of this, they are not
required by the deployment unit to properly operate.

For these, we plan to use
RuntimeCapability.addAdditionalOptionalPackages. We can't make them
"optional" in JBoss module. When galleon provision a subset of the
server we don't include all optional dependencies.

This brings the case of provisioning of optional dependencies present in
JBOSS module.xml.

We have identified multiple kind of optional dependencies.

1) The optional dependencies that are referencing modules only in use
when a feature is present in the configuration of the subsystem (eg: jmx
subsystem optional dep on org.jboss.remoting-jmx due to
<remoting-connector/>, remoting subsystem optional dep on
io.undertow.core for <http-connector/>, elytron subsystem optional dep
on org.picketbox for <jacc-policy>)

In order to have these dependencies to be provisioned with the
subsystem, we can attach thanks to
RuntimeCapability.addAdditionalRequiredPackages the modules to the
feature. When the feature is present in the configuration, the module is
no more optional but required.

2) The optional dependencies that reference modules that are part of
another subsystems and only use if this other subsystem is present (eg:
org.jboss.as.jpa optional dep on org.jboss.as.ejb3,
org.jboss.as.transactions optional dep on org.jboss.remoting). These
ones are simply not taken into account

3)  The optional dependencies that reference modules that are not part
of another subsystem (so are not provisioned by another source) but are
only meaningful if the other subsystem is present. We call these ones
"passive" (eg: org.jboss.as.weld optional dependency on
org.jboss.as.weld.ejb|jpa|beanvalidation|webservices|transactions).
"passive" are analyzed. If all their dependencies are present, then they
are included. Some implicit optional dependencies can fall into this
category (eg: org.jboss.jaxrs optional dep on
org.jboss.resteasy.resteasy-validator-provider-11).

The passive optional dependencies would be advertised with
RuntimeCapability.addAdditionalPassiveOptionalPackages(optional
dependency package name).

So to summarize:

- RuntimeCapability.addAdditionalRequiredPackages for all required
implicit modules

- RuntimeCapability.addAdditionalRequiredPackages to associate optional
dependencies to a feature

- RuntimeCapability.addAdditionalOptionalPackages for all optional
implicit dependencies that are not passive

- RuntimeCapability.addAdditionalPassiveOptionalPackages for all
optional dependencies (implicit or not) that are passive

A good question Alexey raised is whether the capability is the right concept for encapsulating this information.

In the end the key point is that the information ends up in the galleon feature-spec. That's the thing that encapsulates the configuration elements and the packages (i.e. fs content) that are needed to provide a feature. It's the feature that needs the package. We generate the galleon feature specs from our management model information. When in WF 13 we hit the first must-implement use case for getting additional package info into a feature spec I kind of arbitrarily picked 'RuntimeCapability' as the place to record that. It was a reasonable enough choice for that one use case but isn't great otherwise.

The alternative is to record this data in the ResourceDefinition, which is the primary source of the data that's used in the feature spec generation.

The conceptual argument is a capability is all about a named provider of a contract that other elements of the server can rely on; e.g. some MSC Service with a particular value type that will be exposed.  But this additional package stuff isn't really about that contract, it's about other things that will be done that likely are not relevant to other elements of the system.

A hint of a design smell would be if we end up creating RuntimeCapability instances for no other reason than to record this additional package information. Something I could see happening with this logging case you mentioned at the start.

I don't think ResourceDefinition has this problem.
I have introduced artificial capabilities in ee, jaxrs and weld (subsystems that are targeted by the demo) to add additional packages. It seems that it would be common case. So it is actually smelly.

Note though that all 3 of those subsystems are lacking capabilities for things they provide that other subsystems use. Or perhaps in the jaxrs case a private one that lets it require/use capabilities from other subsystems. (Yeray is working on weld.) IOW it's possible you're not really adding artificial capabilities, you're just bumping into the missing ones. ;)

^^^ is just an FYI comment though. I have little doubt we'll find artificial capabilities and it's possible ee, jaxrs, weld would have them.





Is it something that subsystems owner would be ready to put in place to
help galleon in this task? It would require a bit of analysis of the
dependencies of your modules but the gain could be quite important.

Please don't let my point above derail consideration of this question.  Use of ResourceDefinition vs RuntimeCapability is a relatively small detail. 

Some
early experimentation of this has shown a big reduction of the server
filesystem footprint (web server + cdi has been reduced from 156MB to
46MB). The runtime memory usage reduction is not that big, but less
modules being loaded we have a gain.

Thanks for reading.

JF

_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev



--
Brian Stansberry
Manager, Senior Principal Software Engineer
Red Hat




--
Brian Stansberry
Manager, Senior Principal Software Engineer
Red Hat

_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev
Reply | Threaded
Open this post in threaded view
|

Re: Subsystems to advertise implicit and optional dependencies to galleon

Jeff Mesnil
In reply to this post by Jean-Francois Denise

comments inlined

> On 09/11/2018 18:32, Brian Stansberry wrote:
>>
>> The passive optional dependencies would be advertised with
>> RuntimeCapability.addAdditionalPassiveOptionalPackages(optional
>> dependency package name).
>>
>> So to summarize:
>>
>> - RuntimeCapability.addAdditionalRequiredPackages for all required
>> implicit modules
>>
>> - RuntimeCapability.addAdditionalRequiredPackages to associate optional
>> dependencies to a feature
>>
>> - RuntimeCapability.addAdditionalOptionalPackages for all optional
>> implicit dependencies that are not passive
>>
>> - RuntimeCapability.addAdditionalPassiveOptionalPackages for all
>> optional dependencies (implicit or not) that are passive
>>
>> A good question Alexey raised is whether the capability is the right concept for encapsulating this information.

The fact you have to create artificial capabilities to provide this information makes me think Alexey is right.

>> The alternative is to record this data in the ResourceDefinition, which is the primary source of the data that's used in the feature spec generation.

That’s a good idea.
This would solve the logging issues where the resource that registers the DUP (i.e. the /subsystem=logging resource) will indicate the packages that are required for it.

The question is does storing them in ResourceDefinition is enough for all cases or do we have to able to keep them attached to a RuntimeCapability.

>> I don't think ResourceDefinition has this problem.
> I have introduced artificial capabilities in ee, jaxrs and weld (subsystems that are targeted by the demo) to add additional packages. It seems that it would be common case. So it is actually smelly.

We should also ensure that we do not introduce artificial resource definition to provide these packages information but that is unlikely.
(That reminds me how we used to create artificial subresources to store complex configuration that was only used by the parent resource).

>> Is it something that subsystems owner would be ready to put in place to
>> help galleon in this task? It would require a bit of analysis of the
>> dependencies of your modules but the gain could be quite important.

If we want to get buy in from the subsystem developer, we need:
* tools to analyzee the dependencies (you already wrote them)
* good documentation to explain to a subsystem developer what a optional/required/passive/implicit package means (maybe linking to Galleon from the Javadoc is enough but I am not sure…)


Good stuff, Jean-Francois :)

Thanks,
jeff

--
Jeff Mesnil
JBoss, a division of Red Hat
http://jmesnil.net/


_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev