Speeding up WildFly boot time

classic Classic list List threaded Threaded
22 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Speeding up WildFly boot time

Stuart Douglas
When JIRA was being screwy on Friday I used the time to investigate an idea I have had for a while about improving our boot time performance. According to Yourkit the majority of our time is spent in class loading. It seems very unlikely that we will be able to reduce the number of classes we load on boot (or at the very least it would be a massive amount of work) so I investigated a different approach.

I modified ModuleClassLoader to spit out the name and module of every class that is loaded at boot time, and stored this in a properties file. I then created a simple Service that starts immediately that uses two threads to eagerly load every class on this list (I used two threads because that seemed to work well on my laptop, I think Runtime.availableProcessors()/4 is probably the best amount, but that assumption would need to be tested on different hardware). 

The idea behind this is that we know the classes will be used at some point, and we generally do not fully utilise all CPU's during boot, so we can use the unused CPU to pre load these classes so they are ready when they are actually required.

Using this approach I saw the boot time for standalone.xml drop from ~2.9s to ~2.3s on my laptop. The (super hacky) code I used to perform this test is at https://github.com/wildfly/wildfly-core/compare/master...stuartwdouglas:boot-performance-hack

I think these initial results are encouraging, and it is a big enough gain that I think it is worth investigating further.

Firstly it would be great if I could get others to try it out and see if they see similar gains to boot time, it may be that the gain is very system dependent.

Secondly if we do decide to do this there are two approach that we can use that I can see:

1) A hard coded list of class names that we generate before a release (basically what the hack already does), this is simplest, but does add a little bit of additional work to the release process (although if it is missed it would be no big deal, as ClassNotFoundException's would be suppressed, and if a few classes are missing the performance impact is negligible as long as the majority of the list is correct).

2) Generate the list dynamically on first boot, and store it in the temp directory. This would require the addition of a hook into JBoss Modules to generate the list, but is the approach I would prefer (as first boot is always a bit slower anyway).

Thoughts?

Stuart

_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev
Reply | Threaded
Open this post in threaded view
|

Re: Speeding up WildFly boot time

Tomaž Cerar-2
Hey Stuart,

this exact problem we discussed some time ago with David but didn't go as far as implementing a prototype.
At the time one of bigger contention bottleneck was in jdk classes (java.lang.*, java.util.*, etc)
and I think we should do something similar to what you did also in jboss-modules to address speeding up this.
It would probably yield even better results in end run.

--
tomaz



On Mon, May 15, 2017 at 1:36 AM, Stuart Douglas <[hidden email]> wrote:
When JIRA was being screwy on Friday I used the time to investigate an idea I have had for a while about improving our boot time performance. According to Yourkit the majority of our time is spent in class loading. It seems very unlikely that we will be able to reduce the number of classes we load on boot (or at the very least it would be a massive amount of work) so I investigated a different approach.

I modified ModuleClassLoader to spit out the name and module of every class that is loaded at boot time, and stored this in a properties file. I then created a simple Service that starts immediately that uses two threads to eagerly load every class on this list (I used two threads because that seemed to work well on my laptop, I think Runtime.availableProcessors()/4 is probably the best amount, but that assumption would need to be tested on different hardware). 

The idea behind this is that we know the classes will be used at some point, and we generally do not fully utilise all CPU's during boot, so we can use the unused CPU to pre load these classes so they are ready when they are actually required.

Using this approach I saw the boot time for standalone.xml drop from ~2.9s to ~2.3s on my laptop. The (super hacky) code I used to perform this test is at https://github.com/wildfly/wildfly-core/compare/master...stuartwdouglas:boot-performance-hack

I think these initial results are encouraging, and it is a big enough gain that I think it is worth investigating further.

Firstly it would be great if I could get others to try it out and see if they see similar gains to boot time, it may be that the gain is very system dependent.

Secondly if we do decide to do this there are two approach that we can use that I can see:

1) A hard coded list of class names that we generate before a release (basically what the hack already does), this is simplest, but does add a little bit of additional work to the release process (although if it is missed it would be no big deal, as ClassNotFoundException's would be suppressed, and if a few classes are missing the performance impact is negligible as long as the majority of the list is correct).

2) Generate the list dynamically on first boot, and store it in the temp directory. This would require the addition of a hook into JBoss Modules to generate the list, but is the approach I would prefer (as first boot is always a bit slower anyway).

Thoughts?

Stuart

_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev


_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev
Reply | Threaded
Open this post in threaded view
|

Re: Speeding up WildFly boot time

Rostislav Svoboda
In reply to this post by Stuart Douglas
Hi.

I can confirm I see improvements in boot time with your changes.
My HW is Lenovo T440s with Fedora 25, Intel(R) Core(TM) i7-4600U CPU (Base Frequency 2.10 GHz, Max Turbo 3.30 GHz)

I executed 50 iterations of start - stop sequence [1], before execution 5x start - stop for "warmup"

With your changes
       Min: 3116 Max: 3761 Average: 3247.640000

Without:
       Min: 3442 Max: 4081 Average: 3580.840000


> 1) A hard coded list of class names that we generate before a release

This will improve first boot impression, little bit harder for maintaining the list for the final build.

Property files could be located inside properties directory of dedicated module (<resource-root path="properties"/>). Properties directory could contain property files for delivered profiles.

Layered products or customer modifications could deliver own property file.
 e.g. predefined property file for standalone-openshift.xml in EAP image in OpenShift environment, I think they boot the server just once and throw away the whole docker image when something changes.


> 2) Generate the list dynamically on first boot, and store it in the temp

This looks like the most elegant thing to do. Question is how it will slow down the initial boot. People care about first boot impression, some blog writers do the mistake too.
This would also block boot time improvements for use-cases when you start the server just once - e.g. Docker, OpenShift.

Also the logic should take into account which profile is loaded - e.g standalone.xml vs. standalone-full-ha.xml

Rostislav

[1]
rm wildfly-11.0.0.Beta1-SNAPSHOT-preload/standalone/log/server.log
rm wildfly-11.0.0.Beta1-SNAPSHOT/standalone/log/server.log

for i in {1..50}; do
  echo $i
  wildfly-11.0.0.Beta1-SNAPSHOT-preload/bin/standalone.sh 1>/dev/null 2>&1 &
  sleep 8
  wildfly-11.0.0.Beta1-SNAPSHOT-preload/bin/jboss-cli.sh -c :shutdown 1>/dev/null 2>&1
done
grep WFLYSRV0025 wildfly-11.0.0.Beta1-SNAPSHOT-preload/standalone/log/server.log | sed "s/.*\(....\)ms.*/\1/g"  | awk 'NR == 1 { max=$1; min=$1; sum=0 }
   { if ($1>max) max=$1; if ($1<min) min=$1; sum+=$1;}
   END {printf "Min: %d\tMax: %d\tAverage: %f\n", min, max, sum/NR}'


for i in {1..50}; do
  echo $i
  wildfly-11.0.0.Beta1-SNAPSHOT/bin/standalone.sh 1>/dev/null 2>&1 &
  sleep 8
  wildfly-11.0.0.Beta1-SNAPSHOT/bin/jboss-cli.sh -c :shutdown 1>/dev/null 2>&1
done
grep WFLYSRV0025 wildfly-11.0.0.Beta1-SNAPSHOT/standalone/log/server.log | sed "s/.*\(....\)ms.*/\1/g"  | awk 'NR == 1 { max=$1; min=$1; sum=0 }
   { if ($1>max) max=$1; if ($1<min) min=$1; sum+=$1;}
   END {printf "Min: %d\tMax: %d\tAverage: %f\n", min, max, sum/NR}'


----- Original Message -----

> When JIRA was being screwy on Friday I used the time to investigate an idea I
> have had for a while about improving our boot time performance. According to
> Yourkit the majority of our time is spent in class loading. It seems very
> unlikely that we will be able to reduce the number of classes we load on
> boot (or at the very least it would be a massive amount of work) so I
> investigated a different approach.
>
> I modified ModuleClassLoader to spit out the name and module of every class
> that is loaded at boot time, and stored this in a properties file. I then
> created a simple Service that starts immediately that uses two threads to
> eagerly load every class on this list (I used two threads because that
> seemed to work well on my laptop, I think Runtime.availableProcessors()/4 is
> probably the best amount, but that assumption would need to be tested on
> different hardware).
>
> The idea behind this is that we know the classes will be used at some point,
> and we generally do not fully utilise all CPU's during boot, so we can use
> the unused CPU to pre load these classes so they are ready when they are
> actually required.
>
> Using this approach I saw the boot time for standalone.xml drop from ~2.9s to
> ~2.3s on my laptop. The (super hacky) code I used to perform this test is at
> https://github.com/wildfly/wildfly-core/compare/master...stuartwdouglas:boot-performance-hack
>
> I think these initial results are encouraging, and it is a big enough gain
> that I think it is worth investigating further.
>
> Firstly it would be great if I could get others to try it out and see if they
> see similar gains to boot time, it may be that the gain is very system
> dependent.
>
> Secondly if we do decide to do this there are two approach that we can use
> that I can see:
>
> 1) A hard coded list of class names that we generate before a release
> (basically what the hack already does), this is simplest, but does add a
> little bit of additional work to the release process (although if it is
> missed it would be no big deal, as ClassNotFoundException's would be
> suppressed, and if a few classes are missing the performance impact is
> negligible as long as the majority of the list is correct).
>
> 2) Generate the list dynamically on first boot, and store it in the temp
> directory. This would require the addition of a hook into JBoss Modules to
> generate the list, but is the approach I would prefer (as first boot is
> always a bit slower anyway).
>
> Thoughts?
>
> Stuart
>
> _______________________________________________
> wildfly-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/wildfly-dev
_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev
Reply | Threaded
Open this post in threaded view
|

Re: Speeding up WildFly boot time

Brian Stansberry
In reply to this post by Stuart Douglas
Definitely worth investigating. I’d like to have a real good understanding of why it has the benefits it has, so we can see if this is the best way to get them or if something else is better.

This kicks in just before the ModelController starts and begins parsing the config. The config parsing quickly gets into parallel work; as soon as the extension elements are reached the extension modules are loaded concurrently. Then once the parsing is done each subsystem is installed concurrently, so lots of threads doing concurrent classloading.

So why does adding two more make such a big difference?

Is it that they gets lots of work done in that time when the regular boot thread is not doing concurrent work, i.e. the parsing and the non-parallel bits of operation execution?

Is it that these threads are just chugging along doing classloading efficiently while the parallel threads are running along inefficiently getting scheduled and unscheduled?

The latter doesn’t make sense to me as there’s no reason why these threads would be any more efficient than the others.

- Brian

> On May 14, 2017, at 6:36 PM, Stuart Douglas <[hidden email]> wrote:
>
> When JIRA was being screwy on Friday I used the time to investigate an idea I have had for a while about improving our boot time performance. According to Yourkit the majority of our time is spent in class loading. It seems very unlikely that we will be able to reduce the number of classes we load on boot (or at the very least it would be a massive amount of work) so I investigated a different approach.
>
> I modified ModuleClassLoader to spit out the name and module of every class that is loaded at boot time, and stored this in a properties file. I then created a simple Service that starts immediately that uses two threads to eagerly load every class on this list (I used two threads because that seemed to work well on my laptop, I think Runtime.availableProcessors()/4 is probably the best amount, but that assumption would need to be tested on different hardware).
>
> The idea behind this is that we know the classes will be used at some point, and we generally do not fully utilise all CPU's during boot, so we can use the unused CPU to pre load these classes so they are ready when they are actually required.
>
> Using this approach I saw the boot time for standalone.xml drop from ~2.9s to ~2.3s on my laptop. The (super hacky) code I used to perform this test is at https://github.com/wildfly/wildfly-core/compare/master...stuartwdouglas:boot-performance-hack
>
> I think these initial results are encouraging, and it is a big enough gain that I think it is worth investigating further.
>
> Firstly it would be great if I could get others to try it out and see if they see similar gains to boot time, it may be that the gain is very system dependent.
>
> Secondly if we do decide to do this there are two approach that we can use that I can see:
>
> 1) A hard coded list of class names that we generate before a release (basically what the hack already does), this is simplest, but does add a little bit of additional work to the release process (although if it is missed it would be no big deal, as ClassNotFoundException's would be suppressed, and if a few classes are missing the performance impact is negligible as long as the majority of the list is correct).
>
> 2) Generate the list dynamically on first boot, and store it in the temp directory. This would require the addition of a hook into JBoss Modules to generate the list, but is the approach I would prefer (as first boot is always a bit slower anyway).
>
> Thoughts?
>
> Stuart
> _______________________________________________
> wildfly-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/wildfly-dev

--
Brian Stansberry
Manager, Senior Principal Software Engineer
JBoss by Red Hat




_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev
Reply | Threaded
Open this post in threaded view
|

Re: Speeding up WildFly boot time

Brian Stansberry
In reply to this post by Rostislav Svoboda
A disadvantage of a static list is that we have other concerns besides boot speed, i.e. memory footprint. We do not want to be loading a bunch of classes that are not relevant to the configuration. You touch on this issue, Rostislav, in your point about properties file for delivered profiles.

If the lists are per feature-pack (or better yet per package once our feature packs have the package notion) and then the build integrates them, that somewhat mitigates that concern, at least for people who are careful about how they provision. But it doesn’t help people who are not so careful and just rely on standalone.xml trimming or selecting a reasonalbe standarc config to tailor their server footprint.

> On May 15, 2017, at 7:27 AM, Rostislav Svoboda <[hidden email]> wrote:
>
> Hi.
>
> I can confirm I see improvements in boot time with your changes.
> My HW is Lenovo T440s with Fedora 25, Intel(R) Core(TM) i7-4600U CPU (Base Frequency 2.10 GHz, Max Turbo 3.30 GHz)
>
> I executed 50 iterations of start - stop sequence [1], before execution 5x start - stop for "warmup"
>
> With your changes
>       Min: 3116 Max: 3761 Average: 3247.640000
>
> Without:
>       Min: 3442 Max: 4081 Average: 3580.840000
>
>
>> 1) A hard coded list of class names that we generate before a release
>
> This will improve first boot impression, little bit harder for maintaining the list for the final build.
>
> Property files could be located inside properties directory of dedicated module (<resource-root path="properties"/>). Properties directory could contain property files for delivered profiles.
>
> Layered products or customer modifications could deliver own property file.
> e.g. predefined property file for standalone-openshift.xml in EAP image in OpenShift environment, I think they boot the server just once and throw away the whole docker image when something changes.
>
>
>> 2) Generate the list dynamically on first boot, and store it in the temp
>
> This looks like the most elegant thing to do. Question is how it will slow down the initial boot. People care about first boot impression, some blog writers do the mistake too.
> This would also block boot time improvements for use-cases when you start the server just once - e.g. Docker, OpenShift.
>
> Also the logic should take into account which profile is loaded - e.g standalone.xml vs. standalone-full-ha.xml
>
> Rostislav
>
> [1]
> rm wildfly-11.0.0.Beta1-SNAPSHOT-preload/standalone/log/server.log
> rm wildfly-11.0.0.Beta1-SNAPSHOT/standalone/log/server.log
>
> for i in {1..50}; do
>  echo $i
>  wildfly-11.0.0.Beta1-SNAPSHOT-preload/bin/standalone.sh 1>/dev/null 2>&1 &
>  sleep 8
>  wildfly-11.0.0.Beta1-SNAPSHOT-preload/bin/jboss-cli.sh -c :shutdown 1>/dev/null 2>&1
> done
> grep WFLYSRV0025 wildfly-11.0.0.Beta1-SNAPSHOT-preload/standalone/log/server.log | sed "s/.*\(....\)ms.*/\1/g"  | awk 'NR == 1 { max=$1; min=$1; sum=0 }
>   { if ($1>max) max=$1; if ($1<min) min=$1; sum+=$1;}
>   END {printf "Min: %d\tMax: %d\tAverage: %f\n", min, max, sum/NR}'
>
>
> for i in {1..50}; do
>  echo $i
>  wildfly-11.0.0.Beta1-SNAPSHOT/bin/standalone.sh 1>/dev/null 2>&1 &
>  sleep 8
>  wildfly-11.0.0.Beta1-SNAPSHOT/bin/jboss-cli.sh -c :shutdown 1>/dev/null 2>&1
> done
> grep WFLYSRV0025 wildfly-11.0.0.Beta1-SNAPSHOT/standalone/log/server.log | sed "s/.*\(....\)ms.*/\1/g"  | awk 'NR == 1 { max=$1; min=$1; sum=0 }
>   { if ($1>max) max=$1; if ($1<min) min=$1; sum+=$1;}
>   END {printf "Min: %d\tMax: %d\tAverage: %f\n", min, max, sum/NR}'
>
>
> ----- Original Message -----
>> When JIRA was being screwy on Friday I used the time to investigate an idea I
>> have had for a while about improving our boot time performance. According to
>> Yourkit the majority of our time is spent in class loading. It seems very
>> unlikely that we will be able to reduce the number of classes we load on
>> boot (or at the very least it would be a massive amount of work) so I
>> investigated a different approach.
>>
>> I modified ModuleClassLoader to spit out the name and module of every class
>> that is loaded at boot time, and stored this in a properties file. I then
>> created a simple Service that starts immediately that uses two threads to
>> eagerly load every class on this list (I used two threads because that
>> seemed to work well on my laptop, I think Runtime.availableProcessors()/4 is
>> probably the best amount, but that assumption would need to be tested on
>> different hardware).
>>
>> The idea behind this is that we know the classes will be used at some point,
>> and we generally do not fully utilise all CPU's during boot, so we can use
>> the unused CPU to pre load these classes so they are ready when they are
>> actually required.
>>
>> Using this approach I saw the boot time for standalone.xml drop from ~2.9s to
>> ~2.3s on my laptop. The (super hacky) code I used to perform this test is at
>> https://github.com/wildfly/wildfly-core/compare/master...stuartwdouglas:boot-performance-hack
>>
>> I think these initial results are encouraging, and it is a big enough gain
>> that I think it is worth investigating further.
>>
>> Firstly it would be great if I could get others to try it out and see if they
>> see similar gains to boot time, it may be that the gain is very system
>> dependent.
>>
>> Secondly if we do decide to do this there are two approach that we can use
>> that I can see:
>>
>> 1) A hard coded list of class names that we generate before a release
>> (basically what the hack already does), this is simplest, but does add a
>> little bit of additional work to the release process (although if it is
>> missed it would be no big deal, as ClassNotFoundException's would be
>> suppressed, and if a few classes are missing the performance impact is
>> negligible as long as the majority of the list is correct).
>>
>> 2) Generate the list dynamically on first boot, and store it in the temp
>> directory. This would require the addition of a hook into JBoss Modules to
>> generate the list, but is the approach I would prefer (as first boot is
>> always a bit slower anyway).
>>
>> Thoughts?
>>
>> Stuart
>>
>> _______________________________________________
>> wildfly-dev mailing list
>> [hidden email]
>> https://lists.jboss.org/mailman/listinfo/wildfly-dev
> _______________________________________________
> wildfly-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/wildfly-dev

--
Brian Stansberry
Manager, Senior Principal Software Engineer
JBoss by Red Hat




_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev
Reply | Threaded
Open this post in threaded view
|

Re: Speeding up WildFly boot time

Sanne Grinovero
In reply to this post by Rostislav Svoboda
Very interesting.

>From a different perspective, but closely related, I was recently
trying to profile the testsuites of Hibernate projects and also in our
case ClassLoader time is significant portion of the bootstrap time.

In the Hibernate case I noticed it spends quite some time to locate
Service implementations over the ServiceLoader pattern.

The problem seems to be that a lot of internal code has been
refactored in recent versions to be "replaceable" so Hibernate ORM
includes a default implementation for each internal Service but it
will first check if it can find an alternative somewhere else on the
classpath: looking both among its own dependencies and among the
classes provided by the deployment.

I'll see if we can do better in Hibernate ORM (not sure yet!), but
raising it here as I suspect several other libraries could be guilty
of the same approach.

I also hope we'll be able to curate (trim) the dependencies more; the
current JPA subsystem is including many dependencies of questionable
usefulness. That should help?

Thanks,
Sanne



On 15 May 2017 at 13:27, Rostislav Svoboda <[hidden email]> wrote:

> Hi.
>
> I can confirm I see improvements in boot time with your changes.
> My HW is Lenovo T440s with Fedora 25, Intel(R) Core(TM) i7-4600U CPU (Base Frequency 2.10 GHz, Max Turbo 3.30 GHz)
>
> I executed 50 iterations of start - stop sequence [1], before execution 5x start - stop for "warmup"
>
> With your changes
>        Min: 3116        Max: 3761       Average: 3247.640000
>
> Without:
>        Min: 3442        Max: 4081       Average: 3580.840000
>
>
>> 1) A hard coded list of class names that we generate before a release
>
> This will improve first boot impression, little bit harder for maintaining the list for the final build.
>
> Property files could be located inside properties directory of dedicated module (<resource-root path="properties"/>). Properties directory could contain property files for delivered profiles.
>
> Layered products or customer modifications could deliver own property file.
>  e.g. predefined property file for standalone-openshift.xml in EAP image in OpenShift environment, I think they boot the server just once and throw away the whole docker image when something changes.
>
>
>> 2) Generate the list dynamically on first boot, and store it in the temp
>
> This looks like the most elegant thing to do. Question is how it will slow down the initial boot. People care about first boot impression, some blog writers do the mistake too.
> This would also block boot time improvements for use-cases when you start the server just once - e.g. Docker, OpenShift.
>
> Also the logic should take into account which profile is loaded - e.g standalone.xml vs. standalone-full-ha.xml
>
> Rostislav
>
> [1]
> rm wildfly-11.0.0.Beta1-SNAPSHOT-preload/standalone/log/server.log
> rm wildfly-11.0.0.Beta1-SNAPSHOT/standalone/log/server.log
>
> for i in {1..50}; do
>   echo $i
>   wildfly-11.0.0.Beta1-SNAPSHOT-preload/bin/standalone.sh 1>/dev/null 2>&1 &
>   sleep 8
>   wildfly-11.0.0.Beta1-SNAPSHOT-preload/bin/jboss-cli.sh -c :shutdown 1>/dev/null 2>&1
> done
> grep WFLYSRV0025 wildfly-11.0.0.Beta1-SNAPSHOT-preload/standalone/log/server.log | sed "s/.*\(....\)ms.*/\1/g"  | awk 'NR == 1 { max=$1; min=$1; sum=0 }
>    { if ($1>max) max=$1; if ($1<min) min=$1; sum+=$1;}
>    END {printf "Min: %d\tMax: %d\tAverage: %f\n", min, max, sum/NR}'
>
>
> for i in {1..50}; do
>   echo $i
>   wildfly-11.0.0.Beta1-SNAPSHOT/bin/standalone.sh 1>/dev/null 2>&1 &
>   sleep 8
>   wildfly-11.0.0.Beta1-SNAPSHOT/bin/jboss-cli.sh -c :shutdown 1>/dev/null 2>&1
> done
> grep WFLYSRV0025 wildfly-11.0.0.Beta1-SNAPSHOT/standalone/log/server.log | sed "s/.*\(....\)ms.*/\1/g"  | awk 'NR == 1 { max=$1; min=$1; sum=0 }
>    { if ($1>max) max=$1; if ($1<min) min=$1; sum+=$1;}
>    END {printf "Min: %d\tMax: %d\tAverage: %f\n", min, max, sum/NR}'
>
>
> ----- Original Message -----
>> When JIRA was being screwy on Friday I used the time to investigate an idea I
>> have had for a while about improving our boot time performance. According to
>> Yourkit the majority of our time is spent in class loading. It seems very
>> unlikely that we will be able to reduce the number of classes we load on
>> boot (or at the very least it would be a massive amount of work) so I
>> investigated a different approach.
>>
>> I modified ModuleClassLoader to spit out the name and module of every class
>> that is loaded at boot time, and stored this in a properties file. I then
>> created a simple Service that starts immediately that uses two threads to
>> eagerly load every class on this list (I used two threads because that
>> seemed to work well on my laptop, I think Runtime.availableProcessors()/4 is
>> probably the best amount, but that assumption would need to be tested on
>> different hardware).
>>
>> The idea behind this is that we know the classes will be used at some point,
>> and we generally do not fully utilise all CPU's during boot, so we can use
>> the unused CPU to pre load these classes so they are ready when they are
>> actually required.
>>
>> Using this approach I saw the boot time for standalone.xml drop from ~2.9s to
>> ~2.3s on my laptop. The (super hacky) code I used to perform this test is at
>> https://github.com/wildfly/wildfly-core/compare/master...stuartwdouglas:boot-performance-hack
>>
>> I think these initial results are encouraging, and it is a big enough gain
>> that I think it is worth investigating further.
>>
>> Firstly it would be great if I could get others to try it out and see if they
>> see similar gains to boot time, it may be that the gain is very system
>> dependent.
>>
>> Secondly if we do decide to do this there are two approach that we can use
>> that I can see:
>>
>> 1) A hard coded list of class names that we generate before a release
>> (basically what the hack already does), this is simplest, but does add a
>> little bit of additional work to the release process (although if it is
>> missed it would be no big deal, as ClassNotFoundException's would be
>> suppressed, and if a few classes are missing the performance impact is
>> negligible as long as the majority of the list is correct).
>>
>> 2) Generate the list dynamically on first boot, and store it in the temp
>> directory. This would require the addition of a hook into JBoss Modules to
>> generate the list, but is the approach I would prefer (as first boot is
>> always a bit slower anyway).
>>
>> Thoughts?
>>
>> Stuart
>>
>> _______________________________________________
>> wildfly-dev mailing list
>> [hidden email]
>> https://lists.jboss.org/mailman/listinfo/wildfly-dev
> _______________________________________________
> wildfly-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/wildfly-dev
_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev
Reply | Threaded
Open this post in threaded view
|

Re: Speeding up WildFly boot time

Brian Stansberry
+1 re: being careful about ServiceLoader.

We were doing some perf testing work last week and that's one thing that showed up. Unfortunately it was in the app we were testing rather than in the server code but I could easily imagine similar things happening in the server.

In case people are curious, the issue is the javax.json.Json class, which provides a bunch of static utility methods to create JSON related objects. The problem is they are implemented via "JsonProvider.provider().doXXX”. And that JsonProvider.provider() call uses a ServiceLoader to try and find any custom JsonProvider impls before falling back using the default. WildFly doesn’t ship any such impls. Those ServiceLoader calls all result in a FileNotFoundException as the classloader checks for META-INF/services/…JsonProvider. So IO access plus the cost of creating exception.

The solution is to not use  javax.json.Json.xxx all the time in the app, but to call JsonProvider.provider() once and cache it.

> On May 15, 2017, at 9:23 AM, Sanne Grinovero <[hidden email]> wrote:
>
> Very interesting.
>
>> From a different perspective, but closely related, I was recently
> trying to profile the testsuites of Hibernate projects and also in our
> case ClassLoader time is significant portion of the bootstrap time.
>
> In the Hibernate case I noticed it spends quite some time to locate
> Service implementations over the ServiceLoader pattern.
>
> The problem seems to be that a lot of internal code has been
> refactored in recent versions to be "replaceable" so Hibernate ORM
> includes a default implementation for each internal Service but it
> will first check if it can find an alternative somewhere else on the
> classpath: looking both among its own dependencies and among the
> classes provided by the deployment.
>
> I'll see if we can do better in Hibernate ORM (not sure yet!), but
> raising it here as I suspect several other libraries could be guilty
> of the same approach.
>
> I also hope we'll be able to curate (trim) the dependencies more; the
> current JPA subsystem is including many dependencies of questionable
> usefulness. That should help?
>
> Thanks,
> Sanne
>
>
>
> On 15 May 2017 at 13:27, Rostislav Svoboda <[hidden email]> wrote:
>> Hi.
>>
>> I can confirm I see improvements in boot time with your changes.
>> My HW is Lenovo T440s with Fedora 25, Intel(R) Core(TM) i7-4600U CPU (Base Frequency 2.10 GHz, Max Turbo 3.30 GHz)
>>
>> I executed 50 iterations of start - stop sequence [1], before execution 5x start - stop for "warmup"
>>
>> With your changes
>>       Min: 3116        Max: 3761       Average: 3247.640000
>>
>> Without:
>>       Min: 3442        Max: 4081       Average: 3580.840000
>>
>>
>>> 1) A hard coded list of class names that we generate before a release
>>
>> This will improve first boot impression, little bit harder for maintaining the list for the final build.
>>
>> Property files could be located inside properties directory of dedicated module (<resource-root path="properties"/>). Properties directory could contain property files for delivered profiles.
>>
>> Layered products or customer modifications could deliver own property file.
>> e.g. predefined property file for standalone-openshift.xml in EAP image in OpenShift environment, I think they boot the server just once and throw away the whole docker image when something changes.
>>
>>
>>> 2) Generate the list dynamically on first boot, and store it in the temp
>>
>> This looks like the most elegant thing to do. Question is how it will slow down the initial boot. People care about first boot impression, some blog writers do the mistake too.
>> This would also block boot time improvements for use-cases when you start the server just once - e.g. Docker, OpenShift.
>>
>> Also the logic should take into account which profile is loaded - e.g standalone.xml vs. standalone-full-ha.xml
>>
>> Rostislav
>>
>> [1]
>> rm wildfly-11.0.0.Beta1-SNAPSHOT-preload/standalone/log/server.log
>> rm wildfly-11.0.0.Beta1-SNAPSHOT/standalone/log/server.log
>>
>> for i in {1..50}; do
>>  echo $i
>>  wildfly-11.0.0.Beta1-SNAPSHOT-preload/bin/standalone.sh 1>/dev/null 2>&1 &
>>  sleep 8
>>  wildfly-11.0.0.Beta1-SNAPSHOT-preload/bin/jboss-cli.sh -c :shutdown 1>/dev/null 2>&1
>> done
>> grep WFLYSRV0025 wildfly-11.0.0.Beta1-SNAPSHOT-preload/standalone/log/server.log | sed "s/.*\(....\)ms.*/\1/g"  | awk 'NR == 1 { max=$1; min=$1; sum=0 }
>>   { if ($1>max) max=$1; if ($1<min) min=$1; sum+=$1;}
>>   END {printf "Min: %d\tMax: %d\tAverage: %f\n", min, max, sum/NR}'
>>
>>
>> for i in {1..50}; do
>>  echo $i
>>  wildfly-11.0.0.Beta1-SNAPSHOT/bin/standalone.sh 1>/dev/null 2>&1 &
>>  sleep 8
>>  wildfly-11.0.0.Beta1-SNAPSHOT/bin/jboss-cli.sh -c :shutdown 1>/dev/null 2>&1
>> done
>> grep WFLYSRV0025 wildfly-11.0.0.Beta1-SNAPSHOT/standalone/log/server.log | sed "s/.*\(....\)ms.*/\1/g"  | awk 'NR == 1 { max=$1; min=$1; sum=0 }
>>   { if ($1>max) max=$1; if ($1<min) min=$1; sum+=$1;}
>>   END {printf "Min: %d\tMax: %d\tAverage: %f\n", min, max, sum/NR}'
>>
>>
>> ----- Original Message -----
>>> When JIRA was being screwy on Friday I used the time to investigate an idea I
>>> have had for a while about improving our boot time performance. According to
>>> Yourkit the majority of our time is spent in class loading. It seems very
>>> unlikely that we will be able to reduce the number of classes we load on
>>> boot (or at the very least it would be a massive amount of work) so I
>>> investigated a different approach.
>>>
>>> I modified ModuleClassLoader to spit out the name and module of every class
>>> that is loaded at boot time, and stored this in a properties file. I then
>>> created a simple Service that starts immediately that uses two threads to
>>> eagerly load every class on this list (I used two threads because that
>>> seemed to work well on my laptop, I think Runtime.availableProcessors()/4 is
>>> probably the best amount, but that assumption would need to be tested on
>>> different hardware).
>>>
>>> The idea behind this is that we know the classes will be used at some point,
>>> and we generally do not fully utilise all CPU's during boot, so we can use
>>> the unused CPU to pre load these classes so they are ready when they are
>>> actually required.
>>>
>>> Using this approach I saw the boot time for standalone.xml drop from ~2.9s to
>>> ~2.3s on my laptop. The (super hacky) code I used to perform this test is at
>>> https://github.com/wildfly/wildfly-core/compare/master...stuartwdouglas:boot-performance-hack
>>>
>>> I think these initial results are encouraging, and it is a big enough gain
>>> that I think it is worth investigating further.
>>>
>>> Firstly it would be great if I could get others to try it out and see if they
>>> see similar gains to boot time, it may be that the gain is very system
>>> dependent.
>>>
>>> Secondly if we do decide to do this there are two approach that we can use
>>> that I can see:
>>>
>>> 1) A hard coded list of class names that we generate before a release
>>> (basically what the hack already does), this is simplest, but does add a
>>> little bit of additional work to the release process (although if it is
>>> missed it would be no big deal, as ClassNotFoundException's would be
>>> suppressed, and if a few classes are missing the performance impact is
>>> negligible as long as the majority of the list is correct).
>>>
>>> 2) Generate the list dynamically on first boot, and store it in the temp
>>> directory. This would require the addition of a hook into JBoss Modules to
>>> generate the list, but is the approach I would prefer (as first boot is
>>> always a bit slower anyway).
>>>
>>> Thoughts?
>>>
>>> Stuart
>>>
>>> _______________________________________________
>>> wildfly-dev mailing list
>>> [hidden email]
>>> https://lists.jboss.org/mailman/listinfo/wildfly-dev
>> _______________________________________________
>> wildfly-dev mailing list
>> [hidden email]
>> https://lists.jboss.org/mailman/listinfo/wildfly-dev
> _______________________________________________
> wildfly-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/wildfly-dev

--
Brian Stansberry
Manager, Senior Principal Software Engineer
JBoss by Red Hat




_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev
Reply | Threaded
Open this post in threaded view
|

Re: Speeding up WildFly boot time

Tomaž Cerar-2
In reply to this post by Brian Stansberry

On Mon, May 15, 2017 at 4:13 PM, Brian Stansberry <[hidden email]> wrote:
So why does adding two more make such a big difference?

Main reason is that this two threads load most of later required classes which can later be quickly loaded from multiple parallel threads.

Currently concurrency causes that 8 -16 threads (on 4-8 logical core systems) try to load same classes at same time.
this leads to lots of contention as result. "preloading" some of this classes reduces contention.

Looking at the list in the current "hack impl" there are lots of classes that don't need to be there, stuff like subsystem parsers which are only loaded once in any case.

Main pressure is on classes from jboss-modules, controller, server & xml parsers modules, all others are not as problematic.
This is also reason why lots of contention is happening on JDK classes as well as those are shared between all parts of server code.





_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev
Reply | Threaded
Open this post in threaded view
|

Re: Speeding up WildFly boot time

David M. Lloyd
In reply to this post by Stuart Douglas
I have a few thoughts that might be of interest.

Firstly, I'd be interested to see when you are logging the class name
being loaded.  If you are logging it in loadClass, you might not be
seeing the actual correct load order because that method is ultimately
recursive.  To get an accurate picture of what order that classes are
actually defined - and thus what order you can load them in order to
prevent contention on per-class locks within the CL - you should log
immediately _after_ defineClass completes for each class.

Secondly, while debugging a resource iteration performance problem a
user was having with a large number of deployments, I discovered that
contention for the lock on JarFile and ZipFile was a primary cause.  The
workaround I employed was to keep a RAM-based List of the files in the
JAR, which can be iterated over without touching the lock.

When we're preloading classes, we're definitely going to see this same
kind of contention come up, because there's only one lock per JarFile
instance so you can only ever read one entry at a time, thus preventing
any kind of useful concurrency on a per-module basis.

Exploding the files out of the JarFile could expose this contention and
therefore might be useful as a test - but it would also skew the results
a little because you have no decompression overhead, and creating the
separate file streams hypothetically might be somewhat more (or less)
expensive.  I joked about resurrecting jzipfile (which I killed off
because it was something like 20% slower at decompressing entries than
Jar/ZipFile) but it might be worth considering having our own JAR
extractor at some point with a view towards concurrency gains.  If we go
this route, we could go even further and create an optimized module
format, which is an idea I think we've looked at a little bit in the
past; there are a few avenues of exploration here which could be
interesting.

At some point we also need to see how jaotc might improve things.  It
probably won't improve class loading time directly, but it might improve
the processes by which class loading is done because all the one-off
bits would be precompiled.  Also it's worth exploring whether the jimage
format has contention issues like this.

On 05/14/2017 06:36 PM, Stuart Douglas wrote:

> When JIRA was being screwy on Friday I used the time to investigate an
> idea I have had for a while about improving our boot time performance.
> According to Yourkit the majority of our time is spent in class loading.
> It seems very unlikely that we will be able to reduce the number of
> classes we load on boot (or at the very least it would be a massive
> amount of work) so I investigated a different approach.
>
> I modified ModuleClassLoader to spit out the name and module of every
> class that is loaded at boot time, and stored this in a properties file.
> I then created a simple Service that starts immediately that uses two
> threads to eagerly load every class on this list (I used two threads
> because that seemed to work well on my laptop, I think
> Runtime.availableProcessors()/4 is probably the best amount, but that
> assumption would need to be tested on different hardware).
>
> The idea behind this is that we know the classes will be used at some
> point, and we generally do not fully utilise all CPU's during boot, so
> we can use the unused CPU to pre load these classes so they are ready
> when they are actually required.
>
> Using this approach I saw the boot time for standalone.xml drop from
> ~2.9s to ~2.3s on my laptop. The (super hacky) code I used to perform
> this test is at
> https://github.com/wildfly/wildfly-core/compare/master...stuartwdouglas:boot-performance-hack
>
> I think these initial results are encouraging, and it is a big enough
> gain that I think it is worth investigating further.
>
> Firstly it would be great if I could get others to try it out and see if
> they see similar gains to boot time, it may be that the gain is very
> system dependent.
>
> Secondly if we do decide to do this there are two approach that we can
> use that I can see:
>
> 1) A hard coded list of class names that we generate before a release
> (basically what the hack already does), this is simplest, but does add a
> little bit of additional work to the release process (although if it is
> missed it would be no big deal, as ClassNotFoundException's would be
> suppressed, and if a few classes are missing the performance impact is
> negligible as long as the majority of the list is correct).
>
> 2) Generate the list dynamically on first boot, and store it in the temp
> directory. This would require the addition of a hook into JBoss Modules
> to generate the list, but is the approach I would prefer (as first boot
> is always a bit slower anyway).
>
> Thoughts?
>
> Stuart
>
>
> _______________________________________________
> wildfly-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/wildfly-dev
>




--
- DML
_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev
Reply | Threaded
Open this post in threaded view
|

Re: Speeding up WildFly boot time

Brian Stansberry
In reply to this post by Tomaž Cerar-2
Thanks. That’s interesting.

> On May 15, 2017, at 10:04 AM, Tomaž Cerar <[hidden email]> wrote:
>
>
> On Mon, May 15, 2017 at 4:13 PM, Brian Stansberry <[hidden email]> wrote:
> So why does adding two more make such a big difference?
>
> Main reason is that this two threads load most of later required classes which can later be quickly loaded from multiple parallel threads.
>
> Currently concurrency causes that 8 -16 threads (on 4-8 logical core systems) try to load same classes at same time.
> this leads to lots of contention as result. "preloading" some of this classes reduces contention.
>
> Looking at the list in the current "hack impl" there are lots of classes that don't need to be there, stuff like subsystem parsers which are only loaded once in any case.
>
> Main pressure is on classes from jboss-modules, controller, server & xml parsers modules, all others are not as problematic.
> This is also reason why lots of contention is happening on JDK classes as well as those are shared between all parts of server code.
>
>
>
>

--
Brian Stansberry
Manager, Senior Principal Software Engineer
JBoss by Red Hat




_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev
Reply | Threaded
Open this post in threaded view
|

Re: Speeding up WildFly boot time

Stuart Douglas
In reply to this post by Brian Stansberry


On Tue, May 16, 2017 at 12:13 AM, Brian Stansberry <[hidden email]> wrote:
Definitely worth investigating. I’d like to have a real good understanding of why it has the benefits it has, so we can see if this is the best way to get them or if something else is better.

I am pretty sure it is contention related. I modified my hack to load all classes from the same module at once (so once the first class from a module in that properties file is reached, it loads all others from the same module), and this gave another small but significant speedup (so the total gain is ~2.0-2.1s down from ~2.9s).

Looking at the results of monitor profiling in Yourkit it looks like the reason is reduced contention. There is 50% less thread wait time on ModuleLoader$FutureModule, contention on JarFileResourceLoader is no more. I think the reason is that we have a lot of threads active at boot and this results in a lot of contention in module/class loading.

Stuart


 

This kicks in just before the ModelController starts and begins parsing the config. The config parsing quickly gets into parallel work; as soon as the extension elements are reached the extension modules are loaded concurrently. Then once the parsing is done each subsystem is installed concurrently, so lots of threads doing concurrent classloading.

So why does adding two more make such a big difference?

Is it that they gets lots of work done in that time when the regular boot thread is not doing concurrent work, i.e. the parsing and the non-parallel bits of operation execution?

Is it that these threads are just chugging along doing classloading efficiently while the parallel threads are running along inefficiently getting scheduled and unscheduled?

The latter doesn’t make sense to me as there’s no reason why these threads would be any more efficient than the others.

- Brian

> On May 14, 2017, at 6:36 PM, Stuart Douglas <[hidden email]> wrote:
>
> When JIRA was being screwy on Friday I used the time to investigate an idea I have had for a while about improving our boot time performance. According to Yourkit the majority of our time is spent in class loading. It seems very unlikely that we will be able to reduce the number of classes we load on boot (or at the very least it would be a massive amount of work) so I investigated a different approach.
>
> I modified ModuleClassLoader to spit out the name and module of every class that is loaded at boot time, and stored this in a properties file. I then created a simple Service that starts immediately that uses two threads to eagerly load every class on this list (I used two threads because that seemed to work well on my laptop, I think Runtime.availableProcessors()/4 is probably the best amount, but that assumption would need to be tested on different hardware).
>
> The idea behind this is that we know the classes will be used at some point, and we generally do not fully utilise all CPU's during boot, so we can use the unused CPU to pre load these classes so they are ready when they are actually required.
>
> Using this approach I saw the boot time for standalone.xml drop from ~2.9s to ~2.3s on my laptop. The (super hacky) code I used to perform this test is at https://github.com/wildfly/wildfly-core/compare/master...stuartwdouglas:boot-performance-hack
>
> I think these initial results are encouraging, and it is a big enough gain that I think it is worth investigating further.
>
> Firstly it would be great if I could get others to try it out and see if they see similar gains to boot time, it may be that the gain is very system dependent.
>
> Secondly if we do decide to do this there are two approach that we can use that I can see:
>
> 1) A hard coded list of class names that we generate before a release (basically what the hack already does), this is simplest, but does add a little bit of additional work to the release process (although if it is missed it would be no big deal, as ClassNotFoundException's would be suppressed, and if a few classes are missing the performance impact is negligible as long as the majority of the list is correct).
>
> 2) Generate the list dynamically on first boot, and store it in the temp directory. This would require the addition of a hook into JBoss Modules to generate the list, but is the approach I would prefer (as first boot is always a bit slower anyway).
>
> Thoughts?
>
> Stuart
> _______________________________________________
> wildfly-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/wildfly-dev

--
Brian Stansberry
Manager, Senior Principal Software Engineer
JBoss by Red Hat





_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev
Reply | Threaded
Open this post in threaded view
|

Re: Speeding up WildFly boot time

Brian Stansberry
From the time I did parallel boot I’ve always wondered if the level of concurrency was valid, but I never got around to doing any experimentation.

It’s quite naive; a task per extension module load and then one per subystem. I’ve wanted to look into instead dividing the work into X larger tasks with X derived from the number of cores.

But for your fix to be helping things so much it must be loading a lot of these classes during the single-threaded parts of the boot, so I don’t see how my changing it to have fewer tasks would compete with that. It may be beneficial regardless though, e.g. by not spinning up more threads that can be efficiently used.

> On May 15, 2017, at 4:52 PM, Stuart Douglas <[hidden email]> wrote:
>
>
>
> On Tue, May 16, 2017 at 12:13 AM, Brian Stansberry <[hidden email]> wrote:
> Definitely worth investigating. I’d like to have a real good understanding of why it has the benefits it has, so we can see if this is the best way to get them or if something else is better.
>
> I am pretty sure it is contention related. I modified my hack to load all classes from the same module at once (so once the first class from a module in that properties file is reached, it loads all others from the same module), and this gave another small but significant speedup (so the total gain is ~2.0-2.1s down from ~2.9s).
>
> Looking at the results of monitor profiling in Yourkit it looks like the reason is reduced contention. There is 50% less thread wait time on ModuleLoader$FutureModule, contention on JarFileResourceLoader is no more. I think the reason is that we have a lot of threads active at boot and this results in a lot of contention in module/class loading.
>
> Stuart
>
>
>  
>
> This kicks in just before the ModelController starts and begins parsing the config. The config parsing quickly gets into parallel work; as soon as the extension elements are reached the extension modules are loaded concurrently. Then once the parsing is done each subsystem is installed concurrently, so lots of threads doing concurrent classloading.
>
> So why does adding two more make such a big difference?
>
> Is it that they gets lots of work done in that time when the regular boot thread is not doing concurrent work, i.e. the parsing and the non-parallel bits of operation execution?
>
> Is it that these threads are just chugging along doing classloading efficiently while the parallel threads are running along inefficiently getting scheduled and unscheduled?
>
> The latter doesn’t make sense to me as there’s no reason why these threads would be any more efficient than the others.
>
> - Brian
>
> > On May 14, 2017, at 6:36 PM, Stuart Douglas <[hidden email]> wrote:
> >
> > When JIRA was being screwy on Friday I used the time to investigate an idea I have had for a while about improving our boot time performance. According to Yourkit the majority of our time is spent in class loading. It seems very unlikely that we will be able to reduce the number of classes we load on boot (or at the very least it would be a massive amount of work) so I investigated a different approach.
> >
> > I modified ModuleClassLoader to spit out the name and module of every class that is loaded at boot time, and stored this in a properties file. I then created a simple Service that starts immediately that uses two threads to eagerly load every class on this list (I used two threads because that seemed to work well on my laptop, I think Runtime.availableProcessors()/4 is probably the best amount, but that assumption would need to be tested on different hardware).
> >
> > The idea behind this is that we know the classes will be used at some point, and we generally do not fully utilise all CPU's during boot, so we can use the unused CPU to pre load these classes so they are ready when they are actually required.
> >
> > Using this approach I saw the boot time for standalone.xml drop from ~2.9s to ~2.3s on my laptop. The (super hacky) code I used to perform this test is at https://github.com/wildfly/wildfly-core/compare/master...stuartwdouglas:boot-performance-hack
> >
> > I think these initial results are encouraging, and it is a big enough gain that I think it is worth investigating further.
> >
> > Firstly it would be great if I could get others to try it out and see if they see similar gains to boot time, it may be that the gain is very system dependent.
> >
> > Secondly if we do decide to do this there are two approach that we can use that I can see:
> >
> > 1) A hard coded list of class names that we generate before a release (basically what the hack already does), this is simplest, but does add a little bit of additional work to the release process (although if it is missed it would be no big deal, as ClassNotFoundException's would be suppressed, and if a few classes are missing the performance impact is negligible as long as the majority of the list is correct).
> >
> > 2) Generate the list dynamically on first boot, and store it in the temp directory. This would require the addition of a hook into JBoss Modules to generate the list, but is the approach I would prefer (as first boot is always a bit slower anyway).
> >
> > Thoughts?
> >
> > Stuart
> > _______________________________________________
> > wildfly-dev mailing list
> > [hidden email]
> > https://lists.jboss.org/mailman/listinfo/wildfly-dev
>
> --
> Brian Stansberry
> Manager, Senior Principal Software Engineer
> JBoss by Red Hat
>
>
>
>

--
Brian Stansberry
Manager, Senior Principal Software Engineer
JBoss by Red Hat




_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev
Reply | Threaded
Open this post in threaded view
|

Re: Speeding up WildFly boot time

Stuart Douglas
In reply to this post by David M. Lloyd


On Tue, May 16, 2017 at 1:34 AM, David M. Lloyd <[hidden email]> wrote:
I have a few thoughts that might be of interest.

Firstly, I'd be interested to see when you are logging the class name
being loaded.  If you are logging it in loadClass, you might not be
seeing the actual correct load order because that method is ultimately
recursive.  To get an accurate picture of what order that classes are
actually defined - and thus what order you can load them in order to
prevent contention on per-class locks within the CL - you should log
immediately _after_ defineClass completes for each class.

I set a breakpoint in loadClassLocal to print off the information.
 

Secondly, while debugging a resource iteration performance problem a
user was having with a large number of deployments, I discovered that
contention for the lock on JarFile and ZipFile was a primary cause.  The
workaround I employed was to keep a RAM-based List of the files in the
JAR, which can be iterated over without touching the lock.

When we're preloading classes, we're definitely going to see this same
kind of contention come up, because there's only one lock per JarFile
instance so you can only ever read one entry at a time, thus preventing
any kind of useful concurrency on a per-module basis.

I think this is why I see an even bigger gain when pre-loading classes one module at a time.
 

Exploding the files out of the JarFile could expose this contention and
therefore might be useful as a test - but it would also skew the results
a little because you have no decompression overhead, and creating the
separate file streams hypothetically might be somewhat more (or less)
expensive.  I joked about resurrecting jzipfile (which I killed off
because it was something like 20% slower at decompressing entries than
Jar/ZipFile) but it might be worth considering having our own JAR
extractor at some point with a view towards concurrency gains.  If we go
this route, we could go even further and create an optimized module
format, which is an idea I think we've looked at a little bit in the
past; there are a few avenues of exploration here which could be
interesting.

This could be worth investigating. 

Stuart
 

At some point we also need to see how jaotc might improve things.  It
probably won't improve class loading time directly, but it might improve
the processes by which class loading is done because all the one-off
bits would be precompiled.  Also it's worth exploring whether the jimage
format has contention issues like this.

On 05/14/2017 06:36 PM, Stuart Douglas wrote:
> When JIRA was being screwy on Friday I used the time to investigate an
> idea I have had for a while about improving our boot time performance.
> According to Yourkit the majority of our time is spent in class loading.
> It seems very unlikely that we will be able to reduce the number of
> classes we load on boot (or at the very least it would be a massive
> amount of work) so I investigated a different approach.
>
> I modified ModuleClassLoader to spit out the name and module of every
> class that is loaded at boot time, and stored this in a properties file.
> I then created a simple Service that starts immediately that uses two
> threads to eagerly load every class on this list (I used two threads
> because that seemed to work well on my laptop, I think
> Runtime.availableProcessors()/4 is probably the best amount, but that
> assumption would need to be tested on different hardware).
>
> The idea behind this is that we know the classes will be used at some
> point, and we generally do not fully utilise all CPU's during boot, so
> we can use the unused CPU to pre load these classes so they are ready
> when they are actually required.
>
> Using this approach I saw the boot time for standalone.xml drop from
> ~2.9s to ~2.3s on my laptop. The (super hacky) code I used to perform
> this test is at
> https://github.com/wildfly/wildfly-core/compare/master...stuartwdouglas:boot-performance-hack
>
> I think these initial results are encouraging, and it is a big enough
> gain that I think it is worth investigating further.
>
> Firstly it would be great if I could get others to try it out and see if
> they see similar gains to boot time, it may be that the gain is very
> system dependent.
>
> Secondly if we do decide to do this there are two approach that we can
> use that I can see:
>
> 1) A hard coded list of class names that we generate before a release
> (basically what the hack already does), this is simplest, but does add a
> little bit of additional work to the release process (although if it is
> missed it would be no big deal, as ClassNotFoundException's would be
> suppressed, and if a few classes are missing the performance impact is
> negligible as long as the majority of the list is correct).
>
> 2) Generate the list dynamically on first boot, and store it in the temp
> directory. This would require the addition of a hook into JBoss Modules
> to generate the list, but is the approach I would prefer (as first boot
> is always a bit slower anyway).
>
> Thoughts?
>
> Stuart
>
>
> _______________________________________________
> wildfly-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/wildfly-dev
>




--
- DML
_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev


_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev
Reply | Threaded
Open this post in threaded view
|

Re: Speeding up WildFly boot time

Heiko W.Rupp-2
In reply to this post by Rostislav Svoboda
On 16 May 2017, at 0:15, Stuart Douglas wrote:

>>> 2) Generate the list dynamically on first boot, and store it in the temp
>>
>> This looks like the most elegant thing to do. Question is how it will slow
>> down the initial boot. People care about first boot impression, some blog
>> writers do the mistake too.
>>
>
> It will not actually slow down the initial boot (at least not in a
> measurable way), but the first boot would not get the benefit of this
> optimisation.

A mixed mode could be interesting, where the list is
created by tooling (or pseudo-boot) and then written
down. As someone said on Docker/OS every boot
is first boot, so doing the pre-population then would
not help. But creating the list at image creation time
would dynamically create the list and make the
speedup available to all starts of containers from that
image.
_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev
Reply | Threaded
Open this post in threaded view
|

Re: Speeding up WildFly boot time

Jaikiran Pai-2
In reply to this post by Stuart Douglas
Not to undermine these efforts (in fact, this thread has actually brought up a couple of really interesting details), but one of the things I have always seen when we spent time trying to add relatively complex ways to squeeze some milli seconds out of the boot time, is that for the end users, most of the times it really didn’t matter in a noticeable way.I am not talking about the major improvements we have made from AS5/AS6 to the WildFly boot times today.

What I have experienced is that for end users, they are mostly interested in seeing their (usually large) deployments show noticeable improvements in deployment time, not necessarily from a cold boot of the server, but when the server is already up and they either want to deploy something new or re-deploy their application.
 
All in all, as a developer, I will be curiously following how these experiments go, but as an end user, I am not sure this will show up as something noticeable. Of course, the place where this would probably make a difference (even from an end user perspective) is something like maybe WildFly Swarm, but then again I haven’t been following that project to understand if these efforts will directly end up somehow in WildFly Swarm.

-Jaikiran

On 15-May-2017, at 5:06 AM, Stuart Douglas <[hidden email]> wrote:

When JIRA was being screwy on Friday I used the time to investigate an idea I have had for a while about improving our boot time performance. According to Yourkit the majority of our time is spent in class loading. It seems very unlikely that we will be able to reduce the number of classes we load on boot (or at the very least it would be a massive amount of work) so I investigated a different approach.

I modified ModuleClassLoader to spit out the name and module of every class that is loaded at boot time, and stored this in a properties file. I then created a simple Service that starts immediately that uses two threads to eagerly load every class on this list (I used two threads because that seemed to work well on my laptop, I think Runtime.availableProcessors()/4 is probably the best amount, but that assumption would need to be tested on different hardware).

The idea behind this is that we know the classes will be used at some point, and we generally do not fully utilise all CPU's during boot, so we can use the unused CPU to pre load these classes so they are ready when they are actually required.

Using this approach I saw the boot time for standalone.xml drop from ~2.9s to ~2.3s on my laptop. The (super hacky) code I used to perform this test is at https://github.com/wildfly/wildfly-core/compare/master...stuartwdouglas:boot-performance-hack

I think these initial results are encouraging, and it is a big enough gain that I think it is worth investigating further.

Firstly it would be great if I could get others to try it out and see if they see similar gains to boot time, it may be that the gain is very system dependent.

Secondly if we do decide to do this there are two approach that we can use that I can see:

1) A hard coded list of class names that we generate before a release (basically what the hack already does), this is simplest, but does add a little bit of additional work to the release process (although if it is missed it would be no big deal, as ClassNotFoundException's would be suppressed, and if a few classes are missing the performance impact is negligible as long as the majority of the list is correct).

2) Generate the list dynamically on first boot, and store it in the temp directory. This would require the addition of a hook into JBoss Modules to generate the list, but is the approach I would prefer (as first boot is always a bit slower anyway).

Thoughts?

Stuart
_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev


_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev
Reply | Threaded
Open this post in threaded view
|

Re: Speeding up WildFly boot time

David M. Lloyd
In reply to this post by David M. Lloyd
Off list discussion: the NIO.2 JAR file provider appears to have
substantially better performance (and no central locking).  This JIRA
[1] covers using this by default within JBoss Modules.

https://issues.jboss.org/browse/MODULES-285

On 05/15/2017 10:34 AM, David M. Lloyd wrote:

> I have a few thoughts that might be of interest.
>
> Firstly, I'd be interested to see when you are logging the class name
> being loaded.  If you are logging it in loadClass, you might not be
> seeing the actual correct load order because that method is ultimately
> recursive.  To get an accurate picture of what order that classes are
> actually defined - and thus what order you can load them in order to
> prevent contention on per-class locks within the CL - you should log
> immediately _after_ defineClass completes for each class.
>
> Secondly, while debugging a resource iteration performance problem a
> user was having with a large number of deployments, I discovered that
> contention for the lock on JarFile and ZipFile was a primary cause.  The
> workaround I employed was to keep a RAM-based List of the files in the
> JAR, which can be iterated over without touching the lock.
>
> When we're preloading classes, we're definitely going to see this same
> kind of contention come up, because there's only one lock per JarFile
> instance so you can only ever read one entry at a time, thus preventing
> any kind of useful concurrency on a per-module basis.
>
> Exploding the files out of the JarFile could expose this contention and
> therefore might be useful as a test - but it would also skew the results
> a little because you have no decompression overhead, and creating the
> separate file streams hypothetically might be somewhat more (or less)
> expensive.  I joked about resurrecting jzipfile (which I killed off
> because it was something like 20% slower at decompressing entries than
> Jar/ZipFile) but it might be worth considering having our own JAR
> extractor at some point with a view towards concurrency gains.  If we go
> this route, we could go even further and create an optimized module
> format, which is an idea I think we've looked at a little bit in the
> past; there are a few avenues of exploration here which could be
> interesting.
>
> At some point we also need to see how jaotc might improve things.  It
> probably won't improve class loading time directly, but it might improve
> the processes by which class loading is done because all the one-off
> bits would be precompiled.  Also it's worth exploring whether the jimage
> format has contention issues like this.
>
> On 05/14/2017 06:36 PM, Stuart Douglas wrote:
>> When JIRA was being screwy on Friday I used the time to investigate an
>> idea I have had for a while about improving our boot time performance.
>> According to Yourkit the majority of our time is spent in class loading.
>> It seems very unlikely that we will be able to reduce the number of
>> classes we load on boot (or at the very least it would be a massive
>> amount of work) so I investigated a different approach.
>>
>> I modified ModuleClassLoader to spit out the name and module of every
>> class that is loaded at boot time, and stored this in a properties file.
>> I then created a simple Service that starts immediately that uses two
>> threads to eagerly load every class on this list (I used two threads
>> because that seemed to work well on my laptop, I think
>> Runtime.availableProcessors()/4 is probably the best amount, but that
>> assumption would need to be tested on different hardware).
>>
>> The idea behind this is that we know the classes will be used at some
>> point, and we generally do not fully utilise all CPU's during boot, so
>> we can use the unused CPU to pre load these classes so they are ready
>> when they are actually required.
>>
>> Using this approach I saw the boot time for standalone.xml drop from
>> ~2.9s to ~2.3s on my laptop. The (super hacky) code I used to perform
>> this test is at
>> https://github.com/wildfly/wildfly-core/compare/master...stuartwdouglas:boot-performance-hack
>>
>> I think these initial results are encouraging, and it is a big enough
>> gain that I think it is worth investigating further.
>>
>> Firstly it would be great if I could get others to try it out and see if
>> they see similar gains to boot time, it may be that the gain is very
>> system dependent.
>>
>> Secondly if we do decide to do this there are two approach that we can
>> use that I can see:
>>
>> 1) A hard coded list of class names that we generate before a release
>> (basically what the hack already does), this is simplest, but does add a
>> little bit of additional work to the release process (although if it is
>> missed it would be no big deal, as ClassNotFoundException's would be
>> suppressed, and if a few classes are missing the performance impact is
>> negligible as long as the majority of the list is correct).
>>
>> 2) Generate the list dynamically on first boot, and store it in the temp
>> directory. This would require the addition of a hook into JBoss Modules
>> to generate the list, but is the approach I would prefer (as first boot
>> is always a bit slower anyway).
>>
>> Thoughts?
>>
>> Stuart
>>
>>
>> _______________________________________________
>> wildfly-dev mailing list
>> [hidden email]
>> https://lists.jboss.org/mailman/listinfo/wildfly-dev
>>
>
>
>
>


--
- DML
_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev
Reply | Threaded
Open this post in threaded view
|

Re: Speeding up WildFly boot time

Bob McWhirter
In reply to this post by Rostislav Svoboda
From a swarm perspective I'd like something that benefits first boot because we have no place to store stuff for second boot. 

It's all first boots!

Bob

On Mon, May 15, 2017 at 8:29 AM Rostislav Svoboda <[hidden email]> wrote:
Hi.

I can confirm I see improvements in boot time with your changes.
My HW is Lenovo T440s with Fedora 25, Intel(R) Core(TM) i7-4600U CPU (Base Frequency 2.10 GHz, Max Turbo 3.30 GHz)

I executed 50 iterations of start - stop sequence [1], before execution 5x start - stop for "warmup"

With your changes
       Min: 3116        Max: 3761       Average: 3247.640000

Without:
       Min: 3442        Max: 4081       Average: 3580.840000


> 1) A hard coded list of class names that we generate before a release

This will improve first boot impression, little bit harder for maintaining the list for the final build.

Property files could be located inside properties directory of dedicated module (<resource-root path="properties"/>). Properties directory could contain property files for delivered profiles.

Layered products or customer modifications could deliver own property file.
 e.g. predefined property file for standalone-openshift.xml in EAP image in OpenShift environment, I think they boot the server just once and throw away the whole docker image when something changes.


> 2) Generate the list dynamically on first boot, and store it in the temp

This looks like the most elegant thing to do. Question is how it will slow down the initial boot. People care about first boot impression, some blog writers do the mistake too.
This would also block boot time improvements for use-cases when you start the server just once - e.g. Docker, OpenShift.

Also the logic should take into account which profile is loaded - e.g standalone.xml vs. standalone-full-ha.xml

Rostislav

[1]
rm wildfly-11.0.0.Beta1-SNAPSHOT-preload/standalone/log/server.log
rm wildfly-11.0.0.Beta1-SNAPSHOT/standalone/log/server.log

for i in {1..50}; do
  echo $i
  wildfly-11.0.0.Beta1-SNAPSHOT-preload/bin/standalone.sh 1>/dev/null 2>&1 &
  sleep 8
  wildfly-11.0.0.Beta1-SNAPSHOT-preload/bin/jboss-cli.sh -c :shutdown 1>/dev/null 2>&1
done
grep WFLYSRV0025 wildfly-11.0.0.Beta1-SNAPSHOT-preload/standalone/log/server.log | sed "s/.*\(....\)ms.*/\1/g"  | awk 'NR == 1 { max=$1; min=$1; sum=0 }
   { if ($1>max) max=$1; if ($1<min) min=$1; sum+=$1;}
   END {printf "Min: %d\tMax: %d\tAverage: %f\n", min, max, sum/NR}'


for i in {1..50}; do
  echo $i
  wildfly-11.0.0.Beta1-SNAPSHOT/bin/standalone.sh 1>/dev/null 2>&1 &
  sleep 8
  wildfly-11.0.0.Beta1-SNAPSHOT/bin/jboss-cli.sh -c :shutdown 1>/dev/null 2>&1
done
grep WFLYSRV0025 wildfly-11.0.0.Beta1-SNAPSHOT/standalone/log/server.log | sed "s/.*\(....\)ms.*/\1/g"  | awk 'NR == 1 { max=$1; min=$1; sum=0 }
   { if ($1>max) max=$1; if ($1<min) min=$1; sum+=$1;}
   END {printf "Min: %d\tMax: %d\tAverage: %f\n", min, max, sum/NR}'


----- Original Message -----
> When JIRA was being screwy on Friday I used the time to investigate an idea I
> have had for a while about improving our boot time performance. According to
> Yourkit the majority of our time is spent in class loading. It seems very
> unlikely that we will be able to reduce the number of classes we load on
> boot (or at the very least it would be a massive amount of work) so I
> investigated a different approach.
>
> I modified ModuleClassLoader to spit out the name and module of every class
> that is loaded at boot time, and stored this in a properties file. I then
> created a simple Service that starts immediately that uses two threads to
> eagerly load every class on this list (I used two threads because that
> seemed to work well on my laptop, I think Runtime.availableProcessors()/4 is
> probably the best amount, but that assumption would need to be tested on
> different hardware).
>
> The idea behind this is that we know the classes will be used at some point,
> and we generally do not fully utilise all CPU's during boot, so we can use
> the unused CPU to pre load these classes so they are ready when they are
> actually required.
>
> Using this approach I saw the boot time for standalone.xml drop from ~2.9s to
> ~2.3s on my laptop. The (super hacky) code I used to perform this test is at
> https://github.com/wildfly/wildfly-core/compare/master...stuartwdouglas:boot-performance-hack
>
> I think these initial results are encouraging, and it is a big enough gain
> that I think it is worth investigating further.
>
> Firstly it would be great if I could get others to try it out and see if they
> see similar gains to boot time, it may be that the gain is very system
> dependent.
>
> Secondly if we do decide to do this there are two approach that we can use
> that I can see:
>
> 1) A hard coded list of class names that we generate before a release
> (basically what the hack already does), this is simplest, but does add a
> little bit of additional work to the release process (although if it is
> missed it would be no big deal, as ClassNotFoundException's would be
> suppressed, and if a few classes are missing the performance impact is
> negligible as long as the majority of the list is correct).
>
> 2) Generate the list dynamically on first boot, and store it in the temp
> directory. This would require the addition of a hook into JBoss Modules to
> generate the list, but is the approach I would prefer (as first boot is
> always a bit slower anyway).
>
> Thoughts?
>
> Stuart
>
> _______________________________________________
> wildfly-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/wildfly-dev
_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev

_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev
Reply | Threaded
Open this post in threaded view
|

Re: Speeding up WildFly boot time

Brian Stansberry
In reply to this post by Tomaž Cerar-2

> On May 15, 2017, at 10:04 AM, Tomaž Cerar <[hidden email]> wrote:
>
>
> On Mon, May 15, 2017 at 4:13 PM, Brian Stansberry <[hidden email]> wrote:
> So why does adding two more make such a big difference?
>
> Main reason is that this two threads load most of later required classes which can later be quickly loaded from multiple parallel threads.
>
> Currently concurrency causes that 8 -16 threads (on 4-8 logical core systems) try to load same classes at same time.
> this leads to lots of contention as result. "preloading" some of this classes reduces contention.
>
> Looking at the list in the current "hack impl" there are lots of classes that don't need to be there, stuff like subsystem parsers which are only loaded once in any case.
>
> Main pressure is on classes from jboss-modules, controller, server & xml parsers modules, all others are not as problematic.
> This is also reason why lots of contention is happening on JDK classes as well as those are shared between all parts of server code.
>

Stuart/Tomaz —

Please ignore this for now if your thinking has moved on to other approaches, e.g. better concurrency in classloading. :)

Otherwise, are there any numbers on this last point Tomaz made?

I ask because people are asking for a static list since a dynamic list is of no benefit to cloud use cases.

A static list is painful to administer though, and if not administered well can result in loading unneeded classes and wasting memory.

But, a static list limited to modules that are part of the WildFly Core kernel is not particularly hard to administer. So if we can get the bulk of the gains with the minimum of the pain, we might consider that.

--
Brian Stansberry
Manager, Senior Principal Software Engineer
JBoss by Red Hat




_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev
Reply | Threaded
Open this post in threaded view
|

Re: Speeding up WildFly boot time

jtgreene
Administrator

> On May 17, 2017, at 1:42 PM, Brian Stansberry <[hidden email]> wrote:
>
>
>> On May 15, 2017, at 10:04 AM, Tomaž Cerar <[hidden email]> wrote:
>>
>>
>> On Mon, May 15, 2017 at 4:13 PM, Brian Stansberry <[hidden email]> wrote:
>> So why does adding two more make such a big difference?
>>
>> Main reason is that this two threads load most of later required classes which can later be quickly loaded from multiple parallel threads.
>>
>> Currently concurrency causes that 8 -16 threads (on 4-8 logical core systems) try to load same classes at same time.
>> this leads to lots of contention as result. "preloading" some of this classes reduces contention.
>>
>> Looking at the list in the current "hack impl" there are lots of classes that don't need to be there, stuff like subsystem parsers which are only loaded once in any case.
>>
>> Main pressure is on classes from jboss-modules, controller, server & xml parsers modules, all others are not as problematic.
>> This is also reason why lots of contention is happening on JDK classes as well as those are shared between all parts of server code.
>>
>
> Stuart/Tomaz —
>
> Please ignore this for now if your thinking has moved on to other approaches, e.g. better concurrency in classloading. :)
>
> Otherwise, are there any numbers on this last point Tomaz made?
>
> I ask because people are asking for a static list since a dynamic list is of no benefit to cloud use cases.
>
> A static list is painful to administer though, and if not administered well can result in loading unneeded classes and wasting memory.
>
> But, a static list limited to modules that are part of the WildFly Core kernel is not particularly hard to administer. So if we can get the bulk of the gains with the minimum of the pain, we might consider that.
>

We can also just have a dynamic offline list generation, which is ran as a build task.

--
Jason T. Greene
WildFly Lead / JBoss EAP Platform Architect
JBoss, a division of Red Hat


_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev
Reply | Threaded
Open this post in threaded view
|

Re: Speeding up WildFly boot time

Brian Stansberry

> On May 17, 2017, at 3:29 PM, Jason Greene <[hidden email]> wrote:
>
>
>> On May 17, 2017, at 1:42 PM, Brian Stansberry <[hidden email]> wrote:
>>
>>
>>> On May 15, 2017, at 10:04 AM, Tomaž Cerar <[hidden email]> wrote:
>>>
>>>
>>> On Mon, May 15, 2017 at 4:13 PM, Brian Stansberry <[hidden email]> wrote:
>>> So why does adding two more make such a big difference?
>>>
>>> Main reason is that this two threads load most of later required classes which can later be quickly loaded from multiple parallel threads.
>>>
>>> Currently concurrency causes that 8 -16 threads (on 4-8 logical core systems) try to load same classes at same time.
>>> this leads to lots of contention as result. "preloading" some of this classes reduces contention.
>>>
>>> Looking at the list in the current "hack impl" there are lots of classes that don't need to be there, stuff like subsystem parsers which are only loaded once in any case.
>>>
>>> Main pressure is on classes from jboss-modules, controller, server & xml parsers modules, all others are not as problematic.
>>> This is also reason why lots of contention is happening on JDK classes as well as those are shared between all parts of server code.
>>>
>>
>> Stuart/Tomaz —
>>
>> Please ignore this for now if your thinking has moved on to other approaches, e.g. better concurrency in classloading. :)
>>
>> Otherwise, are there any numbers on this last point Tomaz made?
>>
>> I ask because people are asking for a static list since a dynamic list is of no benefit to cloud use cases.
>>
>> A static list is painful to administer though, and if not administered well can result in loading unneeded classes and wasting memory.
>>
>> But, a static list limited to modules that are part of the WildFly Core kernel is not particularly hard to administer. So if we can get the bulk of the gains with the minimum of the pain, we might consider that.
>>
>
> We can also just have a dynamic offline list generation, which is ran as a build task.

Yes, that’s my assumption. When I say “static” I mean static on a given installation.

If it is limited to the kernel (including relevant JDK bits), then there are no issues with ensuring different feature pack maintainers are doing this, no need to combine lists from different parts of the build, no worries about ensuring only those bits relevant to what the user is actually running are loaded, etc.  Those things are the “painful to administer part”. They might very well be worth it but data should demonstrate that.

--
Brian Stansberry
Manager, Senior Principal Software Engineer
JBoss by Red Hat




_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev
12