Pooling EJB Session Beans per default

classic Classic list List threaded Threaded
44 messages Options
123
Reply | Threaded
Open this post in threaded view
|

Re: Pooling EJB Session Beans per default

jtgreene
Administrator

On Aug 4, 2014, at 7:44 PM, James Livingston <[hidden email]> wrote:

> On Mon, 2014-08-04 at 20:22 -0400, Bill Burke wrote:
>> I always liked the ThreadLocal pool.   No synchronization, little to no
>> allocations.
>
> It also can cause massive memory leaks if invoked from threads which
> aren't re-used, like timer threads, and precautions aren't taken. AS/EAP
> 5 suffered from that problem with the default ThreadLocalPool.

Right this is the reason we didn’t preserve it, as it leads to both instance leaks and class loader leaks. However a thread local based pool with a controlled life-cycle attached to deployment is something we have always wanted to do, it’s just been lower priority..

>
> The major problem I've seen in support cases related to StrictMaxPool is
> the very arbitrary default size. I believe it used to be 30, and almost
> no-one knew that they might need to tune it to fit their applications'
> usage patterns and workload. Having a rising wait-time metric for the
> pool indicates that it may be too small, but I believe the statistics
> are disabled by default for performance reasons.

Yes that was a bad default, although the big problem is that any value we pick is a bad default. Ideally you want an auto-tuned construct which is tied into our tuning with worker pools (e.g. like the thread local approach above). Absent that its better to just take the small hit of allocating a few temporary objects per request, and let the user setup the pool config that best fits their app.

--
Jason T. Greene
WildFly Lead / JBoss EAP Platform Architect
JBoss, a division of Red Hat


_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev
Reply | Threaded
Open this post in threaded view
|

Re: Pooling EJB Session Beans per default

jtgreene
Administrator
In reply to this post by Stuart Douglas
I think a thread local solution would be superior. Like strict max,  an unbounded pool has a lot of potential for misconfiguration.
 
On Aug 4, 2014, at 8:51 PM, Stuart Douglas <[hidden email]> wrote:

> I have often thought that a possible solution would be an unbounded pool
> that always keeps n instances around, but just creates new instances
> instead of blocking if the pool is exhausted. In the majority of cases
> new instance creation will be way more performant than blocking.
>
> Stuart
>
> James Livingston wrote:
>> On Mon, 2014-08-04 at 20:22 -0400, Bill Burke wrote:
>>> I always liked the ThreadLocal pool.   No synchronization, little to no
>>> allocations.
>>
>> It also can cause massive memory leaks if invoked from threads which
>> aren't re-used, like timer threads, and precautions aren't taken. AS/EAP
>> 5 suffered from that problem with the default ThreadLocalPool.
>>
>> The major problem I've seen in support cases related to StrictMaxPool is
>> the very arbitrary default size. I believe it used to be 30, and almost
>> no-one knew that they might need to tune it to fit their applications'
>> usage patterns and workload. Having a rising wait-time metric for the
>> pool indicates that it may be too small, but I believe the statistics
>> are disabled by default for performance reasons.
>>
>>
>> InfinitePool obviously doesn't have that problem, but it would have been
>> nicer if it had an idle period with expiry, like the JCA pools.
>>
> _______________________________________________
> wildfly-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/wildfly-dev

--
Jason T. Greene
WildFly Lead / JBoss EAP Platform Architect
JBoss, a division of Red Hat


_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev
Reply | Threaded
Open this post in threaded view
|

Re: Pooling EJB Session Beans per default

Bill Burke
In reply to this post by jtgreene


On 8/5/2014 10:51 AM, Jason Greene wrote:

>
> On Aug 5, 2014, at 9:32 AM, Bill Burke <[hidden email]> wrote:
>
>>
>>
>> On 8/5/2014 10:15 AM, Jason Greene wrote:
>>>> Those silly benchmarks, are indeed silly.  Any workload that doesn't actually do anything with the requests is not very helpful.  You can be really fast on them, and be really slow on a workload that actually does something.  We have already seen this in one case recently.
>>>
>>> It’s all about precision, and limiting variables. I know its fun to criticize micro benchmarks (obviously micro is indeed relative because a great deal of processing goes on even when you have EJB and HTTP requests that don’t do much), however micro benchmarks, provided they are measured correctly, are quick to run and make it easy to tell what you have to fix. Large gigantic apps on the other hand are difficult to analyze, and do not necessarily represent activity that a differently architected app would show. To use an example, if you have pattern matching code, and you want it to be fast, it’s much easier (and accurate) to benchmark the pattern matching code then it is to benchmaark a full EE app that happens to use pattern matching in a few places.
>>>
>>> Arguing against targeted benchmarks is like arguing that you should never write a test case, because only real production load will show problems…. To some extent thats true, but that doesn’t mean that the problems you catch in simulations won’t affect production users.
>>>
>>
>> I'm more worried about focus rather than the actual benchmarks considering how resource strapped every project seems to be.  I'd like to see a discussion started on what are the most important and *ripe* areas for optimization and then have the performance team validate and make recommendations on any priority list that this discussion generates.
>
> The pooling concerns (aside from the occasional expensive post constructs) actually came from a large app/benchmark that the perf team was testing, and they were seeing an optimized strict max pool outperform no pooling at all, and it wasn’t post construct. Their theory is/was GC pressure, because this benchmark/app spends a lot of time in GC, and they see higher GC activity with no pooling. It’s possible it could be an indirect difference though, like the fact that strict max pool acts as a throttle might prevent the system from degrading as a result of no longer being able to keep up with the benchmark load.
>

Its a horrible theory. :)  How many EJB instances of a give type are
created per request?  Generally only 1.  1 instance of one object of one
type!  My $5 bet is that if you went into EJB code and started counting
how many object allocations were made per request, you'd lose count very
quickly.   Better yet, run a single remote EJB request through a perf
tool and let it count the number of allocations for you.  It will be
greater than 1.  :)

Maybe the StrictMaxPool has an effect on performance because it creates
a global synchronization bottleneck.  Throughput is less and you end up
having less concurrent per-request objects being allocated and GC'd.


> Another possibility to look into is that I see we do:
>     interceptorContext.setContextData(new HashMap<String, Object>());
>
> *AND*
>
>      private Map<Object, Object> instanceData = new HashMap<Object, Object>();
>
>
> Aside from the unnecessary overhead, since JDK7, HashMap construction uses the murmur algorithm which requires using a random number generation under global locking. There were plans to optimize this, but it might not be in JDK7 yet. In a few places we use alternative map implementations to work around the issue.
>



IMO, it is more likely that most projects haven't gone through the level
of refactorings that performance-focused projects like Undertow have
gone through.  I'll put another $5 bet down that only the Wildfly family
of projects under you and DML have gone through any real performance
analysis and optimizations.  Just think of all the crappiness that is
happening in the security layer alone!

Also, any optimizations projects have done are probably focused on speed
and throughput which generally is at the expense of memory.

--
Bill Burke
JBoss, a division of Red Hat
http://bill.burkecentral.com
_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev
Reply | Threaded
Open this post in threaded view
|

Re: Pooling EJB Session Beans per default

arjan.tijms
In reply to this post by Bill Burke
On Tuesday, August 5, 2014, Bill Burke <[hidden email]> wrote:
Honestly though, I think this talk of EJB pooling is ridiculous.
Component layers like CDI and JAX-RS don't have pooled architectures for
their per-request objects. 

True, but the problem with EJB is that it has always been implied that stateless beans are pooled. And indeed, many servers have done so.

As a result, various developers have implemented their beans with the assumption that there would be pooling. That code is now seeing (performance) issues.

I think it's reasonable to assume that in many cases pooling is unnecessary, but that occasionally it's needed. One solution to this could be an explicit @Pooled annotation for general usage with (CDI) managed beans.

Kind regards,
Arjan Tijms
 
 Also, in an average application, there are
multiple orders of magnitude more non component classes that are being
instantiated per request.  Just think of all the Strings created by a
simple HTTP request.

You want a better focus?  How about JSON/XML marshalling?  Which could
make things much easier to maintain then the hand-coded parsers that
Wildfly uses to parse config.  And much faster and less memory at
runtime for SOAP and JAX-RS request that currently rely on java reflection.

You could go research perfect hashing algorithms for URL matching with
servlets and JAX-RS.

You could go do some perf analysis of all of our frameworks to make
memory reduction and speed recommendations or even Pull Requests.

You could visit each major project and make sure our automated builds
have and can run automated stress tests and are measured against
previous versions.

Or you could just focus on these silly benchmarks that test no-op HTTP
and EJB requests.


--
Bill Burke
JBoss, a division of Red Hat
http://bill.burkecentral.com
_______________________________________________
wildfly-dev mailing list
<a href="javascript:;" onclick="_e(event, &#39;cvml&#39;, &#39;wildfly-dev@lists.jboss.org&#39;)">wildfly-dev@...
https://lists.jboss.org/mailman/listinfo/wildfly-dev

_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev
Reply | Threaded
Open this post in threaded view
|

Re: Pooling EJB Session Beans per default

Andrig Miller
In reply to this post by jtgreene


----- Original Message -----

> From: "Jason Greene" <[hidden email]>
> To: "Andrig Miller" <[hidden email]>
> Cc: "Bill Burke" <[hidden email]>, [hidden email]
> Sent: Tuesday, August 5, 2014 8:15:47 AM
> Subject: Re: [wildfly-dev] Pooling EJB Session Beans per default
>
>
> On Aug 5, 2014, at 8:43 AM, Andrig Miller <[hidden email]>
> wrote:
>
> >
> >
> > ----- Original Message -----
> >> From: "Bill Burke" <[hidden email]>
> >> To: "James Livingston" <[hidden email]>
> >> Cc: [hidden email]
> >> Sent: Tuesday, August 5, 2014 7:32:04 AM
> >> Subject: Re: [wildfly-dev] Pooling EJB Session Beans per default
> >>
> >>
> >>
> >> On 8/4/2014 8:44 PM, James Livingston wrote:
> >>> On Mon, 2014-08-04 at 20:22 -0400, Bill Burke wrote:
> >>>> I always liked the ThreadLocal pool.   No synchronization,
> >>>> little
> >>>> to no
> >>>> allocations.
> >>>
> >>> It also can cause massive memory leaks if invoked from threads
> >>> which
> >>> aren't re-used, like timer threads, and precautions aren't taken.
> >>> AS/EAP
> >>> 5 suffered from that problem with the default ThreadLocalPool.
> >>>
> >>
> >> Which is also something that could be mitigated by a common thread
> >> facility.  Something AS/EAP 5 didn't have.
> >>
> >> Honestly though, I think this talk of EJB pooling is ridiculous.
> >> Component layers like CDI and JAX-RS don't have pooled
> >> architectures
> >> for
> >> their per-request objects.  Also, in an average application, there
> >> are
> >> multiple orders of magnitude more non component classes that are
> >> being
> >> instantiated per request.  Just think of all the Strings created
> >> by a
> >> simple HTTP request.
> >>
> >> You want a better focus?  How about JSON/XML marshalling?  Which
> >> could
> >> make things much easier to maintain then the hand-coded parsers
> >> that
> >> Wildfly uses to parse config.  And much faster and less memory at
> >> runtime for SOAP and JAX-RS request that currently rely on java
> >> reflection.
> >>
> >> You could go research perfect hashing algorithms for URL matching
> >> with
> >> servlets and JAX-RS.
> >>
> >> You could go do some perf analysis of all of our frameworks to
> >> make
> >> memory reduction and speed recommendations or even Pull Requests.
> >>
> >> You could visit each major project and make sure our automated
> >> builds
> >> have and can run automated stress tests and are measured against
> >> previous versions.
> >>
> >> Or you could just focus on these silly benchmarks that test no-op
> >> HTTP
> >> and EJB requests.
> >>
> >
> > Those silly benchmarks, are indeed silly.  Any workload that
> > doesn't actually do anything with the requests is not very
> > helpful.  You can be really fast on them, and be really slow on a
> > workload that actually does something.  We have already seen this
> > in one case recently.
>
> It’s all about precision, and limiting variables. I know its fun to
> criticize micro benchmarks (obviously micro is indeed relative
> because a great deal of processing goes on even when you have EJB
> and HTTP requests that don’t do much), however micro benchmarks,
> provided they are measured correctly, are quick to run and make it
> easy to tell what you have to fix. Large gigantic apps on the other
> hand are difficult to analyze, and do not necessarily represent
> activity that a differently architected app would show. To use an
> example, if you have pattern matching code, and you want it to be
> fast, it’s much easier (and accurate) to benchmark the pattern
> matching code then it is to benchmaark a full EE app that happens to
> use pattern matching in a few places.
>

I know micro benchmarks can be useful, and the performance has actually written a couple ourselves.  They have to be very carefully designed, so that the results transfer to larger complex workloads.

> Arguing against targeted benchmarks is like arguing that you should
> never write a test case, because only real production load will show
> problems…. To some extent thats true, but that doesn’t mean that the
> problems you catch in simulations won’t affect production users.
>

I don't argue against them, in principal, just that most of the ones I have seen haven't actually exposed problems we continue to find with our larger more complex workloads.

Andy

> --
> Jason T. Greene
> WildFly Lead / JBoss EAP Platform Architect
> JBoss, a division of Red Hat
>
>

_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev
Reply | Threaded
Open this post in threaded view
|

Re: Pooling EJB Session Beans per default

Andrig Miller
In reply to this post by Bill Burke


----- Original Message -----

> From: "Bill Burke" <[hidden email]>
> To: "Jason Greene" <[hidden email]>
> Cc: "Andrig Miller" <[hidden email]>, [hidden email]
> Sent: Tuesday, August 5, 2014 9:49:29 AM
> Subject: Re: [wildfly-dev] Pooling EJB Session Beans per default
>
>
>
> On 8/5/2014 10:51 AM, Jason Greene wrote:
> >
> > On Aug 5, 2014, at 9:32 AM, Bill Burke <[hidden email]> wrote:
> >
> >>
> >>
> >> On 8/5/2014 10:15 AM, Jason Greene wrote:
> >>>> Those silly benchmarks, are indeed silly.  Any workload that
> >>>> doesn't actually do anything with the requests is not very
> >>>> helpful.  You can be really fast on them, and be really slow on
> >>>> a workload that actually does something.  We have already seen
> >>>> this in one case recently.
> >>>
> >>> It’s all about precision, and limiting variables. I know its fun
> >>> to criticize micro benchmarks (obviously micro is indeed
> >>> relative because a great deal of processing goes on even when
> >>> you have EJB and HTTP requests that don’t do much), however
> >>> micro benchmarks, provided they are measured correctly, are
> >>> quick to run and make it easy to tell what you have to fix.
> >>> Large gigantic apps on the other hand are difficult to analyze,
> >>> and do not necessarily represent activity that a differently
> >>> architected app would show. To use an example, if you have
> >>> pattern matching code, and you want it to be fast, it’s much
> >>> easier (and accurate) to benchmark the pattern matching code
> >>> then it is to benchmaark a full EE app that happens to use
> >>> pattern matching in a few places.
> >>>
> >>> Arguing against targeted benchmarks is like arguing that you
> >>> should never write a test case, because only real production
> >>> load will show problems…. To some extent thats true, but that
> >>> doesn’t mean that the problems you catch in simulations won’t
> >>> affect production users.
> >>>
> >>
> >> I'm more worried about focus rather than the actual benchmarks
> >> considering how resource strapped every project seems to be.  I'd
> >> like to see a discussion started on what are the most important
> >> and *ripe* areas for optimization and then have the performance
> >> team validate and make recommendations on any priority list that
> >> this discussion generates.
> >
> > The pooling concerns (aside from the occasional expensive post
> > constructs) actually came from a large app/benchmark that the perf
> > team was testing, and they were seeing an optimized strict max
> > pool outperform no pooling at all, and it wasn’t post construct.
> > Their theory is/was GC pressure, because this benchmark/app spends
> > a lot of time in GC, and they see higher GC activity with no
> > pooling. It’s possible it could be an indirect difference though,
> > like the fact that strict max pool acts as a throttle might
> > prevent the system from degrading as a result of no longer being
> > able to keep up with the benchmark load.
> >
>
> Its a horrible theory. :)  How many EJB instances of a give type are
> created per request?  Generally only 1.  1 instance of one object of
> one
> type!  My $5 bet is that if you went into EJB code and started
> counting
> how many object allocations were made per request, you'd lose count
> very
> quickly.   Better yet, run a single remote EJB request through a perf
> tool and let it count the number of allocations for you.  It will be
> greater than 1.  :)
>
> Maybe the StrictMaxPool has an effect on performance because it
> creates
> a global synchronization bottleneck.  Throughput is less and you end
> up
> having less concurrent per-request objects being allocated and GC'd.
>

The number per request, while relevant is only part of the story.  The number of concurrent requests happening in the server dictates the object allocation rate.  Given enough concurrency, even a very small number of object allocations per request can create an object allocation rate that can no longer be sustained.

Andy

>
> > Another possibility to look into is that I see we do:
> >     interceptorContext.setContextData(new HashMap<String,
> >     Object>());
> >
> > *AND*
> >
> >      private Map<Object, Object> instanceData = new HashMap<Object,
> >      Object>();
> >
> >
> > Aside from the unnecessary overhead, since JDK7, HashMap
> > construction uses the murmur algorithm which requires using a
> > random number generation under global locking. There were plans to
> > optimize this, but it might not be in JDK7 yet. In a few places we
> > use alternative map implementations to work around the issue.
> >
>
>
>
> IMO, it is more likely that most projects haven't gone through the
> level
> of refactorings that performance-focused projects like Undertow have
> gone through.  I'll put another $5 bet down that only the Wildfly
> family
> of projects under you and DML have gone through any real performance
> analysis and optimizations.  Just think of all the crappiness that is
> happening in the security layer alone!
>
> Also, any optimizations projects have done are probably focused on
> speed
> and throughput which generally is at the expense of memory.
>
> --
> Bill Burke
> JBoss, a division of Red Hat
> http://bill.burkecentral.com
>

_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev
Reply | Threaded
Open this post in threaded view
|

Re: Pooling EJB Session Beans per default

Bill Burke


On 8/5/2014 3:54 PM, Andrig Miller wrote:

>> Its a horrible theory. :)  How many EJB instances of a give type are
>> created per request?  Generally only 1.  1 instance of one object of
>> one
>> type!  My $5 bet is that if you went into EJB code and started
>> counting
>> how many object allocations were made per request, you'd lose count
>> very
>> quickly.   Better yet, run a single remote EJB request through a perf
>> tool and let it count the number of allocations for you.  It will be
>> greater than 1.  :)
>>
>> Maybe the StrictMaxPool has an effect on performance because it
>> creates
>> a global synchronization bottleneck.  Throughput is less and you end
>> up
>> having less concurrent per-request objects being allocated and GC'd.
>>
>
> The number per request, while relevant is only part of the story.  The number of concurrent requests happening in the server dictates the object allocation rate.  Given enough concurrency, even a very small number of object allocations per request can create an object allocation rate that can no longer be sustained.
>

I'm saying that the number of concurrent requests might not dictate
object allocation rate.  There are probably a number of allocations that
happen after the EJB instance is obtained.  i.e. interception chains,
contexts, etc.   If StrictMaxPool blocks until a new instance is
available, then there would be less allocations per request as blocking
threads would be serialized.

Whoever is investigating StrictMaxPool, or EJB pooling in general should
stop.  Its pointless.

--
Bill Burke
JBoss, a division of Red Hat
http://bill.burkecentral.com
_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev
Reply | Threaded
Open this post in threaded view
|

Re: Pooling EJB Session Beans per default

Radoslaw Rodak

Am 06.08.2014 um 00:36 schrieb Bill Burke <[hidden email]>:

>
>
> On 8/5/2014 3:54 PM, Andrig Miller wrote:
>>> Its a horrible theory. :)  How many EJB instances of a give type are
>>> created per request?  Generally only 1.  1 instance of one object of
>>> one
>>> type!  My $5 bet is that if you went into EJB code and started
>>> counting
>>> how many object allocations were made per request, you'd lose count
>>> very
>>> quickly.   Better yet, run a single remote EJB request through a perf
>>> tool and let it count the number of allocations for you.  It will be
>>> greater than 1.  :)
>>>
>>> Maybe the StrictMaxPool has an effect on performance because it
>>> creates
>>> a global synchronization bottleneck.  Throughput is less and you end
>>> up
>>> having less concurrent per-request objects being allocated and GC'd.
>>>
>>
>> The number per request, while relevant is only part of the story.  The number of concurrent requests happening in the server dictates the object allocation rate.  Given enough concurrency, even a very small number of object allocations per request can create an object allocation rate that can no longer be sustained.
>>
>
> I'm saying that the number of concurrent requests might not dictate
> object allocation rate.  There are probably a number of allocations that
> happen after the EJB instance is obtained.  i.e. interception chains,
> contexts, etc.   If StrictMaxPool blocks until a new instance is
> available, then there would be less allocations per request as blocking
> threads would be serialized.
>

Scenarion 1 )
------------------
Let say we have a pool of 100 Stateless EJBs and a constant Load of  50 Requests per second  proceeded by 50 EJBs from the pool in one second.
After 1000 seconds how many new EJB Instances will be created having a pool? answer 0 new EJBs  worst case 100 EJB’s in pool… of course object allocation is much higher as of course 1 EJB call leads to many Object from one EJB  but…let see situation without pool.

50 Request/s * 1000 seconds = worst case 50’ 000 EJB Instances on Java heap where 1 EJB might have many objects…   as long as Garbage Collection was not triggered… which sounds to me like faster filling JVM heap and having ofter GC probable depending on GC Strategy.

Scenarion 2)
------------------
Same as before,  Load is still 50 Requests  per second BUT EJB Method call takes 10s.
after 10s we have 500 EJB Instances without pool, after 11s  550 - 10 = 540EJB Instances , after 12s  580 EJBs … after some time very bad perf…full GC …and mabe OutOfMemory..

So… performance advantage could also turn in to disadvantage :-)


> Whoever is investigating StrictMaxPool, or EJB pooling in general should
> stop.  Its pointless.

Agree, pools are outdated…. but something like WorkManager for min, max Threads or even better always not less the X idle Threads would be useful :-)

Radek




_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev
Reply | Threaded
Open this post in threaded view
|

Re: Pooling EJB Session Beans per default

Andrig Miller
In reply to this post by Bill Burke


----- Original Message -----

> From: "Bill Burke" <[hidden email]>
> To: "Andrig Miller" <[hidden email]>
> Cc: [hidden email], "Jason Greene" <[hidden email]>
> Sent: Tuesday, August 5, 2014 4:36:11 PM
> Subject: Re: [wildfly-dev] Pooling EJB Session Beans per default
>
>
>
> On 8/5/2014 3:54 PM, Andrig Miller wrote:
> >> Its a horrible theory. :)  How many EJB instances of a give type
> >> are
> >> created per request?  Generally only 1.  1 instance of one object
> >> of
> >> one
> >> type!  My $5 bet is that if you went into EJB code and started
> >> counting
> >> how many object allocations were made per request, you'd lose
> >> count
> >> very
> >> quickly.   Better yet, run a single remote EJB request through a
> >> perf
> >> tool and let it count the number of allocations for you.  It will
> >> be
> >> greater than 1.  :)
> >>
> >> Maybe the StrictMaxPool has an effect on performance because it
> >> creates
> >> a global synchronization bottleneck.  Throughput is less and you
> >> end
> >> up
> >> having less concurrent per-request objects being allocated and
> >> GC'd.
> >>
> >
> > The number per request, while relevant is only part of the story.
> >  The number of concurrent requests happening in the server
> > dictates the object allocation rate.  Given enough concurrency,
> > even a very small number of object allocations per request can
> > create an object allocation rate that can no longer be sustained.
> >
>
> I'm saying that the number of concurrent requests might not dictate
> object allocation rate.  There are probably a number of allocations
> that
> happen after the EJB instance is obtained.  i.e. interception chains,
> contexts, etc.   If StrictMaxPool blocks until a new instance is
> available, then there would be less allocations per request as
> blocking
> threads would be serialized.
>
> Whoever is investigating StrictMaxPool, or EJB pooling in general
> should
> stop.  Its pointless.
>

Ah, no its not pointless.  We have a new non-blocking implementation of StrictMaxPool, and its upstream in Wildfly 9, and will be in EAP 6.4.  It has helped us increase our throughput, and reduce response times alot!

Andy

> --
> Bill Burke
> JBoss, a division of Red Hat
> http://bill.burkecentral.com
>
_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev
Reply | Threaded
Open this post in threaded view
|

Re: Pooling EJB Session Beans per default

Andrig Miller
In reply to this post by Radoslaw Rodak


----- Original Message -----

> From: "Radoslaw Rodak" <[hidden email]>
> To: [hidden email]
> Sent: Tuesday, August 5, 2014 6:51:03 PM
> Subject: Re: [wildfly-dev] Pooling EJB Session Beans per default
>
>
> Am 06.08.2014 um 00:36 schrieb Bill Burke <[hidden email]>:
>
> >
> >
> > On 8/5/2014 3:54 PM, Andrig Miller wrote:
> >>> Its a horrible theory. :)  How many EJB instances of a give type
> >>> are
> >>> created per request?  Generally only 1.  1 instance of one object
> >>> of
> >>> one
> >>> type!  My $5 bet is that if you went into EJB code and started
> >>> counting
> >>> how many object allocations were made per request, you'd lose
> >>> count
> >>> very
> >>> quickly.   Better yet, run a single remote EJB request through a
> >>> perf
> >>> tool and let it count the number of allocations for you.  It will
> >>> be
> >>> greater than 1.  :)
> >>>
> >>> Maybe the StrictMaxPool has an effect on performance because it
> >>> creates
> >>> a global synchronization bottleneck.  Throughput is less and you
> >>> end
> >>> up
> >>> having less concurrent per-request objects being allocated and
> >>> GC'd.
> >>>
> >>
> >> The number per request, while relevant is only part of the story.
> >>  The number of concurrent requests happening in the server
> >> dictates the object allocation rate.  Given enough concurrency,
> >> even a very small number of object allocations per request can
> >> create an object allocation rate that can no longer be sustained.
> >>
> >
> > I'm saying that the number of concurrent requests might not dictate
> > object allocation rate.  There are probably a number of allocations
> > that
> > happen after the EJB instance is obtained.  i.e. interception
> > chains,
> > contexts, etc.   If StrictMaxPool blocks until a new instance is
> > available, then there would be less allocations per request as
> > blocking
> > threads would be serialized.
> >
>
> Scenarion 1 )
> ------------------
> Let say we have a pool of 100 Stateless EJBs and a constant Load of
>  50 Requests per second  proceeded by 50 EJBs from the pool in one
> second.
> After 1000 seconds how many new EJB Instances will be created having
> a pool? answer 0 new EJBs  worst case 100 EJB’s in pool… of course
> object allocation is much higher as of course 1 EJB call leads to
> many Object from one EJB  but…let see situation without pool.
>
> 50 Request/s * 1000 seconds = worst case 50’ 000 EJB Instances on
> Java heap where 1 EJB might have many objects…   as long as Garbage
> Collection was not triggered… which sounds to me like faster filling
> JVM heap and having ofter GC probable depending on GC Strategy.
>
> Scenarion 2)
> ------------------
> Same as before,  Load is still 50 Requests  per second BUT EJB Method
> call takes 10s.
> after 10s we have 500 EJB Instances without pool, after 11s  550 - 10
> = 540EJB Instances , after 12s  580 EJBs … after some time very bad
> perf…full GC …and mabe OutOfMemory..
>
> So… performance advantage could also turn in to disadvantage :-)
>
>
> > Whoever is investigating StrictMaxPool, or EJB pooling in general
> > should
> > stop.  Its pointless.
>
> Agree, pools are outdated…. but something like WorkManager for min,
> max Threads or even better always not less the X idle Threads would
> be useful :-)
>
> Radek
>

The scenarios above are what is outddated.  Fifty requests per second isn't any load at all!  We have 100's of thousands of clients that we have to scale to, and lots more than 50 requests per second.

Andy

>
>
>
> _______________________________________________
> wildfly-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/wildfly-dev
>

_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev
Reply | Threaded
Open this post in threaded view
|

Re: Pooling EJB Session Beans per default

jtgreene
Administrator
In reply to this post by Radoslaw Rodak

On Aug 5, 2014, at 7:51 PM, Radoslaw Rodak <[hidden email]> wrote:

>
> Am 06.08.2014 um 00:36 schrieb Bill Burke <[hidden email]>:
>
>>
>>
>> On 8/5/2014 3:54 PM, Andrig Miller wrote:
>>>> Its a horrible theory. :)  How many EJB instances of a give type are
>>>> created per request?  Generally only 1.  1 instance of one object of
>>>> one
>>>> type!  My $5 bet is that if you went into EJB code and started
>>>> counting
>>>> how many object allocations were made per request, you'd lose count
>>>> very
>>>> quickly.   Better yet, run a single remote EJB request through a perf
>>>> tool and let it count the number of allocations for you.  It will be
>>>> greater than 1.  :)
>>>>
>>>> Maybe the StrictMaxPool has an effect on performance because it
>>>> creates
>>>> a global synchronization bottleneck.  Throughput is less and you end
>>>> up
>>>> having less concurrent per-request objects being allocated and GC'd.
>>>>
>>>
>>> The number per request, while relevant is only part of the story.  The number of concurrent requests happening in the server dictates the object allocation rate.  Given enough concurrency, even a very small number of object allocations per request can create an object allocation rate that can no longer be sustained.
>>>
>>
>> I'm saying that the number of concurrent requests might not dictate
>> object allocation rate.  There are probably a number of allocations that
>> happen after the EJB instance is obtained.  i.e. interception chains,
>> contexts, etc.   If StrictMaxPool blocks until a new instance is
>> available, then there would be less allocations per request as blocking
>> threads would be serialized.
>>
>
> Scenarion 1 )
> ------------------
> Let say we have a pool of 100 Stateless EJBs and a constant Load of  50 Requests per second  proceeded by 50 EJBs from the pool in one second.
> After 1000 seconds how many new EJB Instances will be created having a pool? answer 0 new EJBs  worst case 100 EJB’s in pool… of course object allocation is much higher as of course 1 EJB call leads to many Object from one EJB  but…let see situation without pool.
>
> 50 Request/s * 1000 seconds = worst case 50’ 000 EJB Instances on Java heap where 1 EJB might have many objects…   as long as Garbage Collection was not triggered… which sounds to me like faster filling JVM heap and having ofter GC probable depending on GC Strategy.

If you think about a single Java EE request invocation that processes data with one EJB in the call, there is typically hundreds of temporary objects created (perhaps even thousands when you are pulling back many rows of data from JPA). Aside from the container API requirements (the container has to create a string for every http header name and value, which can easily be 20+ objects), just writing plain java code that does things like substring creates temporary objects. Now, I don’t have an exact object instance count for an SLSB creation, but glancing at the code it looks ~ 6 objects. So we are talking about a very small percentage of object space, probably around 1-2%.

On the other hand the percentage could be high if you have an ejb method that doesn’t do much (e.g. just returns a constant) and you call it in a big loop as part of a request. Then you could get 6 * N object churn, which could very well end up to be a high percentage (for a large enough value of N).
 
--
Jason T. Greene
WildFly Lead / JBoss EAP Platform Architect
JBoss, a division of Red Hat


_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev
Reply | Threaded
Open this post in threaded view
|

Re: Pooling EJB Session Beans per default

John O'Hara
In reply to this post by Andrig Miller
On 08/06/2014 03:47 PM, Andrig Miller wrote:

----- Original Message -----
From: "Bill Burke" [hidden email]
To: "Andrig Miller" [hidden email]
Cc: [hidden email], "Jason Greene" [hidden email]
Sent: Tuesday, August 5, 2014 4:36:11 PM
Subject: Re: [wildfly-dev] Pooling EJB Session Beans per default



On 8/5/2014 3:54 PM, Andrig Miller wrote:
Its a horrible theory. :)  How many EJB instances of a give type
are
created per request?  Generally only 1.  1 instance of one object
of
one
type!  My $5 bet is that if you went into EJB code and started
counting
how many object allocations were made per request, you'd lose
count
very
quickly.   Better yet, run a single remote EJB request through a
perf
tool and let it count the number of allocations for you.  It will
be
greater than 1.  :)

Maybe the StrictMaxPool has an effect on performance because it
creates
a global synchronization bottleneck.  Throughput is less and you
end
up
having less concurrent per-request objects being allocated and
GC'd.

The number per request, while relevant is only part of the story.
 The number of concurrent requests happening in the server
dictates the object allocation rate.  Given enough concurrency,
even a very small number of object allocations per request can
create an object allocation rate that can no longer be sustained.

I'm saying that the number of concurrent requests might not dictate
object allocation rate.  There are probably a number of allocations
that
happen after the EJB instance is obtained.  i.e. interception chains,
contexts, etc.   If StrictMaxPool blocks until a new instance is
available, then there would be less allocations per request as
blocking
threads would be serialized.

Whoever is investigating StrictMaxPool, or EJB pooling in general
should
stop.  Its pointless.

Ah, no its not pointless.  We have a new non-blocking implementation of StrictMaxPool, and its upstream in Wildfly 9, and will be in EAP 6.4.  It has helped us increase our throughput, and reduce response times alot!

Andy
Some contextual numbers around what Andy is describing. These are results from one of our benchmarks;

Average response times (28600 concurrent users)
Pooled
Non-pooled
Remote EJB invocations
0.114s 2.094s
WS invocations
0.105s 0.332s
HTTP web app invocations


HttpCallTypeA
0.090s 5.589s
HttpCallTypeB 0.042s 2.510s
HttpCallTypeC 0.116s 7.267S

The only difference between these two sets of numbers is EJB pooling.

--
Bill Burke
JBoss, a division of Red Hat
http://bill.burkecentral.com

_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev


-- 
John O'Hara
[hidden email]

JBoss, by Red Hat
Registered Address: Red Hat UK Ltd, Amberley Place, 107-111 Peascod Street, Windsor, Berkshire, SI4 1TE, United Kingdom.
Registered in UK and Wales under Company Registration No. 3798903 Directors: Michael Cunningham (USA), Charlie Peters (USA), Matt Parsons (USA) and Michael O'Neill (Ireland).


_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev
Reply | Threaded
Open this post in threaded view
|

Re: Pooling EJB Session Beans per default

Bill Burke
In reply to this post by Andrig Miller


On 8/6/2014 10:50 AM, Andrig Miller wrote:

>
>
> ----- Original Message -----
>> From: "Radoslaw Rodak" <[hidden email]>
>> To: [hidden email]
>> Sent: Tuesday, August 5, 2014 6:51:03 PM
>> Subject: Re: [wildfly-dev] Pooling EJB Session Beans per default
>>
>>
>> Am 06.08.2014 um 00:36 schrieb Bill Burke <[hidden email]>:
>>
>>>
>>>
>>> On 8/5/2014 3:54 PM, Andrig Miller wrote:
>>>>> Its a horrible theory. :)  How many EJB instances of a give type
>>>>> are
>>>>> created per request?  Generally only 1.  1 instance of one object
>>>>> of
>>>>> one
>>>>> type!  My $5 bet is that if you went into EJB code and started
>>>>> counting
>>>>> how many object allocations were made per request, you'd lose
>>>>> count
>>>>> very
>>>>> quickly.   Better yet, run a single remote EJB request through a
>>>>> perf
>>>>> tool and let it count the number of allocations for you.  It will
>>>>> be
>>>>> greater than 1.  :)
>>>>>
>>>>> Maybe the StrictMaxPool has an effect on performance because it
>>>>> creates
>>>>> a global synchronization bottleneck.  Throughput is less and you
>>>>> end
>>>>> up
>>>>> having less concurrent per-request objects being allocated and
>>>>> GC'd.
>>>>>
>>>>
>>>> The number per request, while relevant is only part of the story.
>>>>   The number of concurrent requests happening in the server
>>>> dictates the object allocation rate.  Given enough concurrency,
>>>> even a very small number of object allocations per request can
>>>> create an object allocation rate that can no longer be sustained.
>>>>
>>>
>>> I'm saying that the number of concurrent requests might not dictate
>>> object allocation rate.  There are probably a number of allocations
>>> that
>>> happen after the EJB instance is obtained.  i.e. interception
>>> chains,
>>> contexts, etc.   If StrictMaxPool blocks until a new instance is
>>> available, then there would be less allocations per request as
>>> blocking
>>> threads would be serialized.
>>>
>>
>> Scenarion 1 )
>> ------------------
>> Let say we have a pool of 100 Stateless EJBs and a constant Load of
>>   50 Requests per second  proceeded by 50 EJBs from the pool in one
>> second.
>> After 1000 seconds how many new EJB Instances will be created having
>> a pool? answer 0 new EJBs  worst case 100 EJB’s in pool… of course
>> object allocation is much higher as of course 1 EJB call leads to
>> many Object from one EJB  but…let see situation without pool.
>>
>> 50 Request/s * 1000 seconds = worst case 50’ 000 EJB Instances on
>> Java heap where 1 EJB might have many objects…   as long as Garbage
>> Collection was not triggered… which sounds to me like faster filling
>> JVM heap and having ofter GC probable depending on GC Strategy.
>>
>> Scenarion 2)
>> ------------------
>> Same as before,  Load is still 50 Requests  per second BUT EJB Method
>> call takes 10s.
>> after 10s we have 500 EJB Instances without pool, after 11s  550 - 10
>> = 540EJB Instances , after 12s  580 EJBs … after some time very bad
>> perf…full GC …and mabe OutOfMemory..
>>
>> So… performance advantage could also turn in to disadvantage :-)
>>
>>
>>> Whoever is investigating StrictMaxPool, or EJB pooling in general
>>> should
>>> stop.  Its pointless.
>>
>> Agree, pools are outdated…. but something like WorkManager for min,
>> max Threads or even better always not less the X idle Threads would
>> be useful :-)
>>
>> Radek
>>
>
> The scenarios above are what is outddated.  Fifty requests per second isn't any load at all!  We have 100's of thousands of clients that we have to scale to, and lots more than 50 requests per second.
>
What you mean to say is that you need to scale to 100's of thousands of
clients on meaningless no-op benchmarks. :)  I do know that that old
SpecJ Java EE benchmarks artifically made EJB pooling important as
process intensive calculation results were cached in these instances.
But real-world apps don't use this feature/anti-pattern.

Also however crappy it was, I did implement an EJB container at one time
in my career.  :)  I know for a fact that there are a number of
per-request internal support objects that need to be allocated.  Let's
count:

* The argument array (for reflection)
* Each argument of the method call
* The response object
* Interceptor context object
* The interceptor context attribute map
* EJBContext
* Subject, Principal, role mappings
* Transaction context
* The message object(s) specific to the remote EJB protocol

Starts to add up huh?   I'm probably missing a bunch more.  Throw in
interaction with JPA and you end up with even more per-request objects
being allocated.  You still believe pooling one EJB instance matters?

--
Bill Burke
JBoss, a division of Red Hat
http://bill.burkecentral.com
_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev
Reply | Threaded
Open this post in threaded view
|

Re: Pooling EJB Session Beans per default

jtgreene
Administrator
In reply to this post by John O'Hara

On Aug 6, 2014, at 10:18 AM, John O'Hara <[hidden email]> wrote:

> On 08/06/2014 03:47 PM, Andrig Miller wrote:
>>
>> ----- Original Message -----
>>
>>> From: "Bill Burke" <[hidden email]>
>>>
>>> To: "Andrig Miller"
>>> <[hidden email]>
>>>
>>> Cc:
>>> [hidden email], "Jason Greene" <[hidden email]>
>>>
>>> Sent: Tuesday, August 5, 2014 4:36:11 PM
>>> Subject: Re: [wildfly-dev] Pooling EJB Session Beans per default
>>>
>>>
>>>
>>> On 8/5/2014 3:54 PM, Andrig Miller wrote:
>>>
>>>>> Its a horrible theory. :)  How many EJB instances of a give type
>>>>> are
>>>>> created per request?  Generally only 1.  1 instance of one object
>>>>> of
>>>>> one
>>>>> type!  My $5 bet is that if you went into EJB code and started
>>>>> counting
>>>>> how many object allocations were made per request, you'd lose
>>>>> count
>>>>> very
>>>>> quickly.   Better yet, run a single remote EJB request through a
>>>>> perf
>>>>> tool and let it count the number of allocations for you.  It will
>>>>> be
>>>>> greater than 1.  :)
>>>>>
>>>>> Maybe the StrictMaxPool has an effect on performance because it
>>>>> creates
>>>>> a global synchronization bottleneck.  Throughput is less and you
>>>>> end
>>>>> up
>>>>> having less concurrent per-request objects being allocated and
>>>>> GC'd.
>>>>>
>>>>>
>>>> The number per request, while relevant is only part of the story.
>>>>  The number of concurrent requests happening in the server
>>>> dictates the object allocation rate.  Given enough concurrency,
>>>> even a very small number of object allocations per request can
>>>> create an object allocation rate that can no longer be sustained.
>>>>
>>>>
>>> I'm saying that the number of concurrent requests might not dictate
>>> object allocation rate.  There are probably a number of allocations
>>> that
>>> happen after the EJB instance is obtained.  i.e. interception chains,
>>> contexts, etc.   If StrictMaxPool blocks until a new instance is
>>> available, then there would be less allocations per request as
>>> blocking
>>> threads would be serialized.
>>>
>>> Whoever is investigating StrictMaxPool, or EJB pooling in general
>>> should
>>> stop.  Its pointless.
>>>
>>>
>> Ah, no its not pointless.  We have a new non-blocking implementation of StrictMaxPool, and its upstream in Wildfly 9, and will be in EAP 6.4.  It has helped us increase our throughput, and reduce response times alot!
>>
>> Andy
>>
> Some contextual numbers around what Andy is describing. These are results from one of our benchmarks;
>
> Average response times (28600 concurrent users)
> Pooled
> Non-pooled
> Remote EJB invocations
> 0.114s 2.094s
> WS invocations
> 0.105s 0.332s
> HTTP web app invocations
>
>
> HttpCallTypeA
> 0.090s 5.589s
> HttpCallTypeB 0.042s 2.510s
> HttpCallTypeC 0.116s 7.267S

I love data, thanks :) Have you by chance taken object histogram samples for these two? It would be useful to see how strong the correlation is, and if a use-pattern shows up in the benchmark that leads to a non-pool implementation creating massive amounts of objects (like the loop scenario I mentioned)

--
Jason T. Greene
WildFly Lead / JBoss EAP Platform Architect
JBoss, a division of Red Hat


_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev
Reply | Threaded
Open this post in threaded view
|

Re: Pooling EJB Session Beans per default

Andrig Miller
In reply to this post by Bill Burke


----- Original Message -----

> From: "Bill Burke" <[hidden email]>
> To: [hidden email]
> Sent: Wednesday, August 6, 2014 9:30:06 AM
> Subject: Re: [wildfly-dev] Pooling EJB Session Beans per default
>
>
>
> On 8/6/2014 10:50 AM, Andrig Miller wrote:
> >
> >
> > ----- Original Message -----
> >> From: "Radoslaw Rodak" <[hidden email]>
> >> To: [hidden email]
> >> Sent: Tuesday, August 5, 2014 6:51:03 PM
> >> Subject: Re: [wildfly-dev] Pooling EJB Session Beans per default
> >>
> >>
> >> Am 06.08.2014 um 00:36 schrieb Bill Burke <[hidden email]>:
> >>
> >>>
> >>>
> >>> On 8/5/2014 3:54 PM, Andrig Miller wrote:
> >>>>> Its a horrible theory. :)  How many EJB instances of a give
> >>>>> type
> >>>>> are
> >>>>> created per request?  Generally only 1.  1 instance of one
> >>>>> object
> >>>>> of
> >>>>> one
> >>>>> type!  My $5 bet is that if you went into EJB code and started
> >>>>> counting
> >>>>> how many object allocations were made per request, you'd lose
> >>>>> count
> >>>>> very
> >>>>> quickly.   Better yet, run a single remote EJB request through
> >>>>> a
> >>>>> perf
> >>>>> tool and let it count the number of allocations for you.  It
> >>>>> will
> >>>>> be
> >>>>> greater than 1.  :)
> >>>>>
> >>>>> Maybe the StrictMaxPool has an effect on performance because it
> >>>>> creates
> >>>>> a global synchronization bottleneck.  Throughput is less and
> >>>>> you
> >>>>> end
> >>>>> up
> >>>>> having less concurrent per-request objects being allocated and
> >>>>> GC'd.
> >>>>>
> >>>>
> >>>> The number per request, while relevant is only part of the
> >>>> story.
> >>>>   The number of concurrent requests happening in the server
> >>>> dictates the object allocation rate.  Given enough concurrency,
> >>>> even a very small number of object allocations per request can
> >>>> create an object allocation rate that can no longer be
> >>>> sustained.
> >>>>
> >>>
> >>> I'm saying that the number of concurrent requests might not
> >>> dictate
> >>> object allocation rate.  There are probably a number of
> >>> allocations
> >>> that
> >>> happen after the EJB instance is obtained.  i.e. interception
> >>> chains,
> >>> contexts, etc.   If StrictMaxPool blocks until a new instance is
> >>> available, then there would be less allocations per request as
> >>> blocking
> >>> threads would be serialized.
> >>>
> >>
> >> Scenarion 1 )
> >> ------------------
> >> Let say we have a pool of 100 Stateless EJBs and a constant Load
> >> of
> >>   50 Requests per second  proceeded by 50 EJBs from the pool in
> >>   one
> >> second.
> >> After 1000 seconds how many new EJB Instances will be created
> >> having
> >> a pool? answer 0 new EJBs  worst case 100 EJB’s in pool… of course
> >> object allocation is much higher as of course 1 EJB call leads to
> >> many Object from one EJB  but…let see situation without pool.
> >>
> >> 50 Request/s * 1000 seconds = worst case 50’ 000 EJB Instances on
> >> Java heap where 1 EJB might have many objects…   as long as
> >> Garbage
> >> Collection was not triggered… which sounds to me like faster
> >> filling
> >> JVM heap and having ofter GC probable depending on GC Strategy.
> >>
> >> Scenarion 2)
> >> ------------------
> >> Same as before,  Load is still 50 Requests  per second BUT EJB
> >> Method
> >> call takes 10s.
> >> after 10s we have 500 EJB Instances without pool, after 11s  550 -
> >> 10
> >> = 540EJB Instances , after 12s  580 EJBs … after some time very
> >> bad
> >> perf…full GC …and mabe OutOfMemory..
> >>
> >> So… performance advantage could also turn in to disadvantage :-)
> >>
> >>
> >>> Whoever is investigating StrictMaxPool, or EJB pooling in general
> >>> should
> >>> stop.  Its pointless.
> >>
> >> Agree, pools are outdated…. but something like WorkManager for
> >> min,
> >> max Threads or even better always not less the X idle Threads
> >> would
> >> be useful :-)
> >>
> >> Radek
> >>
> >
> > The scenarios above are what is outddated.  Fifty requests per
> > second isn't any load at all!  We have 100's of thousands of
> > clients that we have to scale to, and lots more than 50 requests
> > per second.
> >
> What you mean to say is that you need to scale to 100's of thousands
> of
> clients on meaningless no-op benchmarks. :)  I do know that that old
> SpecJ Java EE benchmarks artifically made EJB pooling important as
> process intensive calculation results were cached in these instances.
> But real-world apps don't use this feature/anti-pattern.
>

I am not talking about a meaningless no-op benchmark, but a benchmark that does lots of work.  We don't use meaningless no-op benchmarks on the performance team, with some exception for microbenchmarks that we have carefully crafted that model the interactions for a specific component within the context of how it is actually used for a real application.

> Also however crappy it was, I did implement an EJB container at one
> time
> in my career.  :)  I know for a fact that there are a number of
> per-request internal support objects that need to be allocated.
>  Let's
> count:
>
> * The argument array (for reflection)
> * Each argument of the method call
> * The response object
> * Interceptor context object
> * The interceptor context attribute map
> * EJBContext
> * Subject, Principal, role mappings
> * Transaction context
> * The message object(s) specific to the remote EJB protocol
>
> Starts to add up huh?   I'm probably missing a bunch more.  Throw in
> interaction with JPA and you end up with even more per-request
> objects
> being allocated.  You still believe pooling one EJB instance matters?
>

See John O'Hara's post which shows our non-meaningless benchmark and the difference that pooling makes vs. non-pooling.  It is a dramatic difference to say the least.

This conversation is a perfect example of misinformation that causes us performance and scalability problems within our code bases.

Andy

> --
> Bill Burke
> JBoss, a division of Red Hat
> http://bill.burkecentral.com
> _______________________________________________
> wildfly-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/wildfly-dev

_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev
Reply | Threaded
Open this post in threaded view
|

Re: Pooling EJB Session Beans per default

jtgreene
Administrator

On Aug 6, 2014, at 10:49 AM, Andrig Miller <[hidden email]> wrote:

>
>
> ----- Original Message -----
>> From: "Bill Burke" <[hidden email]>
>> To: [hidden email]
>> Sent: Wednesday, August 6, 2014 9:30:06 AM
>> Subject: Re: [wildfly-dev] Pooling EJB Session Beans per default
>>
>>
>>
>> On 8/6/2014 10:50 AM, Andrig Miller wrote:
>>>
>>>
>>> ----- Original Message -----
>>>> From: "Radoslaw Rodak" <[hidden email]>
>>>> To: [hidden email]
>>>> Sent: Tuesday, August 5, 2014 6:51:03 PM
>>>> Subject: Re: [wildfly-dev] Pooling EJB Session Beans per default
>>>>
>>>>
>>>> Am 06.08.2014 um 00:36 schrieb Bill Burke <[hidden email]>:
>>>>
>>>>>
>>>>>
>>>>> On 8/5/2014 3:54 PM, Andrig Miller wrote:
>>>>>>> Its a horrible theory. :)  How many EJB instances of a give
>>>>>>> type
>>>>>>> are
>>>>>>> created per request?  Generally only 1.  1 instance of one
>>>>>>> object
>>>>>>> of
>>>>>>> one
>>>>>>> type!  My $5 bet is that if you went into EJB code and started
>>>>>>> counting
>>>>>>> how many object allocations were made per request, you'd lose
>>>>>>> count
>>>>>>> very
>>>>>>> quickly.   Better yet, run a single remote EJB request through
>>>>>>> a
>>>>>>> perf
>>>>>>> tool and let it count the number of allocations for you.  It
>>>>>>> will
>>>>>>> be
>>>>>>> greater than 1.  :)
>>>>>>>
>>>>>>> Maybe the StrictMaxPool has an effect on performance because it
>>>>>>> creates
>>>>>>> a global synchronization bottleneck.  Throughput is less and
>>>>>>> you
>>>>>>> end
>>>>>>> up
>>>>>>> having less concurrent per-request objects being allocated and
>>>>>>> GC'd.
>>>>>>>
>>>>>>
>>>>>> The number per request, while relevant is only part of the
>>>>>> story.
>>>>>>  The number of concurrent requests happening in the server
>>>>>> dictates the object allocation rate.  Given enough concurrency,
>>>>>> even a very small number of object allocations per request can
>>>>>> create an object allocation rate that can no longer be
>>>>>> sustained.
>>>>>>
>>>>>
>>>>> I'm saying that the number of concurrent requests might not
>>>>> dictate
>>>>> object allocation rate.  There are probably a number of
>>>>> allocations
>>>>> that
>>>>> happen after the EJB instance is obtained.  i.e. interception
>>>>> chains,
>>>>> contexts, etc.   If StrictMaxPool blocks until a new instance is
>>>>> available, then there would be less allocations per request as
>>>>> blocking
>>>>> threads would be serialized.
>>>>>
>>>>
>>>> Scenarion 1 )
>>>> ------------------
>>>> Let say we have a pool of 100 Stateless EJBs and a constant Load
>>>> of
>>>>  50 Requests per second  proceeded by 50 EJBs from the pool in
>>>>  one
>>>> second.
>>>> After 1000 seconds how many new EJB Instances will be created
>>>> having
>>>> a pool? answer 0 new EJBs  worst case 100 EJB’s in pool… of course
>>>> object allocation is much higher as of course 1 EJB call leads to
>>>> many Object from one EJB  but…let see situation without pool.
>>>>
>>>> 50 Request/s * 1000 seconds = worst case 50’ 000 EJB Instances on
>>>> Java heap where 1 EJB might have many objects…   as long as
>>>> Garbage
>>>> Collection was not triggered… which sounds to me like faster
>>>> filling
>>>> JVM heap and having ofter GC probable depending on GC Strategy.
>>>>
>>>> Scenarion 2)
>>>> ------------------
>>>> Same as before,  Load is still 50 Requests  per second BUT EJB
>>>> Method
>>>> call takes 10s.
>>>> after 10s we have 500 EJB Instances without pool, after 11s  550 -
>>>> 10
>>>> = 540EJB Instances , after 12s  580 EJBs … after some time very
>>>> bad
>>>> perf…full GC …and mabe OutOfMemory..
>>>>
>>>> So… performance advantage could also turn in to disadvantage :-)
>>>>
>>>>
>>>>> Whoever is investigating StrictMaxPool, or EJB pooling in general
>>>>> should
>>>>> stop.  Its pointless.
>>>>
>>>> Agree, pools are outdated…. but something like WorkManager for
>>>> min,
>>>> max Threads or even better always not less the X idle Threads
>>>> would
>>>> be useful :-)
>>>>
>>>> Radek
>>>>
>>>
>>> The scenarios above are what is outddated.  Fifty requests per
>>> second isn't any load at all!  We have 100's of thousands of
>>> clients that we have to scale to, and lots more than 50 requests
>>> per second.
>>>
>> What you mean to say is that you need to scale to 100's of thousands
>> of
>> clients on meaningless no-op benchmarks. :)  I do know that that old
>> SpecJ Java EE benchmarks artifically made EJB pooling important as
>> process intensive calculation results were cached in these instances.
>> But real-world apps don't use this feature/anti-pattern.
>>
>
> I am not talking about a meaningless no-op benchmark, but a benchmark that does lots of work.  We don't use meaningless no-op benchmarks on the performance team, with some exception for microbenchmarks that we have carefully crafted that model the interactions for a specific component within the context of how it is actually used for a real application.
>
>> Also however crappy it was, I did implement an EJB container at one
>> time
>> in my career.  :)  I know for a fact that there are a number of
>> per-request internal support objects that need to be allocated.
>> Let's
>> count:
>>
>> * The argument array (for reflection)
>> * Each argument of the method call
>> * The response object
>> * Interceptor context object
>> * The interceptor context attribute map
>> * EJBContext
>> * Subject, Principal, role mappings
>> * Transaction context
>> * The message object(s) specific to the remote EJB protocol
>>
>> Starts to add up huh?   I'm probably missing a bunch more.  Throw in
>> interaction with JPA and you end up with even more per-request
>> objects
>> being allocated.  You still believe pooling one EJB instance matters?
>>
>
> See John O'Hara's post which shows our non-meaningless benchmark and the difference that pooling makes vs. non-pooling.  It is a dramatic difference to say the least.

There is certainly a correlation identified between the results of this benchmark and the use of pooling. However the underlying cause of the resulting difference is still unknown. If we knew definitively how and why this happens it would help in optimizing this further. As an example, if it turned out to be some secondary factor, like the throttling aspect of the pool, then eliminating these allocations (and others) with a zero-tuning approach, like thread local pooling would offer little to no improvement. If discovered it is indeed extreme object allocation, and that it came from thousands of nested calls in a request, then having a temporary per-request thread local cache would dramatically improve the results, and be cheap/quick to implement vs a full thread local solution. If there is a bug in our code somewhere where under certain situations we create hundreds of objects, when we should be creating 10s, and the pool covers that up, fixing that bug and removing the pool could lead to better results. If it turns out there is only 3% extra churn but that extra churn causes a 10x perf reduction in GC, then we better understand those limits and potentially work with the openjdk team in that area.


>
> This conversation is a perfect example of misinformation that causes us performance and scalability problems within our code bases.

It’s just a surprising result. The pool saves a few allocations, but it also has the cost of concurrency usage which can trigger blocking, additional barriers, and busy looping on CAS. You also still have object churn in the underlying pool data structures that occurs per invocation since every invocation is a check-out and a check-in (requires a new node object instance), and if the semaphore blocks you have additional allocation for the entry in the wait queue. You factor in the remaining allocation savings relative to other allocations that are required for the invocation, and it should be a very small percentage. For that very small percentage to lead to several times a difference in performance to me hints at other factors being involved.

--
Jason T. Greene
WildFly Lead / JBoss EAP Platform Architect
JBoss, a division of Red Hat


_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev
Reply | Threaded
Open this post in threaded view
|

Re: Pooling EJB Session Beans per default

Andrig Miller


----- Original Message -----

> From: "Jason Greene" <[hidden email]>
> To: "Andrig Miller" <[hidden email]>
> Cc: "Bill Burke" <[hidden email]>, [hidden email]
> Sent: Wednesday, August 6, 2014 11:08:02 AM
> Subject: Re: [wildfly-dev] Pooling EJB Session Beans per default
>
>
> On Aug 6, 2014, at 10:49 AM, Andrig Miller <[hidden email]>
> wrote:
>
> >
> >
> > ----- Original Message -----
> >> From: "Bill Burke" <[hidden email]>
> >> To: [hidden email]
> >> Sent: Wednesday, August 6, 2014 9:30:06 AM
> >> Subject: Re: [wildfly-dev] Pooling EJB Session Beans per default
> >>
> >>
> >>
> >> On 8/6/2014 10:50 AM, Andrig Miller wrote:
> >>>
> >>>
> >>> ----- Original Message -----
> >>>> From: "Radoslaw Rodak" <[hidden email]>
> >>>> To: [hidden email]
> >>>> Sent: Tuesday, August 5, 2014 6:51:03 PM
> >>>> Subject: Re: [wildfly-dev] Pooling EJB Session Beans per default
> >>>>
> >>>>
> >>>> Am 06.08.2014 um 00:36 schrieb Bill Burke <[hidden email]>:
> >>>>
> >>>>>
> >>>>>
> >>>>> On 8/5/2014 3:54 PM, Andrig Miller wrote:
> >>>>>>> Its a horrible theory. :)  How many EJB instances of a give
> >>>>>>> type
> >>>>>>> are
> >>>>>>> created per request?  Generally only 1.  1 instance of one
> >>>>>>> object
> >>>>>>> of
> >>>>>>> one
> >>>>>>> type!  My $5 bet is that if you went into EJB code and
> >>>>>>> started
> >>>>>>> counting
> >>>>>>> how many object allocations were made per request, you'd lose
> >>>>>>> count
> >>>>>>> very
> >>>>>>> quickly.   Better yet, run a single remote EJB request
> >>>>>>> through
> >>>>>>> a
> >>>>>>> perf
> >>>>>>> tool and let it count the number of allocations for you.  It
> >>>>>>> will
> >>>>>>> be
> >>>>>>> greater than 1.  :)
> >>>>>>>
> >>>>>>> Maybe the StrictMaxPool has an effect on performance because
> >>>>>>> it
> >>>>>>> creates
> >>>>>>> a global synchronization bottleneck.  Throughput is less and
> >>>>>>> you
> >>>>>>> end
> >>>>>>> up
> >>>>>>> having less concurrent per-request objects being allocated
> >>>>>>> and
> >>>>>>> GC'd.
> >>>>>>>
> >>>>>>
> >>>>>> The number per request, while relevant is only part of the
> >>>>>> story.
> >>>>>>  The number of concurrent requests happening in the server
> >>>>>> dictates the object allocation rate.  Given enough
> >>>>>> concurrency,
> >>>>>> even a very small number of object allocations per request can
> >>>>>> create an object allocation rate that can no longer be
> >>>>>> sustained.
> >>>>>>
> >>>>>
> >>>>> I'm saying that the number of concurrent requests might not
> >>>>> dictate
> >>>>> object allocation rate.  There are probably a number of
> >>>>> allocations
> >>>>> that
> >>>>> happen after the EJB instance is obtained.  i.e. interception
> >>>>> chains,
> >>>>> contexts, etc.   If StrictMaxPool blocks until a new instance
> >>>>> is
> >>>>> available, then there would be less allocations per request as
> >>>>> blocking
> >>>>> threads would be serialized.
> >>>>>
> >>>>
> >>>> Scenarion 1 )
> >>>> ------------------
> >>>> Let say we have a pool of 100 Stateless EJBs and a constant Load
> >>>> of
> >>>>  50 Requests per second  proceeded by 50 EJBs from the pool in
> >>>>  one
> >>>> second.
> >>>> After 1000 seconds how many new EJB Instances will be created
> >>>> having
> >>>> a pool? answer 0 new EJBs  worst case 100 EJB’s in pool… of
> >>>> course
> >>>> object allocation is much higher as of course 1 EJB call leads
> >>>> to
> >>>> many Object from one EJB  but…let see situation without pool.
> >>>>
> >>>> 50 Request/s * 1000 seconds = worst case 50’ 000 EJB Instances
> >>>> on
> >>>> Java heap where 1 EJB might have many objects…   as long as
> >>>> Garbage
> >>>> Collection was not triggered… which sounds to me like faster
> >>>> filling
> >>>> JVM heap and having ofter GC probable depending on GC Strategy.
> >>>>
> >>>> Scenarion 2)
> >>>> ------------------
> >>>> Same as before,  Load is still 50 Requests  per second BUT EJB
> >>>> Method
> >>>> call takes 10s.
> >>>> after 10s we have 500 EJB Instances without pool, after 11s  550
> >>>> -
> >>>> 10
> >>>> = 540EJB Instances , after 12s  580 EJBs … after some time very
> >>>> bad
> >>>> perf…full GC …and mabe OutOfMemory..
> >>>>
> >>>> So… performance advantage could also turn in to disadvantage :-)
> >>>>
> >>>>
> >>>>> Whoever is investigating StrictMaxPool, or EJB pooling in
> >>>>> general
> >>>>> should
> >>>>> stop.  Its pointless.
> >>>>
> >>>> Agree, pools are outdated…. but something like WorkManager for
> >>>> min,
> >>>> max Threads or even better always not less the X idle Threads
> >>>> would
> >>>> be useful :-)
> >>>>
> >>>> Radek
> >>>>
> >>>
> >>> The scenarios above are what is outddated.  Fifty requests per
> >>> second isn't any load at all!  We have 100's of thousands of
> >>> clients that we have to scale to, and lots more than 50 requests
> >>> per second.
> >>>
> >> What you mean to say is that you need to scale to 100's of
> >> thousands
> >> of
> >> clients on meaningless no-op benchmarks. :)  I do know that that
> >> old
> >> SpecJ Java EE benchmarks artifically made EJB pooling important as
> >> process intensive calculation results were cached in these
> >> instances.
> >> But real-world apps don't use this feature/anti-pattern.
> >>
> >
> > I am not talking about a meaningless no-op benchmark, but a
> > benchmark that does lots of work.  We don't use meaningless no-op
> > benchmarks on the performance team, with some exception for
> > microbenchmarks that we have carefully crafted that model the
> > interactions for a specific component within the context of how it
> > is actually used for a real application.
> >
> >> Also however crappy it was, I did implement an EJB container at
> >> one
> >> time
> >> in my career.  :)  I know for a fact that there are a number of
> >> per-request internal support objects that need to be allocated.
> >> Let's
> >> count:
> >>
> >> * The argument array (for reflection)
> >> * Each argument of the method call
> >> * The response object
> >> * Interceptor context object
> >> * The interceptor context attribute map
> >> * EJBContext
> >> * Subject, Principal, role mappings
> >> * Transaction context
> >> * The message object(s) specific to the remote EJB protocol
> >>
> >> Starts to add up huh?   I'm probably missing a bunch more.  Throw
> >> in
> >> interaction with JPA and you end up with even more per-request
> >> objects
> >> being allocated.  You still believe pooling one EJB instance
> >> matters?
> >>
> >
> > See John O'Hara's post which shows our non-meaningless benchmark
> > and the difference that pooling makes vs. non-pooling.  It is a
> > dramatic difference to say the least.
>
> There is certainly a correlation identified between the results of
> this benchmark and the use of pooling. However the underlying cause
> of the resulting difference is still unknown. If we knew
> definitively how and why this happens it would help in optimizing
> this further. As an example, if it turned out to be some secondary
> factor, like the throttling aspect of the pool, then eliminating
> these allocations (and others) with a zero-tuning approach, like
> thread local pooling would offer little to no improvement. If
> discovered it is indeed extreme object allocation, and that it came
> from thousands of nested calls in a request, then having a temporary
> per-request thread local cache would dramatically improve the
> results, and be cheap/quick to implement vs a full thread local
> solution. If there is a bug in our code somewhere where under
> certain situations we create hundreds of objects, when we should be
> creating 10s, and the pool covers that up, fixing that bug and
> removing the pool could lead to better results. If it turns out
> there is only 3% extra churn but that extra churn causes a 10x perf
> reduction in GC, then we better understand those limits and
> potentially work with the openjdk team in that area.
>
>
> >
> > This conversation is a perfect example of misinformation that
> > causes us performance and scalability problems within our code
> > bases.
>
> It’s just a surprising result. The pool saves a few allocations, but
> it also has the cost of concurrency usage which can trigger
> blocking, additional barriers, and busy looping on CAS. You also
> still have object churn in the underlying pool data structures that
> occurs per invocation since every invocation is a check-out and a
> check-in (requires a new node object instance), and if the semaphore
> blocks you have additional allocation for the entry in the wait
> queue. You factor in the remaining allocation savings relative to
> other allocations that are required for the invocation, and it
> should be a very small percentage. For that very small percentage to
> lead to several times a difference in performance to me hints at
> other factors being involved.
>

All logically thought through.  At a 15% lower transaction rate than we are doing now, we saw 4 Gigabytes per second of object allocation.  We, with Sanne doing most of the work, managed to get that down to 3 Gigabytes per second (I would have loved to get it to 2).  Much of that was Hibernate allocations, and of course that was with pooling on.  We have not spent the time to pinpoint the exact differences, memory and other, between having pooling on vs. off.  Our priority has been continue to scale the workload and fix any problems we see as a result.  We have managed to increase the transaction rate another 15% in the last couple of months, but still have another 17+% to go on a single JVM before we start looking at two JVM's for the testing.  

Once we get to our goal, I would love to put this on our list of tasks, so we can get the specific facts, and instead of talking theory, we will no exactly what can and cannot be done, and whether no pooling could ever match pooled.

Andy

> --
> Jason T. Greene
> WildFly Lead / JBoss EAP Platform Architect
> JBoss, a division of Red Hat
>
>

_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev
Reply | Threaded
Open this post in threaded view
|

Re: Pooling EJB Session Beans per default

Radoslaw Rodak
In reply to this post by John O'Hara

Am 06.08.2014 um 17:18 schrieb John O'Hara <[hidden email]>:

On 08/06/2014 03:47 PM, Andrig Miller wrote:
----- Original Message -----
From: "Bill Burke" [hidden email]
To: "Andrig Miller" [hidden email]
Cc: [hidden email], "Jason Greene" [hidden email]
Sent: Tuesday, August 5, 2014 4:36:11 PM
Subject: Re: [wildfly-dev] Pooling EJB Session Beans per default



On 8/5/2014 3:54 PM, Andrig Miller wrote:
Its a horrible theory. :)  How many EJB instances of a give type
are
created per request?  Generally only 1.  1 instance of one object
of
one
type!  My $5 bet is that if you went into EJB code and started
counting
how many object allocations were made per request, you'd lose
count
very
quickly.   Better yet, run a single remote EJB request through a
perf
tool and let it count the number of allocations for you.  It will
be
greater than 1.  :)

Maybe the StrictMaxPool has an effect on performance because it
creates
a global synchronization bottleneck.  Throughput is less and you
end
up
having less concurrent per-request objects being allocated and
GC'd.

The number per request, while relevant is only part of the story.
 The number of concurrent requests happening in the server
dictates the object allocation rate.  Given enough concurrency,
even a very small number of object allocations per request can
create an object allocation rate that can no longer be sustained.

I'm saying that the number of concurrent requests might not dictate
object allocation rate.  There are probably a number of allocations
that
happen after the EJB instance is obtained.  i.e. interception chains,
contexts, etc.   If StrictMaxPool blocks until a new instance is
available, then there would be less allocations per request as
blocking
threads would be serialized.

Whoever is investigating StrictMaxPool, or EJB pooling in general
should
stop.  Its pointless.

Ah, no its not pointless.  We have a new non-blocking implementation of StrictMaxPool, and its upstream in Wildfly 9, and will be in EAP 6.4.  It has helped us increase our throughput, and reduce response times alot!

Andy
Some contextual numbers around what Andy is describing. These are results from one of our benchmarks;

Average response times (28600 concurrent users)
Pooled
Non-pooled
Remote EJB invocations
0.114s 2.094s
WS invocations
0.105s 0.332s
HTTP web app invocations


HttpCallTypeA
0.090s 5.589s
HttpCallTypeB 0.042s 2.510s
HttpCallTypeC 0.116s 7.267S

The only difference between these two sets of numbers is EJB pooling.

I guess this are just average numbers…. do you have graphics with response in function of time during the loadtest?
And how much time JVM spent doing GC… with both tests?
Dynatrace has nice integration with jenkins to generate such reports.


_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev
Reply | Threaded
Open this post in threaded view
|

Re: Pooling EJB Session Beans per default

jtgreene
Administrator
In reply to this post by Bill Burke

On Aug 6, 2014, at 10:30 AM, Bill Burke <[hidden email]> wrote:

> What you mean to say is that you need to scale to 100's of thousands of
> clients on meaningless no-op benchmarks. :)  I do know that that old
> SpecJ Java EE benchmarks artifically made EJB pooling important as
> process intensive calculation results were cached in these instances.
> But real-world apps don't use this feature/anti-pattern.

If the benchmark in question is doing that, that would most definitely explain this. I mean we know the following can perform poorly without pooling:

1. Expensive initialization in @PostConstruct
2. Lazy expensive initialization in a invocation (what you allude to above)
3. Expensive initialization in a constructor

Large allocations of objects count as expensive initialization.

--
Jason T. Greene
WildFly Lead / JBoss EAP Platform Architect
JBoss, a division of Red Hat


_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev
Reply | Threaded
Open this post in threaded view
|

Re: Pooling EJB Session Beans per default

jtgreene
Administrator
In reply to this post by Andrig Miller

On Aug 6, 2014, at 12:21 PM, Andrig Miller <[hidden email]> wrote:

>
>
> ----- Original Message -----
>> From: "Jason Greene" <[hidden email]>
>> To: "Andrig Miller" <[hidden email]>
>> Cc: "Bill Burke" <[hidden email]>, [hidden email]
>> Sent: Wednesday, August 6, 2014 11:08:02 AM
>> Subject: Re: [wildfly-dev] Pooling EJB Session Beans per default
>>
>>
>> On Aug 6, 2014, at 10:49 AM, Andrig Miller <[hidden email]>
>> wrote:
>>
>>>
>>>
>>> ----- Original Message -----
>>>> From: "Bill Burke" <[hidden email]>
>>>> To: [hidden email]
>>>> Sent: Wednesday, August 6, 2014 9:30:06 AM
>>>> Subject: Re: [wildfly-dev] Pooling EJB Session Beans per default
>>>>
>>>


>>> This conversation is a perfect example of misinformation that
>>> causes us performance and scalability problems within our code
>>> bases.
>>
>> It’s just a surprising result. The pool saves a few allocations, but
>> it also has the cost of concurrency usage which can trigger
>> blocking, additional barriers, and busy looping on CAS. You also
>> still have object churn in the underlying pool data structures that
>> occurs per invocation since every invocation is a check-out and a
>> check-in (requires a new node object instance), and if the semaphore
>> blocks you have additional allocation for the entry in the wait
>> queue. You factor in the remaining allocation savings relative to
>> other allocations that are required for the invocation, and it
>> should be a very small percentage. For that very small percentage to
>> lead to several times a difference in performance to me hints at
>> other factors being involved.
>>
>
> All logically thought through.  At a 15% lower transaction rate than we are doing now, we saw 4 Gigabytes per second of object allocation.  We, with Sanne doing most of the work, managed to get that down to 3 Gigabytes per second (I would have loved to get it to 2).  Much of that was Hibernate allocations, and of course that was with pooling on.  We have not spent the time to pinpoint the exact differences, memory and other, between having pooling on vs. off.  Our priority has been continue to scale the workload and fix any problems we see as a result.  We have managed to increase the transaction rate another 15% in the last couple of months, but still have another 17+% to go on a single JVM before we start looking at two JVM's for the testing.  
>
> Once we get to our goal, I would love to put this on our list of tasks, so we can get the specific facts, and instead of talking theory, we will no exactly what can and cannot be done, and whether no pooling could ever match pooled.

Fair enough, and I certainly didn’t mean to imply that such work should be your team, I was just speaking generally. In any case, what I really really would like for us to achieve is a default implementation that performs generally well on all usage patterns, with no tuning required. Since we know that initialization can be costly for some applications usage of SLSB, such an implementation will definitely require a form of pooling.

I suspect that a thread local based design with the pooling tied to worker threads will give us this. Alternatively a shared pool which is auto-tuned to match might be worth looking into.

If there is anyone lurking who wishes to contribute in this area speak up, and I’ll worth with you on it. As doge would say “Such Fun. Much Glory” :)

--
Jason T. Greene
WildFly Lead / JBoss EAP Platform Architect
JBoss, a division of Red Hat


_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev
123