consistent testsuite failures since I synced with master yesterday

classic Classic list List threaded Threaded
25 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Re: ARQ-632 and as7/testsuite/integration failures due to connection not getting closed...

Scott Marlow
No, I still see "java.lang.OutOfMemoryError: unable to create new native
thread".  I'll try a shorter timeout, maybe ten seconds instead of thirty.

Another workaround would be reducing the number of tests in
testsuite/integration (at least until ARQ-632 is fixed).

On 10/20/2011 02:58 PM, Scott Marlow wrote:

> I'll give that a try.
>
> On 10/20/2011 02:36 PM, Brian Stansberry wrote:
>> I'm not seeing this failure, so it's hard to test a workaround. But if
>> someone who is seeing it can give
>> https://github.com/bstansberry/jboss-as/commit/0d422b184760b95e47c886f139426b512341a333
>> a spin and let me know, I'd appreciate it.
>>
>> Probably easiest is just to check out and build
>> https://github.com/bstansberry/jboss-as/tree/jmx-conn-leak
>>
>> On 10/20/11 11:49 AM, Brian Stansberry wrote:
>>> See [1] for a good discussion of this; last post points at a possible
>>> workaround.
>>>
>>> [1] https://forums.oracle.com/forums/thread.jspa?threadID=1665966
>>> On 10/19/11 6:26 PM, Stuart Douglas wrote:
>>>> It is probably to do with the number of tests in the test suite, as
>>>> now all the tests that were previously in preview and spec are
>>>> executed in one run.
>>>>
>>>> Stuart
>>>>
>>>>
>>>> On 20/10/2011, at 10:24 AM, Scott Marlow wrote:
>>>>
>>>>> I agree that we should be closing the connection
>>>>> (https://anonsvn.jboss.org/repos/jbossas/trunk/console/src/main/java/org/jboss/console/twiddle/Twiddle.java
>>>>>
>>>>> is an example of where we handled that in AS6). Probably after we are
>>>>> done with the MBeanServerConnection.
>>>>>
>>>>> I created ARQ-632 for this.
>>>>>
>>>>> I was expecting the cause to be a regression but instead perhaps its
>>>>> the performance improvements (we are running unit tests quicker,
>>>>> causing more "jmx server connection timeout" threads to be created).
>>>>> Or, due to the number of tests in the testsuite now.
>>>>>
>>>>> ----- Original Message -----
>>>>> From: "Brian Stansberry"<[hidden email]>
>>>>> To: [hidden email]
>>>>> Sent: Wednesday, October 19, 2011 5:47:41 PM
>>>>> Subject: Re: [jboss-as7-dev] consistent testsuite failures since I
>>>>> synced with master yesterday
>>>>>
>>>>> org.jboss.arquillian.container.spi.client.protocol.metadata.JMXContext.getConnection()
>>>>>
>>>>>
>>>>> is creating a javax.management.remote.JMXConnector but I don't see
>>>>> anything closing it. I'm not sure, but perhaps one of those is created
>>>>> per test method execution?
>>>>>
>>>>> I believe the "JMX server connection timeout 552" threads are
>>>>> created by
>>>>> the JDK stuff AS7 is using for its JMX connector impl. You can see the
>>>>> relevant code in the stack trace. A quick look at that code leads me to
>>>>> believe those threads will by default live for 120 secs after the last
>>>>> request on the connection.
>>>>>
>>>>> On 10/19/11 11:54 AM, Scott Marlow wrote:
>>>>>> On 10/19/2011 12:13 PM, Scott Marlow wrote:
>>>>>>> On 10/19/2011 11:01 AM, Scott Marlow wrote:
>>>>>>>> I'm using Sun 1.6.0_26-b03. I think Hudson is passing with Sun
>>>>>>>> 1.6.0_24-b07.
>>>>>>>>
>>>>>>>> Jesper also got the failures with jdk 6 u26.
>>>>>>>>
>>>>>>>> I haven't tried running with my old jdk1.6.0_24 yet. I could try
>>>>>>>> that
>>>>>>>> or we could start comparing environment settings.
>>>>>>>>
>>>>>>>> On Hudson, we are setting MAVEN_OPTS="-Xmx1024m -Xms512m
>>>>>>>> -XX:MaxPermSize=128m"
>>>>>>>>
>>>>>>>> I don't have that set locally. I'll try that next...
>>>>>>>
>>>>>>> Testsuite still fails for me with the above MAVEN_OPTS set.
>>>>>>>
>>>>>>> I'll my older jdk 6 (same as hudson is using) and see if that makes
>>>>>>> a diff.
>>>>>>
>>>>>> Same failure still (with MAVEN_OPTS set as above and using
>>>>>> jdk1.6.0_24).
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>> Scott
>>>>>>>>
>>>>>>>> On 10/19/2011 10:49 AM, Jaikiran Pai wrote:
>>>>>>>>> I don't know about the test issues that Ondrej is running into,
>>>>>>>>> but the
>>>>>>>>> ones you mention http://pastebin.com/D3SeJ2rP (OutOfMemory : Cannot
>>>>>>>>> create native thread) were encountered once by Carlo (or on our
>>>>>>>>> Hudson
>>>>>>>>> instance, don't remember). But I haven't seen them after that. It
>>>>>>>>> might
>>>>>>>>> be environment specific. What JDK vendor/version? And are you
>>>>>>>>> running
>>>>>>>>> into this always?
>>>>>>>>>
>>>>>>>>> -Jaikiran
>>>>>>>>> On Wednesday 19 October 2011 08:14 PM, Scott Marlow wrote:
>>>>>>>>>> How did these testsuite failures slip in? From my own experience
>>>>>>>>>> running the updated testsuite, it looks like we now have an
>>>>>>>>>> allTests
>>>>>>>>>> profile that doesn't run the integration tests. Perhaps we
>>>>>>>>>> merged some
>>>>>>>>>> changes and only ran the allTests profile. I think that we need to
>>>>>>>>>> continue to use -DallTests instead (that includes the integration
>>>>>>>>>> tests). Does anyone know more?
>>>>>>>>>>
>>>>>>>>>> I'm not sure why some people aren't seeing the failures but it
>>>>>>>>>> seems we
>>>>>>>>>> need to track backwards from the symptoms (listed below) or
>>>>>>>>>> rollback
>>>>>>>>>> some changes until it goes away.
>>>>>>>>>>
>>>>>>>>>> If anyone is working on tracking the cause down, please speak
>>>>>>>>>> up. I can
>>>>>>>>>> jump in but don't want to duplicate effort.
>>>>>>>>>>
>>>>>>>>>> Scott
>>>>>>>>>>
>>>>>>>>>> On 10/18/2011 02:15 PM, Scott Marlow wrote:
>>>>>>>>>>> The hudson AS7 test just finished with all integration tests
>>>>>>>>>>> passing.
>>>>>>>>>>>
>>>>>>>>>>> When you say that your using a fresh master, is that a new AS7
>>>>>>>>>>> source
>>>>>>>>>>> tree or an existing one freshly synced with AS7 master?
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On 10/18/2011 12:26 PM, Scott Marlow wrote:
>>>>>>>>>>>> On 10/18/2011 08:57 AM, Ondřej Žižka wrote:
>>>>>>>>>>>>> I have a fresh master and testsuite runs fine.
>>>>>>>>>>>>> Except 2 failures (for which I also seek comments):
>>>>>>>>>>>>>
>>>>>>>>>>>>> org.jboss.as.test.integration.osgi.webapp.ServletIntegrationTestCase.testServiceAccess
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> org.jboss.as.test.integration.osgi.jaxrs.RestEasyIntegrationTestCase.org.jboss.as.test.integration.osgi.jaxrs.RestEasyIntegrationTestCase
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> server.log:
>>>>>>>>>>>>> http://pastebin.test.redhat.com/64949
>>>>>>>>>>>>
>>>>>>>>>>>> I didn't see that failure.
>>>>>>>>>>>>
>>>>>>>>>>>> Which folder are you starting the unit test from and what is
>>>>>>>>>>>> your
>>>>>>>>>>>> command line?
>>>>>>>>>>>>
>>>>>>>>>>>> I was trying to use "mvn clean install -DallTests" in the
>>>>>>>>>>>> as7/testsuite/integration folder.
>>>>>>>>>>>>
>>>>>>>>>>>> When I checked earlier, Hudson wasn't running the full test
>>>>>>>>>>>> (was using
>>>>>>>>>>>> -PallTests instead of -DallTests). Looks like someone changed
>>>>>>>>>>>> that but
>>>>>>>>>>>> the test is still in queue.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Anyone seeing that? I thought it could be a race condition
>>>>>>>>>>>>> caused by
>>>>>>>>>>>>> improper use of deployer API, but giving it some time after
>>>>>>>>>>>>> deploy
>>>>>>>>>>>>> didn't help.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Ondra
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Scott Marlow píše v Út 18. 10. 2011 v 08:36 -0400:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> When running the as7/testsuite/integration tests, we run
>>>>>>>>>>>>>> fine for a
>>>>>>>>>>>>>> while but then "OutOfMemoryError: unable to create new
>>>>>>>>>>>>>> native thread"
>>>>>>>>>>>>>> errors start showing up in the server.log.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Failing tests are here http://pastebin.com/D3SeJ2rP
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> AS7 thread dump showing a lot of "JMX server connection
>>>>>>>>>>>>>> timeout NNN"
>>>>>>>>>>>>>> threads is here http://pastebin.com/qDC52WJG
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> To recreate change into as7/testsuite/integration and do a
>>>>>>>>>>>>>> "mvn clean
>>>>>>>>>>>>>> install -DallTests".
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Is anyone not seeing this after syncing with master?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Scott
>>
>>
>
> _______________________________________________
> jboss-as7-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev

_______________________________________________
jboss-as7-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
Reply | Threaded
Open this post in threaded view
|

Re: ARQ-632 and as7/testsuite/integration failures due to connection not getting closed...

Scott Marlow
It probably wasn't using the timeout setting added to
as7/testsuite/integration/pom.xml.  I changed the
o.j.a.j.JMXConnectorService default to ten seconds and made it through
the integration tests.

If we change the default timeout to ten seconds (for a few days), that
might cause test failures on busy/slower machines.

On 10/20/2011 03:21 PM, Scott Marlow wrote:

> No, I still see "java.lang.OutOfMemoryError: unable to create new native
> thread".  I'll try a shorter timeout, maybe ten seconds instead of thirty.
>
> Another workaround would be reducing the number of tests in
> testsuite/integration (at least until ARQ-632 is fixed).
>
> On 10/20/2011 02:58 PM, Scott Marlow wrote:
>> I'll give that a try.
>>
>> On 10/20/2011 02:36 PM, Brian Stansberry wrote:
>>> I'm not seeing this failure, so it's hard to test a workaround. But if
>>> someone who is seeing it can give
>>> https://github.com/bstansberry/jboss-as/commit/0d422b184760b95e47c886f139426b512341a333
>>> a spin and let me know, I'd appreciate it.
>>>
>>> Probably easiest is just to check out and build
>>> https://github.com/bstansberry/jboss-as/tree/jmx-conn-leak
>>>
>>> On 10/20/11 11:49 AM, Brian Stansberry wrote:
>>>> See [1] for a good discussion of this; last post points at a possible
>>>> workaround.
>>>>
>>>> [1] https://forums.oracle.com/forums/thread.jspa?threadID=1665966
>>>> On 10/19/11 6:26 PM, Stuart Douglas wrote:
>>>>> It is probably to do with the number of tests in the test suite, as
>>>>> now all the tests that were previously in preview and spec are
>>>>> executed in one run.
>>>>>
>>>>> Stuart
>>>>>
>>>>>
>>>>> On 20/10/2011, at 10:24 AM, Scott Marlow wrote:
>>>>>
>>>>>> I agree that we should be closing the connection
>>>>>> (https://anonsvn.jboss.org/repos/jbossas/trunk/console/src/main/java/org/jboss/console/twiddle/Twiddle.java
>>>>>>
>>>>>> is an example of where we handled that in AS6). Probably after we are
>>>>>> done with the MBeanServerConnection.
>>>>>>
>>>>>> I created ARQ-632 for this.
>>>>>>
>>>>>> I was expecting the cause to be a regression but instead perhaps its
>>>>>> the performance improvements (we are running unit tests quicker,
>>>>>> causing more "jmx server connection timeout" threads to be created).
>>>>>> Or, due to the number of tests in the testsuite now.
>>>>>>
>>>>>> ----- Original Message -----
>>>>>> From: "Brian Stansberry"<[hidden email]>
>>>>>> To: [hidden email]
>>>>>> Sent: Wednesday, October 19, 2011 5:47:41 PM
>>>>>> Subject: Re: [jboss-as7-dev] consistent testsuite failures since I
>>>>>> synced with master yesterday
>>>>>>
>>>>>> org.jboss.arquillian.container.spi.client.protocol.metadata.JMXContext.getConnection()
>>>>>>
>>>>>>
>>>>>> is creating a javax.management.remote.JMXConnector but I don't see
>>>>>> anything closing it. I'm not sure, but perhaps one of those is created
>>>>>> per test method execution?
>>>>>>
>>>>>> I believe the "JMX server connection timeout 552" threads are
>>>>>> created by
>>>>>> the JDK stuff AS7 is using for its JMX connector impl. You can see the
>>>>>> relevant code in the stack trace. A quick look at that code leads me to
>>>>>> believe those threads will by default live for 120 secs after the last
>>>>>> request on the connection.
>>>>>>
>>>>>> On 10/19/11 11:54 AM, Scott Marlow wrote:
>>>>>>> On 10/19/2011 12:13 PM, Scott Marlow wrote:
>>>>>>>> On 10/19/2011 11:01 AM, Scott Marlow wrote:
>>>>>>>>> I'm using Sun 1.6.0_26-b03. I think Hudson is passing with Sun
>>>>>>>>> 1.6.0_24-b07.
>>>>>>>>>
>>>>>>>>> Jesper also got the failures with jdk 6 u26.
>>>>>>>>>
>>>>>>>>> I haven't tried running with my old jdk1.6.0_24 yet. I could try
>>>>>>>>> that
>>>>>>>>> or we could start comparing environment settings.
>>>>>>>>>
>>>>>>>>> On Hudson, we are setting MAVEN_OPTS="-Xmx1024m -Xms512m
>>>>>>>>> -XX:MaxPermSize=128m"
>>>>>>>>>
>>>>>>>>> I don't have that set locally. I'll try that next...
>>>>>>>>
>>>>>>>> Testsuite still fails for me with the above MAVEN_OPTS set.
>>>>>>>>
>>>>>>>> I'll my older jdk 6 (same as hudson is using) and see if that makes
>>>>>>>> a diff.
>>>>>>>
>>>>>>> Same failure still (with MAVEN_OPTS set as above and using
>>>>>>> jdk1.6.0_24).
>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>>
>>>>>>>>> Scott
>>>>>>>>>
>>>>>>>>> On 10/19/2011 10:49 AM, Jaikiran Pai wrote:
>>>>>>>>>> I don't know about the test issues that Ondrej is running into,
>>>>>>>>>> but the
>>>>>>>>>> ones you mention http://pastebin.com/D3SeJ2rP (OutOfMemory : Cannot
>>>>>>>>>> create native thread) were encountered once by Carlo (or on our
>>>>>>>>>> Hudson
>>>>>>>>>> instance, don't remember). But I haven't seen them after that. It
>>>>>>>>>> might
>>>>>>>>>> be environment specific. What JDK vendor/version? And are you
>>>>>>>>>> running
>>>>>>>>>> into this always?
>>>>>>>>>>
>>>>>>>>>> -Jaikiran
>>>>>>>>>> On Wednesday 19 October 2011 08:14 PM, Scott Marlow wrote:
>>>>>>>>>>> How did these testsuite failures slip in? From my own experience
>>>>>>>>>>> running the updated testsuite, it looks like we now have an
>>>>>>>>>>> allTests
>>>>>>>>>>> profile that doesn't run the integration tests. Perhaps we
>>>>>>>>>>> merged some
>>>>>>>>>>> changes and only ran the allTests profile. I think that we need to
>>>>>>>>>>> continue to use -DallTests instead (that includes the integration
>>>>>>>>>>> tests). Does anyone know more?
>>>>>>>>>>>
>>>>>>>>>>> I'm not sure why some people aren't seeing the failures but it
>>>>>>>>>>> seems we
>>>>>>>>>>> need to track backwards from the symptoms (listed below) or
>>>>>>>>>>> rollback
>>>>>>>>>>> some changes until it goes away.
>>>>>>>>>>>
>>>>>>>>>>> If anyone is working on tracking the cause down, please speak
>>>>>>>>>>> up. I can
>>>>>>>>>>> jump in but don't want to duplicate effort.
>>>>>>>>>>>
>>>>>>>>>>> Scott
>>>>>>>>>>>
>>>>>>>>>>> On 10/18/2011 02:15 PM, Scott Marlow wrote:
>>>>>>>>>>>> The hudson AS7 test just finished with all integration tests
>>>>>>>>>>>> passing.
>>>>>>>>>>>>
>>>>>>>>>>>> When you say that your using a fresh master, is that a new AS7
>>>>>>>>>>>> source
>>>>>>>>>>>> tree or an existing one freshly synced with AS7 master?
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On 10/18/2011 12:26 PM, Scott Marlow wrote:
>>>>>>>>>>>>> On 10/18/2011 08:57 AM, Ondřej Žižka wrote:
>>>>>>>>>>>>>> I have a fresh master and testsuite runs fine.
>>>>>>>>>>>>>> Except 2 failures (for which I also seek comments):
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> org.jboss.as.test.integration.osgi.webapp.ServletIntegrationTestCase.testServiceAccess
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> org.jboss.as.test.integration.osgi.jaxrs.RestEasyIntegrationTestCase.org.jboss.as.test.integration.osgi.jaxrs.RestEasyIntegrationTestCase
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> server.log:
>>>>>>>>>>>>>> http://pastebin.test.redhat.com/64949
>>>>>>>>>>>>>
>>>>>>>>>>>>> I didn't see that failure.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Which folder are you starting the unit test from and what is
>>>>>>>>>>>>> your
>>>>>>>>>>>>> command line?
>>>>>>>>>>>>>
>>>>>>>>>>>>> I was trying to use "mvn clean install -DallTests" in the
>>>>>>>>>>>>> as7/testsuite/integration folder.
>>>>>>>>>>>>>
>>>>>>>>>>>>> When I checked earlier, Hudson wasn't running the full test
>>>>>>>>>>>>> (was using
>>>>>>>>>>>>> -PallTests instead of -DallTests). Looks like someone changed
>>>>>>>>>>>>> that but
>>>>>>>>>>>>> the test is still in queue.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Anyone seeing that? I thought it could be a race condition
>>>>>>>>>>>>>> caused by
>>>>>>>>>>>>>> improper use of deployer API, but giving it some time after
>>>>>>>>>>>>>> deploy
>>>>>>>>>>>>>> didn't help.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Ondra
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Scott Marlow píše v Út 18. 10. 2011 v 08:36 -0400:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> When running the as7/testsuite/integration tests, we run
>>>>>>>>>>>>>>> fine for a
>>>>>>>>>>>>>>> while but then "OutOfMemoryError: unable to create new
>>>>>>>>>>>>>>> native thread"
>>>>>>>>>>>>>>> errors start showing up in the server.log.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Failing tests are here http://pastebin.com/D3SeJ2rP
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> AS7 thread dump showing a lot of "JMX server connection
>>>>>>>>>>>>>>> timeout NNN"
>>>>>>>>>>>>>>> threads is here http://pastebin.com/qDC52WJG
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> To recreate change into as7/testsuite/integration and do a
>>>>>>>>>>>>>>> "mvn clean
>>>>>>>>>>>>>>> install -DallTests".
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Is anyone not seeing this after syncing with master?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Scott
>>>
>>>
>>
>> _______________________________________________
>> jboss-as7-dev mailing list
>> [hidden email]
>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
>
> _______________________________________________
> jboss-as7-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev

_______________________________________________
jboss-as7-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
Reply | Threaded
Open this post in threaded view
|

Re: ARQ-632 and as7/testsuite/integration failures due to connection not getting closed...

Brian Stansberry
I think the AS/ARQ integration (ManagementClient class) can easily
enough cache and re-use the JMXContext and thus the
MBeanServerConnection+JMXConnector. I'll send you a patch that does that
in a bit.

On 10/20/11 3:22 PM, Scott Marlow wrote:

> It probably wasn't using the timeout setting added to
> as7/testsuite/integration/pom.xml. I changed the
> o.j.a.j.JMXConnectorService default to ten seconds and made it through
> the integration tests.
>
> If we change the default timeout to ten seconds (for a few days), that
> might cause test failures on busy/slower machines.
>
> On 10/20/2011 03:21 PM, Scott Marlow wrote:
>> No, I still see "java.lang.OutOfMemoryError: unable to create new native
>> thread". I'll try a shorter timeout, maybe ten seconds instead of thirty.
>>
>> Another workaround would be reducing the number of tests in
>> testsuite/integration (at least until ARQ-632 is fixed).
>>
>> On 10/20/2011 02:58 PM, Scott Marlow wrote:
>>> I'll give that a try.
>>>
>>> On 10/20/2011 02:36 PM, Brian Stansberry wrote:
>>>> I'm not seeing this failure, so it's hard to test a workaround. But if
>>>> someone who is seeing it can give
>>>> https://github.com/bstansberry/jboss-as/commit/0d422b184760b95e47c886f139426b512341a333
>>>>
>>>> a spin and let me know, I'd appreciate it.
>>>>
>>>> Probably easiest is just to check out and build
>>>> https://github.com/bstansberry/jboss-as/tree/jmx-conn-leak
>>>>
>>>> On 10/20/11 11:49 AM, Brian Stansberry wrote:
>>>>> See [1] for a good discussion of this; last post points at a possible
>>>>> workaround.
>>>>>
>>>>> [1] https://forums.oracle.com/forums/thread.jspa?threadID=1665966
>>>>> On 10/19/11 6:26 PM, Stuart Douglas wrote:
>>>>>> It is probably to do with the number of tests in the test suite, as
>>>>>> now all the tests that were previously in preview and spec are
>>>>>> executed in one run.
>>>>>>
>>>>>> Stuart
>>>>>>
>>>>>>
>>>>>> On 20/10/2011, at 10:24 AM, Scott Marlow wrote:
>>>>>>
>>>>>>> I agree that we should be closing the connection
>>>>>>> (https://anonsvn.jboss.org/repos/jbossas/trunk/console/src/main/java/org/jboss/console/twiddle/Twiddle.java
>>>>>>>
>>>>>>>
>>>>>>> is an example of where we handled that in AS6). Probably after we
>>>>>>> are
>>>>>>> done with the MBeanServerConnection.
>>>>>>>
>>>>>>> I created ARQ-632 for this.
>>>>>>>
>>>>>>> I was expecting the cause to be a regression but instead perhaps its
>>>>>>> the performance improvements (we are running unit tests quicker,
>>>>>>> causing more "jmx server connection timeout" threads to be created).
>>>>>>> Or, due to the number of tests in the testsuite now.
>>>>>>>
>>>>>>> ----- Original Message -----
>>>>>>> From: "Brian Stansberry"<[hidden email]>
>>>>>>> To: [hidden email]
>>>>>>> Sent: Wednesday, October 19, 2011 5:47:41 PM
>>>>>>> Subject: Re: [jboss-as7-dev] consistent testsuite failures since I
>>>>>>> synced with master yesterday
>>>>>>>
>>>>>>> org.jboss.arquillian.container.spi.client.protocol.metadata.JMXContext.getConnection()
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> is creating a javax.management.remote.JMXConnector but I don't see
>>>>>>> anything closing it. I'm not sure, but perhaps one of those is
>>>>>>> created
>>>>>>> per test method execution?
>>>>>>>
>>>>>>> I believe the "JMX server connection timeout 552" threads are
>>>>>>> created by
>>>>>>> the JDK stuff AS7 is using for its JMX connector impl. You can
>>>>>>> see the
>>>>>>> relevant code in the stack trace. A quick look at that code leads
>>>>>>> me to
>>>>>>> believe those threads will by default live for 120 secs after the
>>>>>>> last
>>>>>>> request on the connection.
>>>>>>>
>>>>>>> On 10/19/11 11:54 AM, Scott Marlow wrote:
>>>>>>>> On 10/19/2011 12:13 PM, Scott Marlow wrote:
>>>>>>>>> On 10/19/2011 11:01 AM, Scott Marlow wrote:
>>>>>>>>>> I'm using Sun 1.6.0_26-b03. I think Hudson is passing with Sun
>>>>>>>>>> 1.6.0_24-b07.
>>>>>>>>>>
>>>>>>>>>> Jesper also got the failures with jdk 6 u26.
>>>>>>>>>>
>>>>>>>>>> I haven't tried running with my old jdk1.6.0_24 yet. I could try
>>>>>>>>>> that
>>>>>>>>>> or we could start comparing environment settings.
>>>>>>>>>>
>>>>>>>>>> On Hudson, we are setting MAVEN_OPTS="-Xmx1024m -Xms512m
>>>>>>>>>> -XX:MaxPermSize=128m"
>>>>>>>>>>
>>>>>>>>>> I don't have that set locally. I'll try that next...
>>>>>>>>>
>>>>>>>>> Testsuite still fails for me with the above MAVEN_OPTS set.
>>>>>>>>>
>>>>>>>>> I'll my older jdk 6 (same as hudson is using) and see if that
>>>>>>>>> makes
>>>>>>>>> a diff.
>>>>>>>>
>>>>>>>> Same failure still (with MAVEN_OPTS set as above and using
>>>>>>>> jdk1.6.0_24).
>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Scott
>>>>>>>>>>
>>>>>>>>>> On 10/19/2011 10:49 AM, Jaikiran Pai wrote:
>>>>>>>>>>> I don't know about the test issues that Ondrej is running into,
>>>>>>>>>>> but the
>>>>>>>>>>> ones you mention http://pastebin.com/D3SeJ2rP (OutOfMemory :
>>>>>>>>>>> Cannot
>>>>>>>>>>> create native thread) were encountered once by Carlo (or on our
>>>>>>>>>>> Hudson
>>>>>>>>>>> instance, don't remember). But I haven't seen them after
>>>>>>>>>>> that. It
>>>>>>>>>>> might
>>>>>>>>>>> be environment specific. What JDK vendor/version? And are you
>>>>>>>>>>> running
>>>>>>>>>>> into this always?
>>>>>>>>>>>
>>>>>>>>>>> -Jaikiran
>>>>>>>>>>> On Wednesday 19 October 2011 08:14 PM, Scott Marlow wrote:
>>>>>>>>>>>> How did these testsuite failures slip in? From my own
>>>>>>>>>>>> experience
>>>>>>>>>>>> running the updated testsuite, it looks like we now have an
>>>>>>>>>>>> allTests
>>>>>>>>>>>> profile that doesn't run the integration tests. Perhaps we
>>>>>>>>>>>> merged some
>>>>>>>>>>>> changes and only ran the allTests profile. I think that we
>>>>>>>>>>>> need to
>>>>>>>>>>>> continue to use -DallTests instead (that includes the
>>>>>>>>>>>> integration
>>>>>>>>>>>> tests). Does anyone know more?
>>>>>>>>>>>>
>>>>>>>>>>>> I'm not sure why some people aren't seeing the failures but it
>>>>>>>>>>>> seems we
>>>>>>>>>>>> need to track backwards from the symptoms (listed below) or
>>>>>>>>>>>> rollback
>>>>>>>>>>>> some changes until it goes away.
>>>>>>>>>>>>
>>>>>>>>>>>> If anyone is working on tracking the cause down, please speak
>>>>>>>>>>>> up. I can
>>>>>>>>>>>> jump in but don't want to duplicate effort.
>>>>>>>>>>>>
>>>>>>>>>>>> Scott
>>>>>>>>>>>>
>>>>>>>>>>>> On 10/18/2011 02:15 PM, Scott Marlow wrote:
>>>>>>>>>>>>> The hudson AS7 test just finished with all integration tests
>>>>>>>>>>>>> passing.
>>>>>>>>>>>>>
>>>>>>>>>>>>> When you say that your using a fresh master, is that a new AS7
>>>>>>>>>>>>> source
>>>>>>>>>>>>> tree or an existing one freshly synced with AS7 master?
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On 10/18/2011 12:26 PM, Scott Marlow wrote:
>>>>>>>>>>>>>> On 10/18/2011 08:57 AM, Ondřej Žižka wrote:
>>>>>>>>>>>>>>> I have a fresh master and testsuite runs fine.
>>>>>>>>>>>>>>> Except 2 failures (for which I also seek comments):
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> org.jboss.as.test.integration.osgi.webapp.ServletIntegrationTestCase.testServiceAccess
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> org.jboss.as.test.integration.osgi.jaxrs.RestEasyIntegrationTestCase.org.jboss.as.test.integration.osgi.jaxrs.RestEasyIntegrationTestCase
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> server.log:
>>>>>>>>>>>>>>> http://pastebin.test.redhat.com/64949
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I didn't see that failure.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Which folder are you starting the unit test from and what is
>>>>>>>>>>>>>> your
>>>>>>>>>>>>>> command line?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I was trying to use "mvn clean install -DallTests" in the
>>>>>>>>>>>>>> as7/testsuite/integration folder.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> When I checked earlier, Hudson wasn't running the full test
>>>>>>>>>>>>>> (was using
>>>>>>>>>>>>>> -PallTests instead of -DallTests). Looks like someone changed
>>>>>>>>>>>>>> that but
>>>>>>>>>>>>>> the test is still in queue.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Anyone seeing that? I thought it could be a race condition
>>>>>>>>>>>>>>> caused by
>>>>>>>>>>>>>>> improper use of deployer API, but giving it some time after
>>>>>>>>>>>>>>> deploy
>>>>>>>>>>>>>>> didn't help.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Ondra
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Scott Marlow píše v Út 18. 10. 2011 v 08:36 -0400:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> When running the as7/testsuite/integration tests, we run
>>>>>>>>>>>>>>>> fine for a
>>>>>>>>>>>>>>>> while but then "OutOfMemoryError: unable to create new
>>>>>>>>>>>>>>>> native thread"
>>>>>>>>>>>>>>>> errors start showing up in the server.log.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Failing tests are here http://pastebin.com/D3SeJ2rP
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> AS7 thread dump showing a lot of "JMX server connection
>>>>>>>>>>>>>>>> timeout NNN"
>>>>>>>>>>>>>>>> threads is here http://pastebin.com/qDC52WJG
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> To recreate change into as7/testsuite/integration and do a
>>>>>>>>>>>>>>>> "mvn clean
>>>>>>>>>>>>>>>> install -DallTests".
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Is anyone not seeing this after syncing with master?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Scott
>>>>
>>>>
>>>
>>> _______________________________________________
>>> jboss-as7-dev mailing list
>>> [hidden email]
>>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
>>
>> _______________________________________________
>> jboss-as7-dev mailing list
>> [hidden email]
>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
>


--
Brian Stansberry
Principal Software Engineer
JBoss by Red Hat
_______________________________________________
jboss-as7-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
Reply | Threaded
Open this post in threaded view
|

Re: ARQ-632 and as7/testsuite/integration failures due to connection not getting closed...

Brian Stansberry
Can you try this:

https://github.com/bstansberry/jboss-as/commit/a3dc2b2730c6c0aedf36257470d0ea8281802088

It's the right thing to do anyway, as it should improve testsuite perf a
fair bit by not opening connections all the time.

On 10/20/11 3:43 PM, Brian Stansberry wrote:

> I think the AS/ARQ integration (ManagementClient class) can easily
> enough cache and re-use the JMXContext and thus the
> MBeanServerConnection+JMXConnector. I'll send you a patch that does that
> in a bit.
>
> On 10/20/11 3:22 PM, Scott Marlow wrote:
>> It probably wasn't using the timeout setting added to
>> as7/testsuite/integration/pom.xml. I changed the
>> o.j.a.j.JMXConnectorService default to ten seconds and made it through
>> the integration tests.
>>
>> If we change the default timeout to ten seconds (for a few days), that
>> might cause test failures on busy/slower machines.
>>
>> On 10/20/2011 03:21 PM, Scott Marlow wrote:
>>> No, I still see "java.lang.OutOfMemoryError: unable to create new native
>>> thread". I'll try a shorter timeout, maybe ten seconds instead of
>>> thirty.
>>>
>>> Another workaround would be reducing the number of tests in
>>> testsuite/integration (at least until ARQ-632 is fixed).
>>>
>>> On 10/20/2011 02:58 PM, Scott Marlow wrote:
>>>> I'll give that a try.
>>>>
>>>> On 10/20/2011 02:36 PM, Brian Stansberry wrote:
>>>>> I'm not seeing this failure, so it's hard to test a workaround. But if
>>>>> someone who is seeing it can give
>>>>> https://github.com/bstansberry/jboss-as/commit/0d422b184760b95e47c886f139426b512341a333
>>>>>
>>>>>
>>>>> a spin and let me know, I'd appreciate it.
>>>>>
>>>>> Probably easiest is just to check out and build
>>>>> https://github.com/bstansberry/jboss-as/tree/jmx-conn-leak
>>>>>
>>>>> On 10/20/11 11:49 AM, Brian Stansberry wrote:
>>>>>> See [1] for a good discussion of this; last post points at a possible
>>>>>> workaround.
>>>>>>
>>>>>> [1] https://forums.oracle.com/forums/thread.jspa?threadID=1665966
>>>>>> On 10/19/11 6:26 PM, Stuart Douglas wrote:
>>>>>>> It is probably to do with the number of tests in the test suite, as
>>>>>>> now all the tests that were previously in preview and spec are
>>>>>>> executed in one run.
>>>>>>>
>>>>>>> Stuart
>>>>>>>
>>>>>>>
>>>>>>> On 20/10/2011, at 10:24 AM, Scott Marlow wrote:
>>>>>>>
>>>>>>>> I agree that we should be closing the connection
>>>>>>>> (https://anonsvn.jboss.org/repos/jbossas/trunk/console/src/main/java/org/jboss/console/twiddle/Twiddle.java
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> is an example of where we handled that in AS6). Probably after we
>>>>>>>> are
>>>>>>>> done with the MBeanServerConnection.
>>>>>>>>
>>>>>>>> I created ARQ-632 for this.
>>>>>>>>
>>>>>>>> I was expecting the cause to be a regression but instead perhaps
>>>>>>>> its
>>>>>>>> the performance improvements (we are running unit tests quicker,
>>>>>>>> causing more "jmx server connection timeout" threads to be
>>>>>>>> created).
>>>>>>>> Or, due to the number of tests in the testsuite now.
>>>>>>>>
>>>>>>>> ----- Original Message -----
>>>>>>>> From: "Brian Stansberry"<[hidden email]>
>>>>>>>> To: [hidden email]
>>>>>>>> Sent: Wednesday, October 19, 2011 5:47:41 PM
>>>>>>>> Subject: Re: [jboss-as7-dev] consistent testsuite failures since I
>>>>>>>> synced with master yesterday
>>>>>>>>
>>>>>>>> org.jboss.arquillian.container.spi.client.protocol.metadata.JMXContext.getConnection()
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> is creating a javax.management.remote.JMXConnector but I don't see
>>>>>>>> anything closing it. I'm not sure, but perhaps one of those is
>>>>>>>> created
>>>>>>>> per test method execution?
>>>>>>>>
>>>>>>>> I believe the "JMX server connection timeout 552" threads are
>>>>>>>> created by
>>>>>>>> the JDK stuff AS7 is using for its JMX connector impl. You can
>>>>>>>> see the
>>>>>>>> relevant code in the stack trace. A quick look at that code leads
>>>>>>>> me to
>>>>>>>> believe those threads will by default live for 120 secs after the
>>>>>>>> last
>>>>>>>> request on the connection.
>>>>>>>>
>>>>>>>> On 10/19/11 11:54 AM, Scott Marlow wrote:
>>>>>>>>> On 10/19/2011 12:13 PM, Scott Marlow wrote:
>>>>>>>>>> On 10/19/2011 11:01 AM, Scott Marlow wrote:
>>>>>>>>>>> I'm using Sun 1.6.0_26-b03. I think Hudson is passing with Sun
>>>>>>>>>>> 1.6.0_24-b07.
>>>>>>>>>>>
>>>>>>>>>>> Jesper also got the failures with jdk 6 u26.
>>>>>>>>>>>
>>>>>>>>>>> I haven't tried running with my old jdk1.6.0_24 yet. I could try
>>>>>>>>>>> that
>>>>>>>>>>> or we could start comparing environment settings.
>>>>>>>>>>>
>>>>>>>>>>> On Hudson, we are setting MAVEN_OPTS="-Xmx1024m -Xms512m
>>>>>>>>>>> -XX:MaxPermSize=128m"
>>>>>>>>>>>
>>>>>>>>>>> I don't have that set locally. I'll try that next...
>>>>>>>>>>
>>>>>>>>>> Testsuite still fails for me with the above MAVEN_OPTS set.
>>>>>>>>>>
>>>>>>>>>> I'll my older jdk 6 (same as hudson is using) and see if that
>>>>>>>>>> makes
>>>>>>>>>> a diff.
>>>>>>>>>
>>>>>>>>> Same failure still (with MAVEN_OPTS set as above and using
>>>>>>>>> jdk1.6.0_24).
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Scott
>>>>>>>>>>>
>>>>>>>>>>> On 10/19/2011 10:49 AM, Jaikiran Pai wrote:
>>>>>>>>>>>> I don't know about the test issues that Ondrej is running into,
>>>>>>>>>>>> but the
>>>>>>>>>>>> ones you mention http://pastebin.com/D3SeJ2rP (OutOfMemory :
>>>>>>>>>>>> Cannot
>>>>>>>>>>>> create native thread) were encountered once by Carlo (or on our
>>>>>>>>>>>> Hudson
>>>>>>>>>>>> instance, don't remember). But I haven't seen them after
>>>>>>>>>>>> that. It
>>>>>>>>>>>> might
>>>>>>>>>>>> be environment specific. What JDK vendor/version? And are you
>>>>>>>>>>>> running
>>>>>>>>>>>> into this always?
>>>>>>>>>>>>
>>>>>>>>>>>> -Jaikiran
>>>>>>>>>>>> On Wednesday 19 October 2011 08:14 PM, Scott Marlow wrote:
>>>>>>>>>>>>> How did these testsuite failures slip in? From my own
>>>>>>>>>>>>> experience
>>>>>>>>>>>>> running the updated testsuite, it looks like we now have an
>>>>>>>>>>>>> allTests
>>>>>>>>>>>>> profile that doesn't run the integration tests. Perhaps we
>>>>>>>>>>>>> merged some
>>>>>>>>>>>>> changes and only ran the allTests profile. I think that we
>>>>>>>>>>>>> need to
>>>>>>>>>>>>> continue to use -DallTests instead (that includes the
>>>>>>>>>>>>> integration
>>>>>>>>>>>>> tests). Does anyone know more?
>>>>>>>>>>>>>
>>>>>>>>>>>>> I'm not sure why some people aren't seeing the failures but it
>>>>>>>>>>>>> seems we
>>>>>>>>>>>>> need to track backwards from the symptoms (listed below) or
>>>>>>>>>>>>> rollback
>>>>>>>>>>>>> some changes until it goes away.
>>>>>>>>>>>>>
>>>>>>>>>>>>> If anyone is working on tracking the cause down, please speak
>>>>>>>>>>>>> up. I can
>>>>>>>>>>>>> jump in but don't want to duplicate effort.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Scott
>>>>>>>>>>>>>
>>>>>>>>>>>>> On 10/18/2011 02:15 PM, Scott Marlow wrote:
>>>>>>>>>>>>>> The hudson AS7 test just finished with all integration tests
>>>>>>>>>>>>>> passing.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> When you say that your using a fresh master, is that a new
>>>>>>>>>>>>>> AS7
>>>>>>>>>>>>>> source
>>>>>>>>>>>>>> tree or an existing one freshly synced with AS7 master?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On 10/18/2011 12:26 PM, Scott Marlow wrote:
>>>>>>>>>>>>>>> On 10/18/2011 08:57 AM, Ondřej Žižka wrote:
>>>>>>>>>>>>>>>> I have a fresh master and testsuite runs fine.
>>>>>>>>>>>>>>>> Except 2 failures (for which I also seek comments):
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> org.jboss.as.test.integration.osgi.webapp.ServletIntegrationTestCase.testServiceAccess
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> org.jboss.as.test.integration.osgi.jaxrs.RestEasyIntegrationTestCase.org.jboss.as.test.integration.osgi.jaxrs.RestEasyIntegrationTestCase
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> server.log:
>>>>>>>>>>>>>>>> http://pastebin.test.redhat.com/64949
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I didn't see that failure.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Which folder are you starting the unit test from and what is
>>>>>>>>>>>>>>> your
>>>>>>>>>>>>>>> command line?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I was trying to use "mvn clean install -DallTests" in the
>>>>>>>>>>>>>>> as7/testsuite/integration folder.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> When I checked earlier, Hudson wasn't running the full test
>>>>>>>>>>>>>>> (was using
>>>>>>>>>>>>>>> -PallTests instead of -DallTests). Looks like someone
>>>>>>>>>>>>>>> changed
>>>>>>>>>>>>>>> that but
>>>>>>>>>>>>>>> the test is still in queue.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Anyone seeing that? I thought it could be a race condition
>>>>>>>>>>>>>>>> caused by
>>>>>>>>>>>>>>>> improper use of deployer API, but giving it some time after
>>>>>>>>>>>>>>>> deploy
>>>>>>>>>>>>>>>> didn't help.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Ondra
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Scott Marlow píše v Út 18. 10. 2011 v 08:36 -0400:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> When running the as7/testsuite/integration tests, we run
>>>>>>>>>>>>>>>>> fine for a
>>>>>>>>>>>>>>>>> while but then "OutOfMemoryError: unable to create new
>>>>>>>>>>>>>>>>> native thread"
>>>>>>>>>>>>>>>>> errors start showing up in the server.log.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Failing tests are here http://pastebin.com/D3SeJ2rP
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> AS7 thread dump showing a lot of "JMX server connection
>>>>>>>>>>>>>>>>> timeout NNN"
>>>>>>>>>>>>>>>>> threads is here http://pastebin.com/qDC52WJG
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> To recreate change into as7/testsuite/integration and do a
>>>>>>>>>>>>>>>>> "mvn clean
>>>>>>>>>>>>>>>>> install -DallTests".
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Is anyone not seeing this after syncing with master?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Scott
>>>>>
>>>>>
>>>>
>>>> _______________________________________________
>>>> jboss-as7-dev mailing list
>>>> [hidden email]
>>>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
>>>
>>> _______________________________________________
>>> jboss-as7-dev mailing list
>>> [hidden email]
>>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
>>
>
>


--
Brian Stansberry
Principal Software Engineer
JBoss by Red Hat
_______________________________________________
jboss-as7-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
Reply | Threaded
Open this post in threaded view
|

Re: ARQ-632 and as7/testsuite/integration failures due to connection not getting closed...

Scott Marlow
On 10/20/2011 05:42 PM, Brian Stansberry wrote:
> Can you try this:
>
> https://github.com/bstansberry/jboss-as/commit/a3dc2b2730c6c0aedf36257470d0ea8281802088

I updated my local copy of your branch and gave it a whirl.  The
integration tests passed!  Nice patch!  :)


>
>
> It's the right thing to do anyway, as it should improve testsuite perf a
> fair bit by not opening connections all the time.
>
> On 10/20/11 3:43 PM, Brian Stansberry wrote:
>> I think the AS/ARQ integration (ManagementClient class) can easily
>> enough cache and re-use the JMXContext and thus the
>> MBeanServerConnection+JMXConnector. I'll send you a patch that does that
>> in a bit.
>>
>> On 10/20/11 3:22 PM, Scott Marlow wrote:
>>> It probably wasn't using the timeout setting added to
>>> as7/testsuite/integration/pom.xml. I changed the
>>> o.j.a.j.JMXConnectorService default to ten seconds and made it through
>>> the integration tests.
>>>
>>> If we change the default timeout to ten seconds (for a few days), that
>>> might cause test failures on busy/slower machines.
>>>
>>> On 10/20/2011 03:21 PM, Scott Marlow wrote:
>>>> No, I still see "java.lang.OutOfMemoryError: unable to create new
>>>> native
>>>> thread". I'll try a shorter timeout, maybe ten seconds instead of
>>>> thirty.
>>>>
>>>> Another workaround would be reducing the number of tests in
>>>> testsuite/integration (at least until ARQ-632 is fixed).
>>>>
>>>> On 10/20/2011 02:58 PM, Scott Marlow wrote:
>>>>> I'll give that a try.
>>>>>
>>>>> On 10/20/2011 02:36 PM, Brian Stansberry wrote:
>>>>>> I'm not seeing this failure, so it's hard to test a workaround.
>>>>>> But if
>>>>>> someone who is seeing it can give
>>>>>> https://github.com/bstansberry/jboss-as/commit/0d422b184760b95e47c886f139426b512341a333
>>>>>>
>>>>>>
>>>>>>
>>>>>> a spin and let me know, I'd appreciate it.
>>>>>>
>>>>>> Probably easiest is just to check out and build
>>>>>> https://github.com/bstansberry/jboss-as/tree/jmx-conn-leak
>>>>>>
>>>>>> On 10/20/11 11:49 AM, Brian Stansberry wrote:
>>>>>>> See [1] for a good discussion of this; last post points at a
>>>>>>> possible
>>>>>>> workaround.
>>>>>>>
>>>>>>> [1] https://forums.oracle.com/forums/thread.jspa?threadID=1665966
>>>>>>> On 10/19/11 6:26 PM, Stuart Douglas wrote:
>>>>>>>> It is probably to do with the number of tests in the test suite, as
>>>>>>>> now all the tests that were previously in preview and spec are
>>>>>>>> executed in one run.
>>>>>>>>
>>>>>>>> Stuart
>>>>>>>>
>>>>>>>>
>>>>>>>> On 20/10/2011, at 10:24 AM, Scott Marlow wrote:
>>>>>>>>
>>>>>>>>> I agree that we should be closing the connection
>>>>>>>>> (https://anonsvn.jboss.org/repos/jbossas/trunk/console/src/main/java/org/jboss/console/twiddle/Twiddle.java
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> is an example of where we handled that in AS6). Probably after we
>>>>>>>>> are
>>>>>>>>> done with the MBeanServerConnection.
>>>>>>>>>
>>>>>>>>> I created ARQ-632 for this.
>>>>>>>>>
>>>>>>>>> I was expecting the cause to be a regression but instead perhaps
>>>>>>>>> its
>>>>>>>>> the performance improvements (we are running unit tests quicker,
>>>>>>>>> causing more "jmx server connection timeout" threads to be
>>>>>>>>> created).
>>>>>>>>> Or, due to the number of tests in the testsuite now.
>>>>>>>>>
>>>>>>>>> ----- Original Message -----
>>>>>>>>> From: "Brian Stansberry"<[hidden email]>
>>>>>>>>> To: [hidden email]
>>>>>>>>> Sent: Wednesday, October 19, 2011 5:47:41 PM
>>>>>>>>> Subject: Re: [jboss-as7-dev] consistent testsuite failures since I
>>>>>>>>> synced with master yesterday
>>>>>>>>>
>>>>>>>>> org.jboss.arquillian.container.spi.client.protocol.metadata.JMXContext.getConnection()
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> is creating a javax.management.remote.JMXConnector but I don't see
>>>>>>>>> anything closing it. I'm not sure, but perhaps one of those is
>>>>>>>>> created
>>>>>>>>> per test method execution?
>>>>>>>>>
>>>>>>>>> I believe the "JMX server connection timeout 552" threads are
>>>>>>>>> created by
>>>>>>>>> the JDK stuff AS7 is using for its JMX connector impl. You can
>>>>>>>>> see the
>>>>>>>>> relevant code in the stack trace. A quick look at that code leads
>>>>>>>>> me to
>>>>>>>>> believe those threads will by default live for 120 secs after the
>>>>>>>>> last
>>>>>>>>> request on the connection.
>>>>>>>>>
>>>>>>>>> On 10/19/11 11:54 AM, Scott Marlow wrote:
>>>>>>>>>> On 10/19/2011 12:13 PM, Scott Marlow wrote:
>>>>>>>>>>> On 10/19/2011 11:01 AM, Scott Marlow wrote:
>>>>>>>>>>>> I'm using Sun 1.6.0_26-b03. I think Hudson is passing with Sun
>>>>>>>>>>>> 1.6.0_24-b07.
>>>>>>>>>>>>
>>>>>>>>>>>> Jesper also got the failures with jdk 6 u26.
>>>>>>>>>>>>
>>>>>>>>>>>> I haven't tried running with my old jdk1.6.0_24 yet. I could
>>>>>>>>>>>> try
>>>>>>>>>>>> that
>>>>>>>>>>>> or we could start comparing environment settings.
>>>>>>>>>>>>
>>>>>>>>>>>> On Hudson, we are setting MAVEN_OPTS="-Xmx1024m -Xms512m
>>>>>>>>>>>> -XX:MaxPermSize=128m"
>>>>>>>>>>>>
>>>>>>>>>>>> I don't have that set locally. I'll try that next...
>>>>>>>>>>>
>>>>>>>>>>> Testsuite still fails for me with the above MAVEN_OPTS set.
>>>>>>>>>>>
>>>>>>>>>>> I'll my older jdk 6 (same as hudson is using) and see if that
>>>>>>>>>>> makes
>>>>>>>>>>> a diff.
>>>>>>>>>>
>>>>>>>>>> Same failure still (with MAVEN_OPTS set as above and using
>>>>>>>>>> jdk1.6.0_24).
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Scott
>>>>>>>>>>>>
>>>>>>>>>>>> On 10/19/2011 10:49 AM, Jaikiran Pai wrote:
>>>>>>>>>>>>> I don't know about the test issues that Ondrej is running
>>>>>>>>>>>>> into,
>>>>>>>>>>>>> but the
>>>>>>>>>>>>> ones you mention http://pastebin.com/D3SeJ2rP (OutOfMemory :
>>>>>>>>>>>>> Cannot
>>>>>>>>>>>>> create native thread) were encountered once by Carlo (or on
>>>>>>>>>>>>> our
>>>>>>>>>>>>> Hudson
>>>>>>>>>>>>> instance, don't remember). But I haven't seen them after
>>>>>>>>>>>>> that. It
>>>>>>>>>>>>> might
>>>>>>>>>>>>> be environment specific. What JDK vendor/version? And are you
>>>>>>>>>>>>> running
>>>>>>>>>>>>> into this always?
>>>>>>>>>>>>>
>>>>>>>>>>>>> -Jaikiran
>>>>>>>>>>>>> On Wednesday 19 October 2011 08:14 PM, Scott Marlow wrote:
>>>>>>>>>>>>>> How did these testsuite failures slip in? From my own
>>>>>>>>>>>>>> experience
>>>>>>>>>>>>>> running the updated testsuite, it looks like we now have an
>>>>>>>>>>>>>> allTests
>>>>>>>>>>>>>> profile that doesn't run the integration tests. Perhaps we
>>>>>>>>>>>>>> merged some
>>>>>>>>>>>>>> changes and only ran the allTests profile. I think that we
>>>>>>>>>>>>>> need to
>>>>>>>>>>>>>> continue to use -DallTests instead (that includes the
>>>>>>>>>>>>>> integration
>>>>>>>>>>>>>> tests). Does anyone know more?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I'm not sure why some people aren't seeing the failures
>>>>>>>>>>>>>> but it
>>>>>>>>>>>>>> seems we
>>>>>>>>>>>>>> need to track backwards from the symptoms (listed below) or
>>>>>>>>>>>>>> rollback
>>>>>>>>>>>>>> some changes until it goes away.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> If anyone is working on tracking the cause down, please speak
>>>>>>>>>>>>>> up. I can
>>>>>>>>>>>>>> jump in but don't want to duplicate effort.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Scott
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On 10/18/2011 02:15 PM, Scott Marlow wrote:
>>>>>>>>>>>>>>> The hudson AS7 test just finished with all integration tests
>>>>>>>>>>>>>>> passing.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> When you say that your using a fresh master, is that a new
>>>>>>>>>>>>>>> AS7
>>>>>>>>>>>>>>> source
>>>>>>>>>>>>>>> tree or an existing one freshly synced with AS7 master?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On 10/18/2011 12:26 PM, Scott Marlow wrote:
>>>>>>>>>>>>>>>> On 10/18/2011 08:57 AM, Ondřej Žižka wrote:
>>>>>>>>>>>>>>>>> I have a fresh master and testsuite runs fine.
>>>>>>>>>>>>>>>>> Except 2 failures (for which I also seek comments):
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> org.jboss.as.test.integration.osgi.webapp.ServletIntegrationTestCase.testServiceAccess
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> org.jboss.as.test.integration.osgi.jaxrs.RestEasyIntegrationTestCase.org.jboss.as.test.integration.osgi.jaxrs.RestEasyIntegrationTestCase
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> server.log:
>>>>>>>>>>>>>>>>> http://pastebin.test.redhat.com/64949
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I didn't see that failure.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Which folder are you starting the unit test from and
>>>>>>>>>>>>>>>> what is
>>>>>>>>>>>>>>>> your
>>>>>>>>>>>>>>>> command line?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I was trying to use "mvn clean install -DallTests" in the
>>>>>>>>>>>>>>>> as7/testsuite/integration folder.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> When I checked earlier, Hudson wasn't running the full test
>>>>>>>>>>>>>>>> (was using
>>>>>>>>>>>>>>>> -PallTests instead of -DallTests). Looks like someone
>>>>>>>>>>>>>>>> changed
>>>>>>>>>>>>>>>> that but
>>>>>>>>>>>>>>>> the test is still in queue.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Anyone seeing that? I thought it could be a race condition
>>>>>>>>>>>>>>>>> caused by
>>>>>>>>>>>>>>>>> improper use of deployer API, but giving it some time
>>>>>>>>>>>>>>>>> after
>>>>>>>>>>>>>>>>> deploy
>>>>>>>>>>>>>>>>> didn't help.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Ondra
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Scott Marlow píše v Út 18. 10. 2011 v 08:36 -0400:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> When running the as7/testsuite/integration tests, we run
>>>>>>>>>>>>>>>>>> fine for a
>>>>>>>>>>>>>>>>>> while but then "OutOfMemoryError: unable to create new
>>>>>>>>>>>>>>>>>> native thread"
>>>>>>>>>>>>>>>>>> errors start showing up in the server.log.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Failing tests are here http://pastebin.com/D3SeJ2rP
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> AS7 thread dump showing a lot of "JMX server connection
>>>>>>>>>>>>>>>>>> timeout NNN"
>>>>>>>>>>>>>>>>>> threads is here http://pastebin.com/qDC52WJG
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> To recreate change into as7/testsuite/integration and
>>>>>>>>>>>>>>>>>> do a
>>>>>>>>>>>>>>>>>> "mvn clean
>>>>>>>>>>>>>>>>>> install -DallTests".
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Is anyone not seeing this after syncing with master?
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Scott
>>>>>>
>>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> jboss-as7-dev mailing list
>>>>> [hidden email]
>>>>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
>>>>
>>>> _______________________________________________
>>>> jboss-as7-dev mailing list
>>>> [hidden email]
>>>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
>>>
>>
>>
>
>

_______________________________________________
jboss-as7-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
12