consistent testsuite failures since I synced with master yesterday

classic Classic list List threaded Threaded
25 messages Options
12
Reply | Threaded
Open this post in threaded view
|

consistent testsuite failures since I synced with master yesterday

Scott Marlow
When running the as7/testsuite/integration tests, we run fine for a
while but then "OutOfMemoryError: unable to create new native thread"
errors start showing up in the server.log.

Failing tests are here http://pastebin.com/D3SeJ2rP

AS7 thread dump showing a lot of "JMX server connection timeout NNN"
threads is here http://pastebin.com/qDC52WJG

To recreate change into as7/testsuite/integration and do a "mvn clean
install -DallTests".

Is anyone not seeing this after syncing with master?

Scott
_______________________________________________
jboss-as7-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
Reply | Threaded
Open this post in threaded view
|

Re: consistent testsuite failures since I synced with master yesterday

Ondrej Zizka
I have a fresh master and testsuite runs fine.
Except 2 failures (for which I also seek comments):

org.jboss.as.test.integration.osgi.webapp.ServletIntegrationTestCase.testServiceAccess
org.jboss.as.test.integration.osgi.jaxrs.RestEasyIntegrationTestCase.org.jboss.as.test.integration.osgi.jaxrs.RestEasyIntegrationTestCase

server.log:
http://pastebin.test.redhat.com/64949

Anyone seeing that? I thought it could be a race condition caused by improper use of deployer API, but giving it some time after deploy didn't help.

Ondra



Scott Marlow píše v Út 18. 10. 2011 v 08:36 -0400:
When running the as7/testsuite/integration tests, we run fine for a 
while but then "OutOfMemoryError: unable to create new native thread" 
errors start showing up in the server.log.

Failing tests are here http://pastebin.com/D3SeJ2rP

AS7 thread dump showing a lot of "JMX server connection timeout NNN" 
threads is here http://pastebin.com/qDC52WJG

To recreate change into as7/testsuite/integration and do a "mvn clean 
install -DallTests".

Is anyone not seeing this after syncing with master?

Scott
_______________________________________________
jboss-as7-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/jboss-as7-dev

_______________________________________________
jboss-as7-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
Reply | Threaded
Open this post in threaded view
|

Re: consistent testsuite failures since I synced with master yesterday

Scott Marlow
I couldn't tell if hudson is getting the same failures as we don't
appear to be running the testsuite/integration tests (judging by the
https://hudson.qa.jboss.com/hudson/view/JBoss%20AS/job/JBoss-AS-7.0.x-testSuite-sun16/776/console 
partial output).

The hudson job appears to use this command line:

./build.sh clean install -PallTests -Dskip-download-sources -U -B
-Dmaven.test.failure.ignore=true

Locally, I'm using -DallTests which is different than using the allTests
profile (which doesn't seem to run the integration tests).


On 10/18/2011 08:57 AM, Ondřej Žižka wrote:

> I have a fresh master and testsuite runs fine.
> Except 2 failures (for which I also seek comments):
>
> org.jboss.as.test.integration.osgi.webapp.ServletIntegrationTestCase.testServiceAccess
> org.jboss.as.test.integration.osgi.jaxrs.RestEasyIntegrationTestCase.org.jboss.as.test.integration.osgi.jaxrs.RestEasyIntegrationTestCase
>
> server.log:
> http://pastebin.test.redhat.com/64949
>
> Anyone seeing that? I thought it could be a race condition caused by
> improper use of deployer API, but giving it some time after deploy
> didn't help.
>
> Ondra
>
>
>
> Scott Marlow píše v Út 18. 10. 2011 v 08:36 -0400:
>
>> When running the as7/testsuite/integration tests, we run fine for a
>> while but then "OutOfMemoryError: unable to create new native thread"
>> errors start showing up in the server.log.
>>
>> Failing tests are here http://pastebin.com/D3SeJ2rP
>>
>> AS7 thread dump showing a lot of "JMX server connection timeout NNN"
>> threads is here http://pastebin.com/qDC52WJG
>>
>> To recreate change into as7/testsuite/integration and do a "mvn clean
>> install -DallTests".
>>
>> Is anyone not seeing this after syncing with master?
>>
>> Scott
>> _______________________________________________
>> jboss-as7-dev mailing list
>> [hidden email]
>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
>

_______________________________________________
jboss-as7-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
Reply | Threaded
Open this post in threaded view
|

Re: consistent testsuite failures since I synced with master yesterday

Scott Marlow
In reply to this post by Ondrej Zizka
On 10/18/2011 08:57 AM, Ondřej Žižka wrote:
> I have a fresh master and testsuite runs fine.
> Except 2 failures (for which I also seek comments):
>
> org.jboss.as.test.integration.osgi.webapp.ServletIntegrationTestCase.testServiceAccess
> org.jboss.as.test.integration.osgi.jaxrs.RestEasyIntegrationTestCase.org.jboss.as.test.integration.osgi.jaxrs.RestEasyIntegrationTestCase
>
> server.log:
> http://pastebin.test.redhat.com/64949

I didn't see that failure.

Which folder are you starting the unit test from and what is your
command line?

I was trying to use "mvn clean install -DallTests" in the
as7/testsuite/integration folder.

When I checked earlier, Hudson wasn't running the full test (was using
-PallTests instead of -DallTests).  Looks like someone changed that but
the test is still in queue.


>
> Anyone seeing that? I thought it could be a race condition caused by
> improper use of deployer API, but giving it some time after deploy
> didn't help.
>
> Ondra
>
>
>
> Scott Marlow píše v Út 18. 10. 2011 v 08:36 -0400:
>
>> When running the as7/testsuite/integration tests, we run fine for a
>> while but then "OutOfMemoryError: unable to create new native thread"
>> errors start showing up in the server.log.
>>
>> Failing tests are here http://pastebin.com/D3SeJ2rP
>>
>> AS7 thread dump showing a lot of "JMX server connection timeout NNN"
>> threads is here http://pastebin.com/qDC52WJG
>>
>> To recreate change into as7/testsuite/integration and do a "mvn clean
>> install -DallTests".
>>
>> Is anyone not seeing this after syncing with master?
>>
>> Scott
>> _______________________________________________
>> jboss-as7-dev mailing list
>> [hidden email]
>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
>

_______________________________________________
jboss-as7-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
Reply | Threaded
Open this post in threaded view
|

Re: consistent testsuite failures since I synced with master yesterday

Scott Marlow
The hudson AS7 test just finished with all integration tests passing.

When you say that your using a fresh master, is that a new AS7 source
tree or an existing one freshly synced with AS7 master?


On 10/18/2011 12:26 PM, Scott Marlow wrote:

> On 10/18/2011 08:57 AM, Ondřej Žižka wrote:
>> I have a fresh master and testsuite runs fine.
>> Except 2 failures (for which I also seek comments):
>>
>> org.jboss.as.test.integration.osgi.webapp.ServletIntegrationTestCase.testServiceAccess
>> org.jboss.as.test.integration.osgi.jaxrs.RestEasyIntegrationTestCase.org.jboss.as.test.integration.osgi.jaxrs.RestEasyIntegrationTestCase
>>
>> server.log:
>> http://pastebin.test.redhat.com/64949
>
> I didn't see that failure.
>
> Which folder are you starting the unit test from and what is your
> command line?
>
> I was trying to use "mvn clean install -DallTests" in the
> as7/testsuite/integration folder.
>
> When I checked earlier, Hudson wasn't running the full test (was using
> -PallTests instead of -DallTests).  Looks like someone changed that but
> the test is still in queue.
>
>
>>
>> Anyone seeing that? I thought it could be a race condition caused by
>> improper use of deployer API, but giving it some time after deploy
>> didn't help.
>>
>> Ondra
>>
>>
>>
>> Scott Marlow píše v Út 18. 10. 2011 v 08:36 -0400:
>>
>>> When running the as7/testsuite/integration tests, we run fine for a
>>> while but then "OutOfMemoryError: unable to create new native thread"
>>> errors start showing up in the server.log.
>>>
>>> Failing tests are here http://pastebin.com/D3SeJ2rP
>>>
>>> AS7 thread dump showing a lot of "JMX server connection timeout NNN"
>>> threads is here http://pastebin.com/qDC52WJG
>>>
>>> To recreate change into as7/testsuite/integration and do a "mvn clean
>>> install -DallTests".
>>>
>>> Is anyone not seeing this after syncing with master?
>>>
>>> Scott
>>> _______________________________________________
>>> jboss-as7-dev mailing list
>>> [hidden email]
>>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
>>
>
> _______________________________________________
> jboss-as7-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev

_______________________________________________
jboss-as7-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
Reply | Threaded
Open this post in threaded view
|

Re: consistent testsuite failures since I synced with master yesterday

Scott Marlow
How did these testsuite failures slip in?  From my own experience
running the updated testsuite, it looks like we now have an allTests
profile that doesn't run the integration tests.  Perhaps we merged some
changes and only ran the allTests profile.  I think that we need to
continue to use -DallTests instead (that includes the integration
tests).  Does anyone know more?

I'm not sure why some people aren't seeing the failures but it seems we
need to track backwards from the symptoms (listed below) or rollback
some changes until it goes away.

If anyone is working on tracking the cause down, please speak up.  I can
jump in but don't want to duplicate effort.

Scott

On 10/18/2011 02:15 PM, Scott Marlow wrote:

> The hudson AS7 test just finished with all integration tests passing.
>
> When you say that your using a fresh master, is that a new AS7 source
> tree or an existing one freshly synced with AS7 master?
>
>
> On 10/18/2011 12:26 PM, Scott Marlow wrote:
>> On 10/18/2011 08:57 AM, Ondřej Žižka wrote:
>>> I have a fresh master and testsuite runs fine.
>>> Except 2 failures (for which I also seek comments):
>>>
>>> org.jboss.as.test.integration.osgi.webapp.ServletIntegrationTestCase.testServiceAccess
>>> org.jboss.as.test.integration.osgi.jaxrs.RestEasyIntegrationTestCase.org.jboss.as.test.integration.osgi.jaxrs.RestEasyIntegrationTestCase
>>>
>>> server.log:
>>> http://pastebin.test.redhat.com/64949
>>
>> I didn't see that failure.
>>
>> Which folder are you starting the unit test from and what is your
>> command line?
>>
>> I was trying to use "mvn clean install -DallTests" in the
>> as7/testsuite/integration folder.
>>
>> When I checked earlier, Hudson wasn't running the full test (was using
>> -PallTests instead of -DallTests).  Looks like someone changed that but
>> the test is still in queue.
>>
>>
>>>
>>> Anyone seeing that? I thought it could be a race condition caused by
>>> improper use of deployer API, but giving it some time after deploy
>>> didn't help.
>>>
>>> Ondra
>>>
>>>
>>>
>>> Scott Marlow píše v Út 18. 10. 2011 v 08:36 -0400:
>>>
>>>> When running the as7/testsuite/integration tests, we run fine for a
>>>> while but then "OutOfMemoryError: unable to create new native thread"
>>>> errors start showing up in the server.log.
>>>>
>>>> Failing tests are here http://pastebin.com/D3SeJ2rP
>>>>
>>>> AS7 thread dump showing a lot of "JMX server connection timeout NNN"
>>>> threads is here http://pastebin.com/qDC52WJG
>>>>
>>>> To recreate change into as7/testsuite/integration and do a "mvn clean
>>>> install -DallTests".
>>>>
>>>> Is anyone not seeing this after syncing with master?
>>>>
>>>> Scott
>>>> _______________________________________________
>>>> jboss-as7-dev mailing list
>>>> [hidden email]
>>>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
>>>
>>
>> _______________________________________________
>> jboss-as7-dev mailing list
>> [hidden email]
>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
>
> _______________________________________________
> jboss-as7-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev

_______________________________________________
jboss-as7-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
Reply | Threaded
Open this post in threaded view
|

Re: consistent testsuite failures since I synced with master yesterday

Jaikiran Pai
I don't know about the test issues that Ondrej is running into, but the
ones you mention http://pastebin.com/D3SeJ2rP (OutOfMemory : Cannot
create native thread) were encountered once by Carlo (or on our Hudson
instance, don't remember). But I haven't seen them after that. It might
be environment specific. What JDK vendor/version? And are you running
into this always?

-Jaikiran
On Wednesday 19 October 2011 08:14 PM, Scott Marlow wrote:

> How did these testsuite failures slip in?  From my own experience
> running the updated testsuite, it looks like we now have an allTests
> profile that doesn't run the integration tests.  Perhaps we merged some
> changes and only ran the allTests profile.  I think that we need to
> continue to use -DallTests instead (that includes the integration
> tests).  Does anyone know more?
>
> I'm not sure why some people aren't seeing the failures but it seems we
> need to track backwards from the symptoms (listed below) or rollback
> some changes until it goes away.
>
> If anyone is working on tracking the cause down, please speak up.  I can
> jump in but don't want to duplicate effort.
>
> Scott
>
> On 10/18/2011 02:15 PM, Scott Marlow wrote:
>> The hudson AS7 test just finished with all integration tests passing.
>>
>> When you say that your using a fresh master, is that a new AS7 source
>> tree or an existing one freshly synced with AS7 master?
>>
>>
>> On 10/18/2011 12:26 PM, Scott Marlow wrote:
>>> On 10/18/2011 08:57 AM, Ondřej Žižka wrote:
>>>> I have a fresh master and testsuite runs fine.
>>>> Except 2 failures (for which I also seek comments):
>>>>
>>>> org.jboss.as.test.integration.osgi.webapp.ServletIntegrationTestCase.testServiceAccess
>>>> org.jboss.as.test.integration.osgi.jaxrs.RestEasyIntegrationTestCase.org.jboss.as.test.integration.osgi.jaxrs.RestEasyIntegrationTestCase
>>>>
>>>> server.log:
>>>> http://pastebin.test.redhat.com/64949
>>>
>>> I didn't see that failure.
>>>
>>> Which folder are you starting the unit test from and what is your
>>> command line?
>>>
>>> I was trying to use "mvn clean install -DallTests" in the
>>> as7/testsuite/integration folder.
>>>
>>> When I checked earlier, Hudson wasn't running the full test (was using
>>> -PallTests instead of -DallTests).  Looks like someone changed that but
>>> the test is still in queue.
>>>
>>>
>>>>
>>>> Anyone seeing that? I thought it could be a race condition caused by
>>>> improper use of deployer API, but giving it some time after deploy
>>>> didn't help.
>>>>
>>>> Ondra
>>>>
>>>>
>>>>
>>>> Scott Marlow píše v Út 18. 10. 2011 v 08:36 -0400:
>>>>
>>>>> When running the as7/testsuite/integration tests, we run fine for a
>>>>> while but then "OutOfMemoryError: unable to create new native thread"
>>>>> errors start showing up in the server.log.
>>>>>
>>>>> Failing tests are here http://pastebin.com/D3SeJ2rP
>>>>>
>>>>> AS7 thread dump showing a lot of "JMX server connection timeout NNN"
>>>>> threads is here http://pastebin.com/qDC52WJG
>>>>>
>>>>> To recreate change into as7/testsuite/integration and do a "mvn clean
>>>>> install -DallTests".
>>>>>
>>>>> Is anyone not seeing this after syncing with master?
>>>>>
>>>>> Scott
>>>>> _______________________________________________
>>>>> jboss-as7-dev mailing list
>>>>> [hidden email]
>>>>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
>>>>
>>>
>>> _______________________________________________
>>> jboss-as7-dev mailing list
>>> [hidden email]
>>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
>>
>> _______________________________________________
>> jboss-as7-dev mailing list
>> [hidden email]
>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
>
> _______________________________________________
> jboss-as7-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev

_______________________________________________
jboss-as7-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
Reply | Threaded
Open this post in threaded view
|

Re: consistent testsuite failures since I synced with master yesterday

Brian Stansberry
In reply to this post by Scott Marlow
On 10/19/11 9:44 AM, Scott Marlow wrote:
> How did these testsuite failures slip in?  From my own experience
> running the updated testsuite, it looks like we now have an allTests
> profile that doesn't run the integration tests.  Perhaps we merged some
> changes and only ran the allTests profile.  I think that we need to
> continue to use -DallTests instead (that includes the integration
> tests).  Does anyone know more?
>

Everyone who merges pull requests uses -DallTests.

> I'm not sure why some people aren't seeing the failures but it seems we
> need to track backwards from the symptoms (listed below) or rollback
> some changes until it goes away.
>
> If anyone is working on tracking the cause down, please speak up.  I can
> jump in but don't want to duplicate effort.
>
> Scott
>
> On 10/18/2011 02:15 PM, Scott Marlow wrote:
>> The hudson AS7 test just finished with all integration tests passing.
>>
>> When you say that your using a fresh master, is that a new AS7 source
>> tree or an existing one freshly synced with AS7 master?
>>
>>
>> On 10/18/2011 12:26 PM, Scott Marlow wrote:
>>> On 10/18/2011 08:57 AM, Ondřej Žižka wrote:
>>>> I have a fresh master and testsuite runs fine.
>>>> Except 2 failures (for which I also seek comments):
>>>>
>>>> org.jboss.as.test.integration.osgi.webapp.ServletIntegrationTestCase.testServiceAccess
>>>> org.jboss.as.test.integration.osgi.jaxrs.RestEasyIntegrationTestCase.org.jboss.as.test.integration.osgi.jaxrs.RestEasyIntegrationTestCase
>>>>
>>>> server.log:
>>>> http://pastebin.test.redhat.com/64949
>>>
>>> I didn't see that failure.
>>>
>>> Which folder are you starting the unit test from and what is your
>>> command line?
>>>
>>> I was trying to use "mvn clean install -DallTests" in the
>>> as7/testsuite/integration folder.
>>>
>>> When I checked earlier, Hudson wasn't running the full test (was using
>>> -PallTests instead of -DallTests).  Looks like someone changed that but
>>> the test is still in queue.
>>>
>>>
>>>>
>>>> Anyone seeing that? I thought it could be a race condition caused by
>>>> improper use of deployer API, but giving it some time after deploy
>>>> didn't help.
>>>>
>>>> Ondra
>>>>
>>>>
>>>>
>>>> Scott Marlow píše v Út 18. 10. 2011 v 08:36 -0400:
>>>>
>>>>> When running the as7/testsuite/integration tests, we run fine for a
>>>>> while but then "OutOfMemoryError: unable to create new native thread"
>>>>> errors start showing up in the server.log.
>>>>>
>>>>> Failing tests are here http://pastebin.com/D3SeJ2rP
>>>>>
>>>>> AS7 thread dump showing a lot of "JMX server connection timeout NNN"
>>>>> threads is here http://pastebin.com/qDC52WJG
>>>>>
>>>>> To recreate change into as7/testsuite/integration and do a "mvn clean
>>>>> install -DallTests".
>>>>>
>>>>> Is anyone not seeing this after syncing with master?
>>>>>
>>>>> Scott
>>>>> _______________________________________________
>>>>> jboss-as7-dev mailing list
>>>>> [hidden email]
>>>>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
>>>>
>>>
>>> _______________________________________________
>>> jboss-as7-dev mailing list
>>> [hidden email]
>>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
>>
>> _______________________________________________
>> jboss-as7-dev mailing list
>> [hidden email]
>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
>
> _______________________________________________
> jboss-as7-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev


--
Brian Stansberry
Principal Software Engineer
JBoss by Red Hat
_______________________________________________
jboss-as7-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
Reply | Threaded
Open this post in threaded view
|

Re: consistent testsuite failures since I synced with master yesterday

Scott Marlow
On 10/19/2011 10:49 AM, Brian Stansberry wrote:

> On 10/19/11 9:44 AM, Scott Marlow wrote:
>> How did these testsuite failures slip in?  From my own experience
>> running the updated testsuite, it looks like we now have an allTests
>> profile that doesn't run the integration tests.  Perhaps we merged some
>> changes and only ran the allTests profile.  I think that we need to
>> continue to use -DallTests instead (that includes the integration
>> tests).  Does anyone know more?
>>
>
> Everyone who merges pull requests uses -DallTests.

Good to know, so it might be environmental...

>
>> I'm not sure why some people aren't seeing the failures but it seems we
>> need to track backwards from the symptoms (listed below) or rollback
>> some changes until it goes away.
>>
>> If anyone is working on tracking the cause down, please speak up.  I can
>> jump in but don't want to duplicate effort.
>>
>> Scott
>>
>> On 10/18/2011 02:15 PM, Scott Marlow wrote:
>>> The hudson AS7 test just finished with all integration tests passing.
>>>
>>> When you say that your using a fresh master, is that a new AS7 source
>>> tree or an existing one freshly synced with AS7 master?
>>>
>>>
>>> On 10/18/2011 12:26 PM, Scott Marlow wrote:
>>>> On 10/18/2011 08:57 AM, Ondřej Žižka wrote:
>>>>> I have a fresh master and testsuite runs fine.
>>>>> Except 2 failures (for which I also seek comments):
>>>>>
>>>>> org.jboss.as.test.integration.osgi.webapp.ServletIntegrationTestCase.testServiceAccess
>>>>> org.jboss.as.test.integration.osgi.jaxrs.RestEasyIntegrationTestCase.org.jboss.as.test.integration.osgi.jaxrs.RestEasyIntegrationTestCase
>>>>>
>>>>> server.log:
>>>>> http://pastebin.test.redhat.com/64949
>>>>
>>>> I didn't see that failure.
>>>>
>>>> Which folder are you starting the unit test from and what is your
>>>> command line?
>>>>
>>>> I was trying to use "mvn clean install -DallTests" in the
>>>> as7/testsuite/integration folder.
>>>>
>>>> When I checked earlier, Hudson wasn't running the full test (was using
>>>> -PallTests instead of -DallTests).  Looks like someone changed that but
>>>> the test is still in queue.
>>>>
>>>>
>>>>>
>>>>> Anyone seeing that? I thought it could be a race condition caused by
>>>>> improper use of deployer API, but giving it some time after deploy
>>>>> didn't help.
>>>>>
>>>>> Ondra
>>>>>
>>>>>
>>>>>
>>>>> Scott Marlow píše v Út 18. 10. 2011 v 08:36 -0400:
>>>>>
>>>>>> When running the as7/testsuite/integration tests, we run fine for a
>>>>>> while but then "OutOfMemoryError: unable to create new native thread"
>>>>>> errors start showing up in the server.log.
>>>>>>
>>>>>> Failing tests are here http://pastebin.com/D3SeJ2rP
>>>>>>
>>>>>> AS7 thread dump showing a lot of "JMX server connection timeout NNN"
>>>>>> threads is here http://pastebin.com/qDC52WJG
>>>>>>
>>>>>> To recreate change into as7/testsuite/integration and do a "mvn clean
>>>>>> install -DallTests".
>>>>>>
>>>>>> Is anyone not seeing this after syncing with master?
>>>>>>
>>>>>> Scott
>>>>>> _______________________________________________
>>>>>> jboss-as7-dev mailing list
>>>>>> [hidden email]
>>>>>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
>>>>>
>>>>
>>>> _______________________________________________
>>>> jboss-as7-dev mailing list
>>>> [hidden email]
>>>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
>>>
>>> _______________________________________________
>>> jboss-as7-dev mailing list
>>> [hidden email]
>>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
>>
>> _______________________________________________
>> jboss-as7-dev mailing list
>> [hidden email]
>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
>
>

_______________________________________________
jboss-as7-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
Reply | Threaded
Open this post in threaded view
|

Re: consistent testsuite failures since I synced with master yesterday

Scott Marlow
In reply to this post by Jaikiran Pai
I'm using Sun 1.6.0_26-b03.  I think Hudson is passing with Sun
1.6.0_24-b07.

Jesper also got the failures with jdk 6 u26.

I haven't tried running with my old jdk1.6.0_24 yet.  I could try that
or we could start comparing environment settings.

On Hudson, we are setting MAVEN_OPTS="-Xmx1024m -Xms512m
-XX:MaxPermSize=128m"

I don't have that set locally.  I'll try that next...

Scott

On 10/19/2011 10:49 AM, Jaikiran Pai wrote:

> I don't know about the test issues that Ondrej is running into, but the
> ones you mention http://pastebin.com/D3SeJ2rP (OutOfMemory : Cannot
> create native thread) were encountered once by Carlo (or on our Hudson
> instance, don't remember). But I haven't seen them after that. It might
> be environment specific. What JDK vendor/version? And are you running
> into this always?
>
> -Jaikiran
> On Wednesday 19 October 2011 08:14 PM, Scott Marlow wrote:
>> How did these testsuite failures slip in?  From my own experience
>> running the updated testsuite, it looks like we now have an allTests
>> profile that doesn't run the integration tests.  Perhaps we merged some
>> changes and only ran the allTests profile.  I think that we need to
>> continue to use -DallTests instead (that includes the integration
>> tests).  Does anyone know more?
>>
>> I'm not sure why some people aren't seeing the failures but it seems we
>> need to track backwards from the symptoms (listed below) or rollback
>> some changes until it goes away.
>>
>> If anyone is working on tracking the cause down, please speak up.  I can
>> jump in but don't want to duplicate effort.
>>
>> Scott
>>
>> On 10/18/2011 02:15 PM, Scott Marlow wrote:
>>> The hudson AS7 test just finished with all integration tests passing.
>>>
>>> When you say that your using a fresh master, is that a new AS7 source
>>> tree or an existing one freshly synced with AS7 master?
>>>
>>>
>>> On 10/18/2011 12:26 PM, Scott Marlow wrote:
>>>> On 10/18/2011 08:57 AM, Ondřej Žižka wrote:
>>>>> I have a fresh master and testsuite runs fine.
>>>>> Except 2 failures (for which I also seek comments):
>>>>>
>>>>> org.jboss.as.test.integration.osgi.webapp.ServletIntegrationTestCase.testServiceAccess
>>>>> org.jboss.as.test.integration.osgi.jaxrs.RestEasyIntegrationTestCase.org.jboss.as.test.integration.osgi.jaxrs.RestEasyIntegrationTestCase
>>>>>
>>>>> server.log:
>>>>> http://pastebin.test.redhat.com/64949
>>>>
>>>> I didn't see that failure.
>>>>
>>>> Which folder are you starting the unit test from and what is your
>>>> command line?
>>>>
>>>> I was trying to use "mvn clean install -DallTests" in the
>>>> as7/testsuite/integration folder.
>>>>
>>>> When I checked earlier, Hudson wasn't running the full test (was using
>>>> -PallTests instead of -DallTests).  Looks like someone changed that but
>>>> the test is still in queue.
>>>>
>>>>
>>>>>
>>>>> Anyone seeing that? I thought it could be a race condition caused by
>>>>> improper use of deployer API, but giving it some time after deploy
>>>>> didn't help.
>>>>>
>>>>> Ondra
>>>>>
>>>>>
>>>>>
>>>>> Scott Marlow píše v Út 18. 10. 2011 v 08:36 -0400:
>>>>>
>>>>>> When running the as7/testsuite/integration tests, we run fine for a
>>>>>> while but then "OutOfMemoryError: unable to create new native thread"
>>>>>> errors start showing up in the server.log.
>>>>>>
>>>>>> Failing tests are here http://pastebin.com/D3SeJ2rP
>>>>>>
>>>>>> AS7 thread dump showing a lot of "JMX server connection timeout NNN"
>>>>>> threads is here http://pastebin.com/qDC52WJG
>>>>>>
>>>>>> To recreate change into as7/testsuite/integration and do a "mvn clean
>>>>>> install -DallTests".
>>>>>>
>>>>>> Is anyone not seeing this after syncing with master?
>>>>>>
>>>>>> Scott
>>>>>> _______________________________________________
>>>>>> jboss-as7-dev mailing list
>>>>>> [hidden email]
>>>>>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
>>>>>
>>>>
>>>> _______________________________________________
>>>> jboss-as7-dev mailing list
>>>> [hidden email]
>>>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
>>>
>>> _______________________________________________
>>> jboss-as7-dev mailing list
>>> [hidden email]
>>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
>>
>> _______________________________________________
>> jboss-as7-dev mailing list
>> [hidden email]
>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
>
> _______________________________________________
> jboss-as7-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev

_______________________________________________
jboss-as7-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
Reply | Threaded
Open this post in threaded view
|

Re: consistent testsuite failures since I synced with master yesterday

David Lloyd-2
In reply to this post by Jaikiran Pai
If we're still leaking I/O threads at the management layer, then I would
not have any expectation that our test suite would finish anywhere.

On 10/19/2011 09:49 AM, Jaikiran Pai wrote:

> I don't know about the test issues that Ondrej is running into, but the
> ones you mention http://pastebin.com/D3SeJ2rP (OutOfMemory : Cannot
> create native thread) were encountered once by Carlo (or on our Hudson
> instance, don't remember). But I haven't seen them after that. It might
> be environment specific. What JDK vendor/version? And are you running
> into this always?
>
> -Jaikiran
> On Wednesday 19 October 2011 08:14 PM, Scott Marlow wrote:
>> How did these testsuite failures slip in?  From my own experience
>> running the updated testsuite, it looks like we now have an allTests
>> profile that doesn't run the integration tests.  Perhaps we merged some
>> changes and only ran the allTests profile.  I think that we need to
>> continue to use -DallTests instead (that includes the integration
>> tests).  Does anyone know more?
>>
>> I'm not sure why some people aren't seeing the failures but it seems we
>> need to track backwards from the symptoms (listed below) or rollback
>> some changes until it goes away.
>>
>> If anyone is working on tracking the cause down, please speak up.  I can
>> jump in but don't want to duplicate effort.
>>
>> Scott
>>
>> On 10/18/2011 02:15 PM, Scott Marlow wrote:
>>> The hudson AS7 test just finished with all integration tests passing.
>>>
>>> When you say that your using a fresh master, is that a new AS7 source
>>> tree or an existing one freshly synced with AS7 master?
>>>
>>>
>>> On 10/18/2011 12:26 PM, Scott Marlow wrote:
>>>> On 10/18/2011 08:57 AM, Ondřej Žižka wrote:
>>>>> I have a fresh master and testsuite runs fine.
>>>>> Except 2 failures (for which I also seek comments):
>>>>>
>>>>> org.jboss.as.test.integration.osgi.webapp.ServletIntegrationTestCase.testServiceAccess
>>>>> org.jboss.as.test.integration.osgi.jaxrs.RestEasyIntegrationTestCase.org.jboss.as.test.integration.osgi.jaxrs.RestEasyIntegrationTestCase
>>>>>
>>>>> server.log:
>>>>> http://pastebin.test.redhat.com/64949
>>>>
>>>> I didn't see that failure.
>>>>
>>>> Which folder are you starting the unit test from and what is your
>>>> command line?
>>>>
>>>> I was trying to use "mvn clean install -DallTests" in the
>>>> as7/testsuite/integration folder.
>>>>
>>>> When I checked earlier, Hudson wasn't running the full test (was using
>>>> -PallTests instead of -DallTests).  Looks like someone changed that but
>>>> the test is still in queue.
>>>>
>>>>
>>>>>
>>>>> Anyone seeing that? I thought it could be a race condition caused by
>>>>> improper use of deployer API, but giving it some time after deploy
>>>>> didn't help.
>>>>>
>>>>> Ondra
>>>>>
>>>>>
>>>>>
>>>>> Scott Marlow píše v Út 18. 10. 2011 v 08:36 -0400:
>>>>>
>>>>>> When running the as7/testsuite/integration tests, we run fine for a
>>>>>> while but then "OutOfMemoryError: unable to create new native thread"
>>>>>> errors start showing up in the server.log.
>>>>>>
>>>>>> Failing tests are here http://pastebin.com/D3SeJ2rP
>>>>>>
>>>>>> AS7 thread dump showing a lot of "JMX server connection timeout NNN"
>>>>>> threads is here http://pastebin.com/qDC52WJG
>>>>>>
>>>>>> To recreate change into as7/testsuite/integration and do a "mvn clean
>>>>>> install -DallTests".
>>>>>>
>>>>>> Is anyone not seeing this after syncing with master?
>>>>>>
>>>>>> Scott
>>>>>> _______________________________________________
>>>>>> jboss-as7-dev mailing list
>>>>>> [hidden email]
>>>>>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
>>>>>
>>>>
>>>> _______________________________________________
>>>> jboss-as7-dev mailing list
>>>> [hidden email]
>>>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
>>>
>>> _______________________________________________
>>> jboss-as7-dev mailing list
>>> [hidden email]
>>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
>>
>> _______________________________________________
>> jboss-as7-dev mailing list
>> [hidden email]
>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
>
> _______________________________________________
> jboss-as7-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev


--
- DML
_______________________________________________
jboss-as7-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
Reply | Threaded
Open this post in threaded view
|

Re: consistent testsuite failures since I synced with master yesterday

Scott Marlow
In reply to this post by Scott Marlow
On 10/19/2011 11:01 AM, Scott Marlow wrote:

> I'm using Sun 1.6.0_26-b03.  I think Hudson is passing with Sun
> 1.6.0_24-b07.
>
> Jesper also got the failures with jdk 6 u26.
>
> I haven't tried running with my old jdk1.6.0_24 yet.  I could try that
> or we could start comparing environment settings.
>
> On Hudson, we are setting MAVEN_OPTS="-Xmx1024m -Xms512m
> -XX:MaxPermSize=128m"
>
> I don't have that set locally.  I'll try that next...

Testsuite still fails for me with the above MAVEN_OPTS set.

I'll my older jdk 6 (same as hudson is using) and see if that makes a diff.


>
> Scott
>
> On 10/19/2011 10:49 AM, Jaikiran Pai wrote:
>> I don't know about the test issues that Ondrej is running into, but the
>> ones you mention http://pastebin.com/D3SeJ2rP (OutOfMemory : Cannot
>> create native thread) were encountered once by Carlo (or on our Hudson
>> instance, don't remember). But I haven't seen them after that. It might
>> be environment specific. What JDK vendor/version? And are you running
>> into this always?
>>
>> -Jaikiran
>> On Wednesday 19 October 2011 08:14 PM, Scott Marlow wrote:
>>> How did these testsuite failures slip in?  From my own experience
>>> running the updated testsuite, it looks like we now have an allTests
>>> profile that doesn't run the integration tests.  Perhaps we merged some
>>> changes and only ran the allTests profile.  I think that we need to
>>> continue to use -DallTests instead (that includes the integration
>>> tests).  Does anyone know more?
>>>
>>> I'm not sure why some people aren't seeing the failures but it seems we
>>> need to track backwards from the symptoms (listed below) or rollback
>>> some changes until it goes away.
>>>
>>> If anyone is working on tracking the cause down, please speak up.  I can
>>> jump in but don't want to duplicate effort.
>>>
>>> Scott
>>>
>>> On 10/18/2011 02:15 PM, Scott Marlow wrote:
>>>> The hudson AS7 test just finished with all integration tests passing.
>>>>
>>>> When you say that your using a fresh master, is that a new AS7 source
>>>> tree or an existing one freshly synced with AS7 master?
>>>>
>>>>
>>>> On 10/18/2011 12:26 PM, Scott Marlow wrote:
>>>>> On 10/18/2011 08:57 AM, Ondřej Žižka wrote:
>>>>>> I have a fresh master and testsuite runs fine.
>>>>>> Except 2 failures (for which I also seek comments):
>>>>>>
>>>>>> org.jboss.as.test.integration.osgi.webapp.ServletIntegrationTestCase.testServiceAccess
>>>>>> org.jboss.as.test.integration.osgi.jaxrs.RestEasyIntegrationTestCase.org.jboss.as.test.integration.osgi.jaxrs.RestEasyIntegrationTestCase
>>>>>>
>>>>>> server.log:
>>>>>> http://pastebin.test.redhat.com/64949
>>>>>
>>>>> I didn't see that failure.
>>>>>
>>>>> Which folder are you starting the unit test from and what is your
>>>>> command line?
>>>>>
>>>>> I was trying to use "mvn clean install -DallTests" in the
>>>>> as7/testsuite/integration folder.
>>>>>
>>>>> When I checked earlier, Hudson wasn't running the full test (was using
>>>>> -PallTests instead of -DallTests).  Looks like someone changed that but
>>>>> the test is still in queue.
>>>>>
>>>>>
>>>>>>
>>>>>> Anyone seeing that? I thought it could be a race condition caused by
>>>>>> improper use of deployer API, but giving it some time after deploy
>>>>>> didn't help.
>>>>>>
>>>>>> Ondra
>>>>>>
>>>>>>
>>>>>>
>>>>>> Scott Marlow píše v Út 18. 10. 2011 v 08:36 -0400:
>>>>>>
>>>>>>> When running the as7/testsuite/integration tests, we run fine for a
>>>>>>> while but then "OutOfMemoryError: unable to create new native thread"
>>>>>>> errors start showing up in the server.log.
>>>>>>>
>>>>>>> Failing tests are here http://pastebin.com/D3SeJ2rP
>>>>>>>
>>>>>>> AS7 thread dump showing a lot of "JMX server connection timeout NNN"
>>>>>>> threads is here http://pastebin.com/qDC52WJG
>>>>>>>
>>>>>>> To recreate change into as7/testsuite/integration and do a "mvn clean
>>>>>>> install -DallTests".
>>>>>>>
>>>>>>> Is anyone not seeing this after syncing with master?
>>>>>>>
>>>>>>> Scott
>>>>>>> _______________________________________________
>>>>>>> jboss-as7-dev mailing list
>>>>>>> [hidden email]
>>>>>>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
>>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> jboss-as7-dev mailing list
>>>>> [hidden email]
>>>>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
>>>>
>>>> _______________________________________________
>>>> jboss-as7-dev mailing list
>>>> [hidden email]
>>>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
>>>
>>> _______________________________________________
>>> jboss-as7-dev mailing list
>>> [hidden email]
>>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
>>
>> _______________________________________________
>> jboss-as7-dev mailing list
>> [hidden email]
>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
>
> _______________________________________________
> jboss-as7-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev

_______________________________________________
jboss-as7-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
Reply | Threaded
Open this post in threaded view
|

Re: consistent testsuite failures since I synced with master yesterday

Scott Marlow
On 10/19/2011 12:13 PM, Scott Marlow wrote:

> On 10/19/2011 11:01 AM, Scott Marlow wrote:
>> I'm using Sun 1.6.0_26-b03.  I think Hudson is passing with Sun
>> 1.6.0_24-b07.
>>
>> Jesper also got the failures with jdk 6 u26.
>>
>> I haven't tried running with my old jdk1.6.0_24 yet.  I could try that
>> or we could start comparing environment settings.
>>
>> On Hudson, we are setting MAVEN_OPTS="-Xmx1024m -Xms512m
>> -XX:MaxPermSize=128m"
>>
>> I don't have that set locally.  I'll try that next...
>
> Testsuite still fails for me with the above MAVEN_OPTS set.
>
> I'll my older jdk 6 (same as hudson is using) and see if that makes a diff.

Same failure still (with MAVEN_OPTS set as above and using jdk1.6.0_24).

>
>
>>
>> Scott
>>
>> On 10/19/2011 10:49 AM, Jaikiran Pai wrote:
>>> I don't know about the test issues that Ondrej is running into, but the
>>> ones you mention http://pastebin.com/D3SeJ2rP (OutOfMemory : Cannot
>>> create native thread) were encountered once by Carlo (or on our Hudson
>>> instance, don't remember). But I haven't seen them after that. It might
>>> be environment specific. What JDK vendor/version? And are you running
>>> into this always?
>>>
>>> -Jaikiran
>>> On Wednesday 19 October 2011 08:14 PM, Scott Marlow wrote:
>>>> How did these testsuite failures slip in?  From my own experience
>>>> running the updated testsuite, it looks like we now have an allTests
>>>> profile that doesn't run the integration tests.  Perhaps we merged some
>>>> changes and only ran the allTests profile.  I think that we need to
>>>> continue to use -DallTests instead (that includes the integration
>>>> tests).  Does anyone know more?
>>>>
>>>> I'm not sure why some people aren't seeing the failures but it seems we
>>>> need to track backwards from the symptoms (listed below) or rollback
>>>> some changes until it goes away.
>>>>
>>>> If anyone is working on tracking the cause down, please speak up.  I can
>>>> jump in but don't want to duplicate effort.
>>>>
>>>> Scott
>>>>
>>>> On 10/18/2011 02:15 PM, Scott Marlow wrote:
>>>>> The hudson AS7 test just finished with all integration tests passing.
>>>>>
>>>>> When you say that your using a fresh master, is that a new AS7 source
>>>>> tree or an existing one freshly synced with AS7 master?
>>>>>
>>>>>
>>>>> On 10/18/2011 12:26 PM, Scott Marlow wrote:
>>>>>> On 10/18/2011 08:57 AM, Ondřej Žižka wrote:
>>>>>>> I have a fresh master and testsuite runs fine.
>>>>>>> Except 2 failures (for which I also seek comments):
>>>>>>>
>>>>>>> org.jboss.as.test.integration.osgi.webapp.ServletIntegrationTestCase.testServiceAccess
>>>>>>> org.jboss.as.test.integration.osgi.jaxrs.RestEasyIntegrationTestCase.org.jboss.as.test.integration.osgi.jaxrs.RestEasyIntegrationTestCase
>>>>>>>
>>>>>>> server.log:
>>>>>>> http://pastebin.test.redhat.com/64949
>>>>>>
>>>>>> I didn't see that failure.
>>>>>>
>>>>>> Which folder are you starting the unit test from and what is your
>>>>>> command line?
>>>>>>
>>>>>> I was trying to use "mvn clean install -DallTests" in the
>>>>>> as7/testsuite/integration folder.
>>>>>>
>>>>>> When I checked earlier, Hudson wasn't running the full test (was using
>>>>>> -PallTests instead of -DallTests).  Looks like someone changed that but
>>>>>> the test is still in queue.
>>>>>>
>>>>>>
>>>>>>>
>>>>>>> Anyone seeing that? I thought it could be a race condition caused by
>>>>>>> improper use of deployer API, but giving it some time after deploy
>>>>>>> didn't help.
>>>>>>>
>>>>>>> Ondra
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Scott Marlow píše v Út 18. 10. 2011 v 08:36 -0400:
>>>>>>>
>>>>>>>> When running the as7/testsuite/integration tests, we run fine for a
>>>>>>>> while but then "OutOfMemoryError: unable to create new native thread"
>>>>>>>> errors start showing up in the server.log.
>>>>>>>>
>>>>>>>> Failing tests are here http://pastebin.com/D3SeJ2rP
>>>>>>>>
>>>>>>>> AS7 thread dump showing a lot of "JMX server connection timeout NNN"
>>>>>>>> threads is here http://pastebin.com/qDC52WJG
>>>>>>>>
>>>>>>>> To recreate change into as7/testsuite/integration and do a "mvn clean
>>>>>>>> install -DallTests".
>>>>>>>>
>>>>>>>> Is anyone not seeing this after syncing with master?
>>>>>>>>
>>>>>>>> Scott
>>>>>>>> _______________________________________________
>>>>>>>> jboss-as7-dev mailing list
>>>>>>>> [hidden email]
>>>>>>>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
>>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> jboss-as7-dev mailing list
>>>>>> [hidden email]
>>>>>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
>>>>>
>>>>> _______________________________________________
>>>>> jboss-as7-dev mailing list
>>>>> [hidden email]
>>>>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
>>>>
>>>> _______________________________________________
>>>> jboss-as7-dev mailing list
>>>> [hidden email]
>>>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
>>>
>>> _______________________________________________
>>> jboss-as7-dev mailing list
>>> [hidden email]
>>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
>>
>> _______________________________________________
>> jboss-as7-dev mailing list
>> [hidden email]
>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
>
> _______________________________________________
> jboss-as7-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev

_______________________________________________
jboss-as7-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
Reply | Threaded
Open this post in threaded view
|

Re: consistent testsuite failures since I synced with master yesterday

Brian Stansberry
org.jboss.arquillian.container.spi.client.protocol.metadata.JMXContext.getConnection()
is creating a javax.management.remote.JMXConnector but I don't see
anything closing it. I'm not sure, but perhaps one of those is created
per test method execution?

I believe the "JMX server connection timeout 552" threads are created by
the JDK stuff AS7 is using for its JMX connector impl. You can see the
relevant code in the stack trace. A quick look at that code leads me to
believe those threads will by default live for 120 secs after the last
request on the connection.

On 10/19/11 11:54 AM, Scott Marlow wrote:

> On 10/19/2011 12:13 PM, Scott Marlow wrote:
>> On 10/19/2011 11:01 AM, Scott Marlow wrote:
>>> I'm using Sun 1.6.0_26-b03.  I think Hudson is passing with Sun
>>> 1.6.0_24-b07.
>>>
>>> Jesper also got the failures with jdk 6 u26.
>>>
>>> I haven't tried running with my old jdk1.6.0_24 yet.  I could try that
>>> or we could start comparing environment settings.
>>>
>>> On Hudson, we are setting MAVEN_OPTS="-Xmx1024m -Xms512m
>>> -XX:MaxPermSize=128m"
>>>
>>> I don't have that set locally.  I'll try that next...
>>
>> Testsuite still fails for me with the above MAVEN_OPTS set.
>>
>> I'll my older jdk 6 (same as hudson is using) and see if that makes a diff.
>
> Same failure still (with MAVEN_OPTS set as above and using jdk1.6.0_24).
>
>>
>>
>>>
>>> Scott
>>>
>>> On 10/19/2011 10:49 AM, Jaikiran Pai wrote:
>>>> I don't know about the test issues that Ondrej is running into, but the
>>>> ones you mention http://pastebin.com/D3SeJ2rP (OutOfMemory : Cannot
>>>> create native thread) were encountered once by Carlo (or on our Hudson
>>>> instance, don't remember). But I haven't seen them after that. It might
>>>> be environment specific. What JDK vendor/version? And are you running
>>>> into this always?
>>>>
>>>> -Jaikiran
>>>> On Wednesday 19 October 2011 08:14 PM, Scott Marlow wrote:
>>>>> How did these testsuite failures slip in?  From my own experience
>>>>> running the updated testsuite, it looks like we now have an allTests
>>>>> profile that doesn't run the integration tests.  Perhaps we merged some
>>>>> changes and only ran the allTests profile.  I think that we need to
>>>>> continue to use -DallTests instead (that includes the integration
>>>>> tests).  Does anyone know more?
>>>>>
>>>>> I'm not sure why some people aren't seeing the failures but it seems we
>>>>> need to track backwards from the symptoms (listed below) or rollback
>>>>> some changes until it goes away.
>>>>>
>>>>> If anyone is working on tracking the cause down, please speak up.  I can
>>>>> jump in but don't want to duplicate effort.
>>>>>
>>>>> Scott
>>>>>
>>>>> On 10/18/2011 02:15 PM, Scott Marlow wrote:
>>>>>> The hudson AS7 test just finished with all integration tests passing.
>>>>>>
>>>>>> When you say that your using a fresh master, is that a new AS7 source
>>>>>> tree or an existing one freshly synced with AS7 master?
>>>>>>
>>>>>>
>>>>>> On 10/18/2011 12:26 PM, Scott Marlow wrote:
>>>>>>> On 10/18/2011 08:57 AM, Ondřej Žižka wrote:
>>>>>>>> I have a fresh master and testsuite runs fine.
>>>>>>>> Except 2 failures (for which I also seek comments):
>>>>>>>>
>>>>>>>> org.jboss.as.test.integration.osgi.webapp.ServletIntegrationTestCase.testServiceAccess
>>>>>>>> org.jboss.as.test.integration.osgi.jaxrs.RestEasyIntegrationTestCase.org.jboss.as.test.integration.osgi.jaxrs.RestEasyIntegrationTestCase
>>>>>>>>
>>>>>>>> server.log:
>>>>>>>> http://pastebin.test.redhat.com/64949
>>>>>>>
>>>>>>> I didn't see that failure.
>>>>>>>
>>>>>>> Which folder are you starting the unit test from and what is your
>>>>>>> command line?
>>>>>>>
>>>>>>> I was trying to use "mvn clean install -DallTests" in the
>>>>>>> as7/testsuite/integration folder.
>>>>>>>
>>>>>>> When I checked earlier, Hudson wasn't running the full test (was using
>>>>>>> -PallTests instead of -DallTests).  Looks like someone changed that but
>>>>>>> the test is still in queue.
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>> Anyone seeing that? I thought it could be a race condition caused by
>>>>>>>> improper use of deployer API, but giving it some time after deploy
>>>>>>>> didn't help.
>>>>>>>>
>>>>>>>> Ondra
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Scott Marlow píše v Út 18. 10. 2011 v 08:36 -0400:
>>>>>>>>
>>>>>>>>> When running the as7/testsuite/integration tests, we run fine for a
>>>>>>>>> while but then "OutOfMemoryError: unable to create new native thread"
>>>>>>>>> errors start showing up in the server.log.
>>>>>>>>>
>>>>>>>>> Failing tests are here http://pastebin.com/D3SeJ2rP
>>>>>>>>>
>>>>>>>>> AS7 thread dump showing a lot of "JMX server connection timeout NNN"
>>>>>>>>> threads is here http://pastebin.com/qDC52WJG
>>>>>>>>>
>>>>>>>>> To recreate change into as7/testsuite/integration and do a "mvn clean
>>>>>>>>> install -DallTests".
>>>>>>>>>
>>>>>>>>> Is anyone not seeing this after syncing with master?
>>>>>>>>>
>>>>>>>>> Scott
>>>>>>>>> _______________________________________________
>>>>>>>>> jboss-as7-dev mailing list
>>>>>>>>> [hidden email]
>>>>>>>>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
>>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> jboss-as7-dev mailing list
>>>>>>> [hidden email]
>>>>>>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
>>>>>>
>>>>>> _______________________________________________
>>>>>> jboss-as7-dev mailing list
>>>>>> [hidden email]
>>>>>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
>>>>>
>>>>> _______________________________________________
>>>>> jboss-as7-dev mailing list
>>>>> [hidden email]
>>>>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
>>>>
>>>> _______________________________________________
>>>> jboss-as7-dev mailing list
>>>> [hidden email]
>>>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
>>>
>>> _______________________________________________
>>> jboss-as7-dev mailing list
>>> [hidden email]
>>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
>>
>> _______________________________________________
>> jboss-as7-dev mailing list
>> [hidden email]
>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
>
> _______________________________________________
> jboss-as7-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev


--
Brian Stansberry
Principal Software Engineer
JBoss by Red Hat
_______________________________________________
jboss-as7-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
Reply | Threaded
Open this post in threaded view
|

Re: consistent testsuite failures since I synced with master yesterday

Scott Marlow
I agree that we should be closing the connection (https://anonsvn.jboss.org/repos/jbossas/trunk/console/src/main/java/org/jboss/console/twiddle/Twiddle.java is an example of where we handled that in AS6).  Probably after we are done with the MBeanServerConnection.

I created ARQ-632 for this.

I was expecting the cause to be a regression but instead perhaps its the performance improvements (we are running unit tests quicker, causing more "jmx server connection timeout" threads to be created).  Or, due to the number of tests in the testsuite now.

----- Original Message -----
From: "Brian Stansberry" <[hidden email]>
To: [hidden email]
Sent: Wednesday, October 19, 2011 5:47:41 PM
Subject: Re: [jboss-as7-dev] consistent testsuite failures since I synced with master yesterday

org.jboss.arquillian.container.spi.client.protocol.metadata.JMXContext.getConnection()
is creating a javax.management.remote.JMXConnector but I don't see
anything closing it. I'm not sure, but perhaps one of those is created
per test method execution?

I believe the "JMX server connection timeout 552" threads are created by
the JDK stuff AS7 is using for its JMX connector impl. You can see the
relevant code in the stack trace. A quick look at that code leads me to
believe those threads will by default live for 120 secs after the last
request on the connection.

On 10/19/11 11:54 AM, Scott Marlow wrote:

> On 10/19/2011 12:13 PM, Scott Marlow wrote:
>> On 10/19/2011 11:01 AM, Scott Marlow wrote:
>>> I'm using Sun 1.6.0_26-b03.  I think Hudson is passing with Sun
>>> 1.6.0_24-b07.
>>>
>>> Jesper also got the failures with jdk 6 u26.
>>>
>>> I haven't tried running with my old jdk1.6.0_24 yet.  I could try that
>>> or we could start comparing environment settings.
>>>
>>> On Hudson, we are setting MAVEN_OPTS="-Xmx1024m -Xms512m
>>> -XX:MaxPermSize=128m"
>>>
>>> I don't have that set locally.  I'll try that next...
>>
>> Testsuite still fails for me with the above MAVEN_OPTS set.
>>
>> I'll my older jdk 6 (same as hudson is using) and see if that makes a diff.
>
> Same failure still (with MAVEN_OPTS set as above and using jdk1.6.0_24).
>
>>
>>
>>>
>>> Scott
>>>
>>> On 10/19/2011 10:49 AM, Jaikiran Pai wrote:
>>>> I don't know about the test issues that Ondrej is running into, but the
>>>> ones you mention http://pastebin.com/D3SeJ2rP (OutOfMemory : Cannot
>>>> create native thread) were encountered once by Carlo (or on our Hudson
>>>> instance, don't remember). But I haven't seen them after that. It might
>>>> be environment specific. What JDK vendor/version? And are you running
>>>> into this always?
>>>>
>>>> -Jaikiran
>>>> On Wednesday 19 October 2011 08:14 PM, Scott Marlow wrote:
>>>>> How did these testsuite failures slip in?  From my own experience
>>>>> running the updated testsuite, it looks like we now have an allTests
>>>>> profile that doesn't run the integration tests.  Perhaps we merged some
>>>>> changes and only ran the allTests profile.  I think that we need to
>>>>> continue to use -DallTests instead (that includes the integration
>>>>> tests).  Does anyone know more?
>>>>>
>>>>> I'm not sure why some people aren't seeing the failures but it seems we
>>>>> need to track backwards from the symptoms (listed below) or rollback
>>>>> some changes until it goes away.
>>>>>
>>>>> If anyone is working on tracking the cause down, please speak up.  I can
>>>>> jump in but don't want to duplicate effort.
>>>>>
>>>>> Scott
>>>>>
>>>>> On 10/18/2011 02:15 PM, Scott Marlow wrote:
>>>>>> The hudson AS7 test just finished with all integration tests passing.
>>>>>>
>>>>>> When you say that your using a fresh master, is that a new AS7 source
>>>>>> tree or an existing one freshly synced with AS7 master?
>>>>>>
>>>>>>
>>>>>> On 10/18/2011 12:26 PM, Scott Marlow wrote:
>>>>>>> On 10/18/2011 08:57 AM, Ondřej Žižka wrote:
>>>>>>>> I have a fresh master and testsuite runs fine.
>>>>>>>> Except 2 failures (for which I also seek comments):
>>>>>>>>
>>>>>>>> org.jboss.as.test.integration.osgi.webapp.ServletIntegrationTestCase.testServiceAccess
>>>>>>>> org.jboss.as.test.integration.osgi.jaxrs.RestEasyIntegrationTestCase.org.jboss.as.test.integration.osgi.jaxrs.RestEasyIntegrationTestCase
>>>>>>>>
>>>>>>>> server.log:
>>>>>>>> http://pastebin.test.redhat.com/64949
>>>>>>>
>>>>>>> I didn't see that failure.
>>>>>>>
>>>>>>> Which folder are you starting the unit test from and what is your
>>>>>>> command line?
>>>>>>>
>>>>>>> I was trying to use "mvn clean install -DallTests" in the
>>>>>>> as7/testsuite/integration folder.
>>>>>>>
>>>>>>> When I checked earlier, Hudson wasn't running the full test (was using
>>>>>>> -PallTests instead of -DallTests).  Looks like someone changed that but
>>>>>>> the test is still in queue.
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>> Anyone seeing that? I thought it could be a race condition caused by
>>>>>>>> improper use of deployer API, but giving it some time after deploy
>>>>>>>> didn't help.
>>>>>>>>
>>>>>>>> Ondra
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Scott Marlow píše v Út 18. 10. 2011 v 08:36 -0400:
>>>>>>>>
>>>>>>>>> When running the as7/testsuite/integration tests, we run fine for a
>>>>>>>>> while but then "OutOfMemoryError: unable to create new native thread"
>>>>>>>>> errors start showing up in the server.log.
>>>>>>>>>
>>>>>>>>> Failing tests are here http://pastebin.com/D3SeJ2rP
>>>>>>>>>
>>>>>>>>> AS7 thread dump showing a lot of "JMX server connection timeout NNN"
>>>>>>>>> threads is here http://pastebin.com/qDC52WJG
>>>>>>>>>
>>>>>>>>> To recreate change into as7/testsuite/integration and do a "mvn clean
>>>>>>>>> install -DallTests".
>>>>>>>>>
>>>>>>>>> Is anyone not seeing this after syncing with master?
>>>>>>>>>
>>>>>>>>> Scott
>>>>>>>>> _______________________________________________
>>>>>>>>> jboss-as7-dev mailing list
>>>>>>>>> [hidden email]
>>>>>>>>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
>>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> jboss-as7-dev mailing list
>>>>>>> [hidden email]
>>>>>>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
>>>>>>
>>>>>> _______________________________________________
>>>>>> jboss-as7-dev mailing list
>>>>>> [hidden email]
>>>>>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
>>>>>
>>>>> _______________________________________________
>>>>> jboss-as7-dev mailing list
>>>>> [hidden email]
>>>>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
>>>>
>>>> _______________________________________________
>>>> jboss-as7-dev mailing list
>>>> [hidden email]
>>>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
>>>
>>> _______________________________________________
>>> jboss-as7-dev mailing list
>>> [hidden email]
>>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
>>
>> _______________________________________________
>> jboss-as7-dev mailing list
>> [hidden email]
>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
>
> _______________________________________________
> jboss-as7-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev


--
Brian Stansberry
Principal Software Engineer
JBoss by Red Hat
_______________________________________________
jboss-as7-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/jboss-as7-dev

_______________________________________________
jboss-as7-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
Reply | Threaded
Open this post in threaded view
|

Re: consistent testsuite failures since I synced with master yesterday

Stuart Douglas
It is probably to do with the number of tests in the test suite, as now all the tests that were previously in preview and spec are executed in one run.

Stuart


On 20/10/2011, at 10:24 AM, Scott Marlow wrote:

> I agree that we should be closing the connection (https://anonsvn.jboss.org/repos/jbossas/trunk/console/src/main/java/org/jboss/console/twiddle/Twiddle.java is an example of where we handled that in AS6).  Probably after we are done with the MBeanServerConnection.
>
> I created ARQ-632 for this.
>
> I was expecting the cause to be a regression but instead perhaps its the performance improvements (we are running unit tests quicker, causing more "jmx server connection timeout" threads to be created).  Or, due to the number of tests in the testsuite now.
>
> ----- Original Message -----
> From: "Brian Stansberry" <[hidden email]>
> To: [hidden email]
> Sent: Wednesday, October 19, 2011 5:47:41 PM
> Subject: Re: [jboss-as7-dev] consistent testsuite failures since I synced with master yesterday
>
> org.jboss.arquillian.container.spi.client.protocol.metadata.JMXContext.getConnection()
> is creating a javax.management.remote.JMXConnector but I don't see
> anything closing it. I'm not sure, but perhaps one of those is created
> per test method execution?
>
> I believe the "JMX server connection timeout 552" threads are created by
> the JDK stuff AS7 is using for its JMX connector impl. You can see the
> relevant code in the stack trace. A quick look at that code leads me to
> believe those threads will by default live for 120 secs after the last
> request on the connection.
>
> On 10/19/11 11:54 AM, Scott Marlow wrote:
>> On 10/19/2011 12:13 PM, Scott Marlow wrote:
>>> On 10/19/2011 11:01 AM, Scott Marlow wrote:
>>>> I'm using Sun 1.6.0_26-b03.  I think Hudson is passing with Sun
>>>> 1.6.0_24-b07.
>>>>
>>>> Jesper also got the failures with jdk 6 u26.
>>>>
>>>> I haven't tried running with my old jdk1.6.0_24 yet.  I could try that
>>>> or we could start comparing environment settings.
>>>>
>>>> On Hudson, we are setting MAVEN_OPTS="-Xmx1024m -Xms512m
>>>> -XX:MaxPermSize=128m"
>>>>
>>>> I don't have that set locally.  I'll try that next...
>>>
>>> Testsuite still fails for me with the above MAVEN_OPTS set.
>>>
>>> I'll my older jdk 6 (same as hudson is using) and see if that makes a diff.
>>
>> Same failure still (with MAVEN_OPTS set as above and using jdk1.6.0_24).
>>
>>>
>>>
>>>>
>>>> Scott
>>>>
>>>> On 10/19/2011 10:49 AM, Jaikiran Pai wrote:
>>>>> I don't know about the test issues that Ondrej is running into, but the
>>>>> ones you mention http://pastebin.com/D3SeJ2rP (OutOfMemory : Cannot
>>>>> create native thread) were encountered once by Carlo (or on our Hudson
>>>>> instance, don't remember). But I haven't seen them after that. It might
>>>>> be environment specific. What JDK vendor/version? And are you running
>>>>> into this always?
>>>>>
>>>>> -Jaikiran
>>>>> On Wednesday 19 October 2011 08:14 PM, Scott Marlow wrote:
>>>>>> How did these testsuite failures slip in?  From my own experience
>>>>>> running the updated testsuite, it looks like we now have an allTests
>>>>>> profile that doesn't run the integration tests.  Perhaps we merged some
>>>>>> changes and only ran the allTests profile.  I think that we need to
>>>>>> continue to use -DallTests instead (that includes the integration
>>>>>> tests).  Does anyone know more?
>>>>>>
>>>>>> I'm not sure why some people aren't seeing the failures but it seems we
>>>>>> need to track backwards from the symptoms (listed below) or rollback
>>>>>> some changes until it goes away.
>>>>>>
>>>>>> If anyone is working on tracking the cause down, please speak up.  I can
>>>>>> jump in but don't want to duplicate effort.
>>>>>>
>>>>>> Scott
>>>>>>
>>>>>> On 10/18/2011 02:15 PM, Scott Marlow wrote:
>>>>>>> The hudson AS7 test just finished with all integration tests passing.
>>>>>>>
>>>>>>> When you say that your using a fresh master, is that a new AS7 source
>>>>>>> tree or an existing one freshly synced with AS7 master?
>>>>>>>
>>>>>>>
>>>>>>> On 10/18/2011 12:26 PM, Scott Marlow wrote:
>>>>>>>> On 10/18/2011 08:57 AM, Ondřej Žižka wrote:
>>>>>>>>> I have a fresh master and testsuite runs fine.
>>>>>>>>> Except 2 failures (for which I also seek comments):
>>>>>>>>>
>>>>>>>>> org.jboss.as.test.integration.osgi.webapp.ServletIntegrationTestCase.testServiceAccess
>>>>>>>>> org.jboss.as.test.integration.osgi.jaxrs.RestEasyIntegrationTestCase.org.jboss.as.test.integration.osgi.jaxrs.RestEasyIntegrationTestCase
>>>>>>>>>
>>>>>>>>> server.log:
>>>>>>>>> http://pastebin.test.redhat.com/64949
>>>>>>>>
>>>>>>>> I didn't see that failure.
>>>>>>>>
>>>>>>>> Which folder are you starting the unit test from and what is your
>>>>>>>> command line?
>>>>>>>>
>>>>>>>> I was trying to use "mvn clean install -DallTests" in the
>>>>>>>> as7/testsuite/integration folder.
>>>>>>>>
>>>>>>>> When I checked earlier, Hudson wasn't running the full test (was using
>>>>>>>> -PallTests instead of -DallTests).  Looks like someone changed that but
>>>>>>>> the test is still in queue.
>>>>>>>>
>>>>>>>>
>>>>>>>>>
>>>>>>>>> Anyone seeing that? I thought it could be a race condition caused by
>>>>>>>>> improper use of deployer API, but giving it some time after deploy
>>>>>>>>> didn't help.
>>>>>>>>>
>>>>>>>>> Ondra
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Scott Marlow píše v Út 18. 10. 2011 v 08:36 -0400:
>>>>>>>>>
>>>>>>>>>> When running the as7/testsuite/integration tests, we run fine for a
>>>>>>>>>> while but then "OutOfMemoryError: unable to create new native thread"
>>>>>>>>>> errors start showing up in the server.log.
>>>>>>>>>>
>>>>>>>>>> Failing tests are here http://pastebin.com/D3SeJ2rP
>>>>>>>>>>
>>>>>>>>>> AS7 thread dump showing a lot of "JMX server connection timeout NNN"
>>>>>>>>>> threads is here http://pastebin.com/qDC52WJG
>>>>>>>>>>
>>>>>>>>>> To recreate change into as7/testsuite/integration and do a "mvn clean
>>>>>>>>>> install -DallTests".
>>>>>>>>>>
>>>>>>>>>> Is anyone not seeing this after syncing with master?
>>>>>>>>>>
>>>>>>>>>> Scott
>>>>>>>>>> _______________________________________________
>>>>>>>>>> jboss-as7-dev mailing list
>>>>>>>>>> [hidden email]
>>>>>>>>>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
>>>>>>>>>
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> jboss-as7-dev mailing list
>>>>>>>> [hidden email]
>>>>>>>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> jboss-as7-dev mailing list
>>>>>>> [hidden email]
>>>>>>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
>>>>>>
>>>>>> _______________________________________________
>>>>>> jboss-as7-dev mailing list
>>>>>> [hidden email]
>>>>>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
>>>>>
>>>>> _______________________________________________
>>>>> jboss-as7-dev mailing list
>>>>> [hidden email]
>>>>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
>>>>
>>>> _______________________________________________
>>>> jboss-as7-dev mailing list
>>>> [hidden email]
>>>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
>>>
>>> _______________________________________________
>>> jboss-as7-dev mailing list
>>> [hidden email]
>>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
>>
>> _______________________________________________
>> jboss-as7-dev mailing list
>> [hidden email]
>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
>
>
> --
> Brian Stansberry
> Principal Software Engineer
> JBoss by Red Hat
> _______________________________________________
> jboss-as7-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
>
> _______________________________________________
> jboss-as7-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev


_______________________________________________
jboss-as7-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
Reply | Threaded
Open this post in threaded view
|

Re: consistent testsuite failures since I synced with master yesterday

Dimitris Andreadis
I have trouble with the testsuite harness, too:

http://pastebin.com/tCRnHHEd

On 20/10/2011 02:26, Stuart Douglas wrote:

> It is probably to do with the number of tests in the test suite, as now all the tests that were previously in preview and spec are executed in one run.
>
> Stuart
>
>
> On 20/10/2011, at 10:24 AM, Scott Marlow wrote:
>
>> I agree that we should be closing the connection (https://anonsvn.jboss.org/repos/jbossas/trunk/console/src/main/java/org/jboss/console/twiddle/Twiddle.java is an example of where we handled that in AS6).  Probably after we are done with the MBeanServerConnection.
>>
>> I created ARQ-632 for this.
>>
>> I was expecting the cause to be a regression but instead perhaps its the performance improvements (we are running unit tests quicker, causing more "jmx server connection timeout" threads to be created).  Or, due to the number of tests in the testsuite now.
>>
>> ----- Original Message -----
>> From: "Brian Stansberry"<[hidden email]>
>> To: [hidden email]
>> Sent: Wednesday, October 19, 2011 5:47:41 PM
>> Subject: Re: [jboss-as7-dev] consistent testsuite failures since I synced with master yesterday
>>
>> org.jboss.arquillian.container.spi.client.protocol.metadata.JMXContext.getConnection()
>> is creating a javax.management.remote.JMXConnector but I don't see
>> anything closing it. I'm not sure, but perhaps one of those is created
>> per test method execution?
>>
>> I believe the "JMX server connection timeout 552" threads are created by
>> the JDK stuff AS7 is using for its JMX connector impl. You can see the
>> relevant code in the stack trace. A quick look at that code leads me to
>> believe those threads will by default live for 120 secs after the last
>> request on the connection.
>>
>> On 10/19/11 11:54 AM, Scott Marlow wrote:
>>> On 10/19/2011 12:13 PM, Scott Marlow wrote:
>>>> On 10/19/2011 11:01 AM, Scott Marlow wrote:
>>>>> I'm using Sun 1.6.0_26-b03.  I think Hudson is passing with Sun
>>>>> 1.6.0_24-b07.
>>>>>
>>>>> Jesper also got the failures with jdk 6 u26.
>>>>>
>>>>> I haven't tried running with my old jdk1.6.0_24 yet.  I could try that
>>>>> or we could start comparing environment settings.
>>>>>
>>>>> On Hudson, we are setting MAVEN_OPTS="-Xmx1024m -Xms512m
>>>>> -XX:MaxPermSize=128m"
>>>>>
>>>>> I don't have that set locally.  I'll try that next...
>>>>
>>>> Testsuite still fails for me with the above MAVEN_OPTS set.
>>>>
>>>> I'll my older jdk 6 (same as hudson is using) and see if that makes a diff.
>>>
>>> Same failure still (with MAVEN_OPTS set as above and using jdk1.6.0_24).
>>>
>>>>
>>>>
>>>>>
>>>>> Scott
>>>>>
>>>>> On 10/19/2011 10:49 AM, Jaikiran Pai wrote:
>>>>>> I don't know about the test issues that Ondrej is running into, but the
>>>>>> ones you mention http://pastebin.com/D3SeJ2rP (OutOfMemory : Cannot
>>>>>> create native thread) were encountered once by Carlo (or on our Hudson
>>>>>> instance, don't remember). But I haven't seen them after that. It might
>>>>>> be environment specific. What JDK vendor/version? And are you running
>>>>>> into this always?
>>>>>>
>>>>>> -Jaikiran
>>>>>> On Wednesday 19 October 2011 08:14 PM, Scott Marlow wrote:
>>>>>>> How did these testsuite failures slip in?  From my own experience
>>>>>>> running the updated testsuite, it looks like we now have an allTests
>>>>>>> profile that doesn't run the integration tests.  Perhaps we merged some
>>>>>>> changes and only ran the allTests profile.  I think that we need to
>>>>>>> continue to use -DallTests instead (that includes the integration
>>>>>>> tests).  Does anyone know more?
>>>>>>>
>>>>>>> I'm not sure why some people aren't seeing the failures but it seems we
>>>>>>> need to track backwards from the symptoms (listed below) or rollback
>>>>>>> some changes until it goes away.
>>>>>>>
>>>>>>> If anyone is working on tracking the cause down, please speak up.  I can
>>>>>>> jump in but don't want to duplicate effort.
>>>>>>>
>>>>>>> Scott
>>>>>>>
>>>>>>> On 10/18/2011 02:15 PM, Scott Marlow wrote:
>>>>>>>> The hudson AS7 test just finished with all integration tests passing.
>>>>>>>>
>>>>>>>> When you say that your using a fresh master, is that a new AS7 source
>>>>>>>> tree or an existing one freshly synced with AS7 master?
>>>>>>>>
>>>>>>>>
>>>>>>>> On 10/18/2011 12:26 PM, Scott Marlow wrote:
>>>>>>>>> On 10/18/2011 08:57 AM, Ondřej Žižka wrote:
>>>>>>>>>> I have a fresh master and testsuite runs fine.
>>>>>>>>>> Except 2 failures (for which I also seek comments):
>>>>>>>>>>
>>>>>>>>>> org.jboss.as.test.integration.osgi.webapp.ServletIntegrationTestCase.testServiceAccess
>>>>>>>>>> org.jboss.as.test.integration.osgi.jaxrs.RestEasyIntegrationTestCase.org.jboss.as.test.integration.osgi.jaxrs.RestEasyIntegrationTestCase
>>>>>>>>>>
>>>>>>>>>> server.log:
>>>>>>>>>> http://pastebin.test.redhat.com/64949
>>>>>>>>>
>>>>>>>>> I didn't see that failure.
>>>>>>>>>
>>>>>>>>> Which folder are you starting the unit test from and what is your
>>>>>>>>> command line?
>>>>>>>>>
>>>>>>>>> I was trying to use "mvn clean install -DallTests" in the
>>>>>>>>> as7/testsuite/integration folder.
>>>>>>>>>
>>>>>>>>> When I checked earlier, Hudson wasn't running the full test (was using
>>>>>>>>> -PallTests instead of -DallTests).  Looks like someone changed that but
>>>>>>>>> the test is still in queue.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Anyone seeing that? I thought it could be a race condition caused by
>>>>>>>>>> improper use of deployer API, but giving it some time after deploy
>>>>>>>>>> didn't help.
>>>>>>>>>>
>>>>>>>>>> Ondra
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Scott Marlow píše v Út 18. 10. 2011 v 08:36 -0400:
>>>>>>>>>>
>>>>>>>>>>> When running the as7/testsuite/integration tests, we run fine for a
>>>>>>>>>>> while but then "OutOfMemoryError: unable to create new native thread"
>>>>>>>>>>> errors start showing up in the server.log.
>>>>>>>>>>>
>>>>>>>>>>> Failing tests are here http://pastebin.com/D3SeJ2rP
>>>>>>>>>>>
>>>>>>>>>>> AS7 thread dump showing a lot of "JMX server connection timeout NNN"
>>>>>>>>>>> threads is here http://pastebin.com/qDC52WJG
>>>>>>>>>>>
>>>>>>>>>>> To recreate change into as7/testsuite/integration and do a "mvn clean
>>>>>>>>>>> install -DallTests".
>>>>>>>>>>>
>>>>>>>>>>> Is anyone not seeing this after syncing with master?
>>>>>>>>>>>
>>>>>>>>>>> Scott
>>>>>>>>>>> _______________________________________________
>>>>>>>>>>> jboss-as7-dev mailing list
>>>>>>>>>>> [hidden email]
>>>>>>>>>>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> jboss-as7-dev mailing list
>>>>>>>>> [hidden email]
>>>>>>>>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> jboss-as7-dev mailing list
>>>>>>>> [hidden email]
>>>>>>>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> jboss-as7-dev mailing list
>>>>>>> [hidden email]
>>>>>>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
>>>>>>
>>>>>> _______________________________________________
>>>>>> jboss-as7-dev mailing list
>>>>>> [hidden email]
>>>>>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
>>>>>
>>>>> _______________________________________________
>>>>> jboss-as7-dev mailing list
>>>>> [hidden email]
>>>>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
>>>>
>>>> _______________________________________________
>>>> jboss-as7-dev mailing list
>>>> [hidden email]
>>>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
>>>
>>> _______________________________________________
>>> jboss-as7-dev mailing list
>>> [hidden email]
>>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
>>
>>
>> --
>> Brian Stansberry
>> Principal Software Engineer
>> JBoss by Red Hat
>> _______________________________________________
>> jboss-as7-dev mailing list
>> [hidden email]
>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
>>
>> _______________________________________________
>> jboss-as7-dev mailing list
>> [hidden email]
>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
>
>
> _______________________________________________
> jboss-as7-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev

--
xxxxxxxxxxxxxxxxxxxxxxxxxxxx
Dimitris Andreadis
Software Engineering Manager
JBoss Application Server
by Red Hat
xxxxxxxxxxxxxxxxxxxxxxxxxxxx

http://dandreadis.blogspot.com/
_______________________________________________
jboss-as7-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
Reply | Threaded
Open this post in threaded view
|

Re: consistent testsuite failures since I synced with master yesterday

Brian Stansberry
In reply to this post by Stuart Douglas
See [1] for a good discussion of this; last post points at a possible
workaround.

[1] https://forums.oracle.com/forums/thread.jspa?threadID=1665966
On 10/19/11 6:26 PM, Stuart Douglas wrote:

> It is probably to do with the number of tests in the test suite, as now all the tests that were previously in preview and spec are executed in one run.
>
> Stuart
>
>
> On 20/10/2011, at 10:24 AM, Scott Marlow wrote:
>
>> I agree that we should be closing the connection (https://anonsvn.jboss.org/repos/jbossas/trunk/console/src/main/java/org/jboss/console/twiddle/Twiddle.java is an example of where we handled that in AS6).  Probably after we are done with the MBeanServerConnection.
>>
>> I created ARQ-632 for this.
>>
>> I was expecting the cause to be a regression but instead perhaps its the performance improvements (we are running unit tests quicker, causing more "jmx server connection timeout" threads to be created).  Or, due to the number of tests in the testsuite now.
>>
>> ----- Original Message -----
>> From: "Brian Stansberry"<[hidden email]>
>> To: [hidden email]
>> Sent: Wednesday, October 19, 2011 5:47:41 PM
>> Subject: Re: [jboss-as7-dev] consistent testsuite failures since I synced with master yesterday
>>
>> org.jboss.arquillian.container.spi.client.protocol.metadata.JMXContext.getConnection()
>> is creating a javax.management.remote.JMXConnector but I don't see
>> anything closing it. I'm not sure, but perhaps one of those is created
>> per test method execution?
>>
>> I believe the "JMX server connection timeout 552" threads are created by
>> the JDK stuff AS7 is using for its JMX connector impl. You can see the
>> relevant code in the stack trace. A quick look at that code leads me to
>> believe those threads will by default live for 120 secs after the last
>> request on the connection.
>>
>> On 10/19/11 11:54 AM, Scott Marlow wrote:
>>> On 10/19/2011 12:13 PM, Scott Marlow wrote:
>>>> On 10/19/2011 11:01 AM, Scott Marlow wrote:
>>>>> I'm using Sun 1.6.0_26-b03.  I think Hudson is passing with Sun
>>>>> 1.6.0_24-b07.
>>>>>
>>>>> Jesper also got the failures with jdk 6 u26.
>>>>>
>>>>> I haven't tried running with my old jdk1.6.0_24 yet.  I could try that
>>>>> or we could start comparing environment settings.
>>>>>
>>>>> On Hudson, we are setting MAVEN_OPTS="-Xmx1024m -Xms512m
>>>>> -XX:MaxPermSize=128m"
>>>>>
>>>>> I don't have that set locally.  I'll try that next...
>>>>
>>>> Testsuite still fails for me with the above MAVEN_OPTS set.
>>>>
>>>> I'll my older jdk 6 (same as hudson is using) and see if that makes a diff.
>>>
>>> Same failure still (with MAVEN_OPTS set as above and using jdk1.6.0_24).
>>>
>>>>
>>>>
>>>>>
>>>>> Scott
>>>>>
>>>>> On 10/19/2011 10:49 AM, Jaikiran Pai wrote:
>>>>>> I don't know about the test issues that Ondrej is running into, but the
>>>>>> ones you mention http://pastebin.com/D3SeJ2rP (OutOfMemory : Cannot
>>>>>> create native thread) were encountered once by Carlo (or on our Hudson
>>>>>> instance, don't remember). But I haven't seen them after that. It might
>>>>>> be environment specific. What JDK vendor/version? And are you running
>>>>>> into this always?
>>>>>>
>>>>>> -Jaikiran
>>>>>> On Wednesday 19 October 2011 08:14 PM, Scott Marlow wrote:
>>>>>>> How did these testsuite failures slip in?  From my own experience
>>>>>>> running the updated testsuite, it looks like we now have an allTests
>>>>>>> profile that doesn't run the integration tests.  Perhaps we merged some
>>>>>>> changes and only ran the allTests profile.  I think that we need to
>>>>>>> continue to use -DallTests instead (that includes the integration
>>>>>>> tests).  Does anyone know more?
>>>>>>>
>>>>>>> I'm not sure why some people aren't seeing the failures but it seems we
>>>>>>> need to track backwards from the symptoms (listed below) or rollback
>>>>>>> some changes until it goes away.
>>>>>>>
>>>>>>> If anyone is working on tracking the cause down, please speak up.  I can
>>>>>>> jump in but don't want to duplicate effort.
>>>>>>>
>>>>>>> Scott
>>>>>>>
>>>>>>> On 10/18/2011 02:15 PM, Scott Marlow wrote:
>>>>>>>> The hudson AS7 test just finished with all integration tests passing.
>>>>>>>>
>>>>>>>> When you say that your using a fresh master, is that a new AS7 source
>>>>>>>> tree or an existing one freshly synced with AS7 master?
>>>>>>>>
>>>>>>>>
>>>>>>>> On 10/18/2011 12:26 PM, Scott Marlow wrote:
>>>>>>>>> On 10/18/2011 08:57 AM, Ondřej Žižka wrote:
>>>>>>>>>> I have a fresh master and testsuite runs fine.
>>>>>>>>>> Except 2 failures (for which I also seek comments):
>>>>>>>>>>
>>>>>>>>>> org.jboss.as.test.integration.osgi.webapp.ServletIntegrationTestCase.testServiceAccess
>>>>>>>>>> org.jboss.as.test.integration.osgi.jaxrs.RestEasyIntegrationTestCase.org.jboss.as.test.integration.osgi.jaxrs.RestEasyIntegrationTestCase
>>>>>>>>>>
>>>>>>>>>> server.log:
>>>>>>>>>> http://pastebin.test.redhat.com/64949
>>>>>>>>>
>>>>>>>>> I didn't see that failure.
>>>>>>>>>
>>>>>>>>> Which folder are you starting the unit test from and what is your
>>>>>>>>> command line?
>>>>>>>>>
>>>>>>>>> I was trying to use "mvn clean install -DallTests" in the
>>>>>>>>> as7/testsuite/integration folder.
>>>>>>>>>
>>>>>>>>> When I checked earlier, Hudson wasn't running the full test (was using
>>>>>>>>> -PallTests instead of -DallTests).  Looks like someone changed that but
>>>>>>>>> the test is still in queue.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Anyone seeing that? I thought it could be a race condition caused by
>>>>>>>>>> improper use of deployer API, but giving it some time after deploy
>>>>>>>>>> didn't help.
>>>>>>>>>>
>>>>>>>>>> Ondra
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Scott Marlow píše v Út 18. 10. 2011 v 08:36 -0400:
>>>>>>>>>>
>>>>>>>>>>> When running the as7/testsuite/integration tests, we run fine for a
>>>>>>>>>>> while but then "OutOfMemoryError: unable to create new native thread"
>>>>>>>>>>> errors start showing up in the server.log.
>>>>>>>>>>>
>>>>>>>>>>> Failing tests are here http://pastebin.com/D3SeJ2rP
>>>>>>>>>>>
>>>>>>>>>>> AS7 thread dump showing a lot of "JMX server connection timeout NNN"
>>>>>>>>>>> threads is here http://pastebin.com/qDC52WJG
>>>>>>>>>>>
>>>>>>>>>>> To recreate change into as7/testsuite/integration and do a "mvn clean
>>>>>>>>>>> install -DallTests".
>>>>>>>>>>>
>>>>>>>>>>> Is anyone not seeing this after syncing with master?
>>>>>>>>>>>
>>>>>>>>>>> Scott
>>>>>>>>>>> _______________________________________________
>>>>>>>>>>> jboss-as7-dev mailing list
>>>>>>>>>>> [hidden email]
>>>>>>>>>>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> jboss-as7-dev mailing list
>>>>>>>>> [hidden email]
>>>>>>>>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> jboss-as7-dev mailing list
>>>>>>>> [hidden email]
>>>>>>>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> jboss-as7-dev mailing list
>>>>>>> [hidden email]
>>>>>>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
>>>>>>
>>>>>> _______________________________________________
>>>>>> jboss-as7-dev mailing list
>>>>>> [hidden email]
>>>>>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
>>>>>
>>>>> _______________________________________________
>>>>> jboss-as7-dev mailing list
>>>>> [hidden email]
>>>>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
>>>>
>>>> _______________________________________________
>>>> jboss-as7-dev mailing list
>>>> [hidden email]
>>>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
>>>
>>> _______________________________________________
>>> jboss-as7-dev mailing list
>>> [hidden email]
>>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
>>
>>
>> --
>> Brian Stansberry
>> Principal Software Engineer
>> JBoss by Red Hat
>> _______________________________________________
>> jboss-as7-dev mailing list
>> [hidden email]
>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
>>
>> _______________________________________________
>> jboss-as7-dev mailing list
>> [hidden email]
>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
>


--
Brian Stansberry
Principal Software Engineer
JBoss by Red Hat
_______________________________________________
jboss-as7-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
Reply | Threaded
Open this post in threaded view
|

Re: consistent testsuite failures since I synced with master yesterday

Brian Stansberry
I'm not seeing this failure, so it's hard to test a workaround. But if
someone who is seeing it can give
https://github.com/bstansberry/jboss-as/commit/0d422b184760b95e47c886f139426b512341a333 
a spin and let me know, I'd appreciate it.

Probably easiest is just to check out and build
https://github.com/bstansberry/jboss-as/tree/jmx-conn-leak

On 10/20/11 11:49 AM, Brian Stansberry wrote:

> See [1] for a good discussion of this; last post points at a possible
> workaround.
>
> [1] https://forums.oracle.com/forums/thread.jspa?threadID=1665966
> On 10/19/11 6:26 PM, Stuart Douglas wrote:
>> It is probably to do with the number of tests in the test suite, as
>> now all the tests that were previously in preview and spec are
>> executed in one run.
>>
>> Stuart
>>
>>
>> On 20/10/2011, at 10:24 AM, Scott Marlow wrote:
>>
>>> I agree that we should be closing the connection
>>> (https://anonsvn.jboss.org/repos/jbossas/trunk/console/src/main/java/org/jboss/console/twiddle/Twiddle.java
>>> is an example of where we handled that in AS6). Probably after we are
>>> done with the MBeanServerConnection.
>>>
>>> I created ARQ-632 for this.
>>>
>>> I was expecting the cause to be a regression but instead perhaps its
>>> the performance improvements (we are running unit tests quicker,
>>> causing more "jmx server connection timeout" threads to be created).
>>> Or, due to the number of tests in the testsuite now.
>>>
>>> ----- Original Message -----
>>> From: "Brian Stansberry"<[hidden email]>
>>> To: [hidden email]
>>> Sent: Wednesday, October 19, 2011 5:47:41 PM
>>> Subject: Re: [jboss-as7-dev] consistent testsuite failures since I
>>> synced with master yesterday
>>>
>>> org.jboss.arquillian.container.spi.client.protocol.metadata.JMXContext.getConnection()
>>>
>>> is creating a javax.management.remote.JMXConnector but I don't see
>>> anything closing it. I'm not sure, but perhaps one of those is created
>>> per test method execution?
>>>
>>> I believe the "JMX server connection timeout 552" threads are created by
>>> the JDK stuff AS7 is using for its JMX connector impl. You can see the
>>> relevant code in the stack trace. A quick look at that code leads me to
>>> believe those threads will by default live for 120 secs after the last
>>> request on the connection.
>>>
>>> On 10/19/11 11:54 AM, Scott Marlow wrote:
>>>> On 10/19/2011 12:13 PM, Scott Marlow wrote:
>>>>> On 10/19/2011 11:01 AM, Scott Marlow wrote:
>>>>>> I'm using Sun 1.6.0_26-b03. I think Hudson is passing with Sun
>>>>>> 1.6.0_24-b07.
>>>>>>
>>>>>> Jesper also got the failures with jdk 6 u26.
>>>>>>
>>>>>> I haven't tried running with my old jdk1.6.0_24 yet. I could try that
>>>>>> or we could start comparing environment settings.
>>>>>>
>>>>>> On Hudson, we are setting MAVEN_OPTS="-Xmx1024m -Xms512m
>>>>>> -XX:MaxPermSize=128m"
>>>>>>
>>>>>> I don't have that set locally. I'll try that next...
>>>>>
>>>>> Testsuite still fails for me with the above MAVEN_OPTS set.
>>>>>
>>>>> I'll my older jdk 6 (same as hudson is using) and see if that makes
>>>>> a diff.
>>>>
>>>> Same failure still (with MAVEN_OPTS set as above and using
>>>> jdk1.6.0_24).
>>>>
>>>>>
>>>>>
>>>>>>
>>>>>> Scott
>>>>>>
>>>>>> On 10/19/2011 10:49 AM, Jaikiran Pai wrote:
>>>>>>> I don't know about the test issues that Ondrej is running into,
>>>>>>> but the
>>>>>>> ones you mention http://pastebin.com/D3SeJ2rP (OutOfMemory : Cannot
>>>>>>> create native thread) were encountered once by Carlo (or on our
>>>>>>> Hudson
>>>>>>> instance, don't remember). But I haven't seen them after that. It
>>>>>>> might
>>>>>>> be environment specific. What JDK vendor/version? And are you
>>>>>>> running
>>>>>>> into this always?
>>>>>>>
>>>>>>> -Jaikiran
>>>>>>> On Wednesday 19 October 2011 08:14 PM, Scott Marlow wrote:
>>>>>>>> How did these testsuite failures slip in? From my own experience
>>>>>>>> running the updated testsuite, it looks like we now have an
>>>>>>>> allTests
>>>>>>>> profile that doesn't run the integration tests. Perhaps we
>>>>>>>> merged some
>>>>>>>> changes and only ran the allTests profile. I think that we need to
>>>>>>>> continue to use -DallTests instead (that includes the integration
>>>>>>>> tests). Does anyone know more?
>>>>>>>>
>>>>>>>> I'm not sure why some people aren't seeing the failures but it
>>>>>>>> seems we
>>>>>>>> need to track backwards from the symptoms (listed below) or
>>>>>>>> rollback
>>>>>>>> some changes until it goes away.
>>>>>>>>
>>>>>>>> If anyone is working on tracking the cause down, please speak
>>>>>>>> up. I can
>>>>>>>> jump in but don't want to duplicate effort.
>>>>>>>>
>>>>>>>> Scott
>>>>>>>>
>>>>>>>> On 10/18/2011 02:15 PM, Scott Marlow wrote:
>>>>>>>>> The hudson AS7 test just finished with all integration tests
>>>>>>>>> passing.
>>>>>>>>>
>>>>>>>>> When you say that your using a fresh master, is that a new AS7
>>>>>>>>> source
>>>>>>>>> tree or an existing one freshly synced with AS7 master?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 10/18/2011 12:26 PM, Scott Marlow wrote:
>>>>>>>>>> On 10/18/2011 08:57 AM, Ondřej Žižka wrote:
>>>>>>>>>>> I have a fresh master and testsuite runs fine.
>>>>>>>>>>> Except 2 failures (for which I also seek comments):
>>>>>>>>>>>
>>>>>>>>>>> org.jboss.as.test.integration.osgi.webapp.ServletIntegrationTestCase.testServiceAccess
>>>>>>>>>>>
>>>>>>>>>>> org.jboss.as.test.integration.osgi.jaxrs.RestEasyIntegrationTestCase.org.jboss.as.test.integration.osgi.jaxrs.RestEasyIntegrationTestCase
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> server.log:
>>>>>>>>>>> http://pastebin.test.redhat.com/64949
>>>>>>>>>>
>>>>>>>>>> I didn't see that failure.
>>>>>>>>>>
>>>>>>>>>> Which folder are you starting the unit test from and what is your
>>>>>>>>>> command line?
>>>>>>>>>>
>>>>>>>>>> I was trying to use "mvn clean install -DallTests" in the
>>>>>>>>>> as7/testsuite/integration folder.
>>>>>>>>>>
>>>>>>>>>> When I checked earlier, Hudson wasn't running the full test
>>>>>>>>>> (was using
>>>>>>>>>> -PallTests instead of -DallTests). Looks like someone changed
>>>>>>>>>> that but
>>>>>>>>>> the test is still in queue.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Anyone seeing that? I thought it could be a race condition
>>>>>>>>>>> caused by
>>>>>>>>>>> improper use of deployer API, but giving it some time after
>>>>>>>>>>> deploy
>>>>>>>>>>> didn't help.
>>>>>>>>>>>
>>>>>>>>>>> Ondra
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Scott Marlow píše v Út 18. 10. 2011 v 08:36 -0400:
>>>>>>>>>>>
>>>>>>>>>>>> When running the as7/testsuite/integration tests, we run
>>>>>>>>>>>> fine for a
>>>>>>>>>>>> while but then "OutOfMemoryError: unable to create new
>>>>>>>>>>>> native thread"
>>>>>>>>>>>> errors start showing up in the server.log.
>>>>>>>>>>>>
>>>>>>>>>>>> Failing tests are here http://pastebin.com/D3SeJ2rP
>>>>>>>>>>>>
>>>>>>>>>>>> AS7 thread dump showing a lot of "JMX server connection
>>>>>>>>>>>> timeout NNN"
>>>>>>>>>>>> threads is here http://pastebin.com/qDC52WJG
>>>>>>>>>>>>
>>>>>>>>>>>> To recreate change into as7/testsuite/integration and do a
>>>>>>>>>>>> "mvn clean
>>>>>>>>>>>> install -DallTests".
>>>>>>>>>>>>
>>>>>>>>>>>> Is anyone not seeing this after syncing with master?
>>>>>>>>>>>>
>>>>>>>>>>>> Scott


--
Brian Stansberry
Principal Software Engineer
JBoss by Red Hat
_______________________________________________
jboss-as7-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
Reply | Threaded
Open this post in threaded view
|

Re: consistent testsuite failures since I synced with master yesterday

Scott Marlow
I'll give that a try.

On 10/20/2011 02:36 PM, Brian Stansberry wrote:

> I'm not seeing this failure, so it's hard to test a workaround. But if
> someone who is seeing it can give
> https://github.com/bstansberry/jboss-as/commit/0d422b184760b95e47c886f139426b512341a333
> a spin and let me know, I'd appreciate it.
>
> Probably easiest is just to check out and build
> https://github.com/bstansberry/jboss-as/tree/jmx-conn-leak
>
> On 10/20/11 11:49 AM, Brian Stansberry wrote:
>> See [1] for a good discussion of this; last post points at a possible
>> workaround.
>>
>> [1] https://forums.oracle.com/forums/thread.jspa?threadID=1665966
>> On 10/19/11 6:26 PM, Stuart Douglas wrote:
>>> It is probably to do with the number of tests in the test suite, as
>>> now all the tests that were previously in preview and spec are
>>> executed in one run.
>>>
>>> Stuart
>>>
>>>
>>> On 20/10/2011, at 10:24 AM, Scott Marlow wrote:
>>>
>>>> I agree that we should be closing the connection
>>>> (https://anonsvn.jboss.org/repos/jbossas/trunk/console/src/main/java/org/jboss/console/twiddle/Twiddle.java
>>>>
>>>> is an example of where we handled that in AS6). Probably after we are
>>>> done with the MBeanServerConnection.
>>>>
>>>> I created ARQ-632 for this.
>>>>
>>>> I was expecting the cause to be a regression but instead perhaps its
>>>> the performance improvements (we are running unit tests quicker,
>>>> causing more "jmx server connection timeout" threads to be created).
>>>> Or, due to the number of tests in the testsuite now.
>>>>
>>>> ----- Original Message -----
>>>> From: "Brian Stansberry"<[hidden email]>
>>>> To: [hidden email]
>>>> Sent: Wednesday, October 19, 2011 5:47:41 PM
>>>> Subject: Re: [jboss-as7-dev] consistent testsuite failures since I
>>>> synced with master yesterday
>>>>
>>>> org.jboss.arquillian.container.spi.client.protocol.metadata.JMXContext.getConnection()
>>>>
>>>>
>>>> is creating a javax.management.remote.JMXConnector but I don't see
>>>> anything closing it. I'm not sure, but perhaps one of those is created
>>>> per test method execution?
>>>>
>>>> I believe the "JMX server connection timeout 552" threads are
>>>> created by
>>>> the JDK stuff AS7 is using for its JMX connector impl. You can see the
>>>> relevant code in the stack trace. A quick look at that code leads me to
>>>> believe those threads will by default live for 120 secs after the last
>>>> request on the connection.
>>>>
>>>> On 10/19/11 11:54 AM, Scott Marlow wrote:
>>>>> On 10/19/2011 12:13 PM, Scott Marlow wrote:
>>>>>> On 10/19/2011 11:01 AM, Scott Marlow wrote:
>>>>>>> I'm using Sun 1.6.0_26-b03. I think Hudson is passing with Sun
>>>>>>> 1.6.0_24-b07.
>>>>>>>
>>>>>>> Jesper also got the failures with jdk 6 u26.
>>>>>>>
>>>>>>> I haven't tried running with my old jdk1.6.0_24 yet. I could try
>>>>>>> that
>>>>>>> or we could start comparing environment settings.
>>>>>>>
>>>>>>> On Hudson, we are setting MAVEN_OPTS="-Xmx1024m -Xms512m
>>>>>>> -XX:MaxPermSize=128m"
>>>>>>>
>>>>>>> I don't have that set locally. I'll try that next...
>>>>>>
>>>>>> Testsuite still fails for me with the above MAVEN_OPTS set.
>>>>>>
>>>>>> I'll my older jdk 6 (same as hudson is using) and see if that makes
>>>>>> a diff.
>>>>>
>>>>> Same failure still (with MAVEN_OPTS set as above and using
>>>>> jdk1.6.0_24).
>>>>>
>>>>>>
>>>>>>
>>>>>>>
>>>>>>> Scott
>>>>>>>
>>>>>>> On 10/19/2011 10:49 AM, Jaikiran Pai wrote:
>>>>>>>> I don't know about the test issues that Ondrej is running into,
>>>>>>>> but the
>>>>>>>> ones you mention http://pastebin.com/D3SeJ2rP (OutOfMemory : Cannot
>>>>>>>> create native thread) were encountered once by Carlo (or on our
>>>>>>>> Hudson
>>>>>>>> instance, don't remember). But I haven't seen them after that. It
>>>>>>>> might
>>>>>>>> be environment specific. What JDK vendor/version? And are you
>>>>>>>> running
>>>>>>>> into this always?
>>>>>>>>
>>>>>>>> -Jaikiran
>>>>>>>> On Wednesday 19 October 2011 08:14 PM, Scott Marlow wrote:
>>>>>>>>> How did these testsuite failures slip in? From my own experience
>>>>>>>>> running the updated testsuite, it looks like we now have an
>>>>>>>>> allTests
>>>>>>>>> profile that doesn't run the integration tests. Perhaps we
>>>>>>>>> merged some
>>>>>>>>> changes and only ran the allTests profile. I think that we need to
>>>>>>>>> continue to use -DallTests instead (that includes the integration
>>>>>>>>> tests). Does anyone know more?
>>>>>>>>>
>>>>>>>>> I'm not sure why some people aren't seeing the failures but it
>>>>>>>>> seems we
>>>>>>>>> need to track backwards from the symptoms (listed below) or
>>>>>>>>> rollback
>>>>>>>>> some changes until it goes away.
>>>>>>>>>
>>>>>>>>> If anyone is working on tracking the cause down, please speak
>>>>>>>>> up. I can
>>>>>>>>> jump in but don't want to duplicate effort.
>>>>>>>>>
>>>>>>>>> Scott
>>>>>>>>>
>>>>>>>>> On 10/18/2011 02:15 PM, Scott Marlow wrote:
>>>>>>>>>> The hudson AS7 test just finished with all integration tests
>>>>>>>>>> passing.
>>>>>>>>>>
>>>>>>>>>> When you say that your using a fresh master, is that a new AS7
>>>>>>>>>> source
>>>>>>>>>> tree or an existing one freshly synced with AS7 master?
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On 10/18/2011 12:26 PM, Scott Marlow wrote:
>>>>>>>>>>> On 10/18/2011 08:57 AM, Ondřej Žižka wrote:
>>>>>>>>>>>> I have a fresh master and testsuite runs fine.
>>>>>>>>>>>> Except 2 failures (for which I also seek comments):
>>>>>>>>>>>>
>>>>>>>>>>>> org.jboss.as.test.integration.osgi.webapp.ServletIntegrationTestCase.testServiceAccess
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> org.jboss.as.test.integration.osgi.jaxrs.RestEasyIntegrationTestCase.org.jboss.as.test.integration.osgi.jaxrs.RestEasyIntegrationTestCase
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> server.log:
>>>>>>>>>>>> http://pastebin.test.redhat.com/64949
>>>>>>>>>>>
>>>>>>>>>>> I didn't see that failure.
>>>>>>>>>>>
>>>>>>>>>>> Which folder are you starting the unit test from and what is
>>>>>>>>>>> your
>>>>>>>>>>> command line?
>>>>>>>>>>>
>>>>>>>>>>> I was trying to use "mvn clean install -DallTests" in the
>>>>>>>>>>> as7/testsuite/integration folder.
>>>>>>>>>>>
>>>>>>>>>>> When I checked earlier, Hudson wasn't running the full test
>>>>>>>>>>> (was using
>>>>>>>>>>> -PallTests instead of -DallTests). Looks like someone changed
>>>>>>>>>>> that but
>>>>>>>>>>> the test is still in queue.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Anyone seeing that? I thought it could be a race condition
>>>>>>>>>>>> caused by
>>>>>>>>>>>> improper use of deployer API, but giving it some time after
>>>>>>>>>>>> deploy
>>>>>>>>>>>> didn't help.
>>>>>>>>>>>>
>>>>>>>>>>>> Ondra
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Scott Marlow píše v Út 18. 10. 2011 v 08:36 -0400:
>>>>>>>>>>>>
>>>>>>>>>>>>> When running the as7/testsuite/integration tests, we run
>>>>>>>>>>>>> fine for a
>>>>>>>>>>>>> while but then "OutOfMemoryError: unable to create new
>>>>>>>>>>>>> native thread"
>>>>>>>>>>>>> errors start showing up in the server.log.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Failing tests are here http://pastebin.com/D3SeJ2rP
>>>>>>>>>>>>>
>>>>>>>>>>>>> AS7 thread dump showing a lot of "JMX server connection
>>>>>>>>>>>>> timeout NNN"
>>>>>>>>>>>>> threads is here http://pastebin.com/qDC52WJG
>>>>>>>>>>>>>
>>>>>>>>>>>>> To recreate change into as7/testsuite/integration and do a
>>>>>>>>>>>>> "mvn clean
>>>>>>>>>>>>> install -DallTests".
>>>>>>>>>>>>>
>>>>>>>>>>>>> Is anyone not seeing this after syncing with master?
>>>>>>>>>>>>>
>>>>>>>>>>>>> Scott
>
>

_______________________________________________
jboss-as7-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
12