Testsuite fails early, hence has insufficient regression detection

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Testsuite fails early, hence has insufficient regression detection

Thomas Diesler
Folks,

when running

./integration-tests.sh -Dbasic-tests clean install

I see failures like this
Results :

Failed tests:   testChangeMessagePriority(org.jboss.as.test.integration.messaging.mgmt.CoreQueueManagementTestCase): expected:<3> but was:<1>

Tests in error:
  testAddJndi(org.jboss.as.test.integration.messaging.mgmt.JMSQueueManagementTestCase): java.io.IOException: JBAS010625: Channel closed

Tests run: 210, Failures: 1, Errors: 1, Skipped: 5
After that the testrun stops and the larger portion of the test suite is not executed. A cannot see if I have regression in jax, osgi, etc
If the basic tests must always succeed (i.e. like the smoke tests) they should be smoke tests to start of with.

Perhaps there is some magic to run the full set of basic integration tests, is there?

cheers
-thomas

PS: -fae does not make a difference


-- 
xxxxxxxxxxxxxxxxxxxxxxxxxxxx
Thomas Diesler
JBoss OSGi Lead
JBoss, a division of Red Hat
xxxxxxxxxxxxxxxxxxxxxxxxxxxx 

_______________________________________________
jboss-as7-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
Reply | Threaded
Open this post in threaded view
|

Re: Testsuite fails early, hence has insufficient regression detection

Darran Lofthouse
+1 With some of the testsuite instability I experienced in the last
couple of weeks a single flag to force a full run with a summary at the
very end would have helped a lot.

On 11/25/2011 10:52 AM, Thomas Diesler wrote:

> Folks,
>
> when running
>
> ./integration-tests.sh -Dbasic-tests clean install
>
> I see failures like this
>
>     Results :
>
>     Failed tests:
>     testChangeMessagePriority(org.jboss.as.test.integration.messaging.mgmt.CoreQueueManagementTestCase):
>     expected:<3> but was:<1>
>
>     Tests in error:
>     testAddJndi(org.jboss.as.test.integration.messaging.mgmt.JMSQueueManagementTestCase):
>     java.io.IOException: JBAS010625: Channel closed
>
>     Tests run: 210, Failures: 1, Errors: 1, Skipped: 5
>
> After that the testrun stops and the larger portion of the test suite is
> not executed. A cannot see if I have regression in jax, osgi, etc
> If the basic tests must always succeed (i.e. like the smoke tests) they
> should be smoke tests to start of with.
>
> Perhaps there is some magic to run the full set of basic integration
> tests, is there?
>
> cheers
> -thomas
>
> PS: -fae does not make a difference
>
> --
> xxxxxxxxxxxxxxxxxxxxxxxxxxxx
> Thomas Diesler
> JBoss OSGi Lead
> JBoss, a division of Red Hat
> xxxxxxxxxxxxxxxxxxxxxxxxxxxx
>
>
>
> _______________________________________________
> jboss-as7-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
_______________________________________________
jboss-as7-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
Reply | Threaded
Open this post in threaded view
|

Re: Testsuite fails early, hence has insufficient regression detection

kkhan
In reply to this post by Thomas Diesler
I just pushed the latest testsuite patch, it should now be possible to do mvn install -DallTests again.

To get a full testsuite run you can pass in the standard mvn flag '-fae', e.g.:

mvn install -DallTests=true -fae

I am not sure if that works with the integration tests script or not but that should no longer be necessary

On 25 Nov 2011, at 10:52, Thomas Diesler wrote:

> Folks,
>
> when running
>
> ./integration-tests.sh -Dbasic-tests clean install
>
> I see failures like this
> Results :
>
> Failed tests:   testChangeMessagePriority(org.jboss.as.test.integration.messaging.mgmt.CoreQueueManagementTestCase): expected:<3> but was:<1>
>
> Tests in error:
>   testAddJndi(org.jboss.as.test.integration.messaging.mgmt.JMSQueueManagementTestCase): java.io.IOException: JBAS010625: Channel closed
>
> Tests run: 210, Failures: 1, Errors: 1, Skipped: 5
> After that the testrun stops and the larger portion of the test suite is not executed. A cannot see if I have regression in jax, osgi, etc
> If the basic tests must always succeed (i.e. like the smoke tests) they should be smoke tests to start of with.
>
> Perhaps there is some magic to run the full set of basic integration tests, is there?
>
> cheers
> -thomas
>
> PS: -fae does not make a difference
>
>  
> --
> xxxxxxxxxxxxxxxxxxxxxxxxxxxx
> Thomas Diesler
> JBoss OSGi Lead
> JBoss, a division of Red Hat
> xxxxxxxxxxxxxxxxxxxxxxxxxxxx
>
> _______________________________________________
> jboss-as7-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev


_______________________________________________
jboss-as7-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
Reply | Threaded
Open this post in threaded view
|

Re: Testsuite fails early, hence has insufficient regression detection

Ondrej Zizka
In reply to this post by Darran Lofthouse
Last week, I've had a testsuite which had ALL FEATURES which you guys talk about constantly.
It was ignored.

Ondra





Darran Lofthouse píše v Pá 25. 11. 2011 v 11:02 +0000:
+1 With some of the testsuite instability I experienced in the last 
couple of weeks a single flag to force a full run with a summary at the 
very end would have helped a lot.

On 11/25/2011 10:52 AM, Thomas Diesler wrote:
> Folks,
>
> when running
>
> ./integration-tests.sh -Dbasic-tests clean install
>
> I see failures like this
>
>     Results :
>
>     Failed tests:
>     testChangeMessagePriority(org.jboss.as.test.integration.messaging.mgmt.CoreQueueManagementTestCase):
>     expected:<3> but was:<1>
>
>     Tests in error:
>     testAddJndi(org.jboss.as.test.integration.messaging.mgmt.JMSQueueManagementTestCase):
>     java.io.IOException: JBAS010625: Channel closed
>
>     Tests run: 210, Failures: 1, Errors: 1, Skipped: 5
>
> After that the testrun stops and the larger portion of the test suite is
> not executed. A cannot see if I have regression in jax, osgi, etc
> If the basic tests must always succeed (i.e. like the smoke tests) they
> should be smoke tests to start of with.
>
> Perhaps there is some magic to run the full set of basic integration
> tests, is there?
>
> cheers
> -thomas
>
> PS: -fae does not make a difference
>
> --
> xxxxxxxxxxxxxxxxxxxxxxxxxxxx
> Thomas Diesler
> JBoss OSGi Lead
> JBoss, a division of Red Hat
> xxxxxxxxxxxxxxxxxxxxxxxxxxxx
>
>
>
> _______________________________________________
> jboss-as7-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
_______________________________________________
jboss-as7-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/jboss-as7-dev

_______________________________________________
jboss-as7-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
Reply | Threaded
Open this post in threaded view
|

Re: Testsuite fails early, hence has insufficient regression detection

Ondrej Zizka
Okay I sent thiis before reading the rest of today's posts.

So there's a new harness merged, which was testsed thoroughly last week; I'll need to re-test it after rebasing (whether it works as should & satisfie all requirements).
ETA Monday evening.

Regards,
Ondra



Ondřej Žižka píše v Pá 25. 11. 2011 v 21:50 +0100:
Last week, I've had a testsuite which had ALL FEATURES which you guys talk about constantly.
It was ignored.

Ondra





Darran Lofthouse píše v Pá 25. 11. 2011 v 11:02 +0000:
+1 With some of the testsuite instability I experienced in the last 
couple of weeks a single flag to force a full run with a summary at the 
very end would have helped a lot.

On 11/25/2011 10:52 AM, Thomas Diesler wrote:
> Folks,
>
> when running
>
> ./integration-tests.sh -Dbasic-tests clean install
>
> I see failures like this
>
>     Results :
>
>     Failed tests:
>     testChangeMessagePriority(org.jboss.as.test.integration.messaging.mgmt.CoreQueueManagementTestCase):
>     expected:<3> but was:<1>
>
>     Tests in error:
>     testAddJndi(org.jboss.as.test.integration.messaging.mgmt.JMSQueueManagementTestCase):
>     java.io.IOException: JBAS010625: Channel closed
>
>     Tests run: 210, Failures: 1, Errors: 1, Skipped: 5
>
> After that the testrun stops and the larger portion of the test suite is
> not executed. A cannot see if I have regression in jax, osgi, etc
> If the basic tests must always succeed (i.e. like the smoke tests) they
> should be smoke tests to start of with.
>
> Perhaps there is some magic to run the full set of basic integration
> tests, is there?
>
> cheers
> -thomas
>
> PS: -fae does not make a difference
>
> --
> xxxxxxxxxxxxxxxxxxxxxxxxxxxx
> Thomas Diesler
> JBoss OSGi Lead
> JBoss, a division of Red Hat
> xxxxxxxxxxxxxxxxxxxxxxxxxxxx
>
>
>
> _______________________________________________
> jboss-as7-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
_______________________________________________
jboss-as7-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
_______________________________________________
jboss-as7-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/jboss-as7-dev

_______________________________________________
jboss-as7-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
Reply | Threaded
Open this post in threaded view
|

Re: Testsuite fails early, hence has insufficient regression detection

Thomas Diesler
In reply to this post by Ondrej Zizka
Hi Ondra,

a valid solution is one that addresses the requirements.
The requirements are listed here: http://community.jboss.org/wiki/ASTestsuiteRequirements

If you can show that the solution addresses these adequately in the documented way - I'm sure it will not get rejected.

cheers
-thomas

On 11/25/2011 09:50 PM, Ondřej Žižka wrote:
Last week, I've had a testsuite which had ALL FEATURES which you guys talk about constantly.
It was ignored.

Ondra





Darran Lofthouse píše v Pá 25. 11. 2011 v 11:02 +0000:
+1 With some of the testsuite instability I experienced in the last 
couple of weeks a single flag to force a full run with a summary at the 
very end would have helped a lot.

On 11/25/2011 10:52 AM, Thomas Diesler wrote:
> Folks,
>
> when running
>
> ./integration-tests.sh -Dbasic-tests clean install
>
> I see failures like this
>
>     Results :
>
>     Failed tests:
>     testChangeMessagePriority(org.jboss.as.test.integration.messaging.mgmt.CoreQueueManagementTestCase):
>     expected:<3> but was:<1>
>
>     Tests in error:
>     testAddJndi(org.jboss.as.test.integration.messaging.mgmt.JMSQueueManagementTestCase):
>     java.io.IOException: JBAS010625: Channel closed
>
>     Tests run: 210, Failures: 1, Errors: 1, Skipped: 5
>
> After that the testrun stops and the larger portion of the test suite is
> not executed. A cannot see if I have regression in jax, osgi, etc
> If the basic tests must always succeed (i.e. like the smoke tests) they
> should be smoke tests to start of with.
>
> Perhaps there is some magic to run the full set of basic integration
> tests, is there?
>
> cheers
> -thomas
>
> PS: -fae does not make a difference
>
> --
> xxxxxxxxxxxxxxxxxxxxxxxxxxxx
> Thomas Diesler
> JBoss OSGi Lead
> JBoss, a division of Red Hat
> xxxxxxxxxxxxxxxxxxxxxxxxxxxx
>
>
>
> _______________________________________________
> jboss-as7-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
_______________________________________________
jboss-as7-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/jboss-as7-dev


_______________________________________________
jboss-as7-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/jboss-as7-dev

-- 
xxxxxxxxxxxxxxxxxxxxxxxxxxxx
Thomas Diesler
JBoss OSGi Lead
JBoss, a division of Red Hat
xxxxxxxxxxxxxxxxxxxxxxxxxxxx 

_______________________________________________
jboss-as7-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/jboss-as7-dev