Testsuite excludes most tests

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Testsuite excludes most tests

Thomas Diesler
Folks,

I believe the recent changes to the testsuite effectively exclude most integration tests. We have
$ ls  testsuite/integration/src/test/java/org/jboss/as/test/integration/
beanvalidation deployment ejb jaxr jmx jsf messaging pojo sar transaction weld  xerces
common ee jaxb jaxrs jpa management osgi rar security web ws
The basic integration tests only run these in the first execution
<includes>
<include>org/jboss/as/test/integration/**/*SecondTestCase.java</include>
<include>org/jboss/as/test/integration/ejb/iiop/**/*TestCase*.java</include>
<include>org/jboss/as/test/integration/ejb/mdb/**/*TestCase*.java</include>
<include>org/jboss/as/test/integration/ejb/entity/cmp/**/*TestCase*.java</include>
<include>org/jboss/as/test/integration/ejb/remote/entity/cmp/**/*TestCase*.java</include>
<include>org/jboss/as/test/integration/ejb/management/deployments/EjbJarInEarRuntimeResourcesTestCase.java</include>
<include>org/jboss/as/test/integration/ejb/management/deployments/EjbJarRuntimeResourcesTestCase.java</include>
<include>org/jboss/as/test/integration/messaging/**/*TestCase*.java</include>
<include>org/jboss/as/test/integration/management/cli/HelpTestCase.java</include>
<include>org/jboss/as/test/integration/management/cli/JmsTestCase.java</include>
</includes>

I suppose later executions would run stuff excluding the above this is never reached (if there are failures above).

Please fix - being able to run the integration tests is important.

BTW, -DallTests is no option because that includes benchmarks, stress, clustering, which I would want only Jenkins to run.

cheers
-thomas
-- 
xxxxxxxxxxxxxxxxxxxxxxxxxxxx
Thomas Diesler
JBoss OSGi Lead
JBoss, a division of Red Hat
xxxxxxxxxxxxxxxxxxxxxxxxxxxx 

_______________________________________________
jboss-as7-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
Reply | Threaded
Open this post in threaded view
|

Re: Testsuite excludes most tests

kkhan
This is a problem coming from the full/web split - they need to be run as different surefire executions so that the tests target the appropriate configuration.
Stuart and I have talked about making this more friendly to use
  mvn install -Dtest=xx.yy.MyTest
by splitting into further sub modules, one for web and one for full with only one execution in each but that is empty
On 25 Nov 2011, at 17:24, Thomas Diesler wrote:

> Folks,
>
> I believe the recent changes to the testsuite effectively exclude most integration tests. We have
> $ ls  testsuite/integration/src/test/java/org/jboss/as/test/integration/
> beanvalidation deployment ejb jaxr jmx jsf messaging pojo sar transaction weld  xerces
> common ee jaxb jaxrs jpa management osgi rar security web ws
> The basic integration tests only run these in the first execution
>  
> <includes>
> <include>org/jboss/as/test/integration/**/*SecondTestCase.java</include>
> <include>org/jboss/as/test/integration/ejb/iiop/**/
> *TestCase*
> .java</include>
> <include>org/jboss/as/test/integration/ejb/mdb/**/
> *TestCase*
> .java</include>
> <include>org/jboss/as/test/integration/ejb/entity/cmp/**/
> *TestCase*
> .java</include>
> <include>org/jboss/as/test/integration/ejb/remote/entity/cmp/**/
> *TestCase*
> .java</include>
> <include>org/jboss/as/test/integration/ejb/management/deployments/EjbJarInEarRuntimeResourcesTestCase.java</include>
> <include>org/jboss/as/test/integration/ejb/management/deployments/EjbJarRuntimeResourcesTestCase.java</include>
> <include>org/jboss/as/test/integration/messaging/**/
> *TestCase*
> .java</include>
> <include>org/jboss/as/test/integration/management/cli/HelpTestCase.java</include>
> <include>org/jboss/as/test/integration/management/cli/JmsTestCase.java</include>
> </includes>
>
> I suppose later executions would run stuff excluding the above this is never reached (if there are failures above).
>
> Please fix - being able to run the integration tests is important.
>
> BTW, -DallTests is no option because that includes benchmarks, stress, clustering, which I would want only Jenkins to run.
>
> cheers
> -thomas
>
> --
> xxxxxxxxxxxxxxxxxxxxxxxxxxxx
> Thomas Diesler
> JBoss OSGi Lead
> JBoss, a division of Red Hat
> xxxxxxxxxxxxxxxxxxxxxxxxxxxx
>
> _______________________________________________
> jboss-as7-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev


_______________________________________________
jboss-as7-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
Reply | Threaded
Open this post in threaded view
|

Re: Testsuite excludes most tests

kkhan

> This is a problem coming from the full/web split - they need to be run as different surefire executions so that the tests target the appropriate configuration.
> Stuart and I have talked about making this more friendly to use
>  mvn install -Dtest=xx.yy.MyTest
> by splitting into further sub modules, one for web and one for full with only one execution in each but that is empty
I meant to say :-):
...but that has the same problem. If something goes wrong in the first module you won't get the second unless you use -fae

> On 25 Nov 2011, at 17:24, Thomas Diesler wrote:
>
>> Folks,
>>
>> I believe the recent changes to the testsuite effectively exclude most integration tests. We have
>> $ ls  testsuite/integration/src/test/java/org/jboss/as/test/integration/
>> beanvalidation deployment ejb jaxr jmx jsf messaging pojo sar transaction weld  xerces
>> common ee jaxb jaxrs jpa management osgi rar security web ws
>> The basic integration tests only run these in the first execution
>>
>> <includes>
>> <include>org/jboss/as/test/integration/**/*SecondTestCase.java</include>
>> <include>org/jboss/as/test/integration/ejb/iiop/**/
>> *TestCase*
>> .java</include>
>> <include>org/jboss/as/test/integration/ejb/mdb/**/
>> *TestCase*
>> .java</include>
>> <include>org/jboss/as/test/integration/ejb/entity/cmp/**/
>> *TestCase*
>> .java</include>
>> <include>org/jboss/as/test/integration/ejb/remote/entity/cmp/**/
>> *TestCase*
>> .java</include>
>> <include>org/jboss/as/test/integration/ejb/management/deployments/EjbJarInEarRuntimeResourcesTestCase.java</include>
>> <include>org/jboss/as/test/integration/ejb/management/deployments/EjbJarRuntimeResourcesTestCase.java</include>
>> <include>org/jboss/as/test/integration/messaging/**/
>> *TestCase*
>> .java</include>
>> <include>org/jboss/as/test/integration/management/cli/HelpTestCase.java</include>
>> <include>org/jboss/as/test/integration/management/cli/JmsTestCase.java</include>
>> </includes>
>>
>> I suppose later executions would run stuff excluding the above this is never reached (if there are failures above).
>>
>> Please fix - being able to run the integration tests is important.
>>
>> BTW, -DallTests is no option because that includes benchmarks, stress, clustering, which I would want only Jenkins to run.
>>
>> cheers
>> -thomas
>>
>> --
>> xxxxxxxxxxxxxxxxxxxxxxxxxxxx
>> Thomas Diesler
>> JBoss OSGi Lead
>> JBoss, a division of Red Hat
>> xxxxxxxxxxxxxxxxxxxxxxxxxxxx
>>
>> _______________________________________________
>> jboss-as7-dev mailing list
>> [hidden email]
>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
>
>
> _______________________________________________
> jboss-as7-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev


_______________________________________________
jboss-as7-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
Reply | Threaded
Open this post in threaded view
|

Re: Testsuite excludes most tests

Ondrej Zizka
In reply to this post by Thomas Diesler
Regarding this - I have added new -DallInteg for the purpose of running all integration without benchmark, stress etc.  (Again, need to re-test that after rebasing.)
See https://docs.jboss.org/author/display/AS71/AS+7+Integration+Testsuite+User+Guide .

Ondra



BTW, -DallTests is no option because that includes benchmarks, stress, clustering, which I would want only Jenkins to run.

cheers
-thomas
-- 
xxxxxxxxxxxxxxxxxxxxxxxxxxxx
Thomas Diesler
JBoss OSGi Lead
JBoss, a division of Red Hat
xxxxxxxxxxxxxxxxxxxxxxxxxxxx 

_______________________________________________
jboss-as7-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
Reply | Threaded
Open this post in threaded view
|

Re: Testsuite excludes most tests

Thomas Diesler
The use of multiple surefire executions per module is inherently problematic AFAICS. Maven addresses the need to fail at the end by the conveniently called -fae switch - it works across multiple modules.

I'd say a better approach would be to split the testsuite in sets of one server configuration each. You'd have a single surefire configuration/execution per module. Test sources can be reused between modules via standard dependencies. Multiple modules can be executed together using -pl. The full testrun can fail at the end with -fae

Ondřej, could you please check whether the multiple executions approach is even valid. E2 must run even if you have failures in E1. Especially if E2 contains the majority of tests.

cheers
-thomas

On 11/25/2011 10:00 PM, Ondřej Žižka wrote:
Regarding this - I have added new -DallInteg for the purpose of running all integration without benchmark, stress etc.  (Again, need to re-test that after rebasing.)
See https://docs.jboss.org/author/display/AS71/AS+7+Integration+Testsuite+User+Guide .

Ondra


BTW, -DallTests is no option because that includes benchmarks, stress, clustering, which I would want only Jenkins to run.

cheers
-thomas
-- 
xxxxxxxxxxxxxxxxxxxxxxxxxxxx
Thomas Diesler
JBoss OSGi Lead
JBoss, a division of Red Hat
xxxxxxxxxxxxxxxxxxxxxxxxxxxx 


_______________________________________________
jboss-as7-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/jboss-as7-dev

-- 
xxxxxxxxxxxxxxxxxxxxxxxxxxxx
Thomas Diesler
JBoss OSGi Lead
JBoss, a division of Red Hat
xxxxxxxxxxxxxxxxxxxxxxxxxxxx 

_______________________________________________
jboss-as7-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/jboss-as7-dev