Testsuite update

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Testsuite update

Ondrej Zizka
Hi everyone,

as was discussed on this list, the current testsuite harness had some problems fulfilling even basic requirements for various use cases, including but not limited to:
* Using from IDE through pure maven
* -DallTests vs. individial modules vs. distinctive groups vs. even overlaying groups
* Configuring AS using kind of hierarchical templates
* Imposing certain test groups order
* Invoking only some steps of the testsuite
* Running same tests with different AS configuration
etc etc.

I've been working on an updated concept, which is currently at https://github.com/OndraZizka/jboss-as/commits/TS-modules-tmp .
I would like you to review it, run it, and send me some comments.

     cd testsuite/integration
     mvn clean install -f pom2.xml -DallTests -fae
     mvn clean install -f pom2.xml -Dts.iiop

(There are 2 pom.xml's so I can compare two harnesses easily.)
The scripts are not updated yet so they do not work.
CLI and OSGi tests fail. To be sorted out.


The big news are:

* Execution split to modules
* Code and resources remained in the same location
* Added few system properties which provide paths to important directories - see https://docs.jboss.org/author/display/AS71/AS+7+Testsuite+Test+Developer+Guide
* Work directory is now in target/workdir

These cahnges exposed some issues in test code which need to be fixed - hardcoded relative paths.
Since there were no system properties to use, there was no other way, but from now on please use those properties, and fix your existing tests.
(This would happen anyway because some tests clutter workdir (which happens to be module's root dir) with various temp files.)


Thanks,
Ondra




_______________________________________________
jboss-as7-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
Reply | Threaded
Open this post in threaded view
|

Re: Testsuite update

Ondrej Zizka
Update on the update :)

The concept of having sources in different dir than the module running it may provoke some controversy.
Here's my brief explanation:

(05:54:13) stuartdouglas: Why is it good for running from the IDE?
(05:54:35) stuartdouglas: I would have though that having the source in the wrong module would stop that from working althogether
(05:59:07) stuartdouglas: I really dislike having the sources in the wrong module, it makes things considerably more complicated and non-standard, and I don't really see the advantage of it
(06:00:22) OndrejZizka1: Which ide?
(06:00:34) OndrejZizka1: The idea is to run the integration module with different profiles
(06:00:45) OndrejZizka1: That way it's transparent
(06:02:17) OndrejZizka1: In other words, running mvn install would still run the smoke tests and results would be in surefire-reports,
(06:02:22) OndrejZizka1: + HTML report,
(06:02:33) OndrejZizka1: and mvn install -DallTests would run all,
(06:02:58) OndrejZizka1: and also going to testsuite/integration and running mvn install -DallTests would work
(06:03:32) OndrejZizka1: The devs wouldn't touch submodules

Maven/IDE experts please correct me if my assumptions are wrong. I use NetBeans which work fine with this concept.
But after all, the code can be separated, I am not strictly against nor for.

Ondra




Ondřej Žižka píše v Pá 11. 11. 2011 v 05:50 +0100:
Hi everyone,

as was discussed on this list, the current testsuite harness had some problems fulfilling even basic requirements for various use cases, including but not limited to:
* Using from IDE through pure maven
* -DallTests vs. individial modules vs. distinctive groups vs. even overlaying groups
* Configuring AS using kind of hierarchical templates
* Imposing certain test groups order
* Invoking only some steps of the testsuite
* Running same tests with different AS configuration
etc etc.

I've been working on an updated concept, which is currently at https://github.com/OndraZizka/jboss-as/commits/TS-modules-tmp .
I would like you to review it, run it, and send me some comments.

     cd testsuite/integration
     mvn clean install -f pom2.xml -DallTests -fae
     mvn clean install -f pom2.xml -Dts.iiop

(There are 2 pom.xml's so I can compare two harnesses easily.)
The scripts are not updated yet so they do not work.
CLI and OSGi tests fail. To be sorted out.


The big news are:

* Execution split to modules
* Code and resources remained in the same location
* Added few system properties which provide paths to important directories - see https://docs.jboss.org/author/display/AS71/AS+7+Testsuite+Test+Developer+Guide
* Work directory is now in target/workdir

These cahnges exposed some issues in test code which need to be fixed - hardcoded relative paths.
Since there were no system properties to use, there was no other way, but from now on please use those properties, and fix your existing tests.
(This would happen anyway because some tests clutter workdir (which happens to be module's root dir) with various temp files.)


Thanks,
Ondra



_______________________________________________
jboss-as7-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/jboss-as7-dev

_______________________________________________
jboss-as7-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
Reply | Threaded
Open this post in threaded view
|

Re: Testsuite update

Thomas Diesler
In reply to this post by Ondrej Zizka
Would it be possible to create a design document (wiki) that lists the current testsuite requirements together with which of those reqs are fulfilled already? In the past we often had changes to the testsuite happen that introduced new stuff but at the same time broke working use cases.

-thomas    


On 11/11/2011 04:50 AM, Ondřej Žižka wrote:
Hi everyone,

as was discussed on this list, the current testsuite harness had some problems fulfilling even basic requirements for various use cases, including but not limited to:
* Using from IDE through pure maven
* -DallTests vs. individial modules vs. distinctive groups vs. even overlaying groups
* Configuring AS using kind of hierarchical templates
* Imposing certain test groups order
* Invoking only some steps of the testsuite
* Running same tests with different AS configuration
etc etc.

I've been working on an updated concept, which is currently at https://github.com/OndraZizka/jboss-as/commits/TS-modules-tmp .
I would like you to review it, run it, and send me some comments.

     cd testsuite/integration
     mvn clean install -f pom2.xml -DallTests -fae
     mvn clean install -f pom2.xml -Dts.iiop

(There are 2 pom.xml's so I can compare two harnesses easily.)
The scripts are not updated yet so they do not work.
CLI and OSGi tests fail. To be sorted out.


The big news are:

* Execution split to modules
* Code and resources remained in the same location
* Added few system properties which provide paths to important directories - see https://docs.jboss.org/author/display/AS71/AS+7+Testsuite+Test+Developer+Guide
* Work directory is now in target/workdir

These cahnges exposed some issues in test code which need to be fixed - hardcoded relative paths.
Since there were no system properties to use, there was no other way, but from now on please use those properties, and fix your existing tests.
(This would happen anyway because some tests clutter workdir (which happens to be module's root dir) with various temp files.)


Thanks,
Ondra





_______________________________________________
jboss-as7-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/jboss-as7-dev

-- 
xxxxxxxxxxxxxxxxxxxxxxxxxxxx
Thomas Diesler
JBoss OSGi Lead
JBoss, a division of Red Hat
xxxxxxxxxxxxxxxxxxxxxxxxxxxx 

_______________________________________________
jboss-as7-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
Reply | Threaded
Open this post in threaded view
|

AS Testsuite Requirements page // Re: Testsuite update

Ondrej Zizka
Here you are:

AS Testsuite Requirements page
http://community.jboss.org/wiki/ASTestsuiteRequirements

Consider that as a workbench, the document shall evolve into something shared between all stakeholders.

Regards,
Ondra



Thomas Diesler píše v Út 15. 11. 2011 v 14:10 +0000:
Would it be possible to create a design document (wiki) that lists the current testsuite requirements together with which of those reqs are fulfilled already? In the past we often had changes to the testsuite happen that introduced new stuff but at the same time broke working use cases.

-thomas    

On 11/11/2011 04:50 AM, Ondřej Žižka wrote:
Hi everyone,

as was discussed on this list, the current testsuite harness had some problems fulfilling even basic requirements for various use cases, including but not limited to:
* Using from IDE through pure maven
* -DallTests vs. individial modules vs. distinctive groups vs. even overlaying groups
* Configuring AS using kind of hierarchical templates
* Imposing certain test groups order
* Invoking only some steps of the testsuite
* Running same tests with different AS configuration
etc etc.

I've been working on an updated concept, which is currently at https://github.com/OndraZizka/jboss-as/commits/TS-modules-tmp .
I would like you to review it, run it, and send me some comments.

     cd testsuite/integration
     mvn clean install -f pom2.xml -DallTests -fae
     mvn clean install -f pom2.xml -Dts.iiop

(There are 2 pom.xml's so I can compare two harnesses easily.)
The scripts are not updated yet so they do not work.
CLI and OSGi tests fail. To be sorted out.


The big news are:

* Execution split to modules
* Code and resources remained in the same location
* Added few system properties which provide paths to important directories - see https://docs.jboss.org/author/display/AS71/AS+7+Testsuite+Test+Developer+Guide
* Work directory is now in target/workdir

These cahnges exposed some issues in test code which need to be fixed - hardcoded relative paths.
Since there were no system properties to use, there was no other way, but from now on please use those properties, and fix your existing tests.
(This would happen anyway because some tests clutter workdir (which happens to be module's root dir) with various temp files.)


Thanks,
Ondra





_______________________________________________
jboss-as7-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/jboss-as7-dev

-- 
xxxxxxxxxxxxxxxxxxxxxxxxxxxx
Thomas Diesler
JBoss OSGi Lead
JBoss, a division of Red Hat
xxxxxxxxxxxxxxxxxxxxxxxxxxxx 

_______________________________________________
jboss-as7-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
Reply | Threaded
Open this post in threaded view
|

Re: AS Testsuite Requirements page // Re: Testsuite update

Thomas Diesler
Thanks.

- It should be possible to run individual tests from IDE - please show how
- It should be possible to run individual tests against an already running server. This used to work. There is an option in arquillian.xml for that.

-thomas

On 11/16/2011 04:37 AM, Ondřej Žižka wrote:
Here you are:

AS Testsuite Requirements page
http://community.jboss.org/wiki/ASTestsuiteRequirements

Consider that as a workbench, the document shall evolve into something shared between all stakeholders.

Regards,
Ondra



Thomas Diesler píše v Út 15. 11. 2011 v 14:10 +0000:
Would it be possible to create a design document (wiki) that lists the current testsuite requirements together with which of those reqs are fulfilled already? In the past we often had changes to the testsuite happen that introduced new stuff but at the same time broke working use cases.

-thomas    

On 11/11/2011 04:50 AM, Ondřej Žižka wrote:
Hi everyone,

as was discussed on this list, the current testsuite harness had some problems fulfilling even basic requirements for various use cases, including but not limited to:
* Using from IDE through pure maven
* -DallTests vs. individial modules vs. distinctive groups vs. even overlaying groups
* Configuring AS using kind of hierarchical templates
* Imposing certain test groups order
* Invoking only some steps of the testsuite
* Running same tests with different AS configuration
etc etc.

I've been working on an updated concept, which is currently at https://github.com/OndraZizka/jboss-as/commits/TS-modules-tmp .
I would like you to review it, run it, and send me some comments.

     cd testsuite/integration
     mvn clean install -f pom2.xml -DallTests -fae
     mvn clean install -f pom2.xml -Dts.iiop

(There are 2 pom.xml's so I can compare two harnesses easily.)
The scripts are not updated yet so they do not work.
CLI and OSGi tests fail. To be sorted out.


The big news are:

* Execution split to modules
* Code and resources remained in the same location
* Added few system properties which provide paths to important directories - see https://docs.jboss.org/author/display/AS71/AS+7+Testsuite+Test+Developer+Guide
* Work directory is now in target/workdir

These cahnges exposed some issues in test code which need to be fixed - hardcoded relative paths.
Since there were no system properties to use, there was no other way, but from now on please use those properties, and fix your existing tests.
(This would happen anyway because some tests clutter workdir (which happens to be module's root dir) with various temp files.)


Thanks,
Ondra





_______________________________________________
jboss-as7-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/jboss-as7-dev

-- 
xxxxxxxxxxxxxxxxxxxxxxxxxxxx
Thomas Diesler
JBoss OSGi Lead
JBoss, a division of Red Hat
xxxxxxxxxxxxxxxxxxxxxxxxxxxx 

-- 
xxxxxxxxxxxxxxxxxxxxxxxxxxxx
Thomas Diesler
JBoss OSGi Lead
JBoss, a division of Red Hat
xxxxxxxxxxxxxxxxxxxxxxxxxxxx 

_______________________________________________
jboss-as7-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
Reply | Threaded
Open this post in threaded view
|

AS Testsuite Requirements page, round 2

Ondrej Zizka
In reply to this post by Ondrej Zizka
Hi all,

let's review http://community.jboss.org/wiki/ASTestsuiteRequirements .
To prevent future misunderstandings:

I expect this page to contain everything sacred regarding AS 7 testsuite "API". Even `mvn clean` should be there.
Because without it, many cases are left just to my guess, and when I decide to pick a solution which breaks someone's favorite param, like mvn ... -pl ..., it may start some late usability discussions.

Therefore - please, fill in every use case you want to preserve.

Thanks,
Ondra

PS:
On the other side, I don't expect much to change in the future. (Depends on requirements.)



On Wed, 2011-11-16 at 05:37 +0100, Ondřej Žižka wrote:
Here you are:

AS Testsuite Requirements page
http://community.jboss.org/wiki/ASTestsuiteRequirements

Consider that as a workbench, the document shall evolve into something shared between all stakeholders.

Regards,
Ondra



Thomas Diesler píše v Út 15. 11. 2011 v 14:10 +0000:
Would it be possible to create a design document (wiki) that lists the current testsuite requirements together with which of those reqs are fulfilled already? In the past we often had changes to the testsuite happen that introduced new stuff but at the same time broke working use cases.

-thomas    

On 11/11/2011 04:50 AM, Ondřej Žižka wrote:
Hi everyone,

as was discussed on this list, the current testsuite harness had some problems fulfilling even basic requirements for various use cases, including but not limited to:
* Using from IDE through pure maven
* -DallTests vs. individial modules vs. distinctive groups vs. even overlaying groups
* Configuring AS using kind of hierarchical templates
* Imposing certain test groups order
* Invoking only some steps of the testsuite
* Running same tests with different AS configuration
etc etc.

I've been working on an updated concept, which is currently at https://github.com/OndraZizka/jboss-as/commits/TS-modules-tmp .
I would like you to review it, run it, and send me some comments.

     cd testsuite/integration
     mvn clean install -f pom2.xml -DallTests -fae
     mvn clean install -f pom2.xml -Dts.iiop

(There are 2 pom.xml's so I can compare two harnesses easily.)
The scripts are not updated yet so they do not work.
CLI and OSGi tests fail. To be sorted out.


The big news are:

* Execution split to modules
* Code and resources remained in the same location
* Added few system properties which provide paths to important directories - see https://docs.jboss.org/author/display/AS71/AS+7+Testsuite+Test+Developer+Guide
* Work directory is now in target/workdir

These cahnges exposed some issues in test code which need to be fixed - hardcoded relative paths.
Since there were no system properties to use, there was no other way, but from now on please use those properties, and fix your existing tests.
(This would happen anyway because some tests clutter workdir (which happens to be module's root dir) with various temp files.)


Thanks,
Ondra





_______________________________________________
jboss-as7-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/jboss-as7-dev

-- 
xxxxxxxxxxxxxxxxxxxxxxxxxxxx
Thomas Diesler
JBoss OSGi Lead
JBoss, a division of Red Hat
xxxxxxxxxxxxxxxxxxxxxxxxxxxx 


_______________________________________________
jboss-as7-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/jboss-as7-dev