Revisited: Integration TestSuite Organization and Maintenance

classic Classic list List threaded Threaded
35 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Revisited: Integration TestSuite Organization and Maintenance

Andrew Lee Rubinger
Hi guys:

I'd like to reopen the discussion regarding the testsuite organization
and its ongoing maintenance.  This issue dates back a few months with
some debates and differing opinions, so I'll do my best to outline the
guiding principles I'd like to see put in place concisely.

To start off, I've a Proof-of-Concept for many of the following points
now located:

https://github.com/ALRubinger/jboss-as/tree/AS7-999

The relevant JIRA I've been using to track things:

https://issues.jboss.org/browse/AS7-999

So:

1) TestSuite Organization

I believe we need a single top-level categorization by which we may
organize integration tests which are deployment-based and run within the
context of the server.  Because we use Maven modules (which are bound to
a dependency structure), it makes sense to file these modules by the
compile-time dependencies they require.  So in place I've put:

testsuite/spec - Java SE and Java EE APIs only
testsuite/api - AS7 APIs + Spec
testsuite/internals - Use anything in the AS7 runtime in your
deployments; not guaranteed to be back-compat across releases

The primary motivation here is to ensure that the dependencies we export
(ie. "spec-api", and "api" modules) are complete enough for users to
create their own deployments.  In this setup, we act as *users* of our
own APIs, and everything in src/main is limited to the relevant
dependencies.

I know the source of some disagreements earlier centered around placing
the tests right next to the deployments, and some folks consider the
deployments as part of the test itself.  That's not a bad argument at
all, but again consider that we then lose the ability to validate our
tests in the context of our exported APIs.

2) Run Modes, Test Subsets

Because the primary organizational criteria proposed in 1) is by
dependency, these modules will grow large over time.  The AS build over
time will take longer and longer to run.  Additionally, there are
runtime options to consider when starting tests.  So consider the
following requirements:

   * Running the testsuite in IPv6
   * Running only a subset of tests as part of the main build

These lend themselves well to using build profiles.  By default, I think
the "smoke tests" should simply be a set of tests we deem important or
indicative of the general health of AS7 with respect to each subsystem.
  As it stands now, "smoke" is its own module with a bunch of
Embedded-based tests, and I think these should move to the
organizational structure in 1) and instead we can apply some filtering
to make the "smoke" some default set of includes.

3) An authoritative maintainer

I'd like to treat the Arquillian and TestSuite modules as true
subsystems of the Application Server, and as such we'll need someone to
assume the responsibility to review incoming commits/pull requests and
ensure they fit the criteria for acceptance.  Simple things like
consistent package names, using ARQ correctly, and not leaking
dependencies are very important.

So assuming we come to agreement on these points, I'd like to request
push access to the AS7 repo to field testsuite and ARQ-related pull
requests.

...there's much more to discuss (I've more issues to raise alongside the
upcoming EAP requirements), but let's start with those first 3 major
points and my POC, and run from there.

S,
ALR
_______________________________________________
jboss-as7-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
Reply | Threaded
Open this post in threaded view
|

Re: Revisited: Integration TestSuite Organization and Maintenance

Andrew Lee Rubinger
I'll add this link, too:

http://jboss.hudson.alrubinger.com/job/AS7-999/1/

Just to show that my Big Ol Patch Which Moves Everything Around is
working. :)

S,
ALR

On 08/11/2011 07:38 AM, Andrew Lee Rubinger wrote:

> Hi guys:
>
> I'd like to reopen the discussion regarding the testsuite organization
> and its ongoing maintenance.  This issue dates back a few months with
> some debates and differing opinions, so I'll do my best to outline the
> guiding principles I'd like to see put in place concisely.
>
> To start off, I've a Proof-of-Concept for many of the following points
> now located:
>
> https://github.com/ALRubinger/jboss-as/tree/AS7-999
>
> The relevant JIRA I've been using to track things:
>
> https://issues.jboss.org/browse/AS7-999
>
> So:
>
> 1) TestSuite Organization
>
> I believe we need a single top-level categorization by which we may
> organize integration tests which are deployment-based and run within the
> context of the server.  Because we use Maven modules (which are bound to
> a dependency structure), it makes sense to file these modules by the
> compile-time dependencies they require.  So in place I've put:
>
> testsuite/spec - Java SE and Java EE APIs only
> testsuite/api - AS7 APIs + Spec
> testsuite/internals - Use anything in the AS7 runtime in your
> deployments; not guaranteed to be back-compat across releases
>
> The primary motivation here is to ensure that the dependencies we export
> (ie. "spec-api", and "api" modules) are complete enough for users to
> create their own deployments.  In this setup, we act as *users* of our
> own APIs, and everything in src/main is limited to the relevant
> dependencies.
>
> I know the source of some disagreements earlier centered around placing
> the tests right next to the deployments, and some folks consider the
> deployments as part of the test itself.  That's not a bad argument at
> all, but again consider that we then lose the ability to validate our
> tests in the context of our exported APIs.
>
> 2) Run Modes, Test Subsets
>
> Because the primary organizational criteria proposed in 1) is by
> dependency, these modules will grow large over time.  The AS build over
> time will take longer and longer to run.  Additionally, there are
> runtime options to consider when starting tests.  So consider the
> following requirements:
>
>     * Running the testsuite in IPv6
>     * Running only a subset of tests as part of the main build
>
> These lend themselves well to using build profiles.  By default, I think
> the "smoke tests" should simply be a set of tests we deem important or
> indicative of the general health of AS7 with respect to each subsystem.
>    As it stands now, "smoke" is its own module with a bunch of
> Embedded-based tests, and I think these should move to the
> organizational structure in 1) and instead we can apply some filtering
> to make the "smoke" some default set of includes.
>
> 3) An authoritative maintainer
>
> I'd like to treat the Arquillian and TestSuite modules as true
> subsystems of the Application Server, and as such we'll need someone to
> assume the responsibility to review incoming commits/pull requests and
> ensure they fit the criteria for acceptance.  Simple things like
> consistent package names, using ARQ correctly, and not leaking
> dependencies are very important.
>
> So assuming we come to agreement on these points, I'd like to request
> push access to the AS7 repo to field testsuite and ARQ-related pull
> requests.
>
> ...there's much more to discuss (I've more issues to raise alongside the
> upcoming EAP requirements), but let's start with those first 3 major
> points and my POC, and run from there.
>
> S,
> ALR
> _______________________________________________
> jboss-as7-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
_______________________________________________
jboss-as7-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
Reply | Threaded
Open this post in threaded view
|

Re: Revisited: Integration TestSuite Organization and Maintenance

Scott Marlow
In reply to this post by Andrew Lee Rubinger
If this reorganization doesn't slow us down in the creation of tests,
then I don't mind the extra constraints.

On 08/11/2011 07:38 AM, Andrew Lee Rubinger wrote:

> Hi guys:
>
> I'd like to reopen the discussion regarding the testsuite organization
> and its ongoing maintenance.  This issue dates back a few months with
> some debates and differing opinions, so I'll do my best to outline the
> guiding principles I'd like to see put in place concisely.
>
> To start off, I've a Proof-of-Concept for many of the following points
> now located:
>
> https://github.com/ALRubinger/jboss-as/tree/AS7-999
>
> The relevant JIRA I've been using to track things:
>
> https://issues.jboss.org/browse/AS7-999
>
> So:
>
> 1) TestSuite Organization
>
> I believe we need a single top-level categorization by which we may
> organize integration tests which are deployment-based and run within the
> context of the server.  Because we use Maven modules (which are bound to
> a dependency structure), it makes sense to file these modules by the
> compile-time dependencies they require.  So in place I've put:
>
> testsuite/spec - Java SE and Java EE APIs only
> testsuite/api - AS7 APIs + Spec
> testsuite/internals - Use anything in the AS7 runtime in your
> deployments; not guaranteed to be back-compat across releases
>

What happens to the testsuite/compat stuff that I added for testing
older versions of Hibernate?

Currently, on the problems list with "compat" is lack of support for
testing custom AS7 modules being added to the AS7/modules folder by unit
tests.  There should be a way to test custom modules with a unit test
somehow.  It would be nice if this reorganization, added support for
that.  ;)


> The primary motivation here is to ensure that the dependencies we export
> (ie. "spec-api", and "api" modules) are complete enough for users to
> create their own deployments.  In this setup, we act as *users* of our
> own APIs, and everything in src/main is limited to the relevant
> dependencies.
>
> I know the source of some disagreements earlier centered around placing
> the tests right next to the deployments, and some folks consider the
> deployments as part of the test itself.  That's not a bad argument at
> all, but again consider that we then lose the ability to validate our
> tests in the context of our exported APIs.
>
> 2) Run Modes, Test Subsets
>
> Because the primary organizational criteria proposed in 1) is by
> dependency, these modules will grow large over time.  The AS build over
> time will take longer and longer to run.  Additionally, there are
> runtime options to consider when starting tests.  So consider the
> following requirements:
>
>     * Running the testsuite in IPv6
>     * Running only a subset of tests as part of the main build

I would like to see the ability to run with different persistence
providers as well.  So that I can run the current JPA tests against
Hibernate 3 or OGM or something else.

>
> These lend themselves well to using build profiles.  By default, I think
> the "smoke tests" should simply be a set of tests we deem important or
> indicative of the general health of AS7 with respect to each subsystem.
>    As it stands now, "smoke" is its own module with a bunch of
> Embedded-based tests, and I think these should move to the
> organizational structure in 1) and instead we can apply some filtering
> to make the "smoke" some default set of includes.
>
> 3) An authoritative maintainer
>
> I'd like to treat the Arquillian and TestSuite modules as true
> subsystems of the Application Server, and as such we'll need someone to
> assume the responsibility to review incoming commits/pull requests and
> ensure they fit the criteria for acceptance.  Simple things like
> consistent package names, using ARQ correctly, and not leaking
> dependencies are very important.

Currently, we have a few (five?) AS7 committers, they review everything.
  Are you proposing separation of the AS7 committer task from the
reviewing of changes?  Or adding more specialized AS7 committers that
only handle certain source modules in AS7?

>
> So assuming we come to agreement on these points, I'd like to request
> push access to the AS7 repo to field testsuite and ARQ-related pull
> requests.
>
> ...there's much more to discuss (I've more issues to raise alongside the
> upcoming EAP requirements), but let's start with those first 3 major
> points and my POC, and run from there.
>
> S,
> ALR
> _______________________________________________
> jboss-as7-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev

_______________________________________________
jboss-as7-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
Reply | Threaded
Open this post in threaded view
|

Re: Revisited: Integration TestSuite Organization and Maintenance

Andrew Lee Rubinger
On 08/11/2011 09:50 AM, Scott Marlow wrote:
> If this reorganization doesn't slow us down in the creation of tests,
> then I don't mind the extra constraints.

Can't see why it would, really.  The biggest difference would simply be
in separating out deployable components (src/main) from the test itself
(src/test).

S,
ALR

>
> On 08/11/2011 07:38 AM, Andrew Lee Rubinger wrote:
>> Hi guys:
>>
>> I'd like to reopen the discussion regarding the testsuite organization
>> and its ongoing maintenance.  This issue dates back a few months with
>> some debates and differing opinions, so I'll do my best to outline the
>> guiding principles I'd like to see put in place concisely.
>>
>> To start off, I've a Proof-of-Concept for many of the following points
>> now located:
>>
>> https://github.com/ALRubinger/jboss-as/tree/AS7-999
>>
>> The relevant JIRA I've been using to track things:
>>
>> https://issues.jboss.org/browse/AS7-999
>>
>> So:
>>
>> 1) TestSuite Organization
>>
>> I believe we need a single top-level categorization by which we may
>> organize integration tests which are deployment-based and run within the
>> context of the server.  Because we use Maven modules (which are bound to
>> a dependency structure), it makes sense to file these modules by the
>> compile-time dependencies they require.  So in place I've put:
>>
>> testsuite/spec - Java SE and Java EE APIs only
>> testsuite/api - AS7 APIs + Spec
>> testsuite/internals - Use anything in the AS7 runtime in your
>> deployments; not guaranteed to be back-compat across releases
>>
>
> What happens to the testsuite/compat stuff that I added for testing
> older versions of Hibernate?
>
> Currently, on the problems list with "compat" is lack of support for
> testing custom AS7 modules being added to the AS7/modules folder by unit
> tests.  There should be a way to test custom modules with a unit test
> somehow.  It would be nice if this reorganization, added support for
> that.  ;)
>
>
>> The primary motivation here is to ensure that the dependencies we export
>> (ie. "spec-api", and "api" modules) are complete enough for users to
>> create their own deployments.  In this setup, we act as *users* of our
>> own APIs, and everything in src/main is limited to the relevant
>> dependencies.
>>
>> I know the source of some disagreements earlier centered around placing
>> the tests right next to the deployments, and some folks consider the
>> deployments as part of the test itself.  That's not a bad argument at
>> all, but again consider that we then lose the ability to validate our
>> tests in the context of our exported APIs.
>>
>> 2) Run Modes, Test Subsets
>>
>> Because the primary organizational criteria proposed in 1) is by
>> dependency, these modules will grow large over time.  The AS build over
>> time will take longer and longer to run.  Additionally, there are
>> runtime options to consider when starting tests.  So consider the
>> following requirements:
>>
>>      * Running the testsuite in IPv6
>>      * Running only a subset of tests as part of the main build
>
> I would like to see the ability to run with different persistence
> providers as well.  So that I can run the current JPA tests against
> Hibernate 3 or OGM or something else.
>
>>
>> These lend themselves well to using build profiles.  By default, I think
>> the "smoke tests" should simply be a set of tests we deem important or
>> indicative of the general health of AS7 with respect to each subsystem.
>>     As it stands now, "smoke" is its own module with a bunch of
>> Embedded-based tests, and I think these should move to the
>> organizational structure in 1) and instead we can apply some filtering
>> to make the "smoke" some default set of includes.
>>
>> 3) An authoritative maintainer
>>
>> I'd like to treat the Arquillian and TestSuite modules as true
>> subsystems of the Application Server, and as such we'll need someone to
>> assume the responsibility to review incoming commits/pull requests and
>> ensure they fit the criteria for acceptance.  Simple things like
>> consistent package names, using ARQ correctly, and not leaking
>> dependencies are very important.
>
> Currently, we have a few (five?) AS7 committers, they review everything.
>    Are you proposing separation of the AS7 committer task from the
> reviewing of changes?  Or adding more specialized AS7 committers that
> only handle certain source modules in AS7?
>
>>
>> So assuming we come to agreement on these points, I'd like to request
>> push access to the AS7 repo to field testsuite and ARQ-related pull
>> requests.
>>
>> ...there's much more to discuss (I've more issues to raise alongside the
>> upcoming EAP requirements), but let's start with those first 3 major
>> points and my POC, and run from there.
>>
>> S,
>> ALR
>> _______________________________________________
>> jboss-as7-dev mailing list
>> [hidden email]
>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
>
> _______________________________________________
> jboss-as7-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
_______________________________________________
jboss-as7-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
Reply | Threaded
Open this post in threaded view
|

Re: Revisited: Integration TestSuite Organization and Maintenance

Richard Achmatowicz
In reply to this post by Andrew Lee Rubinger
Andrew

Can you give a simple, concrete example of a spec test case (citing part
of the J2EE spec, say), which illustrates:
- what dependencies you want to be on the classpath, as well as those
you don't want to be only the classpath
- what sample artifacts you would put in src/main and what you would put
in src/test
- what compile and test execution time constraints you would want to enforce
This would help me to see the motivation for the refactoring and your
planned organization of artifacts in src/main and src/test, which
differ from the standard organization of putting all test related
artifacts in src/test.

Also, is it not possible to control what is on the classpath by maven
elements like <classpathDependencyExcludes/> and the like?

Richard

On 08/11/2011 07:38 AM, Andrew Lee Rubinger wrote:

> Hi guys:
>
> I'd like to reopen the discussion regarding the testsuite organization
> and its ongoing maintenance.  This issue dates back a few months with
> some debates and differing opinions, so I'll do my best to outline the
> guiding principles I'd like to see put in place concisely.
>
> To start off, I've a Proof-of-Concept for many of the following points
> now located:
>
> https://github.com/ALRubinger/jboss-as/tree/AS7-999
>
> The relevant JIRA I've been using to track things:
>
> https://issues.jboss.org/browse/AS7-999
>
> So:
>
> 1) TestSuite Organization
>
> I believe we need a single top-level categorization by which we may
> organize integration tests which are deployment-based and run within the
> context of the server.  Because we use Maven modules (which are bound to
> a dependency structure), it makes sense to file these modules by the
> compile-time dependencies they require.  So in place I've put:
>
> testsuite/spec - Java SE and Java EE APIs only
> testsuite/api - AS7 APIs + Spec
> testsuite/internals - Use anything in the AS7 runtime in your
> deployments; not guaranteed to be back-compat across releases
>
> The primary motivation here is to ensure that the dependencies we export
> (ie. "spec-api", and "api" modules) are complete enough for users to
> create their own deployments.  In this setup, we act as *users* of our
> own APIs, and everything in src/main is limited to the relevant
> dependencies.
>
> I know the source of some disagreements earlier centered around placing
> the tests right next to the deployments, and some folks consider the
> deployments as part of the test itself.  That's not a bad argument at
> all, but again consider that we then lose the ability to validate our
> tests in the context of our exported APIs.
>
> 2) Run Modes, Test Subsets
>
> Because the primary organizational criteria proposed in 1) is by
> dependency, these modules will grow large over time.  The AS build over
> time will take longer and longer to run.  Additionally, there are
> runtime options to consider when starting tests.  So consider the
> following requirements:
>
>     * Running the testsuite in IPv6
>     * Running only a subset of tests as part of the main build
>
> These lend themselves well to using build profiles.  By default, I think
> the "smoke tests" should simply be a set of tests we deem important or
> indicative of the general health of AS7 with respect to each subsystem.
>    As it stands now, "smoke" is its own module with a bunch of
> Embedded-based tests, and I think these should move to the
> organizational structure in 1) and instead we can apply some filtering
> to make the "smoke" some default set of includes.
>
> 3) An authoritative maintainer
>
> I'd like to treat the Arquillian and TestSuite modules as true
> subsystems of the Application Server, and as such we'll need someone to
> assume the responsibility to review incoming commits/pull requests and
> ensure they fit the criteria for acceptance.  Simple things like
> consistent package names, using ARQ correctly, and not leaking
> dependencies are very important.
>
> So assuming we come to agreement on these points, I'd like to request
> push access to the AS7 repo to field testsuite and ARQ-related pull
> requests.
>
> ...there's much more to discuss (I've more issues to raise alongside the
> upcoming EAP requirements), but let's start with those first 3 major
> points and my POC, and run from there.
>
> S,
> ALR
> _______________________________________________
> jboss-as7-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev

_______________________________________________
jboss-as7-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
Reply | Threaded
Open this post in threaded view
|

Re: Revisited: Integration TestSuite Organization and Maintenance

Andrew Lee Rubinger
Inline.

On 08/11/2011 05:12 PM, Richard Achmatowicz wrote:
> Andrew
>
> Can you give a simple, concrete example of a spec test case (citing part
> of the J2EE spec, say), which illustrates:

Hehe, my PoC is filled with 'em. :)  But keep in mind we're not directly
necessarily citing the EE specs, because we're *not* building a TCK in AS7.

> - what dependencies you want to be on the classpath, as well as those
> you don't want to be only the classpath

For this discussion, I'm primarily concerned with the *compilation*
ClassPath, as this is the view users see when building their
applications.  It's about explicitly defining what the AS7 API is.  So
the deps for the compilation ClassPath should be as they are in by
AS7-999 branch and as described in the last email:

* spec - Java SE and Java EE
* api - JBoss-specific extensions to "spec" which comprise our AS7 API
* internals - Anything in the AS7 runtime

> - what sample artifacts you would put in src/main and what you would put
> in src/test

* src/main - Stuff that is part of the deployment.  Anything that you'd
be including in the @Deployment archive defined by an Arquillian-based
test case.
* src/test - The test infrastructure itself, like the JUnit test and
other util/helpers

> - what compile and test execution time constraints you would want to enforce
> This would help me to see the motivation for the refactoring and your
> planned organization of artifacts in src/main and src/test, which
> differ from the standard organization of putting all test related
> artifacts in src/test.

Test execution time is an orthogonal concern, and relates instead back
to using "smoke" tests to run by default instead of the entire AS7
integration test suite (which won't scale to run on every build over
time).  IMO it's already taking too long for standard builds.

Again, the motivation for moving/organizing in this fashion is to honor
dependencies and assert that the APIs we export ("api" and "spec-api"
modules) are complete.  For instance, while moving tests around I
discovered that our POMs were incompete:

https://issues.jboss.org/browse/AS7-1489
https://issues.jboss.org/browse/AS7-1493
https://issues.jboss.org/browse/AS7-1479
https://issues.jboss.org/browse/AS7-1478

> Also, is it not possible to control what is on the classpath by maven
> elements like<classpathDependencyExcludes/>  and the like?

That's for test runtime.  Because Arquillian is starting (or connecting
to) the container in a remote process, we're less concerned with the
client runtime ClassPath, so long as it's enough for Arquillian to do
its thing.

S,
ALR

>
> Richard
>
> On 08/11/2011 07:38 AM, Andrew Lee Rubinger wrote:
>> Hi guys:
>>
>> I'd like to reopen the discussion regarding the testsuite organization
>> and its ongoing maintenance.  This issue dates back a few months with
>> some debates and differing opinions, so I'll do my best to outline the
>> guiding principles I'd like to see put in place concisely.
>>
>> To start off, I've a Proof-of-Concept for many of the following points
>> now located:
>>
>> https://github.com/ALRubinger/jboss-as/tree/AS7-999
>>
>> The relevant JIRA I've been using to track things:
>>
>> https://issues.jboss.org/browse/AS7-999
>>
>> So:
>>
>> 1) TestSuite Organization
>>
>> I believe we need a single top-level categorization by which we may
>> organize integration tests which are deployment-based and run within the
>> context of the server.  Because we use Maven modules (which are bound to
>> a dependency structure), it makes sense to file these modules by the
>> compile-time dependencies they require.  So in place I've put:
>>
>> testsuite/spec - Java SE and Java EE APIs only
>> testsuite/api - AS7 APIs + Spec
>> testsuite/internals - Use anything in the AS7 runtime in your
>> deployments; not guaranteed to be back-compat across releases
>>
>> The primary motivation here is to ensure that the dependencies we export
>> (ie. "spec-api", and "api" modules) are complete enough for users to
>> create their own deployments.  In this setup, we act as *users* of our
>> own APIs, and everything in src/main is limited to the relevant
>> dependencies.
>>
>> I know the source of some disagreements earlier centered around placing
>> the tests right next to the deployments, and some folks consider the
>> deployments as part of the test itself.  That's not a bad argument at
>> all, but again consider that we then lose the ability to validate our
>> tests in the context of our exported APIs.
>>
>> 2) Run Modes, Test Subsets
>>
>> Because the primary organizational criteria proposed in 1) is by
>> dependency, these modules will grow large over time.  The AS build over
>> time will take longer and longer to run.  Additionally, there are
>> runtime options to consider when starting tests.  So consider the
>> following requirements:
>>
>>      * Running the testsuite in IPv6
>>      * Running only a subset of tests as part of the main build
>>
>> These lend themselves well to using build profiles.  By default, I think
>> the "smoke tests" should simply be a set of tests we deem important or
>> indicative of the general health of AS7 with respect to each subsystem.
>>     As it stands now, "smoke" is its own module with a bunch of
>> Embedded-based tests, and I think these should move to the
>> organizational structure in 1) and instead we can apply some filtering
>> to make the "smoke" some default set of includes.
>>
>> 3) An authoritative maintainer
>>
>> I'd like to treat the Arquillian and TestSuite modules as true
>> subsystems of the Application Server, and as such we'll need someone to
>> assume the responsibility to review incoming commits/pull requests and
>> ensure they fit the criteria for acceptance.  Simple things like
>> consistent package names, using ARQ correctly, and not leaking
>> dependencies are very important.
>>
>> So assuming we come to agreement on these points, I'd like to request
>> push access to the AS7 repo to field testsuite and ARQ-related pull
>> requests.
>>
>> ...there's much more to discuss (I've more issues to raise alongside the
>> upcoming EAP requirements), but let's start with those first 3 major
>> points and my POC, and run from there.
>>
>> S,
>> ALR
>> _______________________________________________
>> jboss-as7-dev mailing list
>> [hidden email]
>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
>
> _______________________________________________
> jboss-as7-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
_______________________________________________
jboss-as7-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
Reply | Threaded
Open this post in threaded view
|

Re: Revisited: Integration TestSuite Organization and Maintenance

Stuart Douglas

On 12/08/2011, at 9:06 AM, Andrew Lee Rubinger wrote:

Inline.

On 08/11/2011 05:12 PM, Richard Achmatowicz wrote:
Andrew

Can you give a simple, concrete example of a spec test case (citing part
of the J2EE spec, say), which illustrates:

Hehe, my PoC is filled with 'em. :)  But keep in mind we're not directly
necessarily citing the EE specs, because we're *not* building a TCK in AS7.

- what dependencies you want to be on the classpath, as well as those
you don't want to be only the classpath

For this discussion, I'm primarily concerned with the *compilation*
ClassPath, as this is the view users see when building their
applications.  It's about explicitly defining what the AS7 API is.  So
the deps for the compilation ClassPath should be as they are in by
AS7-999 branch and as described in the last email:

* spec - Java SE and Java EE
* api - JBoss-specific extensions to "spec" which comprise our AS7 API
* internals - Anything in the AS7 runtime

I am ok with splitting the tests up in this manner, but I think we will need additional test suite modules for purely technical reasons. 

For instance, at the moment we also have compat for hibernate 3 testing, and timer service, to test the ejb timer service.

Compat is needed because maven cannot handle having multiple hibernate versions in the same module, and timer service 
is separate because it needs the preview configuration, and need to run multiple times in order to test restoring persistent timers.


- what sample artifacts you would put in src/main and what you would put
in src/test

* src/main - Stuff that is part of the deployment.  Anything that you'd
be including in the @Deployment archive defined by an Arquillian-based
test case.
* src/test - The test infrastructure itself, like the JUnit test and
other util/helpers

I am very much opposed to this split. IMHO splitting the test classes into two different places makes it much harder to read and write tests. 

I also don't see how having additional test scoped dependencies on the class path is a problem, as in practice this works out to being Arquillian and JUnit. 

Stuart


- what compile and test execution time constraints you would want to enforce
This would help me to see the motivation for the refactoring and your
planned organization of artifacts in src/main and src/test, which
differ from the standard organization of putting all test related
artifacts in src/test.

Test execution time is an orthogonal concern, and relates instead back
to using "smoke" tests to run by default instead of the entire AS7
integration test suite (which won't scale to run on every build over
time).  IMO it's already taking too long for standard builds.

Again, the motivation for moving/organizing in this fashion is to honor
dependencies and assert that the APIs we export ("api" and "spec-api"
modules) are complete.  For instance, while moving tests around I
discovered that our POMs were incompete:

https://issues.jboss.org/browse/AS7-1489
https://issues.jboss.org/browse/AS7-1493
https://issues.jboss.org/browse/AS7-1479
https://issues.jboss.org/browse/AS7-1478

Also, is it not possible to control what is on the classpath by maven
elements like<classpathDependencyExcludes/>  and the like?

That's for test runtime.  Because Arquillian is starting (or connecting
to) the container in a remote process, we're less concerned with the
client runtime ClassPath, so long as it's enough for Arquillian to do
its thing.

S,
ALR


Richard

On 08/11/2011 07:38 AM, Andrew Lee Rubinger wrote:
Hi guys:

I'd like to reopen the discussion regarding the testsuite organization
and its ongoing maintenance.  This issue dates back a few months with
some debates and differing opinions, so I'll do my best to outline the
guiding principles I'd like to see put in place concisely.

To start off, I've a Proof-of-Concept for many of the following points
now located:

https://github.com/ALRubinger/jboss-as/tree/AS7-999

The relevant JIRA I've been using to track things:

https://issues.jboss.org/browse/AS7-999

So:

1) TestSuite Organization

I believe we need a single top-level categorization by which we may
organize integration tests which are deployment-based and run within the
context of the server.  Because we use Maven modules (which are bound to
a dependency structure), it makes sense to file these modules by the
compile-time dependencies they require.  So in place I've put:

testsuite/spec - Java SE and Java EE APIs only
testsuite/api - AS7 APIs + Spec
testsuite/internals - Use anything in the AS7 runtime in your
deployments; not guaranteed to be back-compat across releases

The primary motivation here is to ensure that the dependencies we export
(ie. "spec-api", and "api" modules) are complete enough for users to
create their own deployments.  In this setup, we act as *users* of our
own APIs, and everything in src/main is limited to the relevant
dependencies.

I know the source of some disagreements earlier centered around placing
the tests right next to the deployments, and some folks consider the
deployments as part of the test itself.  That's not a bad argument at
all, but again consider that we then lose the ability to validate our
tests in the context of our exported APIs.

2) Run Modes, Test Subsets

Because the primary organizational criteria proposed in 1) is by
dependency, these modules will grow large over time.  The AS build over
time will take longer and longer to run.  Additionally, there are
runtime options to consider when starting tests.  So consider the
following requirements:

    * Running the testsuite in IPv6
    * Running only a subset of tests as part of the main build

These lend themselves well to using build profiles.  By default, I think
the "smoke tests" should simply be a set of tests we deem important or
indicative of the general health of AS7 with respect to each subsystem.
   As it stands now, "smoke" is its own module with a bunch of
Embedded-based tests, and I think these should move to the
organizational structure in 1) and instead we can apply some filtering
to make the "smoke" some default set of includes.

3) An authoritative maintainer

I'd like to treat the Arquillian and TestSuite modules as true
subsystems of the Application Server, and as such we'll need someone to
assume the responsibility to review incoming commits/pull requests and
ensure they fit the criteria for acceptance.  Simple things like
consistent package names, using ARQ correctly, and not leaking
dependencies are very important.

So assuming we come to agreement on these points, I'd like to request
push access to the AS7 repo to field testsuite and ARQ-related pull
requests.

...there's much more to discuss (I've more issues to raise alongside the
upcoming EAP requirements), but let's start with those first 3 major
points and my POC, and run from there.

S,
ALR
_______________________________________________
jboss-as7-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/jboss-as7-dev

_______________________________________________
jboss-as7-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
_______________________________________________
jboss-as7-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/jboss-as7-dev


_______________________________________________
jboss-as7-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
Reply | Threaded
Open this post in threaded view
|

Re: Revisited: Integration TestSuite Organization and Maintenance

kkhan

On 12 Aug 2011, at 04:28, Stuart Douglas wrote:

>
> On 12/08/2011, at 9:06 AM, Andrew Lee Rubinger wrote:
>
>> Inline.
>>
>> On 08/11/2011 05:12 PM, Richard Achmatowicz wrote:
>>> Andrew
>>>
>>> Can you give a simple, concrete example of a spec test case (citing part
>>> of the J2EE spec, say), which illustrates:
>>
>> Hehe, my PoC is filled with 'em. :)  But keep in mind we're not directly
>> necessarily citing the EE specs, because we're *not* building a TCK in AS7.
>>
>>> - what dependencies you want to be on the classpath, as well as those
>>> you don't want to be only the classpath
>>
>> For this discussion, I'm primarily concerned with the *compilation*
>> ClassPath, as this is the view users see when building their
>> applications.  It's about explicitly defining what the AS7 API is.  So
>> the deps for the compilation ClassPath should be as they are in by
>> AS7-999 branch and as described in the last email:
>>
>> * spec - Java SE and Java EE
>> * api - JBoss-specific extensions to "spec" which comprise our AS7 API
>> * internals - Anything in the AS7 runtime
>
> I am ok with splitting the tests up in this manner, but I think we will need additional test suite modules for purely technical reasons.
>
> For instance, at the moment we also have compat for hibernate 3 testing, and timer service, to test the ejb timer service.
>
> Compat is needed because maven cannot handle having multiple hibernate versions in the same module, and timer service
> is separate because it needs the preview configuration, and need to run multiple times in order to test restoring persistent timers.
>
>>
>>> - what sample artifacts you would put in src/main and what you would put
>>> in src/test
>>
>> * src/main - Stuff that is part of the deployment.  Anything that you'd
>> be including in the @Deployment archive defined by an Arquillian-based
>> test case.
>> * src/test - The test infrastructure itself, like the JUnit test and
>> other util/helpers
>
> I am very much opposed to this split. IMHO splitting the test classes into two different places makes it much harder to read and write tests.
>
> I also don't see how having additional test scoped dependencies on the class path is a problem, as in practice this works out to being Arquillian and JUnit.

I agree here, I don't like having tests split up over two source folders

>
> Stuart
>
>>
>>> - what compile and test execution time constraints you would want to enforce
>>> This would help me to see the motivation for the refactoring and your
>>> planned organization of artifacts in src/main and src/test, which
>>> differ from the standard organization of putting all test related
>>> artifacts in src/test.
>>
>> Test execution time is an orthogonal concern, and relates instead back
>> to using "smoke" tests to run by default instead of the entire AS7
>> integration test suite (which won't scale to run on every build over
>> time).  IMO it's already taking too long for standard builds.
>>
>> Again, the motivation for moving/organizing in this fashion is to honor
>> dependencies and assert that the APIs we export ("api" and "spec-api"
>> modules) are complete.  For instance, while moving tests around I
>> discovered that our POMs were incompete:
>>
>> https://issues.jboss.org/browse/AS7-1489
>> https://issues.jboss.org/browse/AS7-1493
>> https://issues.jboss.org/browse/AS7-1479
>> https://issues.jboss.org/browse/AS7-1478
>>
>>> Also, is it not possible to control what is on the classpath by maven
>>> elements like<classpathDependencyExcludes/>  and the like?
>>
>> That's for test runtime.  Because Arquillian is starting (or connecting
>> to) the container in a remote process, we're less concerned with the
>> client runtime ClassPath, so long as it's enough for Arquillian to do
>> its thing.
>>
>> S,
>> ALR
>>
>>>
>>> Richard
>>>
>>> On 08/11/2011 07:38 AM, Andrew Lee Rubinger wrote:
>>>> Hi guys:
>>>>
>>>> I'd like to reopen the discussion regarding the testsuite organization
>>>> and its ongoing maintenance.  This issue dates back a few months with
>>>> some debates and differing opinions, so I'll do my best to outline the
>>>> guiding principles I'd like to see put in place concisely.
>>>>
>>>> To start off, I've a Proof-of-Concept for many of the following points
>>>> now located:
>>>>
>>>> https://github.com/ALRubinger/jboss-as/tree/AS7-999
>>>>
>>>> The relevant JIRA I've been using to track things:
>>>>
>>>> https://issues.jboss.org/browse/AS7-999
>>>>
>>>> So:
>>>>
>>>> 1) TestSuite Organization
>>>>
>>>> I believe we need a single top-level categorization by which we may
>>>> organize integration tests which are deployment-based and run within the
>>>> context of the server.  Because we use Maven modules (which are bound to
>>>> a dependency structure), it makes sense to file these modules by the
>>>> compile-time dependencies they require.  So in place I've put:
>>>>
>>>> testsuite/spec - Java SE and Java EE APIs only
>>>> testsuite/api - AS7 APIs + Spec
>>>> testsuite/internals - Use anything in the AS7 runtime in your
>>>> deployments; not guaranteed to be back-compat across releases
>>>>
>>>> The primary motivation here is to ensure that the dependencies we export
>>>> (ie. "spec-api", and "api" modules) are complete enough for users to
>>>> create their own deployments.  In this setup, we act as *users* of our
>>>> own APIs, and everything in src/main is limited to the relevant
>>>> dependencies.
>>>>
>>>> I know the source of some disagreements earlier centered around placing
>>>> the tests right next to the deployments, and some folks consider the
>>>> deployments as part of the test itself.  That's not a bad argument at
>>>> all, but again consider that we then lose the ability to validate our
>>>> tests in the context of our exported APIs.
>>>>
>>>> 2) Run Modes, Test Subsets
>>>>
>>>> Because the primary organizational criteria proposed in 1) is by
>>>> dependency, these modules will grow large over time.  The AS build over
>>>> time will take longer and longer to run.  Additionally, there are
>>>> runtime options to consider when starting tests.  So consider the
>>>> following requirements:
>>>>
>>>>     * Running the testsuite in IPv6
>>>>     * Running only a subset of tests as part of the main build
>>>>
>>>> These lend themselves well to using build profiles.  By default, I think
>>>> the "smoke tests" should simply be a set of tests we deem important or
>>>> indicative of the general health of AS7 with respect to each subsystem.
>>>>    As it stands now, "smoke" is its own module with a bunch of
>>>> Embedded-based tests, and I think these should move to the
>>>> organizational structure in 1) and instead we can apply some filtering
>>>> to make the "smoke" some default set of includes.
>>>>
>>>> 3) An authoritative maintainer
>>>>
>>>> I'd like to treat the Arquillian and TestSuite modules as true
>>>> subsystems of the Application Server, and as such we'll need someone to
>>>> assume the responsibility to review incoming commits/pull requests and
>>>> ensure they fit the criteria for acceptance.  Simple things like
>>>> consistent package names, using ARQ correctly, and not leaking
>>>> dependencies are very important.
>>>>
>>>> So assuming we come to agreement on these points, I'd like to request
>>>> push access to the AS7 repo to field testsuite and ARQ-related pull
>>>> requests.
>>>>
>>>> ...there's much more to discuss (I've more issues to raise alongside the
>>>> upcoming EAP requirements), but let's start with those first 3 major
>>>> points and my POC, and run from there.
>>>>
>>>> S,
>>>> ALR
>>>> _______________________________________________
>>>> jboss-as7-dev mailing list
>>>> [hidden email]
>>>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
>>>
>>> _______________________________________________
>>> jboss-as7-dev mailing list
>>> [hidden email]
>>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
>> _______________________________________________
>> jboss-as7-dev mailing list
>> [hidden email]
>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
>
> _______________________________________________
> jboss-as7-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev


_______________________________________________
jboss-as7-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
Reply | Threaded
Open this post in threaded view
|

Re: Revisited: Integration TestSuite Organization and Maintenance

Andrew Lee Rubinger-2
Inline.

----- Original Message -----
From: Kabir Khan <[hidden email]>
To: Stuart Douglas <[hidden email]>
Cc: Andrew Lee Rubinger <[hidden email]>, [hidden email]
Sent: Fri, 12 Aug 2011 05:14:25 -0400 (EDT)
Subject: Re: [jboss-as7-dev] Revisited: Integration TestSuite Organization and Maintenance


On 12 Aug 2011, at 04:28, Stuart Douglas wrote:

>
> On 12/08/2011, at 9:06 AM, Andrew Lee Rubinger wrote:
>
>> Inline.
>>
>> On 08/11/2011 05:12 PM, Richard Achmatowicz wrote:
>>> Andrew
>>>
>>> Can you give a simple, concrete example of a spec test case (citing part
>>> of the J2EE spec, say), which illustrates:
>>
>> Hehe, my PoC is filled with 'em. :)  But keep in mind we're not directly
>> necessarily citing the EE specs, because we're *not* building a TCK in AS7.
>>
>>> - what dependencies you want to be on the classpath, as well as those
>>> you don't want to be only the classpath
>>
>> For this discussion, I'm primarily concerned with the *compilation*
>> ClassPath, as this is the view users see when building their
>> applications.  It's about explicitly defining what the AS7 API is.  So
>> the deps for the compilation ClassPath should be as they are in by
>> AS7-999 branch and as described in the last email:
>>
>> * spec - Java SE and Java EE
>> * api - JBoss-specific extensions to "spec" which comprise our AS7 API
>> * internals - Anything in the AS7 runtime
>
> I am ok with splitting the tests up in this manner, but I think we will need additional test suite modules for purely technical reasons.
>
> For instance, at the moment we also have compat for hibernate 3 testing, and timer service, to test the ejb timer service.
>
> Compat is needed because maven cannot handle having multiple hibernate versions in the same module, and timer service
> is separate because it needs the preview configuration, and need to run multiple times in order to test restoring persistent timers.
>
>>
>>> - what sample artifacts you would put in src/main and what you would put
>>> in src/test
>>
>> * src/main - Stuff that is part of the deployment.  Anything that you'd
>> be including in the @Deployment archive defined by an Arquillian-based
>> test case.
>> * src/test - The test infrastructure itself, like the JUnit test and
>> other util/helpers
>
> I am very much opposed to this split. IMHO splitting the test classes into two different places makes it much harder to read and write tests.
>
> I also don't see how having additional test scoped dependencies on the class path is a problem, as in practice this works out to being Arquillian and JUnit.

> I agree here, I don't like having tests split up over two source folders

Sure, it'd be nice to have everything right next to each other; I'll give you that.  But I don't see how we can do that without opening ourselves up to miss errors in dependency management.

Having additional dependencies on the compilation classpath is a very big problem, Stuart.  Take for instance Arquillian in src/test.  You need not only the Arquillian API, but also all the transitive deps needed to fire up the ARQ runtime.  To get a sense of what would leak in, just crack open the "arquillian/container-managed/pom.xml" in Eclipse and switch over to the "Dependency Hierarchy" tab.  By unifying src/main and src/test, everything listed there would leak onto the compilation classpath, killing our ability to validate the "spec-api" and "api" POMs are doing their job.  We'd be checking against a polluted ClassPath.

So yeah, putting things in 2 places is a bit less convenient, but let's be honest - it's not that bad.  Same package names in different source folders makes it really easy to see what's going on.  And taking a more strict approach lets us ensure things are working *correctly*, which is the most important thing.

S,
ALR

>
> Stuart
>
>>
>>> - what compile and test execution time constraints you would want to enforce
>>> This would help me to see the motivation for the refactoring and your
>>> planned organization of artifacts in src/main and src/test, which
>>> differ from the standard organization of putting all test related
>>> artifacts in src/test.
>>
>> Test execution time is an orthogonal concern, and relates instead back
>> to using "smoke" tests to run by default instead of the entire AS7
>> integration test suite (which won't scale to run on every build over
>> time).  IMO it's already taking too long for standard builds.
>>
>> Again, the motivation for moving/organizing in this fashion is to honor
>> dependencies and assert that the APIs we export ("api" and "spec-api"
>> modules) are complete.  For instance, while moving tests around I
>> discovered that our POMs were incompete:
>>
>> https://issues.jboss.org/browse/AS7-1489
>> https://issues.jboss.org/browse/AS7-1493
>> https://issues.jboss.org/browse/AS7-1479
>> https://issues.jboss.org/browse/AS7-1478
>>
>>> Also, is it not possible to control what is on the classpath by maven
>>> elements like<classpathDependencyExcludes/>  and the like?
>>
>> That's for test runtime.  Because Arquillian is starting (or connecting
>> to) the container in a remote process, we're less concerned with the
>> client runtime ClassPath, so long as it's enough for Arquillian to do
>> its thing.
>>
>> S,
>> ALR
>>
>>>
>>> Richard
>>>
>>> On 08/11/2011 07:38 AM, Andrew Lee Rubinger wrote:
>>>> Hi guys:
>>>>
>>>> I'd like to reopen the discussion regarding the testsuite organization
>>>> and its ongoing maintenance.  This issue dates back a few months with
>>>> some debates and differing opinions, so I'll do my best to outline the
>>>> guiding principles I'd like to see put in place concisely.
>>>>
>>>> To start off, I've a Proof-of-Concept for many of the following points
>>>> now located:
>>>>
>>>> https://github.com/ALRubinger/jboss-as/tree/AS7-999
>>>>
>>>> The relevant JIRA I've been using to track things:
>>>>
>>>> https://issues.jboss.org/browse/AS7-999
>>>>
>>>> So:
>>>>
>>>> 1) TestSuite Organization
>>>>
>>>> I believe we need a single top-level categorization by which we may
>>>> organize integration tests which are deployment-based and run within the
>>>> context of the server.  Because we use Maven modules (which are bound to
>>>> a dependency structure), it makes sense to file these modules by the
>>>> compile-time dependencies they require.  So in place I've put:
>>>>
>>>> testsuite/spec - Java SE and Java EE APIs only
>>>> testsuite/api - AS7 APIs + Spec
>>>> testsuite/internals - Use anything in the AS7 runtime in your
>>>> deployments; not guaranteed to be back-compat across releases
>>>>
>>>> The primary motivation here is to ensure that the dependencies we export
>>>> (ie. "spec-api", and "api" modules) are complete enough for users to
>>>> create their own deployments.  In this setup, we act as *users* of our
>>>> own APIs, and everything in src/main is limited to the relevant
>>>> dependencies.
>>>>
>>>> I know the source of some disagreements earlier centered around placing
>>>> the tests right next to the deployments, and some folks consider the
>>>> deployments as part of the test itself.  That's not a bad argument at
>>>> all, but again consider that we then lose the ability to validate our
>>>> tests in the context of our exported APIs.
>>>>
>>>> 2) Run Modes, Test Subsets
>>>>
>>>> Because the primary organizational criteria proposed in 1) is by
>>>> dependency, these modules will grow large over time.  The AS build over
>>>> time will take longer and longer to run.  Additionally, there are
>>>> runtime options to consider when starting tests.  So consider the
>>>> following requirements:
>>>>
>>>>     * Running the testsuite in IPv6
>>>>     * Running only a subset of tests as part of the main build
>>>>
>>>> These lend themselves well to using build profiles.  By default, I think
>>>> the "smoke tests" should simply be a set of tests we deem important or
>>>> indicative of the general health of AS7 with respect to each subsystem.
>>>>    As it stands now, "smoke" is its own module with a bunch of
>>>> Embedded-based tests, and I think these should move to the
>>>> organizational structure in 1) and instead we can apply some filtering
>>>> to make the "smoke" some default set of includes.
>>>>
>>>> 3) An authoritative maintainer
>>>>
>>>> I'd like to treat the Arquillian and TestSuite modules as true
>>>> subsystems of the Application Server, and as such we'll need someone to
>>>> assume the responsibility to review incoming commits/pull requests and
>>>> ensure they fit the criteria for acceptance.  Simple things like
>>>> consistent package names, using ARQ correctly, and not leaking
>>>> dependencies are very important.
>>>>
>>>> So assuming we come to agreement on these points, I'd like to request
>>>> push access to the AS7 repo to field testsuite and ARQ-related pull
>>>> requests.
>>>>
>>>> ...there's much more to discuss (I've more issues to raise alongside the
>>>> upcoming EAP requirements), but let's start with those first 3 major
>>>> points and my POC, and run from there.
>>>>
>>>> S,
>>>> ALR
>>>> _______________________________________________
>>>> jboss-as7-dev mailing list
>>>> [hidden email]
>>>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
>>>
>>> _______________________________________________
>>> jboss-as7-dev mailing list
>>> [hidden email]
>>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
>> _______________________________________________
>> jboss-as7-dev mailing list
>> [hidden email]
>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
>
> _______________________________________________
> jboss-as7-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev


_______________________________________________
jboss-as7-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/jboss-as7-dev

_______________________________________________
jboss-as7-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
Reply | Threaded
Open this post in threaded view
|

Re: Revisited: Integration TestSuite Organization and Maintenance

Rostislav Svoboda
In reply to this post by kkhan
> On 12 Aug 2011, at 04:28, Stuart Douglas wrote:
>
> >
> > On 12/08/2011, at 9:06 AM, Andrew Lee Rubinger wrote:
> >
> >> Inline.
> >>
> >> On 08/11/2011 05:12 PM, Richard Achmatowicz wrote:
> >>> Andrew
> >>>
> >>> Can you give a simple, concrete example of a spec test case
> >>> (citing part
> >>> of the J2EE spec, say), which illustrates:
> >>
> >> Hehe, my PoC is filled with 'em. :) But keep in mind we're not
> >> directly
> >> necessarily citing the EE specs, because we're *not* building a TCK
> >> in AS7.
> >>
> >>> - what dependencies you want to be on the classpath, as well as
> >>> those
> >>> you don't want to be only the classpath
> >>
> >> For this discussion, I'm primarily concerned with the *compilation*
> >> ClassPath, as this is the view users see when building their
> >> applications. It's about explicitly defining what the AS7 API is.
> >> So
> >> the deps for the compilation ClassPath should be as they are in by
> >> AS7-999 branch and as described in the last email:
> >>
> >> * spec - Java SE and Java EE
> >> * api - JBoss-specific extensions to "spec" which comprise our AS7
> >> API
> >> * internals - Anything in the AS7 runtime
> >
> > I am ok with splitting the tests up in this manner, but I think we
> > will need additional test suite modules for purely technical
> > reasons.
> >
> > For instance, at the moment we also have compat for hibernate 3
> > testing, and timer service, to test the ejb timer service.
> >
> > Compat is needed because maven cannot handle having multiple
> > hibernate versions in the same module, and timer service
> > is separate because it needs the preview configuration, and need to
> > run multiple times in order to test restoring persistent timers.

Another example for new module should be tests on several AS7 instances - for some cases you would need 2, 4 and more instances, sometimes you will need to shutdown/kill AS7 instance (failover, replication tests).
How would be AS7 instances managed in this proposal? It seems that everybody is interested only to run one instance of standalone server and run tests against it.

What about 'Clustering tests on multiple AS7 instances proposal' by Karel Piwko (http://lists.jboss.org/pipermail/jboss-as7-dev/2011-August/003434.html) ? Clustering tests are not mentioned in this proposal.

And what about tests for management APIs? Should they be placed in api module? Management APIs are not extension to "spec".

Andrew wants to use profiles to specify parameters for tests and maven plugins + to select tests which should be executed.
In QA we would like to execute all tests with one command (mvn test) and I think that wouldn't be easily possible with this approach.
I can imagine array of parameters, Maven is called in cycles with different parameters. This is complicated to maintain - you must cover all combinations + when new profile is added all test jobs should be reconfigured to use new configurations. And you must be sure that test results are not overwritten with new run. We need to run testsuite on RHEL/Solaris/Windows, one command to execute all tests is necessary.

> >>
> >>> - what sample artifacts you would put in src/main and what you
> >>> would put
> >>> in src/test
> >>
> >> * src/main - Stuff that is part of the deployment. Anything that
> >> you'd
> >> be including in the @Deployment archive defined by an
> >> Arquillian-based
> >> test case.
> >> * src/test - The test infrastructure itself, like the JUnit test
> >> and
> >> other util/helpers
> >
> > I am very much opposed to this split. IMHO splitting the test
> > classes into two different places makes it much harder to read and
> > write tests.
> >
> > I also don't see how having additional test scoped dependencies on
> > the class path is a problem, as in practice this works out to being
> > Arquillian and JUnit.
>
> I agree here, I don't like having tests split up over two source
> folders

+1

Rosta

> >
> > Stuart
> >
> >>
> >>> - what compile and test execution time constraints you would want
> >>> to enforce
> >>> This would help me to see the motivation for the refactoring and
> >>> your
> >>> planned organization of artifacts in src/main and src/test, which
> >>> differ from the standard organization of putting all test related
> >>> artifacts in src/test.
> >>
> >> Test execution time is an orthogonal concern, and relates instead
> >> back
> >> to using "smoke" tests to run by default instead of the entire AS7
> >> integration test suite (which won't scale to run on every build
> >> over
> >> time). IMO it's already taking too long for standard builds.
> >>
> >> Again, the motivation for moving/organizing in this fashion is to
> >> honor
> >> dependencies and assert that the APIs we export ("api" and
> >> "spec-api"
> >> modules) are complete. For instance, while moving tests around I
> >> discovered that our POMs were incompete:
> >>
> >> https://issues.jboss.org/browse/AS7-1489
> >> https://issues.jboss.org/browse/AS7-1493
> >> https://issues.jboss.org/browse/AS7-1479
> >> https://issues.jboss.org/browse/AS7-1478
> >>
> >>> Also, is it not possible to control what is on the classpath by
> >>> maven
> >>> elements like<classpathDependencyExcludes/> and the like?
> >>
> >> That's for test runtime. Because Arquillian is starting (or
> >> connecting
> >> to) the container in a remote process, we're less concerned with
> >> the
> >> client runtime ClassPath, so long as it's enough for Arquillian to
> >> do
> >> its thing.
> >>
> >> S,
> >> ALR
> >>
> >>>
> >>> Richard
> >>>
> >>> On 08/11/2011 07:38 AM, Andrew Lee Rubinger wrote:
> >>>> Hi guys:
> >>>>
> >>>> I'd like to reopen the discussion regarding the testsuite
> >>>> organization
> >>>> and its ongoing maintenance. This issue dates back a few months
> >>>> with
> >>>> some debates and differing opinions, so I'll do my best to
> >>>> outline the
> >>>> guiding principles I'd like to see put in place concisely.
> >>>>
> >>>> To start off, I've a Proof-of-Concept for many of the following
> >>>> points
> >>>> now located:
> >>>>
> >>>> https://github.com/ALRubinger/jboss-as/tree/AS7-999
> >>>>
> >>>> The relevant JIRA I've been using to track things:
> >>>>
> >>>> https://issues.jboss.org/browse/AS7-999
> >>>>
> >>>> So:
> >>>>
> >>>> 1) TestSuite Organization
> >>>>
> >>>> I believe we need a single top-level categorization by which we
> >>>> may
> >>>> organize integration tests which are deployment-based and run
> >>>> within the
> >>>> context of the server. Because we use Maven modules (which are
> >>>> bound to
> >>>> a dependency structure), it makes sense to file these modules by
> >>>> the
> >>>> compile-time dependencies they require. So in place I've put:
> >>>>
> >>>> testsuite/spec - Java SE and Java EE APIs only
> >>>> testsuite/api - AS7 APIs + Spec
> >>>> testsuite/internals - Use anything in the AS7 runtime in your
> >>>> deployments; not guaranteed to be back-compat across releases
> >>>>
> >>>> The primary motivation here is to ensure that the dependencies we
> >>>> export
> >>>> (ie. "spec-api", and "api" modules) are complete enough for users
> >>>> to
> >>>> create their own deployments. In this setup, we act as *users* of
> >>>> our
> >>>> own APIs, and everything in src/main is limited to the relevant
> >>>> dependencies.
> >>>>
> >>>> I know the source of some disagreements earlier centered around
> >>>> placing
> >>>> the tests right next to the deployments, and some folks consider
> >>>> the
> >>>> deployments as part of the test itself. That's not a bad argument
> >>>> at
> >>>> all, but again consider that we then lose the ability to validate
> >>>> our
> >>>> tests in the context of our exported APIs.
> >>>>
> >>>> 2) Run Modes, Test Subsets
> >>>>
> >>>> Because the primary organizational criteria proposed in 1) is by
> >>>> dependency, these modules will grow large over time. The AS build
> >>>> over
> >>>> time will take longer and longer to run. Additionally, there are
> >>>> runtime options to consider when starting tests. So consider the
> >>>> following requirements:
> >>>>
> >>>>     * Running the testsuite in IPv6
> >>>>     * Running only a subset of tests as part of the main build
> >>>>
> >>>> These lend themselves well to using build profiles. By default, I
> >>>> think
> >>>> the "smoke tests" should simply be a set of tests we deem
> >>>> important or
> >>>> indicative of the general health of AS7 with respect to each
> >>>> subsystem.
> >>>>    As it stands now, "smoke" is its own module with a bunch of
> >>>> Embedded-based tests, and I think these should move to the
> >>>> organizational structure in 1) and instead we can apply some
> >>>> filtering
> >>>> to make the "smoke" some default set of includes.
> >>>>
> >>>> 3) An authoritative maintainer
> >>>>
> >>>> I'd like to treat the Arquillian and TestSuite modules as true
> >>>> subsystems of the Application Server, and as such we'll need
> >>>> someone to
> >>>> assume the responsibility to review incoming commits/pull
> >>>> requests and
> >>>> ensure they fit the criteria for acceptance. Simple things like
> >>>> consistent package names, using ARQ correctly, and not leaking
> >>>> dependencies are very important.
> >>>>
> >>>> So assuming we come to agreement on these points, I'd like to
> >>>> request
> >>>> push access to the AS7 repo to field testsuite and ARQ-related
> >>>> pull
> >>>> requests.
> >>>>
> >>>> ...there's much more to discuss (I've more issues to raise
> >>>> alongside the
> >>>> upcoming EAP requirements), but let's start with those first 3
> >>>> major
> >>>> points and my POC, and run from there.
> >>>>
> >>>> S,
> >>>> ALR
> >>>> _______________________________________________
> >>>> jboss-as7-dev mailing list
> >>>> [hidden email]
> >>>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
> >>>
> >>> _______________________________________________
> >>> jboss-as7-dev mailing list
> >>> [hidden email]
> >>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
> >> _______________________________________________
> >> jboss-as7-dev mailing list
> >> [hidden email]
> >> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
> >
> > _______________________________________________
> > jboss-as7-dev mailing list
> > [hidden email]
> > https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
>
>
> _______________________________________________
> jboss-as7-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
_______________________________________________
jboss-as7-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
Reply | Threaded
Open this post in threaded view
|

Re: Revisited: Integration TestSuite Organization and Maintenance

Andrew Lee Rubinger-2
In reply to this post by Andrew Lee Rubinger
Inline.

On 08/11/2011 07:06 PM, Andrew Lee Rubinger wrote:
> On 08/11/2011 05:12 PM, Richard Achmatowicz wrote:
>> Andrew
>>
>> Can you give a simple, concrete example of a spec test case (citing part
>> of the J2EE spec, say), which illustrates:
>
> Hehe, my PoC is filled with 'em. :)  But keep in mind we're not directly
> necessarily citing the EE specs, because we're *not* building a TCK in AS7.

My mistake, I'd meant to throw a link in here as Good Example:

src/main:
https://github.com/ALRubinger/jboss-as/tree/AS7-999/testsuite/spec/src/main/java/org/jboss/as/test/spec/web/ejb
src/test:
https://github.com/ALRubinger/jboss-as/blob/AS7-999/testsuite/spec/src/test/java/org/jboss/as/test/spec/web/ejb/ServletInjectionTestCase.java

S,
ALR


>
>> - what dependencies you want to be on the classpath, as well as those
>> you don't want to be only the classpath
>
> For this discussion, I'm primarily concerned with the *compilation*
> ClassPath, as this is the view users see when building their
> applications.  It's about explicitly defining what the AS7 API is.  So
> the deps for the compilation ClassPath should be as they are in by
> AS7-999 branch and as described in the last email:
>
> * spec - Java SE and Java EE
> * api - JBoss-specific extensions to "spec" which comprise our AS7 API
> * internals - Anything in the AS7 runtime
>
>> - what sample artifacts you would put in src/main and what you would put
>> in src/test
>
> * src/main - Stuff that is part of the deployment.  Anything that you'd
> be including in the @Deployment archive defined by an Arquillian-based
> test case.
> * src/test - The test infrastructure itself, like the JUnit test and
> other util/helpers
>
>> - what compile and test execution time constraints you would want to enforce
>> This would help me to see the motivation for the refactoring and your
>> planned organization of artifacts in src/main and src/test, which
>> differ from the standard organization of putting all test related
>> artifacts in src/test.
>
> Test execution time is an orthogonal concern, and relates instead back
> to using "smoke" tests to run by default instead of the entire AS7
> integration test suite (which won't scale to run on every build over
> time).  IMO it's already taking too long for standard builds.
>
> Again, the motivation for moving/organizing in this fashion is to honor
> dependencies and assert that the APIs we export ("api" and "spec-api"
> modules) are complete.  For instance, while moving tests around I
> discovered that our POMs were incompete:
>
> https://issues.jboss.org/browse/AS7-1489
> https://issues.jboss.org/browse/AS7-1493
> https://issues.jboss.org/browse/AS7-1479
> https://issues.jboss.org/browse/AS7-1478
>
>> Also, is it not possible to control what is on the classpath by maven
>> elements like<classpathDependencyExcludes/>   and the like?
>
> That's for test runtime.  Because Arquillian is starting (or connecting
> to) the container in a remote process, we're less concerned with the
> client runtime ClassPath, so long as it's enough for Arquillian to do
> its thing.
>
> S,
> ALR
>
>>
>> Richard
>>
>> On 08/11/2011 07:38 AM, Andrew Lee Rubinger wrote:
>>> Hi guys:
>>>
>>> I'd like to reopen the discussion regarding the testsuite organization
>>> and its ongoing maintenance.  This issue dates back a few months with
>>> some debates and differing opinions, so I'll do my best to outline the
>>> guiding principles I'd like to see put in place concisely.
>>>
>>> To start off, I've a Proof-of-Concept for many of the following points
>>> now located:
>>>
>>> https://github.com/ALRubinger/jboss-as/tree/AS7-999
>>>
>>> The relevant JIRA I've been using to track things:
>>>
>>> https://issues.jboss.org/browse/AS7-999
>>>
>>> So:
>>>
>>> 1) TestSuite Organization
>>>
>>> I believe we need a single top-level categorization by which we may
>>> organize integration tests which are deployment-based and run within the
>>> context of the server.  Because we use Maven modules (which are bound to
>>> a dependency structure), it makes sense to file these modules by the
>>> compile-time dependencies they require.  So in place I've put:
>>>
>>> testsuite/spec - Java SE and Java EE APIs only
>>> testsuite/api - AS7 APIs + Spec
>>> testsuite/internals - Use anything in the AS7 runtime in your
>>> deployments; not guaranteed to be back-compat across releases
>>>
>>> The primary motivation here is to ensure that the dependencies we export
>>> (ie. "spec-api", and "api" modules) are complete enough for users to
>>> create their own deployments.  In this setup, we act as *users* of our
>>> own APIs, and everything in src/main is limited to the relevant
>>> dependencies.
>>>
>>> I know the source of some disagreements earlier centered around placing
>>> the tests right next to the deployments, and some folks consider the
>>> deployments as part of the test itself.  That's not a bad argument at
>>> all, but again consider that we then lose the ability to validate our
>>> tests in the context of our exported APIs.
>>>
>>> 2) Run Modes, Test Subsets
>>>
>>> Because the primary organizational criteria proposed in 1) is by
>>> dependency, these modules will grow large over time.  The AS build over
>>> time will take longer and longer to run.  Additionally, there are
>>> runtime options to consider when starting tests.  So consider the
>>> following requirements:
>>>
>>>       * Running the testsuite in IPv6
>>>       * Running only a subset of tests as part of the main build
>>>
>>> These lend themselves well to using build profiles.  By default, I think
>>> the "smoke tests" should simply be a set of tests we deem important or
>>> indicative of the general health of AS7 with respect to each subsystem.
>>>      As it stands now, "smoke" is its own module with a bunch of
>>> Embedded-based tests, and I think these should move to the
>>> organizational structure in 1) and instead we can apply some filtering
>>> to make the "smoke" some default set of includes.
>>>
>>> 3) An authoritative maintainer
>>>
>>> I'd like to treat the Arquillian and TestSuite modules as true
>>> subsystems of the Application Server, and as such we'll need someone to
>>> assume the responsibility to review incoming commits/pull requests and
>>> ensure they fit the criteria for acceptance.  Simple things like
>>> consistent package names, using ARQ correctly, and not leaking
>>> dependencies are very important.
>>>
>>> So assuming we come to agreement on these points, I'd like to request
>>> push access to the AS7 repo to field testsuite and ARQ-related pull
>>> requests.
>>>
>>> ...there's much more to discuss (I've more issues to raise alongside the
>>> upcoming EAP requirements), but let's start with those first 3 major
>>> points and my POC, and run from there.
>>>
>>> S,
>>> ALR
>>> _______________________________________________
>>> jboss-as7-dev mailing list
>>> [hidden email]
>>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
>>
>> _______________________________________________
>> jboss-as7-dev mailing list
>> [hidden email]
>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
> _______________________________________________
> jboss-as7-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
_______________________________________________
jboss-as7-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
Reply | Threaded
Open this post in threaded view
|

Re: Revisited: Integration TestSuite Organization and Maintenance

Andrew Lee Rubinger
In reply to this post by Stuart Douglas
Inline again.

On 08/11/2011 11:28 PM, Stuart Douglas wrote:

> I am ok with splitting the tests up in this manner, but I think we will
> need additional test suite modules for purely technical reasons.
>
> For instance, at the moment we also have compat for hibernate 3 testing,
> and timer service, to test the ejb timer service.
>
> Compat is needed because maven cannot handle having multiple hibernate
> versions in the same module, and timer service
> is separate because it needs the preview configuration, and need to run
> multiple times in order to test restoring persistent timers.

Agreed.  At the moment I've addressed only the standard "we need to test
a deployment" type of integration case.  There will be the need for
other modules as you note, also for stuff like:

1) Tests which do manual control of multinode domain
2) CLI

...etc.  QA is feeding us some great requirements for these, and I'd
like to discuss those once we resolve the first 3 points I've raised so
far. :)

S,
ALR
_______________________________________________
jboss-as7-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
Reply | Threaded
Open this post in threaded view
|

Re: Revisited: Integration TestSuite Organization and Maintenance

Andrew Lee Rubinger
Was able to catch up w/ Stuart on #jboss-as7, and thought it'd make
sense to document that conversation here; it clearly identifies the gaps
in our approaches.

(07:49:54 AM) ALR@Freenode: stuartdouglas: At this point, we're mainly
at a gap just on the source folder separation>
(07:49:54 AM) ALR@Freenode: ?
(07:51:13 AM) stuartdouglas: yes, I just don't think that it actually
buys you much in terms of testing the API's, and it makes it harder to
write and read the tests
(07:52:12 AM) stuartdouglas: I also don't think that restricting the
tests to a the API's counts as testing the API's
(07:52:55 AM) ALR@Freenode: stuartdouglas: What else would you propose
to address the testing that the APIs are complete?
(07:53:14 AM) ALR@Freenode: For instance in doing the separation, I
uncovered 4 JIRAs.
(07:53:29 AM) ALR@Freenode: We would have missed these otherwise (and we
had been missing them until now).
(07:53:35 AM) ALR@Freenode: That's proof enough for me. :)
(07:54:02 AM) ALR@Freenode: There's 2 things you need to do to fully
test an exported API:
(07:54:11 AM) ALR@Freenode: 1) Ensure it's complete (which is what this
does)
(07:54:17 AM) tcrawley [~tcrawley@redhat/jboss/tcrawley] entered the room.
(07:54:17 AM) mode (+v tcrawley) by ChanServ
(07:54:24 AM) ALR@Freenode: 2) Ensure it's not leaking out too much
(which is a much tougher problem)
(07:56:45 AM) stuartdouglas: I don't think it really does either, the
tests have not really been designed with the intention of fully
exercising the API, even though it may expose some issues
(07:58:02 AM) ALR@Freenode: stuartdouglas: Which is why we should be
continuing to write them as if we were users :)
(07:58:12 AM) ALR@Freenode: stuartdouglas: Though honestly I think they
do a very good job.
(07:58:20 AM) ALR@Freenode: Arquillian is helping a lot with that.
(07:58:41 AM) ALR@Freenode: Most of them basically just define a
deployment, maybe do an injection, and then perform their assertions.
(07:59:01 AM) stuartdouglas: Which is why I don't see why it is so
important to restrict the class path
(07:59:02 AM) ALR@Freenode: Yours in particular are very clear and
well-written IMO
(07:59:11 AM) ALR@Freenode: stuartdouglas: How can that be?
(07:59:24 AM) ALR@Freenode: I've shown you quite a few reasons why it's
problematic now.
(07:59:55 AM) ALR@Freenode: The JIRAs I discovered for one.  How the ARQ
runtime deps would leak for another.
(08:00:21 AM) ALR@Freenode: So it looks like the basis of your argument
is that you don't refute these points, but just don't consider them
important.
(08:00:34 AM) ALR@Freenode: And are instead placing greater importance
over the convenience of unified source folders.
(08:02:32 AM) stuartdouglas: It's not just convenience, it makes them
easier to read and easier to write, which is important. If people are
disciplined when they are writing tests then the fact that there are
dependencies available on the class path should not matter.
(08:02:45 AM) stuartdouglas: I also don't think that it really counts as
'testing the API'
(08:03:06 AM) pilhuhn is now known as pil-afk-biab
(08:03:39 AM) stuartdouglas: if you want to test the API I think you
should have a test/module for this testing that actually tests it properly
(08:04:22 AM) ALR@Freenode: stuartdouglas: Hehe, I think we've proven
that the testsuite is not approached w/ discipline.
(08:04:39 AM) ALR@Freenode: It's a bit wild west, with many styles mixed
in depending upon the author.
(08:05:26 AM) ALR@Freenode: stuartdouglas: Also we need to settle upon a
categorization criteria.  Mine proposed is by dependency.  Unless you
separate it out into separate source folders, you might as well not
categorize at all.
(08:05:42 AM) ALR@Freenode: Or propose another criteria by which
integration tests should be categorized.
(08:06:35 AM) ALR@Freenode: stuartdouglas: And the key to testing an API
is simply coverage.
(08:06:54 AM) ALR@Freenode: By that measure, I think the integration
testsuite as it stands is a perfect candidate.
(08:07:15 AM) ALR@Freenode: Hitting all corners of the types of apps
that users may write.
(08:07:50 AM) ALR@Freenode: So yeah, I don't want to trade technical
concerns for (IMO relatively) small convenience.
(08:10:28 AM) bbrowning: Is it intentional that the ProcessController is
hard-coded to bind to 127.0.0.1 and a random port? The code at
https://github.com/jbossas/jboss-as/blob/master/process-controller/src/main/java/org/jboss/as/process/Main.java#L210 
looks like maybe it's meant to be configurable?
(08:10:37 AM) bbrowning: lines 210 - 216
(08:16:24 AM) ALR@Freenode: stuartdouglas: I'm considering your points
pretty thoroughly, but I just can't agree.  I've uncovered some serious
issues with regards to our dependency management, and fixed them in my
AS7-999 patch.  I don't see how keeping a unified source folder
structure is worth re-exposing those risks.
(08:16:26 AM) jbossbot: jira [AS7-999] Move tests from
testsuite/integration (legacy) to api/spec/internals [Open (Unresolved)
Task, Major, Andrew Rubinger] https://issues.jboss.org/browse/AS7-999
(08:17:37 AM) ALR@Freenode: Sorry, I'd really like to keep everyone
happy here.
(08:17:54 AM) ALR@Freenode: But I'll favor strict and correct over easy.
(08:18:05 AM) ALR@Freenode: Hehe, just take a look at what a PITA
modules config is!
(08:18:27 AM) ALR@Freenode: Not easy.  But keeps isolation and keeps
things from colliding.

S,
ALR

On 08/12/2011 08:38 AM, Andrew Lee Rubinger wrote:

> Inline again.
>
> On 08/11/2011 11:28 PM, Stuart Douglas wrote:
>> I am ok with splitting the tests up in this manner, but I think we will
>> need additional test suite modules for purely technical reasons.
>>
>> For instance, at the moment we also have compat for hibernate 3 testing,
>> and timer service, to test the ejb timer service.
>>
>> Compat is needed because maven cannot handle having multiple hibernate
>> versions in the same module, and timer service
>> is separate because it needs the preview configuration, and need to run
>> multiple times in order to test restoring persistent timers.
>
> Agreed.  At the moment I've addressed only the standard "we need to test
> a deployment" type of integration case.  There will be the need for
> other modules as you note, also for stuff like:
>
> 1) Tests which do manual control of multinode domain
> 2) CLI
>
> ...etc.  QA is feeding us some great requirements for these, and I'd
> like to discuss those once we resolve the first 3 points I've raised so
> far. :)
>
> S,
> ALR
> _______________________________________________
> jboss-as7-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
_______________________________________________
jboss-as7-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
Reply | Threaded
Open this post in threaded view
|

Re: Revisited: Integration TestSuite Organization and Maintenance

Richard Achmatowicz
In reply to this post by Rostislav Svoboda
On 08/12/2011 07:05 AM, Rostislav Svoboda wrote:

>> On 12 Aug 2011, at 04:28, Stuart Douglas wrote:
>>
>>> On 12/08/2011, at 9:06 AM, Andrew Lee Rubinger wrote:
>>>
>>>> Inline.
>>>>
>>>> On 08/11/2011 05:12 PM, Richard Achmatowicz wrote:
>>>>> Andrew
>>>>>
>>>>> Can you give a simple, concrete example of a spec test case
>>>>> (citing part
>>>>> of the J2EE spec, say), which illustrates:
>>>> Hehe, my PoC is filled with 'em. :) But keep in mind we're not
>>>> directly
>>>> necessarily citing the EE specs, because we're *not* building a TCK
>>>> in AS7.
>>>>
>>>>> - what dependencies you want to be on the classpath, as well as
>>>>> those
>>>>> you don't want to be only the classpath
>>>> For this discussion, I'm primarily concerned with the *compilation*
>>>> ClassPath, as this is the view users see when building their
>>>> applications. It's about explicitly defining what the AS7 API is.
>>>> So
>>>> the deps for the compilation ClassPath should be as they are in by
>>>> AS7-999 branch and as described in the last email:
>>>>
>>>> * spec - Java SE and Java EE
>>>> * api - JBoss-specific extensions to "spec" which comprise our AS7
>>>> API
>>>> * internals - Anything in the AS7 runtime
>>> I am ok with splitting the tests up in this manner, but I think we
>>> will need additional test suite modules for purely technical
>>> reasons.
>>>
>>> For instance, at the moment we also have compat for hibernate 3
>>> testing, and timer service, to test the ejb timer service.
>>>
>>> Compat is needed because maven cannot handle having multiple
>>> hibernate versions in the same module, and timer service
>>> is separate because it needs the preview configuration, and need to
>>> run multiple times in order to test restoring persistent timers.
> Another example for new module should be tests on several AS7 instances - for some cases you would need 2, 4 and more instances, sometimes you will need to shutdown/kill AS7 instance (failover, replication tests).
> How would be AS7 instances managed in this proposal? It seems that everybody is interested only to run one instance of standalone server and run tests against it.

I believe this is not a full proposal but an initial discussion on how
to cater for specific types of dependency management (e.g. for correctly
testing apis) which may be required down the line.

> What about 'Clustering tests on multiple AS7 instances proposal' by Karel Piwko (http://lists.jboss.org/pipermail/jboss-as7-dev/2011-August/003434.html) ? Clustering tests are not mentioned in this proposal.
>
> And what about tests for management APIs? Should they be placed in api module? Management APIs are not extension to "spec".
>
> Andrew wants to use profiles to specify parameters for tests and maven plugins + to select tests which should be executed.
> In QA we would like to execute all tests with one command (mvn test) and I think that wouldn't be easily possible with this approach.
> I can imagine array of parameters, Maven is called in cycles with different parameters. This is complicated to maintain - you must cover all combinations + when new profile is added all test jobs should be reconfigured to use new configurations. And you must be sure that test results are not overwritten with new run. We need to run testsuite on RHEL/Solaris/Windows, one command to execute all tests is necessary.
>
>>>>> - what sample artifacts you would put in src/main and what you
>>>>> would put
>>>>> in src/test
>>>> * src/main - Stuff that is part of the deployment. Anything that
>>>> you'd
>>>> be including in the @Deployment archive defined by an
>>>> Arquillian-based
>>>> test case.
>>>> * src/test - The test infrastructure itself, like the JUnit test
>>>> and
>>>> other util/helpers
>>> I am very much opposed to this split. IMHO splitting the test
>>> classes into two different places makes it much harder to read and
>>> write tests.
>>>
>>> I also don't see how having additional test scoped dependencies on
>>> the class path is a problem, as in practice this works out to being
>>> Arquillian and JUnit.
>> I agree here, I don't like having tests split up over two source
>> folders
> +1
>
> Rosta
>
>>> Stuart
>>>
>>>>> - what compile and test execution time constraints you would want
>>>>> to enforce
>>>>> This would help me to see the motivation for the refactoring and
>>>>> your
>>>>> planned organization of artifacts in src/main and src/test, which
>>>>> differ from the standard organization of putting all test related
>>>>> artifacts in src/test.
>>>> Test execution time is an orthogonal concern, and relates instead
>>>> back
>>>> to using "smoke" tests to run by default instead of the entire AS7
>>>> integration test suite (which won't scale to run on every build
>>>> over
>>>> time). IMO it's already taking too long for standard builds.
>>>>
>>>> Again, the motivation for moving/organizing in this fashion is to
>>>> honor
>>>> dependencies and assert that the APIs we export ("api" and
>>>> "spec-api"
>>>> modules) are complete. For instance, while moving tests around I
>>>> discovered that our POMs were incompete:
>>>>
>>>> https://issues.jboss.org/browse/AS7-1489
>>>> https://issues.jboss.org/browse/AS7-1493
>>>> https://issues.jboss.org/browse/AS7-1479
>>>> https://issues.jboss.org/browse/AS7-1478
>>>>
>>>>> Also, is it not possible to control what is on the classpath by
>>>>> maven
>>>>> elements like<classpathDependencyExcludes/>  and the like?
>>>> That's for test runtime. Because Arquillian is starting (or
>>>> connecting
>>>> to) the container in a remote process, we're less concerned with
>>>> the
>>>> client runtime ClassPath, so long as it's enough for Arquillian to
>>>> do
>>>> its thing.
>>>>
>>>> S,
>>>> ALR
>>>>
>>>>> Richard
>>>>>
>>>>> On 08/11/2011 07:38 AM, Andrew Lee Rubinger wrote:
>>>>>> Hi guys:
>>>>>>
>>>>>> I'd like to reopen the discussion regarding the testsuite
>>>>>> organization
>>>>>> and its ongoing maintenance. This issue dates back a few months
>>>>>> with
>>>>>> some debates and differing opinions, so I'll do my best to
>>>>>> outline the
>>>>>> guiding principles I'd like to see put in place concisely.
>>>>>>
>>>>>> To start off, I've a Proof-of-Concept for many of the following
>>>>>> points
>>>>>> now located:
>>>>>>
>>>>>> https://github.com/ALRubinger/jboss-as/tree/AS7-999
>>>>>>
>>>>>> The relevant JIRA I've been using to track things:
>>>>>>
>>>>>> https://issues.jboss.org/browse/AS7-999
>>>>>>
>>>>>> So:
>>>>>>
>>>>>> 1) TestSuite Organization
>>>>>>
>>>>>> I believe we need a single top-level categorization by which we
>>>>>> may
>>>>>> organize integration tests which are deployment-based and run
>>>>>> within the
>>>>>> context of the server. Because we use Maven modules (which are
>>>>>> bound to
>>>>>> a dependency structure), it makes sense to file these modules by
>>>>>> the
>>>>>> compile-time dependencies they require. So in place I've put:
>>>>>>
>>>>>> testsuite/spec - Java SE and Java EE APIs only
>>>>>> testsuite/api - AS7 APIs + Spec
>>>>>> testsuite/internals - Use anything in the AS7 runtime in your
>>>>>> deployments; not guaranteed to be back-compat across releases
>>>>>>
>>>>>> The primary motivation here is to ensure that the dependencies we
>>>>>> export
>>>>>> (ie. "spec-api", and "api" modules) are complete enough for users
>>>>>> to
>>>>>> create their own deployments. In this setup, we act as *users* of
>>>>>> our
>>>>>> own APIs, and everything in src/main is limited to the relevant
>>>>>> dependencies.
>>>>>>
>>>>>> I know the source of some disagreements earlier centered around
>>>>>> placing
>>>>>> the tests right next to the deployments, and some folks consider
>>>>>> the
>>>>>> deployments as part of the test itself. That's not a bad argument
>>>>>> at
>>>>>> all, but again consider that we then lose the ability to validate
>>>>>> our
>>>>>> tests in the context of our exported APIs.
>>>>>>
>>>>>> 2) Run Modes, Test Subsets
>>>>>>
>>>>>> Because the primary organizational criteria proposed in 1) is by
>>>>>> dependency, these modules will grow large over time. The AS build
>>>>>> over
>>>>>> time will take longer and longer to run. Additionally, there are
>>>>>> runtime options to consider when starting tests. So consider the
>>>>>> following requirements:
>>>>>>
>>>>>>      * Running the testsuite in IPv6
>>>>>>      * Running only a subset of tests as part of the main build
>>>>>>
>>>>>> These lend themselves well to using build profiles. By default, I
>>>>>> think
>>>>>> the "smoke tests" should simply be a set of tests we deem
>>>>>> important or
>>>>>> indicative of the general health of AS7 with respect to each
>>>>>> subsystem.
>>>>>>     As it stands now, "smoke" is its own module with a bunch of
>>>>>> Embedded-based tests, and I think these should move to the
>>>>>> organizational structure in 1) and instead we can apply some
>>>>>> filtering
>>>>>> to make the "smoke" some default set of includes.
>>>>>>
>>>>>> 3) An authoritative maintainer
>>>>>>
>>>>>> I'd like to treat the Arquillian and TestSuite modules as true
>>>>>> subsystems of the Application Server, and as such we'll need
>>>>>> someone to
>>>>>> assume the responsibility to review incoming commits/pull
>>>>>> requests and
>>>>>> ensure they fit the criteria for acceptance. Simple things like
>>>>>> consistent package names, using ARQ correctly, and not leaking
>>>>>> dependencies are very important.
>>>>>>
>>>>>> So assuming we come to agreement on these points, I'd like to
>>>>>> request
>>>>>> push access to the AS7 repo to field testsuite and ARQ-related
>>>>>> pull
>>>>>> requests.
>>>>>>
>>>>>> ...there's much more to discuss (I've more issues to raise
>>>>>> alongside the
>>>>>> upcoming EAP requirements), but let's start with those first 3
>>>>>> major
>>>>>> points and my POC, and run from there.
>>>>>>
>>>>>> S,
>>>>>> ALR
>>>>>> _______________________________________________
>>>>>> jboss-as7-dev mailing list
>>>>>> [hidden email]
>>>>>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
>>>>> _______________________________________________
>>>>> jboss-as7-dev mailing list
>>>>> [hidden email]
>>>>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
>>>> _______________________________________________
>>>> jboss-as7-dev mailing list
>>>> [hidden email]
>>>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
>>> _______________________________________________
>>> jboss-as7-dev mailing list
>>> [hidden email]
>>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
>>
>> _______________________________________________
>> jboss-as7-dev mailing list
>> [hidden email]
>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
> _______________________________________________
> jboss-as7-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev

_______________________________________________
jboss-as7-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
Reply | Threaded
Open this post in threaded view
|

Re: Revisited: Integration TestSuite Organization and Maintenance

Andrew Lee Rubinger
True.  The other issues we've identified I'll raise once we clear some
of the general approach issues blocking testsuite organization. :)

S,
ALR

On 08/12/2011 09:59 AM, Richard Achmatowicz wrote:
>> How would be AS7 instances managed in this proposal? It seems that everybody is interested only to run one instance of standalone server and run tests against it.
> I believe this is not a full proposal but an initial discussion on how
> to cater for specific types of dependency management (e.g. for correctly
> testing apis) which may be required down the line.
_______________________________________________
jboss-as7-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
Reply | Threaded
Open this post in threaded view
|

Re: Revisited: Integration TestSuite Organization and Maintenance

Andrew Lee Rubinger-2
In reply to this post by Andrew Lee Rubinger
A summary of where I believe we stand on the 3 points raised here:

1) TestSuite Organization

Stuart and Kabir have voiced concerns that separating src/main and src/test make authoring more complex.

While I'll concede that may be true to a small extent, I believe the benefits of separation far outweigh the drawbacks.  By separating these out I uncovered 4 JIRAs which proved that our "api" and "spec-api" modules were incomplete.  In short, paying attention to user dependencies to validate against them is very important.

2) Run Modes, Test Subsets

I don't think there's been much discussion here, with the exception of the QE team who have provided some use cases that may not be possible given a standard layout.  For instance the CLI and clustering tests are more than the standard "deploy something and make assertions" format we cover in point 1), so these will likely get their own modules as is appropriate.

3) An authoritative maintainer

No one has commented on this.

By Tuesday evening I need to report back on at least the 3 points above, and also have approval for getting my patch committed.  Failing that, we need to agree on an alternate path forward.  I'm happy with any solution which addresses the respect for dependencies in the testsuite as I've outlined.

S,
ALR

----- Original Message -----
From: "Andrew Lee Rubinger" <[hidden email]>
To: "JBoss AS7 Development" <[hidden email]>
Sent: Thursday, August 11, 2011 7:38:51 AM
Subject: [jboss-as7-dev] Revisited: Integration TestSuite Organization and Maintenance

Hi guys:

I'd like to reopen the discussion regarding the testsuite organization
and its ongoing maintenance.  This issue dates back a few months with
some debates and differing opinions, so I'll do my best to outline the
guiding principles I'd like to see put in place concisely.

To start off, I've a Proof-of-Concept for many of the following points
now located:

https://github.com/ALRubinger/jboss-as/tree/AS7-999

The relevant JIRA I've been using to track things:

https://issues.jboss.org/browse/AS7-999

So:

1) TestSuite Organization

I believe we need a single top-level categorization by which we may
organize integration tests which are deployment-based and run within the
context of the server.  Because we use Maven modules (which are bound to
a dependency structure), it makes sense to file these modules by the
compile-time dependencies they require.  So in place I've put:

testsuite/spec - Java SE and Java EE APIs only
testsuite/api - AS7 APIs + Spec
testsuite/internals - Use anything in the AS7 runtime in your
deployments; not guaranteed to be back-compat across releases

The primary motivation here is to ensure that the dependencies we export
(ie. "spec-api", and "api" modules) are complete enough for users to
create their own deployments.  In this setup, we act as *users* of our
own APIs, and everything in src/main is limited to the relevant
dependencies.

I know the source of some disagreements earlier centered around placing
the tests right next to the deployments, and some folks consider the
deployments as part of the test itself.  That's not a bad argument at
all, but again consider that we then lose the ability to validate our
tests in the context of our exported APIs.

2) Run Modes, Test Subsets

Because the primary organizational criteria proposed in 1) is by
dependency, these modules will grow large over time.  The AS build over
time will take longer and longer to run.  Additionally, there are
runtime options to consider when starting tests.  So consider the
following requirements:

   * Running the testsuite in IPv6
   * Running only a subset of tests as part of the main build

These lend themselves well to using build profiles.  By default, I think
the "smoke tests" should simply be a set of tests we deem important or
indicative of the general health of AS7 with respect to each subsystem.
  As it stands now, "smoke" is its own module with a bunch of
Embedded-based tests, and I think these should move to the
organizational structure in 1) and instead we can apply some filtering
to make the "smoke" some default set of includes.

3) An authoritative maintainer

I'd like to treat the Arquillian and TestSuite modules as true
subsystems of the Application Server, and as such we'll need someone to
assume the responsibility to review incoming commits/pull requests and
ensure they fit the criteria for acceptance.  Simple things like
consistent package names, using ARQ correctly, and not leaking
dependencies are very important.

So assuming we come to agreement on these points, I'd like to request
push access to the AS7 repo to field testsuite and ARQ-related pull
requests.

...there's much more to discuss (I've more issues to raise alongside the
upcoming EAP requirements), but let's start with those first 3 major
points and my POC, and run from there.

S,
ALR
_______________________________________________
jboss-as7-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
_______________________________________________
jboss-as7-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
Reply | Threaded
Open this post in threaded view
|

Re: Revisited: Integration TestSuite Organization and Maintenance

Scott Marlow
On 08/15/2011 03:27 AM, Andrew Lee Rubinger wrote:
> A summary of where I believe we stand on the 3 points raised here:
>
> 1) TestSuite Organization
>
> Stuart and Kabir have voiced concerns that separating src/main and src/test make authoring more complex.

I think keeping it simple/easy to create new tests, is more important
than being able to prove that our api/spec-api modules are incomplete.
We depend on the tests to prove that our AS build works correctly (keep
it simple/easy so people will create the tests).

>
> While I'll concede that may be true to a small extent, I believe the benefits of separation far outweigh the drawbacks.  By separating these out I uncovered 4 JIRAs which proved that our "api" and "spec-api" modules were incomplete.  In short, paying attention to user dependencies to validate against them is very important.
>
> 2) Run Modes, Test Subsets
>
> I don't think there's been much discussion here, with the exception of the QE team who have provided some use cases that may not be possible given a standard layout.  For instance the CLI and clustering tests are more than the standard "deploy something and make assertions" format we cover in point 1), so these will likely get their own modules as is appropriate.
>
> 3) An authoritative maintainer

If someone sets the standard, by creating many of the unit tests by
using ARQ correctly.  Others are more likely to follow their example.
Perhaps someone could tag certain tests as good examples to follow.

>
> No one has commented on this.
>
> By Tuesday evening I need to report back on at least the 3 points above, and also have approval for getting my patch committed.  Failing that, we need to agree on an alternate path forward.  I'm happy with any solution which addresses the respect for dependencies in the testsuite as I've outlined.
>
> S,
> ALR
>
_______________________________________________
jboss-as7-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
Reply | Threaded
Open this post in threaded view
|

Re: Revisited: Integration TestSuite Organization and Maintenance

Andrew Lee Rubinger
On 08/15/2011 08:39 AM, Scott Marlow wrote:

> On 08/15/2011 03:27 AM, Andrew Lee Rubinger wrote:
>> A summary of where I believe we stand on the 3 points raised here:
>>
>> 1) TestSuite Organization
>>
>> Stuart and Kabir have voiced concerns that separating src/main and
>> src/test make authoring more complex.
>
> I think keeping it simple/easy to create new tests, is more important
> than being able to prove that our api/spec-api modules are incomplete.
> We depend on the tests to prove that our AS build works correctly (keep
> it simple/easy so people will create the tests).

Is it really that difficult to split your tests vs. the deployables into
two source folders?  Arquillian makes it all so easy to begin with, I'm
honestly surprised this is such a debate.

And if we're not validating that api/spec-api is complete, what is?

Try to answer this question: What composes the AS7 API?  What artifacts
can users depend upon in order to write software to run on our server?
These are very important to usability, and I want to be sure they're
addressed here.  Because the integration testsuite should by and large
be written as if we are users.

S,
ALR

>
>>
>> While I'll concede that may be true to a small extent, I believe the
>> benefits of separation far outweigh the drawbacks. By separating these
>> out I uncovered 4 JIRAs which proved that our "api" and "spec-api"
>> modules were incomplete. In short, paying attention to user
>> dependencies to validate against them is very important.
>>
>> 2) Run Modes, Test Subsets
>>
>> I don't think there's been much discussion here, with the exception of
>> the QE team who have provided some use cases that may not be possible
>> given a standard layout. For instance the CLI and clustering tests are
>> more than the standard "deploy something and make assertions" format
>> we cover in point 1), so these will likely get their own modules as is
>> appropriate.
>>
>> 3) An authoritative maintainer
>
> If someone sets the standard, by creating many of the unit tests by
> using ARQ correctly. Others are more likely to follow their example.
> Perhaps someone could tag certain tests as good examples to follow.
>
>>
>> No one has commented on this.
>>
>> By Tuesday evening I need to report back on at least the 3 points
>> above, and also have approval for getting my patch committed. Failing
>> that, we need to agree on an alternate path forward. I'm happy with
>> any solution which addresses the respect for dependencies in the
>> testsuite as I've outlined.
>>
>> S,
>> ALR
>>

--
Andrew Lee Rubinger
Senior Software Engineer
JBoss by Red Hat
Twitter: @ALRubinger
http://about.me/alrubinger
_______________________________________________
jboss-as7-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
Reply | Threaded
Open this post in threaded view
|

Re: Revisited: Integration TestSuite Organization and Maintenance

Richard Achmatowicz
In reply to this post by Andrew Lee Rubinger-2
On 08/15/2011 03:27 AM, Andrew Lee Rubinger wrote:
> A summary of where I believe we stand on the 3 points raised here:
>
> 1) TestSuite Organization
>
> Stuart and Kabir have voiced concerns that separating src/main and src/test make authoring more complex.
>
> While I'll concede that may be true to a small extent, I believe the benefits of separation far outweigh the drawbacks.  By separating these out I uncovered 4 JIRAs which proved that our "api" and "spec-api" modules were incomplete.  In short, paying attention to user dependencies to validate against them is very important.
I've been trying to understand in more detail the issues you have raised
concerning creating separate modules for api testing and another for
general integration testing.  I think what is missing here (I may be
missing the full point) is a description of what you are specifically
trying to achieve and we are getting caught up instead in implementation
details. To make things clear, i'm going to refer to this effort as
'spec testing'. Some related questions:

1. I looked at the ServletInjectionTestCase example you gave, and it
seems that your motivation for putting artifacts which may be deployed
during the test case in src/main is simply to be able to:
* compile them against a minimal set of dependencies which are only JDK
jars and J2EE jars (in the case of the spec suite)
* compile them against a minimal set of dependencies which are only JDK
jars, J2EE jars and the AS API jars (in the case of the AS-specific API
suite suite)
Is that all you want to do for your spec testing? Simply compile a set
of java classes representing deployable artifacts against minimal set of
dependencies and check that the compilation succeeds? I suspect that you
have a wider purpose, but, if that is the case, there could be other
ways to achieve the same end.

I also see the word 'complete' being used, which leads me to believe
that you don't just want to perform this process for one specific
deployable artifact, but rather you have some important 'covering' set
of deployable artifacts in mind and maybe test cases as well? What does
the word 'complete' mean with regard to this 'spec testing'? And how
will you know when you have achieved full coverage, whether it is in a
separate module or not?

2. I can see how the JBoss specific 'spec testing' would be include
things not found in the TCK, but how is your spec module different from
what the TCK would do? Does it cover cases which are not covered by the
TCK?

3. Is your ultimate goal to:
(i) test to some level of detail every user-facing API (and every method
within that API), whether that be accessed from deployable components
(like EJBs and Servlets) or components which interact with the AS
remotely (like the CLI or the Domain management interface) and
(ii) for these components, make sure they compile against the minimal
set of dependecies

Or are there other criteria for performing this spec-style testing?
> 2) Run Modes, Test Subsets
>
> I don't think there's been much discussion here, with the exception of the QE team who have provided some use cases that may not be possible given a standard layout.  For instance the CLI and clustering tests are more than the standard "deploy something and make assertions" format we cover in point 1), so these will likely get their own modules as is appropriate.
On the matter of how to concretely organize the testsuite (what we need
to be able to do when executing a test case, where to put sources and
resources for test cases, where to put server configs, where to put
descriptions of sets of test cases, etc), there has been some discussion.

On that front, there are two proposals which have been documented:
* one from QE, found here:
https://docspace.corp.redhat.com/docs/DOC-69049, based on using:
-- submodules to organize test case sources
-- submodules to create server configurations
-- surefire plugin executions to execute sets of test cases
* one from me, found here (parts still under construction) :
https://docspace.corp.redhat.com/docs/DOC-74146, based on using:
-- a single module and package structure to organize test case sources
-- ant snippets to create server configurations (a simple mojo could be
created as well and tied to a lifecycle phase)
-- plugin executions and profile aggregation to structure test cases and
execute sets of test cases
Both approaches will probably work. Essentially, one is based on
modules, the other on flat files. What needs to be decided is which is
simplest to use and maintain. Or maybe there is a third option.

Another issue is the JUnit/testNG split: are we going to support two
test execution frameworks for the next 5 years or simplify down to one.
At the moment, JUnit has been used by default (largely through
intertia), whereas both JGroups and Infinispan converted to testNG years
ago. testNG has many features for grouping test cases which JUnit does
not, for example. In my view, we should scrap JUnit.

> 3) An authoritative maintainer
>
> No one has commented on this.
>
> By Tuesday evening I need to report back on at least the 3 points above, and also have approval for getting my patch committed.  Failing that, we need to agree on an alternate path forward.  I'm happy with any solution which addresses the respect for dependencies in the testsuite as I've outlined.
>
> S,
> ALR
>
> ----- Original Message -----
> From: "Andrew Lee Rubinger"<[hidden email]>
> To: "JBoss AS7 Development"<[hidden email]>
> Sent: Thursday, August 11, 2011 7:38:51 AM
> Subject: [jboss-as7-dev] Revisited: Integration TestSuite Organization and Maintenance
>
> Hi guys:
>
> I'd like to reopen the discussion regarding the testsuite organization
> and its ongoing maintenance.  This issue dates back a few months with
> some debates and differing opinions, so I'll do my best to outline the
> guiding principles I'd like to see put in place concisely.
>
> To start off, I've a Proof-of-Concept for many of the following points
> now located:
>
> https://github.com/ALRubinger/jboss-as/tree/AS7-999
>
> The relevant JIRA I've been using to track things:
>
> https://issues.jboss.org/browse/AS7-999
>
> So:
>
> 1) TestSuite Organization
>
> I believe we need a single top-level categorization by which we may
> organize integration tests which are deployment-based and run within the
> context of the server.  Because we use Maven modules (which are bound to
> a dependency structure), it makes sense to file these modules by the
> compile-time dependencies they require.  So in place I've put:
>
> testsuite/spec - Java SE and Java EE APIs only
> testsuite/api - AS7 APIs + Spec
> testsuite/internals - Use anything in the AS7 runtime in your
> deployments; not guaranteed to be back-compat across releases
>
> The primary motivation here is to ensure that the dependencies we export
> (ie. "spec-api", and "api" modules) are complete enough for users to
> create their own deployments.  In this setup, we act as *users* of our
> own APIs, and everything in src/main is limited to the relevant
> dependencies.
>
> I know the source of some disagreements earlier centered around placing
> the tests right next to the deployments, and some folks consider the
> deployments as part of the test itself.  That's not a bad argument at
> all, but again consider that we then lose the ability to validate our
> tests in the context of our exported APIs.
>
> 2) Run Modes, Test Subsets
>
> Because the primary organizational criteria proposed in 1) is by
> dependency, these modules will grow large over time.  The AS build over
> time will take longer and longer to run.  Additionally, there are
> runtime options to consider when starting tests.  So consider the
> following requirements:
>
>     * Running the testsuite in IPv6
>     * Running only a subset of tests as part of the main build
>
> These lend themselves well to using build profiles.  By default, I think
> the "smoke tests" should simply be a set of tests we deem important or
> indicative of the general health of AS7 with respect to each subsystem.
>    As it stands now, "smoke" is its own module with a bunch of
> Embedded-based tests, and I think these should move to the
> organizational structure in 1) and instead we can apply some filtering
> to make the "smoke" some default set of includes.
>
> 3) An authoritative maintainer
>
> I'd like to treat the Arquillian and TestSuite modules as true
> subsystems of the Application Server, and as such we'll need someone to
> assume the responsibility to review incoming commits/pull requests and
> ensure they fit the criteria for acceptance.  Simple things like
> consistent package names, using ARQ correctly, and not leaking
> dependencies are very important.
>
> So assuming we come to agreement on these points, I'd like to request
> push access to the AS7 repo to field testsuite and ARQ-related pull
> requests.
>
> ...there's much more to discuss (I've more issues to raise alongside the
> upcoming EAP requirements), but let's start with those first 3 major
> points and my POC, and run from there.
>
> S,
> ALR
> _______________________________________________
> jboss-as7-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
> _______________________________________________
> jboss-as7-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev

_______________________________________________
jboss-as7-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
Reply | Threaded
Open this post in threaded view
|

Re: Revisited: Integration TestSuite Organization and Maintenance

Stuart Douglas
In reply to this post by Andrew Lee Rubinger-2

On 15/08/2011, at 5:27 PM, Andrew Lee Rubinger wrote:

> A summary of where I believe we stand on the 3 points raised here:
>
> 1) TestSuite Organization
>
> Stuart and Kabir have voiced concerns that separating src/main and src/test make authoring more complex.
>
> While I'll concede that may be true to a small extent, I believe the benefits of separation far outweigh the drawbacks.  By separating these out I uncovered 4 JIRAs which proved that our "api" and "spec-api" modules were incomplete.  In short, paying attention to user dependencies to validate against them is very important.
>

Personally I still don' t think that the additional testing of the API module makes it worth the extra inconvenience. The API does not change often, and bugs are both easy to identify and work around (e.g. if we are missing an artefact it a end user can for around it by adding the dependency to their pom, not ideal, but but not the end of the world either). On the other hand if this change does result in less tests being written, then this may result in more bugs in the application server code itself, which will be much more difficult to work around.  

With that said though if everyone else agrees with this I am prepared to give it a go, with the caveat that if it causes problems we go back to the current way of doing things.

> 2) Run Modes, Test Subsets
>
> I don't think there's been much discussion here, with the exception of the QE team who have provided some use cases that may not be possible given a standard layout.  For instance the CLI and clustering tests are more than the standard "deploy something and make assertions" format we cover in point 1), so these will likely get their own modules as is appropriate.
>
> 3) An authoritative maintainer
>

Most commits to the test suite come in as part of larger change sets. If there was a maintainer that was supposed to sign off on all test suite changes, then the majority of non-trivial changes would have to be approved by the maintainer, which I do not think will scale very well.

Stuart


> No one has commented on this.
>
> By Tuesday evening I need to report back on at least the 3 points above, and also have approval for getting my patch committed.  Failing that, we need to agree on an alternate path forward.  I'm happy with any solution which addresses the respect for dependencies in the testsuite as I've outlined.
>
> S,
> ALR
>
> ----- Original Message -----
> From: "Andrew Lee Rubinger" <[hidden email]>
> To: "JBoss AS7 Development" <[hidden email]>
> Sent: Thursday, August 11, 2011 7:38:51 AM
> Subject: [jboss-as7-dev] Revisited: Integration TestSuite Organization and Maintenance
>
> Hi guys:
>
> I'd like to reopen the discussion regarding the testsuite organization
> and its ongoing maintenance.  This issue dates back a few months with
> some debates and differing opinions, so I'll do my best to outline the
> guiding principles I'd like to see put in place concisely.
>
> To start off, I've a Proof-of-Concept for many of the following points
> now located:
>
> https://github.com/ALRubinger/jboss-as/tree/AS7-999
>
> The relevant JIRA I've been using to track things:
>
> https://issues.jboss.org/browse/AS7-999
>
> So:
>
> 1) TestSuite Organization
>
> I believe we need a single top-level categorization by which we may
> organize integration tests which are deployment-based and run within the
> context of the server.  Because we use Maven modules (which are bound to
> a dependency structure), it makes sense to file these modules by the
> compile-time dependencies they require.  So in place I've put:
>
> testsuite/spec - Java SE and Java EE APIs only
> testsuite/api - AS7 APIs + Spec
> testsuite/internals - Use anything in the AS7 runtime in your
> deployments; not guaranteed to be back-compat across releases
>
> The primary motivation here is to ensure that the dependencies we export
> (ie. "spec-api", and "api" modules) are complete enough for users to
> create their own deployments.  In this setup, we act as *users* of our
> own APIs, and everything in src/main is limited to the relevant
> dependencies.
>
> I know the source of some disagreements earlier centered around placing
> the tests right next to the deployments, and some folks consider the
> deployments as part of the test itself.  That's not a bad argument at
> all, but again consider that we then lose the ability to validate our
> tests in the context of our exported APIs.
>
> 2) Run Modes, Test Subsets
>
> Because the primary organizational criteria proposed in 1) is by
> dependency, these modules will grow large over time.  The AS build over
> time will take longer and longer to run.  Additionally, there are
> runtime options to consider when starting tests.  So consider the
> following requirements:
>
>   * Running the testsuite in IPv6
>   * Running only a subset of tests as part of the main build
>
> These lend themselves well to using build profiles.  By default, I think
> the "smoke tests" should simply be a set of tests we deem important or
> indicative of the general health of AS7 with respect to each subsystem.
>  As it stands now, "smoke" is its own module with a bunch of
> Embedded-based tests, and I think these should move to the
> organizational structure in 1) and instead we can apply some filtering
> to make the "smoke" some default set of includes.
>
> 3) An authoritative maintainer
>
> I'd like to treat the Arquillian and TestSuite modules as true
> subsystems of the Application Server, and as such we'll need someone to
> assume the responsibility to review incoming commits/pull requests and
> ensure they fit the criteria for acceptance.  Simple things like
> consistent package names, using ARQ correctly, and not leaking
> dependencies are very important.
>
> So assuming we come to agreement on these points, I'd like to request
> push access to the AS7 repo to field testsuite and ARQ-related pull
> requests.
>
> ...there's much more to discuss (I've more issues to raise alongside the
> upcoming EAP requirements), but let's start with those first 3 major
> points and my POC, and run from there.
>
> S,
> ALR
> _______________________________________________
> jboss-as7-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
> _______________________________________________
> jboss-as7-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev


_______________________________________________
jboss-as7-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
12