(AS7-2796) -DskipTests argument no longer preventing the execution of tests on AS7 build

classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|

(AS7-2796) -DskipTests argument no longer preventing the execution of tests on AS7 build

Ondrej Zizka
Hi,

I did this intentionally - the intention was to skip unit tests but run the testsuite - not to get the build failed if some module's unit tests fail.

How about having two params for that, with -DskipTests for unit tests, and, say, -DskipTestsuite for skipping the testsuite module's tests?

Ondra





-------- Přeposlaná zpráva --------
Od: Darran Lofthouse (Created) (JIRA) <[hidden email]>
Komu: [hidden email]
Předmět: [JBoss JIRA] (AS7-2796) -DskipTests argument no longer preventing the execution of tests on AS7 build
Datum: Fri, 25 Nov 2011 10:15:40 -0500 (EST)

-DskipTests argument no longer preventing the execution of tests on AS7 build
-----------------------------------------------------------------------------

                 Key: AS7-2796
                 URL: https://issues.jboss.org/browse/AS7-2796
             Project: Application Server 7
          Issue Type: Bug
          Components: Test Suite
            Reporter: Darran Lofthouse
            Assignee: Ondrej Zizka
             Fix For: 7.1.0.CR1


Since the testsuite changes have been committed the -DskipTests argument no longer skips the tests.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

_______________________________________________
jboss-as7-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
Reply | Threaded
Open this post in threaded view
|

Re: (AS7-2796) -DskipTests argument no longer preventing the execution of tests on AS7 build

Jaikiran Pai
On Monday 28 November 2011 01:35 PM, Ondřej Žižka wrote:
> Hi,
>
> I did this intentionally - the intention was to skip unit tests but
> run the testsuite - not to get the build failed if some module's unit
> tests fail.
If a unit test fails the build should fail. There's
-Dmaven.test.failure.ignore=true if you want to ignore those test
failures. I don't know how the AS7 project handles the
-Dmaven.test.failure.ignore property due to our testsuite
profiles/customizations. But we shouldn't be adding new meaning to
already well known flags.

>
> How about having two params for that, with -DskipTests for unit tests,
> and, say, -DskipTestsuite for skipping the testsuite module's tests?
>
IMO, we shouldn't just be adding all these new flags which make it
complicated to understand what set of flags to use and when. The actual
question for this specific issue is what's the real usecase of skipping
unit tests (which potentially can have failures) but still want to run
the integration tests?

To be clear - IMO, we should have the -DskipTests back without adding a
new meaning to that param.

-Jaikiran

> Ondra
>
>
>
>
>
> -------- Přeposlaná zpráva --------
>> *Od*: Darran Lofthouse (Created) (JIRA) <[hidden email]
>> <mailto:%22Darran%20Lofthouse%20%28Created%29%20%28JIRA%29%22%20%[hidden email]%3e>>
>> *Komu* : [hidden email] <mailto:[hidden email]>
>> *Předmět* : [JBoss JIRA] (AS7-2796) -DskipTests argument no longer
>> preventing the execution of tests on AS7 build
>> *Datum* : Fri, 25 Nov 2011 10:15:40 -0500 (EST)
>>
>> -DskipTests argument no longer preventing the execution of tests on AS7 build
>> -----------------------------------------------------------------------------
>>
>>                   Key: AS7-2796
>>                   URL:https://issues.jboss.org/browse/AS7-2796
>>               Project: Application Server 7
>>            Issue Type: Bug
>>            Components: Test Suite
>>              Reporter: Darran Lofthouse
>>              Assignee: Ondrej Zizka
>>               Fix For: 7.1.0.CR1
>>
>>
>> Since the testsuite changes have been committed the -DskipTests argument no longer skips the tests.
>>
>> --
>> This message is automatically generated by JIRA.
>> If you think it was sent incorrectly, please contact your JIRA administrators:https://issues.jboss.org/secure/ContactAdministrators!default.jspa  <https://issues.jboss.org/secure/ContactAdministrators%21default.jspa>
>> For more information on JIRA, see:http://www.atlassian.com/software/jira
>>
>>
>
>
> _______________________________________________
> jboss-as7-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev

_______________________________________________
jboss-as7-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
Reply | Threaded
Open this post in threaded view
|

Re: (AS7-2796) -DskipTests argument no longer preventing the execution of tests on AS7 build

Jaikiran Pai
On Monday 28 November 2011 01:48 PM, Jaikiran Pai wrote:
>
>> How about having two params for that, with -DskipTests for unit tests,
>> and, say, -DskipTestsuite for skipping the testsuite module's tests?
>>
> IMO, we shouldn't just be adding all these new flags which make it
> complicated to understand what set of flags to use and when. The actual
> question for this specific issue is what's the real usecase of skipping
> unit tests (which potentially can have failures) but still want to run
> the integration tests?
By the way, you could still achieve the goal of running the integration
testsuite and skipping the unit tests, by doing the following 2 steps:

1) From AS7 top level - mvn clean install -DskipTests

This will run the entire build and skip all tests including unit and
integration tests.

2) cd testsuite/integration - mvn clean install

This will run the integration tests

I really don't see a real need for adding another flag for this. But
yes, I might be missing some usecase.

-Jaikiran
_______________________________________________
jboss-as7-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
Reply | Threaded
Open this post in threaded view
|

Re: (AS7-2796) -DskipTests argument no longer preventing the execution of tests on AS7 build

Carlo de Wolf
Note that the Jenkins run (should) use -fae.

But I don't mind the build to fail when module tests fail. Those tests
should be rock stable.

I can see a potential requirement to only run the (full / partial)
testsuite on a completed build. This could be done via profiles.
(Parallel testsuite executions.)

In essence the default behavior of maven should not be tweaked, but
additional behavior defined.

Carlo

On 11/28/2011 09:22 AM, Jaikiran Pai wrote:

> On Monday 28 November 2011 01:48 PM, Jaikiran Pai wrote:
>>> How about having two params for that, with -DskipTests for unit tests,
>>> and, say, -DskipTestsuite for skipping the testsuite module's tests?
>>>
>> IMO, we shouldn't just be adding all these new flags which make it
>> complicated to understand what set of flags to use and when. The actual
>> question for this specific issue is what's the real usecase of skipping
>> unit tests (which potentially can have failures) but still want to run
>> the integration tests?
> By the way, you could still achieve the goal of running the integration
> testsuite and skipping the unit tests, by doing the following 2 steps:
>
> 1) From AS7 top level - mvn clean install -DskipTests
>
> This will run the entire build and skip all tests including unit and
> integration tests.
>
> 2) cd testsuite/integration - mvn clean install
>
> This will run the integration tests
>
> I really don't see a real need for adding another flag for this. But
> yes, I might be missing some usecase.
>
> -Jaikiran
> _______________________________________________
> jboss-as7-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev

_______________________________________________
jboss-as7-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
Reply | Threaded
Open this post in threaded view
|

Re: (AS7-2796) -DskipTests argument no longer preventing the execution of tests on AS7 build

Ondrej Zizka
In reply to this post by Jaikiran Pai
Jaikiran Pai píše v Po 28. 11. 2011 v 13:48 +0530:
On Monday 28 November 2011 01:35 PM, Ondřej Žižka wrote:
> Hi,
>
> I did this intentionally - the intention was to skip unit tests but 
> run the testsuite - not to get the build failed if some module's unit 
> tests fail.
If a unit test fails the build should fail. 
This is still true.
There's 
-Dmaven.test.failure.ignore=true if you want to ignore those test 
failures. I don't know how the AS7 project handles the 
-Dmaven.test.failure.ignore property due to our testsuite 
profiles/customizations. But we shouldn't be adding new meaning to 
already well known flags.
The problem with these is that they apply to whole build.


> How about having two params for that, with -DskipTests for unit tests, 
> and, say, -DskipTestsuite for skipping the testsuite module's tests?
>
IMO, we shouldn't just be adding all these new flags which make it 
complicated to understand what set of flags to use and when.
I have to disappoint you but you'll see more and more flags being added in the future.
Please realize that there are other groups using the testsuite e.g. for EAP testing.
But I can keep you uninformed and keep the testsuite running without changes of "API" if that helps.

 The actual 
question for this specific issue is what's the real usecase of skipping 
unit tests (which potentially can have failures) but still want to run 
the integration tests?
Often it happens that some unit tests fail and block one step AS build & testsuite execution.
And let me ask the other way around - what is the use case of -DskipTests, skipping all tests?

To be clear - IMO, we should have the -DskipTests back without adding a 
new meaning to that param.
Okay, I realized I can set <skipTests> from a different property. So it will be back in a while.


-Jaikiran
> Ondra
>
>
>
>
>
> -------- Přeposlaná zpráva --------
>> *Od*: Darran Lofthouse (Created) (JIRA) <[hidden email] 
>> <[hidden email]%3e>>
>> *Komu* : [hidden email] <[hidden email]>
>> *Předmět* : [JBoss JIRA] (AS7-2796) -DskipTests argument no longer 
>> preventing the execution of tests on AS7 build
>> *Datum* : Fri, 25 Nov 2011 10:15:40 -0500 (EST)
>>
>> -DskipTests argument no longer preventing the execution of tests on AS7 build
>> -----------------------------------------------------------------------------
>>
>>                   Key: AS7-2796
>>                   URL:https://issues.jboss.org/browse/AS7-2796
>>               Project: Application Server 7
>>            Issue Type: Bug
>>            Components: Test Suite
>>              Reporter: Darran Lofthouse
>>              Assignee: Ondrej Zizka
>>               Fix For: 7.1.0.CR1
>>
>>
>> Since the testsuite changes have been committed the -DskipTests argument no longer skips the tests.
>>
>> --
>> This message is automatically generated by JIRA.
>> If you think it was sent incorrectly, please contact your JIRA administrators:https://issues.jboss.org/secure/ContactAdministrators!default.jspa  <https://issues.jboss.org/secure/ContactAdministrators%21default.jspa>
>> For more information on JIRA, see:http://www.atlassian.com/software/jira
>>
>>
>
>
> _______________________________________________
> jboss-as7-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev

_______________________________________________
jboss-as7-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/jboss-as7-dev

_______________________________________________
jboss-as7-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
Reply | Threaded
Open this post in threaded view
|

Re: (AS7-2796) -DskipTests argument no longer preventing the execution of tests on AS7 build

Jaikiran Pai
Comments inline.

On Monday 28 November 2011 02:52 PM, Ondřej Žižka wrote:
> Jaikiran Pai píše v Po 28. 11. 2011 v 13:48 +0530:
>> There's
>> -Dmaven.test.failure.ignore=true if you want to ignore those test
>> failures. I don't know how the AS7 project handles the
>> -Dmaven.test.failure.ignore property due to our testsuite
>> profiles/customizations. But we shouldn't be adding new meaning to
>> already well known flags.
> The problem with these is that they apply to whole build.
That's not a _problem_ since that's what it's expected to do.

>
>
>> >  How about having two params for that, with -DskipTests for unit tests,
>> >  and, say, -DskipTestsuite for skipping the testsuite module's tests?
>> >
>> IMO, we shouldn't just be adding all these new flags which make it
>> complicated to understand what set of flags to use and when.
> I have to disappoint you but you'll see more and more flags being
> added in the future.
> Please realize that there are other groups using the testsuite e.g.
> for EAP testing.
I understand that it's not just the developers who are using these
testsuite. My point is that most of these additional flags that we keep
adding are only relevant for the EAP testing/QA - which really means
that these shouldn't be added to the default build or in other words
should not show any impact on how the other group (i.e. the developers)
uses the current build. Since these flags apply to the EAP testing, why
not have a specific script which deals with all these flags and that
script can then be used for EAP testing? My point is not about adding
these new flags but is more about having to expect the non-QA developers
(for whom those are irrelevant) to understand and use these flags for
the builds.

> But I can keep you uninformed and keep the testsuite running without
> changes of "API" if that helps.
I guess, you meant informed. Yes, that would help - if adding the flags
helps QA testing but doesn't have any impact on the "API" and helps keep
the testsuite running, then I have no objection to those changes/flags.

>
>>   The actual
>> question for this specific issue is what's the real usecase of skipping
>> unit tests (which potentially can have failures) but still want to run
>> the integration tests?
> Often it happens that some unit tests fail and block one step AS build
> & testsuite execution.
So far I haven't seen any unit test fail in AS7 (and if some of them are
failing then those need to be fixed). Yes, there are integration test
failures which are intermittent. But those issues need to be fixed. Like
I said in my earlier reply, building the entire project by using
-DskipTests (to skip all tests) and then cd testsuite/integration; mvn
clean install (to run the integration tests) is what you are looking for
in such cases. Ofcourse it's a 2 step process but to me that's more
logical (and can even be added to a script) than adding and integrating
some new property.

> And let me ask the other way around - what is the use case of
> -DskipTests, skipping all tests?
Yes.
>
>> To be clear - IMO, we should have the -DskipTests back without adding a
>> new meaning to that param.
> Okay, I realized I can set <skipTests> from a different property. So
> it will be back in a while.
Thanks!

-Jaikiran
_______________________________________________
jboss-as7-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
Reply | Threaded
Open this post in threaded view
|

Re: (AS7-2796) -DskipTests argument no longer preventing the execution of tests on AS7 build

Dimitris Andreadis
Jaikiran, just have in mind that we want to keep the AS/EAP diff at the source level minimum.

On 28/11/2011 11:35, Jaikiran Pai wrote:

> Comments inline.
>
> On Monday 28 November 2011 02:52 PM, Ondřej Žižka wrote:
>> Jaikiran Pai píše v Po 28. 11. 2011 v 13:48 +0530:
>>> There's
>>> -Dmaven.test.failure.ignore=true if you want to ignore those test
>>> failures. I don't know how the AS7 project handles the
>>> -Dmaven.test.failure.ignore property due to our testsuite
>>> profiles/customizations. But we shouldn't be adding new meaning to
>>> already well known flags.
>> The problem with these is that they apply to whole build.
> That's not a _problem_ since that's what it's expected to do.
>>
>>
>>>>   How about having two params for that, with -DskipTests for unit tests,
>>>>   and, say, -DskipTestsuite for skipping the testsuite module's tests?
>>>>
>>> IMO, we shouldn't just be adding all these new flags which make it
>>> complicated to understand what set of flags to use and when.
>> I have to disappoint you but you'll see more and more flags being
>> added in the future.
>> Please realize that there are other groups using the testsuite e.g.
>> for EAP testing.
> I understand that it's not just the developers who are using these
> testsuite. My point is that most of these additional flags that we keep
> adding are only relevant for the EAP testing/QA - which really means
> that these shouldn't be added to the default build or in other words
> should not show any impact on how the other group (i.e. the developers)
> uses the current build. Since these flags apply to the EAP testing, why
> not have a specific script which deals with all these flags and that
> script can then be used for EAP testing? My point is not about adding
> these new flags but is more about having to expect the non-QA developers
> (for whom those are irrelevant) to understand and use these flags for
> the builds.
>
>> But I can keep you uninformed and keep the testsuite running without
>> changes of "API" if that helps.
> I guess, you meant informed. Yes, that would help - if adding the flags
> helps QA testing but doesn't have any impact on the "API" and helps keep
> the testsuite running, then I have no objection to those changes/flags.
>
>>
>>>    The actual
>>> question for this specific issue is what's the real usecase of skipping
>>> unit tests (which potentially can have failures) but still want to run
>>> the integration tests?
>> Often it happens that some unit tests fail and block one step AS build
>> &  testsuite execution.
> So far I haven't seen any unit test fail in AS7 (and if some of them are
> failing then those need to be fixed). Yes, there are integration test
> failures which are intermittent. But those issues need to be fixed. Like
> I said in my earlier reply, building the entire project by using
> -DskipTests (to skip all tests) and then cd testsuite/integration; mvn
> clean install (to run the integration tests) is what you are looking for
> in such cases. Ofcourse it's a 2 step process but to me that's more
> logical (and can even be added to a script) than adding and integrating
> some new property.
>
>> And let me ask the other way around - what is the use case of
>> -DskipTests, skipping all tests?
> Yes.
>>
>>> To be clear - IMO, we should have the -DskipTests back without adding a
>>> new meaning to that param.
>> Okay, I realized I can set<skipTests>  from a different property. So
>> it will be back in a while.
> Thanks!
>
> -Jaikiran
> _______________________________________________
> jboss-as7-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev

--
xxxxxxxxxxxxxxxxxxxxxxxxxxxx
Dimitris Andreadis
Software Engineering Manager
JBoss Application Server
by Red Hat
xxxxxxxxxxxxxxxxxxxxxxxxxxxx

http://dandreadis.blogspot.com/
_______________________________________________
jboss-as7-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
Reply | Threaded
Open this post in threaded view
|

Re: (AS7-2796) -DskipTests argument no longer preventing the execution of tests on AS7 build

Darran Lofthouse
In reply to this post by Jaikiran Pai
On 11/28/2011 08:18 AM, Jaikiran Pai wrote:

> On Monday 28 November 2011 01:35 PM, Ondřej Žižka wrote:
>> How about having two params for that, with -DskipTests for unit tests,
>> and, say, -DskipTestsuite for skipping the testsuite module's tests?
>>
> IMO, we shouldn't just be adding all these new flags which make it
> complicated to understand what set of flags to use and when. The actual
> question for this specific issue is what's the real usecase of skipping
> unit tests (which potentially can have failures) but still want to run
> the integration tests?
>
> To be clear - IMO, we should have the -DskipTests back without adding a
> new meaning to that param.

+1 - To me that parameter already had an existing meaning and that was
to skip ALL tests.  Developing AS I can't see where I would want to be
skipping unit tests to allow the rest of the testsuite to run.
_______________________________________________
jboss-as7-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
Reply | Threaded
Open this post in threaded view
|

Re: (AS7-2796) -DskipTests argument no longer preventing the execution of tests on AS7 build

Jaikiran Pai
In reply to this post by Dimitris Andreadis
Agreed, and I'm not opposed to that. It's just that some of these
changes break some well known conventions and do have an alternative
option of implementing the same. So just trying to make sure that we
understand the real usecase before doing the change.

-Jaikiran
On Monday 28 November 2011 03:12 PM, Dimitris Andreadis wrote:

> Jaikiran, just have in mind that we want to keep the AS/EAP diff at the source level minimum.
>
> On 28/11/2011 11:35, Jaikiran Pai wrote:
>> Comments inline.
>>
>> On Monday 28 November 2011 02:52 PM, Ondřej Žižka wrote:
>>> Jaikiran Pai píše v Po 28. 11. 2011 v 13:48 +0530:
>>>> There's
>>>> -Dmaven.test.failure.ignore=true if you want to ignore those test
>>>> failures. I don't know how the AS7 project handles the
>>>> -Dmaven.test.failure.ignore property due to our testsuite
>>>> profiles/customizations. But we shouldn't be adding new meaning to
>>>> already well known flags.
>>> The problem with these is that they apply to whole build.
>> That's not a _problem_ since that's what it's expected to do.
>>>
>>>
>>>>>    How about having two params for that, with -DskipTests for unit tests,
>>>>>    and, say, -DskipTestsuite for skipping the testsuite module's tests?
>>>>>
>>>> IMO, we shouldn't just be adding all these new flags which make it
>>>> complicated to understand what set of flags to use and when.
>>> I have to disappoint you but you'll see more and more flags being
>>> added in the future.
>>> Please realize that there are other groups using the testsuite e.g.
>>> for EAP testing.
>> I understand that it's not just the developers who are using these
>> testsuite. My point is that most of these additional flags that we keep
>> adding are only relevant for the EAP testing/QA - which really means
>> that these shouldn't be added to the default build or in other words
>> should not show any impact on how the other group (i.e. the developers)
>> uses the current build. Since these flags apply to the EAP testing, why
>> not have a specific script which deals with all these flags and that
>> script can then be used for EAP testing? My point is not about adding
>> these new flags but is more about having to expect the non-QA developers
>> (for whom those are irrelevant) to understand and use these flags for
>> the builds.
>>
>>> But I can keep you uninformed and keep the testsuite running without
>>> changes of "API" if that helps.
>> I guess, you meant informed. Yes, that would help - if adding the flags
>> helps QA testing but doesn't have any impact on the "API" and helps keep
>> the testsuite running, then I have no objection to those changes/flags.
>>
>>>
>>>>     The actual
>>>> question for this specific issue is what's the real usecase of skipping
>>>> unit tests (which potentially can have failures) but still want to run
>>>> the integration tests?
>>> Often it happens that some unit tests fail and block one step AS build
>>> &   testsuite execution.
>> So far I haven't seen any unit test fail in AS7 (and if some of them are
>> failing then those need to be fixed). Yes, there are integration test
>> failures which are intermittent. But those issues need to be fixed. Like
>> I said in my earlier reply, building the entire project by using
>> -DskipTests (to skip all tests) and then cd testsuite/integration; mvn
>> clean install (to run the integration tests) is what you are looking for
>> in such cases. Ofcourse it's a 2 step process but to me that's more
>> logical (and can even be added to a script) than adding and integrating
>> some new property.
>>
>>> And let me ask the other way around - what is the use case of
>>> -DskipTests, skipping all tests?
>> Yes.
>>>
>>>> To be clear - IMO, we should have the -DskipTests back without adding a
>>>> new meaning to that param.
>>> Okay, I realized I can set<skipTests>   from a different property. So
>>> it will be back in a while.
>> Thanks!
>>
>> -Jaikiran
>> _______________________________________________
>> jboss-as7-dev mailing list
>> [hidden email]
>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
>

_______________________________________________
jboss-as7-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
Reply | Threaded
Open this post in threaded view
|

Re: (AS7-2796) -DskipTests argument no longer preventing the execution of tests on AS7 build

Alexey Loubyansky
In reply to this post by Darran Lofthouse
Exactly.

Ondra, please, discuss these kind of changes before applying.

On 11/28/2011 10:46 AM, Darran Lofthouse wrote:

> On 11/28/2011 08:18 AM, Jaikiran Pai wrote:
>> On Monday 28 November 2011 01:35 PM, Ondřej Žižka wrote:
>>> How about having two params for that, with -DskipTests for unit tests,
>>> and, say, -DskipTestsuite for skipping the testsuite module's tests?
>>>
>> IMO, we shouldn't just be adding all these new flags which make it
>> complicated to understand what set of flags to use and when. The actual
>> question for this specific issue is what's the real usecase of skipping
>> unit tests (which potentially can have failures) but still want to run
>> the integration tests?
>>
>> To be clear - IMO, we should have the -DskipTests back without adding a
>> new meaning to that param.
>
> +1 - To me that parameter already had an existing meaning and that was
> to skip ALL tests.  Developing AS I can't see where I would want to be
> skipping unit tests to allow the rest of the testsuite to run.
> _______________________________________________
> jboss-as7-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
_______________________________________________
jboss-as7-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
Reply | Threaded
Open this post in threaded view
|

Keeping testsuite "API" consistent // -DskipTests argument no longer ...

Ondrej Zizka
In reply to this post by Jaikiran Pai
Jaikiran Pai píše v Po 28. 11. 2011 v 15:05 +0530:
Comments inline.
-"-


Since these flags apply to the EAP testing, why 
not have a specific script which deals with all these flags and that 
script can then be used for EAP testing? 
Well, since the testsuite is maven-based, the script can only use what's really in pom.xml's.
I could move some steps (e.g. server config) to the script.
But I think that having all in one place is beneficial: If a bug in EAP is found, then it's easy for the developer to reproduce - they'll simply use same set of maven properties.

Maybe my words "more and more" scared you a bit :)   It's not that bad, especially after splitting to modules, the pom.xml's are not that hard to read anymore.
And since we have gone the modularized way, I can keep the functionality outside of scope of the "normal" build, i.e. not include some modules at all - only for EAP builds / test runs.

What I mean is:
testsuite
  + integration
     + configure-database
     + configure-ipv6
     + group-smoke
     + group-basic
     + group-iiop

And activate the submodules on demand.


My point is not about adding 
these new flags but is more about having to expect the non-QA developers 
(for whom those are irrelevant) to understand and use these flags for 
the builds.
Actually, I try not to change the behavior unless someone asks for some change - that's the case of -DskipTests btw.
I didn't assume anyone relying on that.
And, this was not in testsuite requirements, which we discussed for a while on this list.
But as I wrote - it's coming back.


> But I can keep you uninformed and keep the testsuite running without 
> changes of "API" if that helps.
I guess, you meant informed. Yes, that would help - if adding the flags 
helps QA testing but doesn't have any impact on the "API" and helps keep 
the testsuite running, then I have no objection to those changes/flags.
Okay. How about others?


>>   The actual
>> question for this specific issue is what's the real usecase of skipping
>> unit tests (which potentially can have failures) but still want to run
>> the integration tests?
> Often it happens that some unit tests fail and block one step AS build 
> & testsuite execution.
So far I haven't seen any unit test fail in AS7 (and if some of them are 
failing then those need to be fixed). Yes, there are integration test 
failures which are intermittent. But those issues need to be fixed. Like 
I said in my earlier reply, building the entire project by using 
-DskipTests (to skip all tests) and then cd testsuite/integration; mvn 
clean install (to run the integration tests) is what you are looking for 
in such cases. Ofcourse it's a 2 step process but to me that's more 
logical (and can even be added to a script) than adding and integrating 
some new property.
Okay. Someone from QA wanted to be able to run in one step due to some Jenkins issue with Maven-based build, I'll discuss that.


> And let me ask the other way around - what is the use case of 
> -DskipTests, skipping all tests?
Yes.
That was like a part of the question - what is the use case of skipping _all_ tests?
* As you said, unit tests should be rock solid and you rarely see a failure.
* So you skip tests to make the build faster, not because of unit tests failures.
* So you'd probably like to skip testsuite module to make it even faster.
Then I don't understand why you dislike the idea of having a separated flag to skip the testsuite. ?


Ondra
_______________________________________________
jboss-as7-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
Reply | Threaded
Open this post in threaded view
|

Testsuite USE CASES & REQUIREMENTS to preserve

Ondrej Zizka
In reply to this post by Jaikiran Pai
Hi all,

let me remind:

Please add all your testsuite use cases which wou want never to change (if not discussed and decided otherwise) here:
http://community.jboss.org/wiki/ASTestsuiteRequirements

Ondra



Jaikiran Pai píše v Po 28. 11. 2011 v 15:25 +0530:
Agreed, and I'm not opposed to that. It's just that some of these 
changes break some well known conventions and do have an alternative 
option of implementing the same. So just trying to make sure that we 
understand the real usecase before doing the change.

-Jaikiran
On Monday 28 November 2011 03:12 PM, Dimitris Andreadis wrote:
> Jaikiran, just have in mind that we want to keep the AS/EAP diff at the source level minimum.
>


_______________________________________________
jboss-as7-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/jboss-as7-dev