First AS 7 testsuite coverage report available.

classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|

First AS 7 testsuite coverage report available.

Ondrej Zizka
Hi,


this is the first AS 7 coverage report: http://www.qa.jboss.com/~ozizka/coverage-report/html/index.html
Download at: http://www.qa.jboss.com/~ozizka/AS7-coverage-full-2012-01-24.zip
Jira: https://issues.jboss.org/browse/AS7-2022

It's made using the JaCoCo tool.
Whole testsuite was run, except for the integration/compat module.
Only tests launched through Arquillian are counted. Thus, for example, CLI tests are not - since they launch JVM on their own.
To  be addressed later.

Please look at your modules/components (both devs and QA) and take actions to increase the coverage.
If your tests are not being counted, please let me know, and we will figure out how to fix it.


Regards,
Ondra
_______________________________________________
jboss-as7-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
Reply | Threaded
Open this post in threaded view
|

Re: First AS 7 testsuite coverage report available.

kkhan
I think probably tests extending AbstractSubsystemTest should be included as well.
On 24 Jan 2012, at 21:09, Ondrej Zizka wrote:

> Hi,
>
>
> this is the first AS 7 coverage report: http://www.qa.jboss.com/~ozizka/coverage-report/html/index.html
> Download at: http://www.qa.jboss.com/~ozizka/AS7-coverage-full-2012-01-24.zip
> Jira: https://issues.jboss.org/browse/AS7-2022
>
> It's made using the JaCoCo tool.
> Whole testsuite was run, except for the integration/compat module.
> Only tests launched through Arquillian are counted. Thus, for example, CLI tests are not - since they launch JVM on their own.
> To  be addressed later.
>
> Please look at your modules/components (both devs and QA) and take actions to increase the coverage.
> If your tests are not being counted, please let me know, and we will figure out how to fix it.
>
>
> Regards,
> Ondra
> _______________________________________________
> jboss-as7-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev


_______________________________________________
jboss-as7-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
Reply | Threaded
Open this post in threaded view
|

Re: First AS 7 testsuite coverage report available.

Brian Stansberry
In reply to this post by Ondrej Zizka
On 1/24/12 3:09 PM, Ondrej Zizka wrote:
> Hi,
>
>
> this is the first AS 7 coverage report:
> http://www.qa.jboss.com/~ozizka/coverage-report/html/index.html
> Download at:
> http://www.qa.jboss.com/~ozizka/AS7-coverage-full-2012-01-24.zip
> Jira: https://issues.jboss.org/browse/AS7-2022
>

Thanks!

> It's made using the JaCoCo tool.
> Whole testsuite was run, except for the integration/compat module.
> Only tests launched through Arquillian are counted. Thus, for example,
> CLI tests are not - since they launch JVM on their own.
> To be addressed later.
>
> Please look at your modules/components (both devs and QA) and take
> actions to increase the coverage.
> If your tests are not being counted, please let me know, and we will
> figure out how to fix it.
>

Nothing in testsuite/domain is Arquillian based.

Are the unit tests outside the testsuite modules counted?


--
Brian Stansberry
Principal Software Engineer
JBoss by Red Hat
_______________________________________________
jboss-as7-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
Reply | Threaded
Open this post in threaded view
|

Re: First AS 7 testsuite coverage report available.

Carlo de Wolf
In reply to this post by Ondrej Zizka
The drill down into source is not available.
Can that be enabled?

Carlo

On 01/24/2012 10:09 PM, Ondrej Zizka wrote:
Hi,


this is the first AS 7 coverage report:
http://www.qa.jboss.com/~ozizka/coverage-report/html/index.html
Download at:
http://www.qa.jboss.com/~ozizka/AS7-coverage-full-2012-01-24.zip
Jira: https://issues.jboss.org/browse/AS7-2022

It's made using the JaCoCo tool.
Whole testsuite was run, except for the integration/compat module.
Only tests launched through Arquillian are counted. Thus, for example,
CLI tests are not - since they launch JVM on their own.
To  be addressed later.

Please look at your modules/components (both devs and QA) and take
actions to increase the coverage.
If your tests are not being counted, please let me know, and we will
figure out how to fix it.


Regards,
Ondra



_______________________________________________
jboss-as7-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/jboss-as7-dev


_______________________________________________
jboss-as7-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
Reply | Threaded
Open this post in threaded view
|

Re: First AS 7 testsuite coverage report available.

Ondrej Zizka
Source code dirs are provided to Jacoco but for some reason it doesn't pick it up. Will look into that later (when it gets into master).

Ondra



On Wed, 2012-01-25 at 19:07 +0100, Carlo de Wolf wrote:
The drill down into source is not available.
Can that be enabled?

Carlo

On 01/24/2012 10:09 PM, Ondrej Zizka wrote:
Hi,


this is the first AS 7 coverage report:
http://www.qa.jboss.com/~ozizka/coverage-report/html/index.html
Download at:
http://www.qa.jboss.com/~ozizka/AS7-coverage-full-2012-01-24.zip
Jira: https://issues.jboss.org/browse/AS7-2022

It's made using the JaCoCo tool.
Whole testsuite was run, except for the integration/compat module.
Only tests launched through Arquillian are counted. Thus, for example,
CLI tests are not - since they launch JVM on their own.
To  be addressed later.

Please look at your modules/components (both devs and QA) and take
actions to increase the coverage.
If your tests are not being counted, please let me know, and we will
figure out how to fix it.


Regards,
Ondra



_______________________________________________
jboss-as7-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/jboss-as7-dev



_______________________________________________
jboss-as7-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
Reply | Threaded
Open this post in threaded view
|

Re: First AS 7 testsuite coverage report available.

Ondrej Zizka
In reply to this post by kkhan
>From what I see, AbstractSubsystemTest is only used for unit tests.
Unit tests are usually covered by Cobertura or Emma.
This coverage report only covers the testsuite/ module.
We could extend Jacoco to cover modules' unit tests to, but that would need to change surefire's JVM args update in all modules.

Ondra




On Wed, 2012-01-25 at 17:03 +0000, Kabir Khan wrote:
I think probably tests extending AbstractSubsystemTest should be included as well. 
On 24 Jan 2012, at 21:09, Ondrej Zizka wrote:

> Hi,
> 
> 
> this is the first AS 7 coverage report: http://www.qa.jboss.com/~ozizka/coverage-report/html/index.html
> Download at: http://www.qa.jboss.com/~ozizka/AS7-coverage-full-2012-01-24.zip
> Jira: https://issues.jboss.org/browse/AS7-2022
> 
> It's made using the JaCoCo tool.
> Whole testsuite was run, except for the integration/compat module.
> Only tests launched through Arquillian are counted. Thus, for example, CLI tests are not - since they launch JVM on their own.
> To  be addressed later.
> 
> Please look at your modules/components (both devs and QA) and take actions to increase the coverage.
> If your tests are not being counted, please let me know, and we will figure out how to fix it.
> 
> 
> Regards,
> Ondra
> _______________________________________________
> jboss-as7-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev



_______________________________________________
jboss-as7-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
Reply | Threaded
Open this post in threaded view
|

Re: First AS 7 testsuite coverage report available.

jtgreene
Administrator
In reply to this post by Brian Stansberry
On 1/25/12 11:35 AM, Brian Stansberry wrote:
> Nothing in testsuite/domain is Arquillian based.
>
> Are the unit tests outside the testsuite modules counted?

Yeah, the big challenge is that to really find untested code you have to
count every testsuite run against the server. This includes:

1. All AS testsuites
2. All component testsuites
3. All TCK testsuites (standalone and full container)

--
Jason T. Greene
JBoss AS Lead / EAP Platform Architect
JBoss, a division of Red Hat
_______________________________________________
jboss-as7-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
Reply | Threaded
Open this post in threaded view
|

Re: First AS 7 testsuite coverage report available.

kkhan
In reply to this post by Ondrej Zizka

On 25 Jan 2012, at 18:56, Ondrej Zizka wrote:

> From what I see, AbstractSubsystemTest is only used for unit tests.
Yes, my reason for pointing that out is that that is where the subsystems should be put through their paces for lots of different working and broken configurations.

> Unit tests are usually covered by Cobertura or Emma.
> This coverage report only covers the testsuite/ module.
> We could extend Jacoco to cover modules' unit tests to, but that would need to change surefire's JVM args update in all modules.
>
> Ondra
>
>
>
>
> On Wed, 2012-01-25 at 17:03 +0000, Kabir Khan wrote:
>> I think probably tests extending AbstractSubsystemTest should be included as well.
>> On 24 Jan 2012, at 21:09, Ondrej Zizka wrote:
>>
>> > Hi,
>> >
>> >
>> > this is the first AS 7 coverage report:
>> http://www.qa.jboss.com/~ozizka/coverage-report/html/index.html
>>
>> > Download at:
>> http://www.qa.jboss.com/~ozizka/AS7-coverage-full-2012-01-24.zip
>>
>> > Jira:
>> https://issues.jboss.org/browse/AS7-2022
>>
>> >
>> > It's made using the JaCoCo tool.
>> > Whole testsuite was run, except for the integration/compat module.
>> > Only tests launched through Arquillian are counted. Thus, for example, CLI tests are not - since they launch JVM on their own.
>> > To  be addressed later.
>> >
>> > Please look at your modules/components (both devs and QA) and take actions to increase the coverage.
>> > If your tests are not being counted, please let me know, and we will figure out how to fix it.
>> >
>> >
>> > Regards,
>> > Ondra
>> > _______________________________________________
>> > jboss-as7-dev mailing list
>> >
>> [hidden email]
>>
>> >
>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
>>
>>
>>
>


_______________________________________________
jboss-as7-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
Reply | Threaded
Open this post in threaded view
|

Re: First AS 7 testsuite coverage report available.

Jaikiran Pai
In reply to this post by Ondrej Zizka
I'm not sure I understand the report. For example I was looking at the
org.jboss.as.ejb3 package
http://www.qa.jboss.com/~ozizka/coverage-report/html/org.jboss.as.ejb3/index.html#dn-a.
That looks very odd. It seems to be reporting just 6 (mostly of low
relevance) classes from that package. Is this some config issue with the
reporting? Certainly the testsuite would have covered a lot more classes
in the org.jboss.as.ejb3 package.

-Jaikiran
On Wednesday 25 January 2012 02:39 AM, Ondrej Zizka wrote:

> Hi,
>
>
> this is the first AS 7 coverage report:
> http://www.qa.jboss.com/~ozizka/coverage-report/html/index.html 
> <http://www.qa.jboss.com/%7Eozizka/coverage-report/html/index.html>
> Download at:
> http://www.qa.jboss.com/~ozizka/AS7-coverage-full-2012-01-24.zip 
> <http://www.qa.jboss.com/%7Eozizka/AS7-coverage-full-2012-01-24.zip>
> Jira: https://issues.jboss.org/browse/AS7-2022
>
> It's made using the JaCoCo tool.
> Whole testsuite was run, except for the integration/compat module.
> Only tests launched through Arquillian are counted. Thus, for example,
> CLI tests are not - since they launch JVM on their own.
> To  be addressed later.
>
> Please look at your modules/components (both devs and QA) and take
> actions to increase the coverage.
> If your tests are not being counted, please let me know, and we will
> figure out how to fix it.
>
>
> Regards,
> Ondra
>
>
> _______________________________________________
> jboss-as7-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev

_______________________________________________
jboss-as7-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
Reply | Threaded
Open this post in threaded view
|

Re: First AS 7 testsuite coverage report available.

Carlo de Wolf
Either I'm looking at the wrong stuff or there are no other classes in
that package.
https://github.com/jbossas/jboss-as/tree/master/ejb3/src/main/java/org/jboss/as/ejb3

Carlo

On 01/26/2012 08:30 AM, Jaikiran Pai wrote:

> I'm not sure I understand the report. For example I was looking at the
> org.jboss.as.ejb3 package
> http://www.qa.jboss.com/~ozizka/coverage-report/html/org.jboss.as.ejb3/index.html#dn-a.
> That looks very odd. It seems to be reporting just 6 (mostly of low
> relevance) classes from that package. Is this some config issue with the
> reporting? Certainly the testsuite would have covered a lot more classes
> in the org.jboss.as.ejb3 package.
>
> -Jaikiran
> On Wednesday 25 January 2012 02:39 AM, Ondrej Zizka wrote:
>> Hi,
>>
>>
>> this is the first AS 7 coverage report:
>> http://www.qa.jboss.com/~ozizka/coverage-report/html/index.html
>> <http://www.qa.jboss.com/%7Eozizka/coverage-report/html/index.html>
>> Download at:
>> http://www.qa.jboss.com/~ozizka/AS7-coverage-full-2012-01-24.zip
>> <http://www.qa.jboss.com/%7Eozizka/AS7-coverage-full-2012-01-24.zip>
>> Jira: https://issues.jboss.org/browse/AS7-2022
>>
>> It's made using the JaCoCo tool.
>> Whole testsuite was run, except for the integration/compat module.
>> Only tests launched through Arquillian are counted. Thus, for example,
>> CLI tests are not - since they launch JVM on their own.
>> To  be addressed later.
>>
>> Please look at your modules/components (both devs and QA) and take
>> actions to increase the coverage.
>> If your tests are not being counted, please let me know, and we will
>> figure out how to fix it.
>>
>>
>> Regards,
>> Ondra
>>
>>
>> _______________________________________________
>> jboss-as7-dev mailing list
>> [hidden email]
>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
> _______________________________________________
> jboss-as7-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev

_______________________________________________
jboss-as7-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
Reply | Threaded
Open this post in threaded view
|

Re: First AS 7 testsuite coverage report available.

Jaikiran Pai
Ah right! For some reason I kept assuming that org.jboss.as.ejb3 report
would even include the sub-package classes.

-Jaikiran
On Thursday 26 January 2012 02:09 PM, Carlo de Wolf wrote:

> Either I'm looking at the wrong stuff or there are no other classes in
> that package.
> https://github.com/jbossas/jboss-as/tree/master/ejb3/src/main/java/org/jboss/as/ejb3
>
> Carlo
>
> On 01/26/2012 08:30 AM, Jaikiran Pai wrote:
>> I'm not sure I understand the report. For example I was looking at the
>> org.jboss.as.ejb3 package
>> http://www.qa.jboss.com/~ozizka/coverage-report/html/org.jboss.as.ejb3/index.html#dn-a.
>> That looks very odd. It seems to be reporting just 6 (mostly of low
>> relevance) classes from that package. Is this some config issue with the
>> reporting? Certainly the testsuite would have covered a lot more classes
>> in the org.jboss.as.ejb3 package.
>>
>> -Jaikiran
>> On Wednesday 25 January 2012 02:39 AM, Ondrej Zizka wrote:
>>> Hi,
>>>
>>>
>>> this is the first AS 7 coverage report:
>>> http://www.qa.jboss.com/~ozizka/coverage-report/html/index.html
>>> <http://www.qa.jboss.com/%7Eozizka/coverage-report/html/index.html>
>>> Download at:
>>> http://www.qa.jboss.com/~ozizka/AS7-coverage-full-2012-01-24.zip
>>> <http://www.qa.jboss.com/%7Eozizka/AS7-coverage-full-2012-01-24.zip>
>>> Jira: https://issues.jboss.org/browse/AS7-2022
>>>
>>> It's made using the JaCoCo tool.
>>> Whole testsuite was run, except for the integration/compat module.
>>> Only tests launched through Arquillian are counted. Thus, for example,
>>> CLI tests are not - since they launch JVM on their own.
>>> To  be addressed later.
>>>
>>> Please look at your modules/components (both devs and QA) and take
>>> actions to increase the coverage.
>>> If your tests are not being counted, please let me know, and we will
>>> figure out how to fix it.
>>>
>>>
>>> Regards,
>>> Ondra
>>>
>>>
>>> _______________________________________________
>>> jboss-as7-dev mailing list
>>> [hidden email]
>>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
>> _______________________________________________
>> jboss-as7-dev mailing list
>> [hidden email]
>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
>
> _______________________________________________
> jboss-as7-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev

_______________________________________________
jboss-as7-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
Reply | Threaded
Open this post in threaded view
|

Re: First AS 7 testsuite coverage report available. Now with SOURCES!

Ondrej Zizka
In reply to this post by Ondrej Zizka
Hi all,

second AS 7 test suite coverage report is available

  Browse:           http://www.qa.jboss.com/~ozizka/AS7/coverage/2012-03-07/html/
  Download:     http://www.qa.jboss.com/~ozizka/AS7/coverage/2012-03-07/coverage-report.zip
  Jenkins job:   https://hudson.qa.jboss.com/hudson/job/AS7-TS-valid-jacoco/
  Pull request: https://github.com/jbossas/jboss-as/pull/1733

Notes:

1) Domain and compat are not yet included, I'm waiting for other good run.

2) Jenkins job is not stable enough, often it fails due to various bugs, e.g. AS7-3128 or NFS file locks etc.

3) Some jar files had to be excluded because of duplicated classes, see
      Until that is fixed (or JaCoCo supports that), those jars won't be included in JaCoCo report.

  4) You can sort by columns by clicking on their headers.

  5) Sources are included this time!  Only for AS code though.  Example:  http://www.qa.jboss.com/~ozizka/AS7/coverage/2012-03-07/html/org.jboss.as.webservices.deployers/SecurityActions.java.html#L46


Enjoy :)
Ondra




[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] JBoss Application Server Test Suite: Aggregator ... SUCCESS [17.604s]
[INFO] JBoss Application Server Test Suite: Integration .. SUCCESS [2.100s]
[INFO] JBoss Application Server Test Suite: Integration - Smoke  SUCCESS [2:51.394s]
[INFO] JBoss Application Server Test Suite: Integration - Basic  SUCCESS [19:09.726s]
[INFO] JBoss Application Server Test Suite: Integration - Clustering  SUCCESS [9:51.389s]
[INFO] JBoss Application Server Test Suite: Integration - IIOP  SUCCESS [28.699s]
[INFO] JBoss Application Server Test Suite: Integration - XTS  SUCCESS [29.635s]
[INFO] JBoss AS Test Suite: Integration - Multinode Tests  SUCCESS [25.313s]
[INFO] JBoss AS Test Suite: Integration - Manual Mode Tests  SUCCESS [1:07.744s]
[INFO] JBoss Application Server Test Suite: Compatibility Tests  SUCCESS [30.772s]
[INFO] JBoss Application Server Test Suite: Domain Mode Integration Tests  SUCCESS [2:49.866s]
[INFO] JBoss Application Server Test Suite: Benchmark Tests  SUCCESS [7.538s]
[INFO] JBoss Application Server Test Suite: Stress Tests . SUCCESS [0.950s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 38:14.536s
[INFO] Finished at: Wed Mar 07 07:33:41 CET 2012
[INFO] Final Memory: 157M/380M
[INFO] ------------------------------------------------------------------------
[hidden email]:~/work/AS7/ozizka-git$ find . -name *exec
./testsuite/integration/multinode/target/jacoco.exec
./testsuite/integration/manualmode/target/jacoco.exec
./testsuite/integration/iiop/target/jacoco.exec
./testsuite/integration/clust/target/jacoco.exec
./testsuite/integration/smoke/target/jacoco.exec
./testsuite/integration/xts/target/jacoco.exec
./testsuite/integration/basic/target/jacoco.exec







On Tue, 2012-01-24 at 16:09 -0500, Ondrej Zizka wrote:
Hi,


this is the first AS 7 coverage report: http://www.qa.jboss.com/~ozizka/coverage-report/html/index.html
Download at: http://www.qa.jboss.com/~ozizka/AS7-coverage-full-2012-01-24.zip
Jira: https://issues.jboss.org/browse/AS7-2022

It's made using the JaCoCo tool.
Whole testsuite was run, except for the integration/compat module.
Only tests launched through Arquillian are counted. Thus, for example, CLI tests are not - since they launch JVM on their own.
To  be addressed later.

Please look at your modules/components (both devs and QA) and take actions to increase the coverage.
If your tests are not being counted, please let me know, and we will figure out how to fix it.


Regards,
Ondra


_______________________________________________
jboss-as7-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/jboss-as7-dev