Quantcast

Elytron integration tests in WildFly testsuite

classic Classic list List threaded Threaded
24 messages Options
12
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Elytron integration tests in WildFly testsuite

Josef Cacek
Hi,

EAP QE and Elytron Dev teams would like to raise a discussion about how the Elytron integration tests will be added to the WildFly/EAP testsuite.
We would like to agree on a unified approach across the testsuite, so we don't end up in situation where each component comes with its own "best-fitting" solution.

We see following requirements:
1. new features which come with Elytron need to be covered by integration tests
2. existing scenarios have to stay working when Elytron is configured as the default security subsystem
3. legacy security subsystem must stay working in parallel

Our teams came to agreement that the safest way would be to copy whole existing testsuite modules into new ones. E.g.

cp -r testsuite/integration/basic testsuite/integration/basic-legacy-security
# + Reconfigure testsuite/integration/basic to use Elytron
# + Add @Ignore to all tests which fail with Elytron or use directly the old security configuration

The modules would just live side by side - basic would use Elytron configuration, basic-legacy-security would use configuration similar to (or same as) the current server configuration.

In the next step we will go through the Ignored tests (the TODO list) for Elytron and rewrite them or report issues if necessary.

By this way we would keep all the test coverage for legacy security. We would also have all existing scenarios covered (or at least present) for Elytron configuration.

Are there any objections to this suggestion?

Thanks,

-- Josef
_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Elytron integration tests in WildFly testsuite

Tomaž Cerar-2

On Fri, Dec 2, 2016 at 9:11 AM, Josef Cacek <[hidden email]> wrote:
The modules would just live side by side - basic would use Elytron configuration, basic-legacy-security would use configuration similar to (or same as) the current server configuration.


What would this actually mean?
we will have two copies basic tests suites one running with elytron another with legacy security subsystem?

Do I read that right? Please say I am not.


_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Elytron integration tests in WildFly testsuite

Darran Lofthouse
On 02/12/16 11:03, Tomaž Cerar wrote:

>
> On Fri, Dec 2, 2016 at 9:11 AM, Josef Cacek <[hidden email]
> <mailto:[hidden email]>> wrote:
>
>     The modules would just live side by side - basic would use Elytron
>     configuration, basic-legacy-security would use configuration similar
>     to (or same as) the current server configuration.
>
>
>
> What would this actually mean?
> we will have two copies basic tests suites one running with elytron
> another with legacy security subsystem?
>
> Do I read that right? Please say I am not.

That is correct - we have two security implementations they both need
testing.

One needs testing for backwards compatibility and regressions, the other
for equivalent behaviour and then new features and bugs.

Needing to test both was discussed previously so this is more about how
to separate both and also give the Elytron testing a good foundation to
start from.

>
>
> _______________________________________________
> wildfly-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/wildfly-dev
>

_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Elytron integration tests in WildFly testsuite

Darran Lofthouse
Probably should add - any duplication should only be for security tests
- not everything else in there!

On 02/12/16 11:08, Darran Lofthouse wrote:

> On 02/12/16 11:03, Tomaž Cerar wrote:
>>
>> On Fri, Dec 2, 2016 at 9:11 AM, Josef Cacek <[hidden email]
>> <mailto:[hidden email]>> wrote:
>>
>>     The modules would just live side by side - basic would use Elytron
>>     configuration, basic-legacy-security would use configuration similar
>>     to (or same as) the current server configuration.
>>
>>
>>
>> What would this actually mean?
>> we will have two copies basic tests suites one running with elytron
>> another with legacy security subsystem?
>>
>> Do I read that right? Please say I am not.
>
> That is correct - we have two security implementations they both need
> testing.
>
> One needs testing for backwards compatibility and regressions, the other
> for equivalent behaviour and then new features and bugs.
>
> Needing to test both was discussed previously so this is more about how
> to separate both and also give the Elytron testing a good foundation to
> start from.
>
>>
>>
>> _______________________________________________
>> wildfly-dev mailing list
>> [hidden email]
>> https://lists.jboss.org/mailman/listinfo/wildfly-dev
>>
>

_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Elytron integration tests in WildFly testsuite

Josef Cacek
In reply to this post by Tomaž Cerar-2
> On Fri, Dec 2, 2016 at 9:11 AM, Josef Cacek < [hidden email] > wrote:

> > The modules would just live side by side - basic would use Elytron
> > configuration, basic-legacy-security would use configuration similar to (or
> > same as) the current server configuration.
>
> What would this actually mean?
> we will have two copies basic tests suites one running with elytron another
> with legacy security subsystem?

> Do I read that right? Please say I am not.

Not only the "basic". All testsuite modules in ideal case.
_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Elytron integration tests in WildFly testsuite

Tomaž Cerar-2
In reply to this post by Darran Lofthouse
That is probably fine, but! it should be done differently.

instead of duplicating whole testsuite (and adding extra hour to execution and extra headaches with intermittent problems and duplication of maintenance)
I would suggest that all security related tests get extracted to new "security" testsuite module and than only that part is duplicated.

This way we will have all security related stuff in one place.



On Fri, Dec 2, 2016 at 12:09 PM, Darran Lofthouse <[hidden email]> wrote:
Probably should add - any duplication should only be for security tests
- not everything else in there!

On 02/12/16 11:08, Darran Lofthouse wrote:
> On 02/12/16 11:03, Tomaž Cerar wrote:
>>
>> On Fri, Dec 2, 2016 at 9:11 AM, Josef Cacek <[hidden email]
>> <mailto:[hidden email]>> wrote:
>>
>>     The modules would just live side by side - basic would use Elytron
>>     configuration, basic-legacy-security would use configuration similar
>>     to (or same as) the current server configuration.
>>
>>
>>
>> What would this actually mean?
>> we will have two copies basic tests suites one running with elytron
>> another with legacy security subsystem?
>>
>> Do I read that right? Please say I am not.
>
> That is correct - we have two security implementations they both need
> testing.
>
> One needs testing for backwards compatibility and regressions, the other
> for equivalent behaviour and then new features and bugs.
>
> Needing to test both was discussed previously so this is more about how
> to separate both and also give the Elytron testing a good foundation to
> start from.
>
>>
>>
>> _______________________________________________
>> wildfly-dev mailing list
>> [hidden email]
>> https://lists.jboss.org/mailman/listinfo/wildfly-dev
>>
>

_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev


_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Elytron integration tests in WildFly testsuite

Darran Lofthouse
Although after my e-mail I hear from Josef they do want to duplicate all ;-)

 From my perspective I think we want to ensure the security tests are
clearly split in some way so we can identify legacy tests and Elytron
tests independently.

When it comes to code review the legacy tests are much more about
backwards compatibility so any changes to those should be given greater
scrutiny.

Secondly we should hit a point where we can remove the old tests so
separation will make that easier.

A package split would be fine but maybe a project split would make it
easier to define which tests are run and when.

Instead maybe we could split it into three: -

basic, basic-legacy-security, basic-security

The two 'security' projects would be the different form of security
testing, maybe the 'basic' project could have an optional profile to
switch it to a 100% legacy config mode.


On 02/12/16 11:37, Tomaž Cerar wrote:

> That is probably fine, but! it should be done differently.
>
> instead of duplicating whole testsuite (and adding extra hour to
> execution and extra headaches with intermittent problems and duplication
> of maintenance)
> I would suggest that all security related tests get extracted to new
> "security" testsuite module and than only that part is duplicated.
>
> This way we will have all security related stuff in one place.
>
>
>
> On Fri, Dec 2, 2016 at 12:09 PM, Darran Lofthouse
> <[hidden email] <mailto:[hidden email]>> wrote:
>
>     Probably should add - any duplication should only be for security tests
>     - not everything else in there!
>
>     On 02/12/16 11:08, Darran Lofthouse wrote:
>     > On 02/12/16 11:03, Tomaž Cerar wrote:
>     >>
>     >> On Fri, Dec 2, 2016 at 9:11 AM, Josef Cacek <[hidden email]
>     <mailto:[hidden email]>
>     >> <mailto:[hidden email] <mailto:[hidden email]>>> wrote:
>     >>
>     >>     The modules would just live side by side - basic would use
>     Elytron
>     >>     configuration, basic-legacy-security would use configuration
>     similar
>     >>     to (or same as) the current server configuration.
>     >>
>     >>
>     >>
>     >> What would this actually mean?
>     >> we will have two copies basic tests suites one running with elytron
>     >> another with legacy security subsystem?
>     >>
>     >> Do I read that right? Please say I am not.
>     >
>     > That is correct - we have two security implementations they both need
>     > testing.
>     >
>     > One needs testing for backwards compatibility and regressions, the
>     other
>     > for equivalent behaviour and then new features and bugs.
>     >
>     > Needing to test both was discussed previously so this is more
>     about how
>     > to separate both and also give the Elytron testing a good
>     foundation to
>     > start from.
>     >
>     >>
>     >>
>     >> _______________________________________________
>     >> wildfly-dev mailing list
>     >> [hidden email] <mailto:[hidden email]>
>     >> https://lists.jboss.org/mailman/listinfo/wildfly-dev
>     <https://lists.jboss.org/mailman/listinfo/wildfly-dev>
>     >>
>     >
>
>     _______________________________________________
>     wildfly-dev mailing list
>     [hidden email] <mailto:[hidden email]>
>     https://lists.jboss.org/mailman/listinfo/wildfly-dev
>     <https://lists.jboss.org/mailman/listinfo/wildfly-dev>
>
>
_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Elytron integration tests in WildFly testsuite

Josef Cacek
In reply to this post by Tomaž Cerar-2
We prefer duplication of all tests in the testsuite to keep test coverage.

To be sure we don't have regressions with introducing Elytron and at the same time we are able to cover current features with Elytron, we must run all tests with both security subsystems.
Even if a test is not security related on the first glance, the tested component may use a security feature internally. By switching to the new security subsystem the feature may stop work.

Let's keep the current (legacy security based) tests alive until the legacy security subsystem is fully removed.
Then we'll simply remove the *-legacy-security testsuite modules.

-- Josef

----- Original Message -----

> From: "Tomaž Cerar" <[hidden email]>
> To: "Darran Lofthouse" <[hidden email]>
> Cc: [hidden email]
> Sent: Friday, December 2, 2016 12:37:22 PM
> Subject: Re: [wildfly-dev] Elytron integration tests in WildFly testsuite
>
> That is probably fine, but! it should be done differently.
>
> instead of duplicating whole testsuite (and adding extra hour to execution
> and extra headaches with intermittent problems and duplication of
> maintenance)
> I would suggest that all security related tests get extracted to new
> "security" testsuite module and than only that part is duplicated.
>
> This way we will have all security related stuff in one place.
>
>
>
> On Fri, Dec 2, 2016 at 12:09 PM, Darran Lofthouse <
> [hidden email] > wrote:
>
>
> Probably should add - any duplication should only be for security tests
> - not everything else in there!
>
> On 02/12/16 11:08, Darran Lofthouse wrote:
> > On 02/12/16 11:03, Tomaž Cerar wrote:
> >>
> >> On Fri, Dec 2, 2016 at 9:11 AM, Josef Cacek < [hidden email]
> >> <mailto: [hidden email] >> wrote:
> >>
> >> The modules would just live side by side - basic would use Elytron
> >> configuration, basic-legacy-security would use configuration similar
> >> to (or same as) the current server configuration.
> >>
> >>
> >>
> >> What would this actually mean?
> >> we will have two copies basic tests suites one running with elytron
> >> another with legacy security subsystem?
> >>
> >> Do I read that right? Please say I am not.
> >
> > That is correct - we have two security implementations they both need
> > testing.
> >
> > One needs testing for backwards compatibility and regressions, the other
> > for equivalent behaviour and then new features and bugs.
> >
> > Needing to test both was discussed previously so this is more about how
> > to separate both and also give the Elytron testing a good foundation to
> > start from.
> >
> >>
> >>
> >> _______________________________________________
> >> wildfly-dev mailing list
> >> [hidden email]
> >> https://lists.jboss.org/mailman/listinfo/wildfly-dev
> >>
> >
>
> _______________________________________________
> wildfly-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/wildfly-dev
>
>
> _______________________________________________
> wildfly-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/wildfly-dev

_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Elytron integration tests in WildFly testsuite

Jan Kalina
I agree with Josef - it looks like more or less most of integration
tests can be affected by security features.

On Fri, Dec 2, 2016 at 12:58 PM, Josef Cacek <[hidden email]> wrote:

> We prefer duplication of all tests in the testsuite to keep test coverage.
>
> To be sure we don't have regressions with introducing Elytron and at the same time we are able to cover current features with Elytron, we must run all tests with both security subsystems.
> Even if a test is not security related on the first glance, the tested component may use a security feature internally. By switching to the new security subsystem the feature may stop work.
>
> Let's keep the current (legacy security based) tests alive until the legacy security subsystem is fully removed.
> Then we'll simply remove the *-legacy-security testsuite modules.
>
> -- Josef
>
> ----- Original Message -----
>> From: "Tomaž Cerar" <[hidden email]>
>> To: "Darran Lofthouse" <[hidden email]>
>> Cc: [hidden email]
>> Sent: Friday, December 2, 2016 12:37:22 PM
>> Subject: Re: [wildfly-dev] Elytron integration tests in WildFly testsuite
>>
>> That is probably fine, but! it should be done differently.
>>
>> instead of duplicating whole testsuite (and adding extra hour to execution
>> and extra headaches with intermittent problems and duplication of
>> maintenance)
>> I would suggest that all security related tests get extracted to new
>> "security" testsuite module and than only that part is duplicated.
>>
>> This way we will have all security related stuff in one place.
>>
>>
>>
>> On Fri, Dec 2, 2016 at 12:09 PM, Darran Lofthouse <
>> [hidden email] > wrote:
>>
>>
>> Probably should add - any duplication should only be for security tests
>> - not everything else in there!
>>
>> On 02/12/16 11:08, Darran Lofthouse wrote:
>> > On 02/12/16 11:03, Tomaž Cerar wrote:
>> >>
>> >> On Fri, Dec 2, 2016 at 9:11 AM, Josef Cacek < [hidden email]
>> >> <mailto: [hidden email] >> wrote:
>> >>
>> >> The modules would just live side by side - basic would use Elytron
>> >> configuration, basic-legacy-security would use configuration similar
>> >> to (or same as) the current server configuration.
>> >>
>> >>
>> >>
>> >> What would this actually mean?
>> >> we will have two copies basic tests suites one running with elytron
>> >> another with legacy security subsystem?
>> >>
>> >> Do I read that right? Please say I am not.
>> >
>> > That is correct - we have two security implementations they both need
>> > testing.
>> >
>> > One needs testing for backwards compatibility and regressions, the other
>> > for equivalent behaviour and then new features and bugs.
>> >
>> > Needing to test both was discussed previously so this is more about how
>> > to separate both and also give the Elytron testing a good foundation to
>> > start from.
>> >
>> >>
>> >>
>> >> _______________________________________________
>> >> wildfly-dev mailing list
>> >> [hidden email]
>> >> https://lists.jboss.org/mailman/listinfo/wildfly-dev
>> >>
>> >
>>
>> _______________________________________________
>> wildfly-dev mailing list
>> [hidden email]
>> https://lists.jboss.org/mailman/listinfo/wildfly-dev
>>
>>
>> _______________________________________________
>> wildfly-dev mailing list
>> [hidden email]
>> https://lists.jboss.org/mailman/listinfo/wildfly-dev
>
> _______________________________________________
> wildfly-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/wildfly-dev

_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Elytron integration tests in WildFly testsuite

Tomaž Cerar-2
In reply to this post by Josef Cacek
So if I understood all this correctly,
this mail here was more of a heads up what is happening
than a discussion on how could or should be best done.

I just hope you (the team behind decisions) considered all scenarios how will that impact everyone involved.

--
tomaz

On Fri, Dec 2, 2016 at 12:58 PM, Josef Cacek <[hidden email]> wrote:
We prefer duplication of all tests in the testsuite to keep test coverage.

To be sure we don't have regressions with introducing Elytron and at the same time we are able to cover current features with Elytron, we must run all tests with both security subsystems.
Even if a test is not security related on the first glance, the tested component may use a security feature internally. By switching to the new security subsystem the feature may stop work.

Let's keep the current (legacy security based) tests alive until the legacy security subsystem is fully removed.
Then we'll simply remove the *-legacy-security testsuite modules.

-- Josef

----- Original Message -----
> From: "Tomaž Cerar" <[hidden email]>
> To: "Darran Lofthouse" <[hidden email]>
> Cc: [hidden email]
> Sent: Friday, December 2, 2016 12:37:22 PM
> Subject: Re: [wildfly-dev] Elytron integration tests in WildFly testsuite
>
> That is probably fine, but! it should be done differently.
>
> instead of duplicating whole testsuite (and adding extra hour to execution
> and extra headaches with intermittent problems and duplication of
> maintenance)
> I would suggest that all security related tests get extracted to new
> "security" testsuite module and than only that part is duplicated.
>
> This way we will have all security related stuff in one place.
>
>
>
> On Fri, Dec 2, 2016 at 12:09 PM, Darran Lofthouse <
> [hidden email] > wrote:
>
>
> Probably should add - any duplication should only be for security tests
> - not everything else in there!
>
> On 02/12/16 11:08, Darran Lofthouse wrote:
> > On 02/12/16 11:03, Tomaž Cerar wrote:
> >>
> >> On Fri, Dec 2, 2016 at 9:11 AM, Josef Cacek < [hidden email]
> >> <mailto: [hidden email] >> wrote:
> >>
> >> The modules would just live side by side - basic would use Elytron
> >> configuration, basic-legacy-security would use configuration similar
> >> to (or same as) the current server configuration.
> >>
> >>
> >>
> >> What would this actually mean?
> >> we will have two copies basic tests suites one running with elytron
> >> another with legacy security subsystem?
> >>
> >> Do I read that right? Please say I am not.
> >
> > That is correct - we have two security implementations they both need
> > testing.
> >
> > One needs testing for backwards compatibility and regressions, the other
> > for equivalent behaviour and then new features and bugs.
> >
> > Needing to test both was discussed previously so this is more about how
> > to separate both and also give the Elytron testing a good foundation to
> > start from.
> >
> >>
> >>
> >> _______________________________________________
> >> wildfly-dev mailing list
> >> [hidden email]
> >> https://lists.jboss.org/mailman/listinfo/wildfly-dev
> >>
> >
>
> _______________________________________________
> wildfly-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/wildfly-dev
>
>
> _______________________________________________
> wildfly-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/wildfly-dev


_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Elytron integration tests in WildFly testsuite

Darran Lofthouse
No the part we can't avoid is that we have two security implementations
in the server and we have to have tests for both.  The e-mail from Josef
is a suggestion as to how this will be achieved.

Other suggestions are welcome ;-)

But we have two security implementations to test.

One of these is deprecated and will hopefully be removed at some point
so ideally we have something that allows testing to legacy to be removed
without too much pain.

Writing sufficient test coverage is not a small task so anything that
gives us more than starting from scratch is a benefit.

But maybe it would be worth summarising these additional scenarios that
will be affected by changes.

Regards,
Darran Lofthouse.

On 02/12/16 14:47, Tomaž Cerar wrote:

> So if I understood all this correctly,
> this mail here was more of a heads up what is happening
> than a discussion on how could or should be best done.
>
> I just hope you (the team behind decisions) considered all scenarios how
> will that impact everyone involved.
>
> --
> tomaz
>
> On Fri, Dec 2, 2016 at 12:58 PM, Josef Cacek <[hidden email]
> <mailto:[hidden email]>> wrote:
>
>     We prefer duplication of all tests in the testsuite to keep test
>     coverage.
>
>     To be sure we don't have regressions with introducing Elytron and at
>     the same time we are able to cover current features with Elytron, we
>     must run all tests with both security subsystems.
>     Even if a test is not security related on the first glance, the
>     tested component may use a security feature internally. By switching
>     to the new security subsystem the feature may stop work.
>
>     Let's keep the current (legacy security based) tests alive until the
>     legacy security subsystem is fully removed.
>     Then we'll simply remove the *-legacy-security testsuite modules.
>
>     -- Josef
>
>     ----- Original Message -----
>     > From: "Tomaž Cerar" <[hidden email]
>     <mailto:[hidden email]>>
>     > To: "Darran Lofthouse" <[hidden email]
>     <mailto:[hidden email]>>
>     > Cc: [hidden email] <mailto:[hidden email]>
>     > Sent: Friday, December 2, 2016 12:37:22 PM
>     > Subject: Re: [wildfly-dev] Elytron integration tests in WildFly
>     testsuite
>     >
>     > That is probably fine, but! it should be done differently.
>     >
>     > instead of duplicating whole testsuite (and adding extra hour to
>     execution
>     > and extra headaches with intermittent problems and duplication of
>     > maintenance)
>     > I would suggest that all security related tests get extracted to new
>     > "security" testsuite module and than only that part is duplicated.
>     >
>     > This way we will have all security related stuff in one place.
>     >
>     >
>     >
>     > On Fri, Dec 2, 2016 at 12:09 PM, Darran Lofthouse <
>     > [hidden email] <mailto:[hidden email]> >
>     wrote:
>     >
>     >
>     > Probably should add - any duplication should only be for security
>     tests
>     > - not everything else in there!
>     >
>     > On 02/12/16 11:08, Darran Lofthouse wrote:
>     > > On 02/12/16 11:03, Tomaž Cerar wrote:
>     > >>
>     > >> On Fri, Dec 2, 2016 at 9:11 AM, Josef Cacek < [hidden email]
>     <mailto:[hidden email]>
>     > >> <mailto: [hidden email] <mailto:[hidden email]> >> wrote:
>     > >>
>     > >> The modules would just live side by side - basic would use Elytron
>     > >> configuration, basic-legacy-security would use configuration
>     similar
>     > >> to (or same as) the current server configuration.
>     > >>
>     > >>
>     > >>
>     > >> What would this actually mean?
>     > >> we will have two copies basic tests suites one running with elytron
>     > >> another with legacy security subsystem?
>     > >>
>     > >> Do I read that right? Please say I am not.
>     > >
>     > > That is correct - we have two security implementations they both
>     need
>     > > testing.
>     > >
>     > > One needs testing for backwards compatibility and regressions,
>     the other
>     > > for equivalent behaviour and then new features and bugs.
>     > >
>     > > Needing to test both was discussed previously so this is more
>     about how
>     > > to separate both and also give the Elytron testing a good
>     foundation to
>     > > start from.
>     > >
>     > >>
>     > >>
>     > >> _______________________________________________
>     > >> wildfly-dev mailing list
>     > >> [hidden email] <mailto:[hidden email]>
>     > >> https://lists.jboss.org/mailman/listinfo/wildfly-dev
>     <https://lists.jboss.org/mailman/listinfo/wildfly-dev>
>     > >>
>     > >
>     >
>     > _______________________________________________
>     > wildfly-dev mailing list
>     > [hidden email] <mailto:[hidden email]>
>     > https://lists.jboss.org/mailman/listinfo/wildfly-dev
>     <https://lists.jboss.org/mailman/listinfo/wildfly-dev>
>     >
>     >
>     > _______________________________________________
>     > wildfly-dev mailing list
>     > [hidden email] <mailto:[hidden email]>
>     > https://lists.jboss.org/mailman/listinfo/wildfly-dev
>     <https://lists.jboss.org/mailman/listinfo/wildfly-dev>
>
>

_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Elytron integration tests in WildFly testsuite

Brian Stansberry
Duplicating the whole testsuite is basically a minimal effort way of getting coverage by avoiding the task of test coverage analysis.

Only justification for that is expediency. That may be sufficient justification; I don’t know.

But assuming for the sake of argument it’s sufficient, doubling the amount of time needed to run the testsuite is a massive productivity loss and that means cutting that time down to the necessary minimum and nothing more would need to be a critical extreme high priority task.

So, with this proposal, who would be the people assigned to this critical extreme high priority task? I’d like to see names, lots and lots of names. Darran Lofthouse not among them. ;)


> On Dec 2, 2016, at 8:59 AM, Darran Lofthouse <[hidden email]> wrote:
>
> No the part we can't avoid is that we have two security implementations
> in the server and we have to have tests for both.  The e-mail from Josef
> is a suggestion as to how this will be achieved.
>
> Other suggestions are welcome ;-)
>
> But we have two security implementations to test.
>
> One of these is deprecated and will hopefully be removed at some point
> so ideally we have something that allows testing to legacy to be removed
> without too much pain.
>
> Writing sufficient test coverage is not a small task so anything that
> gives us more than starting from scratch is a benefit.
>
> But maybe it would be worth summarising these additional scenarios that
> will be affected by changes.
>
> Regards,
> Darran Lofthouse.
>
> On 02/12/16 14:47, Tomaž Cerar wrote:
>> So if I understood all this correctly,
>> this mail here was more of a heads up what is happening
>> than a discussion on how could or should be best done.
>>
>> I just hope you (the team behind decisions) considered all scenarios how
>> will that impact everyone involved.
>>
>> --
>> tomaz
>>
>> On Fri, Dec 2, 2016 at 12:58 PM, Josef Cacek <[hidden email]
>> <mailto:[hidden email]>> wrote:
>>
>>    We prefer duplication of all tests in the testsuite to keep test
>>    coverage.
>>
>>    To be sure we don't have regressions with introducing Elytron and at
>>    the same time we are able to cover current features with Elytron, we
>>    must run all tests with both security subsystems.
>>    Even if a test is not security related on the first glance, the
>>    tested component may use a security feature internally. By switching
>>    to the new security subsystem the feature may stop work.
>>
>>    Let's keep the current (legacy security based) tests alive until the
>>    legacy security subsystem is fully removed.
>>    Then we'll simply remove the *-legacy-security testsuite modules.
>>
>>    -- Josef
>>
>>    ----- Original Message -----
>>> From: "Tomaž Cerar" <[hidden email]
>>    <mailto:[hidden email]>>
>>> To: "Darran Lofthouse" <[hidden email]
>>    <mailto:[hidden email]>>
>>> Cc: [hidden email] <mailto:[hidden email]>
>>> Sent: Friday, December 2, 2016 12:37:22 PM
>>> Subject: Re: [wildfly-dev] Elytron integration tests in WildFly
>>    testsuite
>>>
>>> That is probably fine, but! it should be done differently.
>>>
>>> instead of duplicating whole testsuite (and adding extra hour to
>>    execution
>>> and extra headaches with intermittent problems and duplication of
>>> maintenance)
>>> I would suggest that all security related tests get extracted to new
>>> "security" testsuite module and than only that part is duplicated.
>>>
>>> This way we will have all security related stuff in one place.
>>>
>>>
>>>
>>> On Fri, Dec 2, 2016 at 12:09 PM, Darran Lofthouse <
>>> [hidden email] <mailto:[hidden email]> >
>>    wrote:
>>>
>>>
>>> Probably should add - any duplication should only be for security
>>    tests
>>> - not everything else in there!
>>>
>>> On 02/12/16 11:08, Darran Lofthouse wrote:
>>>> On 02/12/16 11:03, Tomaž Cerar wrote:
>>>>>
>>>>> On Fri, Dec 2, 2016 at 9:11 AM, Josef Cacek < [hidden email]
>>    <mailto:[hidden email]>
>>>>> <mailto: [hidden email] <mailto:[hidden email]> >> wrote:
>>>>>
>>>>> The modules would just live side by side - basic would use Elytron
>>>>> configuration, basic-legacy-security would use configuration
>>    similar
>>>>> to (or same as) the current server configuration.
>>>>>
>>>>>
>>>>>
>>>>> What would this actually mean?
>>>>> we will have two copies basic tests suites one running with elytron
>>>>> another with legacy security subsystem?
>>>>>
>>>>> Do I read that right? Please say I am not.
>>>>
>>>> That is correct - we have two security implementations they both
>>    need
>>>> testing.
>>>>
>>>> One needs testing for backwards compatibility and regressions,
>>    the other
>>>> for equivalent behaviour and then new features and bugs.
>>>>
>>>> Needing to test both was discussed previously so this is more
>>    about how
>>>> to separate both and also give the Elytron testing a good
>>    foundation to
>>>> start from.
>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> wildfly-dev mailing list
>>>>> [hidden email] <mailto:[hidden email]>
>>>>> https://lists.jboss.org/mailman/listinfo/wildfly-dev
>>    <https://lists.jboss.org/mailman/listinfo/wildfly-dev>
>>>>>
>>>>
>>>
>>> _______________________________________________
>>> wildfly-dev mailing list
>>> [hidden email] <mailto:[hidden email]>
>>> https://lists.jboss.org/mailman/listinfo/wildfly-dev
>>    <https://lists.jboss.org/mailman/listinfo/wildfly-dev>
>>>
>>>
>>> _______________________________________________
>>> wildfly-dev mailing list
>>> [hidden email] <mailto:[hidden email]>
>>> https://lists.jboss.org/mailman/listinfo/wildfly-dev
>>    <https://lists.jboss.org/mailman/listinfo/wildfly-dev>
>>
>>
>
> _______________________________________________
> wildfly-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/wildfly-dev

--
Brian Stansberry
Manager, Senior Principal Software Engineer
JBoss by Red Hat




_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Elytron integration tests in WildFly testsuite

kkhan
+1 doubling the testsuite size/time is not practical. Also, there is the issue that if all tests are duplicated, when people come to enhance an existing test they will need to do so in two places.

Could we not do what was suggested earlier and pull out tests which obviously target security into two legacy/elytron modules, and leave the rest where they are? Then if something later turns out to have 'hidden' implications, that test gets pulled out into the two modules.

Regarding the deprecated legacy stuff, I think we will have to keep that around for many years still due to it being used by the product (and in any case in that branch, although removing it from WF might be possible sooner).


> On 2 Dec 2016, at 15:20, Brian Stansberry <[hidden email]> wrote:
>
> Duplicating the whole testsuite is basically a minimal effort way of getting coverage by avoiding the task of test coverage analysis.
>
> Only justification for that is expediency. That may be sufficient justification; I don’t know.
>
> But assuming for the sake of argument it’s sufficient, doubling the amount of time needed to run the testsuite is a massive productivity loss and that means cutting that time down to the necessary minimum and nothing more would need to be a critical extreme high priority task.
>
> So, with this proposal, who would be the people assigned to this critical extreme high priority task? I’d like to see names, lots and lots of names. Darran Lofthouse not among them. ;)
>
>
>> On Dec 2, 2016, at 8:59 AM, Darran Lofthouse <[hidden email]> wrote:
>>
>> No the part we can't avoid is that we have two security implementations
>> in the server and we have to have tests for both.  The e-mail from Josef
>> is a suggestion as to how this will be achieved.
>>
>> Other suggestions are welcome ;-)
>>
>> But we have two security implementations to test.
>>
>> One of these is deprecated and will hopefully be removed at some point
>> so ideally we have something that allows testing to legacy to be removed
>> without too much pain.
>>
>> Writing sufficient test coverage is not a small task so anything that
>> gives us more than starting from scratch is a benefit.
>>
>> But maybe it would be worth summarising these additional scenarios that
>> will be affected by changes.
>>
>> Regards,
>> Darran Lofthouse.
>>
>> On 02/12/16 14:47, Tomaž Cerar wrote:
>>> So if I understood all this correctly,
>>> this mail here was more of a heads up what is happening
>>> than a discussion on how could or should be best done.
>>>
>>> I just hope you (the team behind decisions) considered all scenarios how
>>> will that impact everyone involved.
>>>
>>> --
>>> tomaz
>>>
>>> On Fri, Dec 2, 2016 at 12:58 PM, Josef Cacek <[hidden email]
>>> <mailto:[hidden email]>> wrote:
>>>
>>>   We prefer duplication of all tests in the testsuite to keep test
>>>   coverage.
>>>
>>>   To be sure we don't have regressions with introducing Elytron and at
>>>   the same time we are able to cover current features with Elytron, we
>>>   must run all tests with both security subsystems.
>>>   Even if a test is not security related on the first glance, the
>>>   tested component may use a security feature internally. By switching
>>>   to the new security subsystem the feature may stop work.
>>>
>>>   Let's keep the current (legacy security based) tests alive until the
>>>   legacy security subsystem is fully removed.
>>>   Then we'll simply remove the *-legacy-security testsuite modules.
>>>
>>>   -- Josef
>>>
>>>   ----- Original Message -----
>>>> From: "Tomaž Cerar" <[hidden email]
>>>   <mailto:[hidden email]>>
>>>> To: "Darran Lofthouse" <[hidden email]
>>>   <mailto:[hidden email]>>
>>>> Cc: [hidden email] <mailto:[hidden email]>
>>>> Sent: Friday, December 2, 2016 12:37:22 PM
>>>> Subject: Re: [wildfly-dev] Elytron integration tests in WildFly
>>>   testsuite
>>>>
>>>> That is probably fine, but! it should be done differently.
>>>>
>>>> instead of duplicating whole testsuite (and adding extra hour to
>>>   execution
>>>> and extra headaches with intermittent problems and duplication of
>>>> maintenance)
>>>> I would suggest that all security related tests get extracted to new
>>>> "security" testsuite module and than only that part is duplicated.
>>>>
>>>> This way we will have all security related stuff in one place.
>>>>
>>>>
>>>>
>>>> On Fri, Dec 2, 2016 at 12:09 PM, Darran Lofthouse <
>>>> [hidden email] <mailto:[hidden email]> >
>>>   wrote:
>>>>
>>>>
>>>> Probably should add - any duplication should only be for security
>>>   tests
>>>> - not everything else in there!
>>>>
>>>> On 02/12/16 11:08, Darran Lofthouse wrote:
>>>>> On 02/12/16 11:03, Tomaž Cerar wrote:
>>>>>>
>>>>>> On Fri, Dec 2, 2016 at 9:11 AM, Josef Cacek < [hidden email]
>>>   <mailto:[hidden email]>
>>>>>> <mailto: [hidden email] <mailto:[hidden email]> >> wrote:
>>>>>>
>>>>>> The modules would just live side by side - basic would use Elytron
>>>>>> configuration, basic-legacy-security would use configuration
>>>   similar
>>>>>> to (or same as) the current server configuration.
>>>>>>
>>>>>>
>>>>>>
>>>>>> What would this actually mean?
>>>>>> we will have two copies basic tests suites one running with elytron
>>>>>> another with legacy security subsystem?
>>>>>>
>>>>>> Do I read that right? Please say I am not.
>>>>>
>>>>> That is correct - we have two security implementations they both
>>>   need
>>>>> testing.
>>>>>
>>>>> One needs testing for backwards compatibility and regressions,
>>>   the other
>>>>> for equivalent behaviour and then new features and bugs.
>>>>>
>>>>> Needing to test both was discussed previously so this is more
>>>   about how
>>>>> to separate both and also give the Elytron testing a good
>>>   foundation to
>>>>> start from.
>>>>>
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> wildfly-dev mailing list
>>>>>> [hidden email] <mailto:[hidden email]>
>>>>>> https://lists.jboss.org/mailman/listinfo/wildfly-dev
>>>   <https://lists.jboss.org/mailman/listinfo/wildfly-dev>
>>>>>>
>>>>>
>>>>
>>>> _______________________________________________
>>>> wildfly-dev mailing list
>>>> [hidden email] <mailto:[hidden email]>
>>>> https://lists.jboss.org/mailman/listinfo/wildfly-dev
>>>   <https://lists.jboss.org/mailman/listinfo/wildfly-dev>
>>>>
>>>>
>>>> _______________________________________________
>>>> wildfly-dev mailing list
>>>> [hidden email] <mailto:[hidden email]>
>>>> https://lists.jboss.org/mailman/listinfo/wildfly-dev
>>>   <https://lists.jboss.org/mailman/listinfo/wildfly-dev>
>>>
>>>
>>
>> _______________________________________________
>> wildfly-dev mailing list
>> [hidden email]
>> https://lists.jboss.org/mailman/listinfo/wildfly-dev
>
> --
> Brian Stansberry
> Manager, Senior Principal Software Engineer
> JBoss by Red Hat
>
>
>
>
> _______________________________________________
> wildfly-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/wildfly-dev


_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Elytron integration tests in WildFly testsuite

James Perkins
In reply to this post by Josef Cacek
Instead of duplicating the entire test suite can we just add a new CI job that runs the tests with the *-elytron.xml configuration file? Duplicating the entire testsuite will end in nothing but headaches IMO. Like Kabir mentioned any bug in a test will need to be fixed in two locations.

On Fri, Dec 2, 2016 at 12:11 AM, Josef Cacek <[hidden email]> wrote:
Hi,

EAP QE and Elytron Dev teams would like to raise a discussion about how the Elytron integration tests will be added to the WildFly/EAP testsuite.
We would like to agree on a unified approach across the testsuite, so we don't end up in situation where each component comes with its own "best-fitting" solution.

We see following requirements:
1. new features which come with Elytron need to be covered by integration tests
2. existing scenarios have to stay working when Elytron is configured as the default security subsystem
3. legacy security subsystem must stay working in parallel

Our teams came to agreement that the safest way would be to copy whole existing testsuite modules into new ones. E.g.

cp -r testsuite/integration/basic testsuite/integration/basic-legacy-security
# + Reconfigure testsuite/integration/basic to use Elytron
# + Add @Ignore to all tests which fail with Elytron or use directly the old security configuration

The modules would just live side by side - basic would use Elytron configuration, basic-legacy-security would use configuration similar to (or same as) the current server configuration.

In the next step we will go through the Ignored tests (the TODO list) for Elytron and rewrite them or report issues if necessary.

By this way we would keep all the test coverage for legacy security. We would also have all existing scenarios covered (or at least present) for Elytron configuration.

Are there any objections to this suggestion?

Thanks,

-- Josef
_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev



--
James R. Perkins
JBoss by Red Hat

_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Elytron integration tests in WildFly testsuite

jtgreene
Administrator
While it’s important that we maintain some legacy coverage, I think its important that our testsuite prioritize coverage of the new vs the legacy, and this can be done organically. New tests should use elytron, and old tests that are updated, which aren’t specifically testing legacy compatibility should be converted over gradually, when convenient. 

My main rationale hare is that code which is continually evolved is more likely to break than that which is largely frozen. It’s really the interop pieces that interact with legacy areas that are of most in need of continued coverage.

On Dec 2, 2016, at 11:22 AM, James Perkins <[hidden email]> wrote:

Instead of duplicating the entire test suite can we just add a new CI job that runs the tests with the *-elytron.xml configuration file? Duplicating the entire testsuite will end in nothing but headaches IMO. Like Kabir mentioned any bug in a test will need to be fixed in two locations.

On Fri, Dec 2, 2016 at 12:11 AM, Josef Cacek <[hidden email]> wrote:
Hi,

EAP QE and Elytron Dev teams would like to raise a discussion about how the Elytron integration tests will be added to the WildFly/EAP testsuite.
We would like to agree on a unified approach across the testsuite, so we don't end up in situation where each component comes with its own "best-fitting" solution.

We see following requirements:
1. new features which come with Elytron need to be covered by integration tests
2. existing scenarios have to stay working when Elytron is configured as the default security subsystem
3. legacy security subsystem must stay working in parallel

Our teams came to agreement that the safest way would be to copy whole existing testsuite modules into new ones. E.g.

cp -r testsuite/integration/basic testsuite/integration/basic-legacy-security
# + Reconfigure testsuite/integration/basic to use Elytron
# + Add @Ignore to all tests which fail with Elytron or use directly the old security configuration

The modules would just live side by side - basic would use Elytron configuration, basic-legacy-security would use configuration similar to (or same as) the current server configuration.

In the next step we will go through the Ignored tests (the TODO list) for Elytron and rewrite them or report issues if necessary.

By this way we would keep all the test coverage for legacy security. We would also have all existing scenarios covered (or at least present) for Elytron configuration.

Are there any objections to this suggestion?

Thanks,

-- Josef
_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev



--
James R. Perkins
JBoss by Red Hat
_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev

--
Jason T. Greene
WildFly Lead / JBoss EAP Platform Architect
JBoss, a division of Red Hat


_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Elytron integration tests in WildFly testsuite

jtgreene
Administrator
In reply to this post by Brian Stansberry
Right we can’t just double the testsuite. Although I could understand someone doing this as a temporary exercise. Not only is this bad from an overhead standpoint, but its also not guaranteed to be effective, since a large portion of our tests either don’t use security, or only the most basic functions. I think we need targeted testing for the new functionality regardless.

> On Dec 2, 2016, at 9:20 AM, Brian Stansberry <[hidden email]> wrote:
>
> Duplicating the whole testsuite is basically a minimal effort way of getting coverage by avoiding the task of test coverage analysis.
>
> Only justification for that is expediency. That may be sufficient justification; I don’t know.
>
> But assuming for the sake of argument it’s sufficient, doubling the amount of time needed to run the testsuite is a massive productivity loss and that means cutting that time down to the necessary minimum and nothing more would need to be a critical extreme high priority task.
>
> So, with this proposal, who would be the people assigned to this critical extreme high priority task? I’d like to see names, lots and lots of names. Darran Lofthouse not among them. ;)
>
>
>> On Dec 2, 2016, at 8:59 AM, Darran Lofthouse <[hidden email]> wrote:
>>
>> No the part we can't avoid is that we have two security implementations
>> in the server and we have to have tests for both.  The e-mail from Josef
>> is a suggestion as to how this will be achieved.
>>
>> Other suggestions are welcome ;-)
>>
>> But we have two security implementations to test.
>>
>> One of these is deprecated and will hopefully be removed at some point
>> so ideally we have something that allows testing to legacy to be removed
>> without too much pain.
>>
>> Writing sufficient test coverage is not a small task so anything that
>> gives us more than starting from scratch is a benefit.
>>
>> But maybe it would be worth summarising these additional scenarios that
>> will be affected by changes.
>>
>> Regards,
>> Darran Lofthouse.
>>
>> On 02/12/16 14:47, Tomaž Cerar wrote:
>>> So if I understood all this correctly,
>>> this mail here was more of a heads up what is happening
>>> than a discussion on how could or should be best done.
>>>
>>> I just hope you (the team behind decisions) considered all scenarios how
>>> will that impact everyone involved.
>>>
>>> --
>>> tomaz
>>>
>>> On Fri, Dec 2, 2016 at 12:58 PM, Josef Cacek <[hidden email]
>>> <mailto:[hidden email]>> wrote:
>>>
>>>   We prefer duplication of all tests in the testsuite to keep test
>>>   coverage.
>>>
>>>   To be sure we don't have regressions with introducing Elytron and at
>>>   the same time we are able to cover current features with Elytron, we
>>>   must run all tests with both security subsystems.
>>>   Even if a test is not security related on the first glance, the
>>>   tested component may use a security feature internally. By switching
>>>   to the new security subsystem the feature may stop work.
>>>
>>>   Let's keep the current (legacy security based) tests alive until the
>>>   legacy security subsystem is fully removed.
>>>   Then we'll simply remove the *-legacy-security testsuite modules.
>>>
>>>   -- Josef
>>>
>>>   ----- Original Message -----
>>>> From: "Tomaž Cerar" <[hidden email]
>>>   <mailto:[hidden email]>>
>>>> To: "Darran Lofthouse" <[hidden email]
>>>   <mailto:[hidden email]>>
>>>> Cc: [hidden email] <mailto:[hidden email]>
>>>> Sent: Friday, December 2, 2016 12:37:22 PM
>>>> Subject: Re: [wildfly-dev] Elytron integration tests in WildFly
>>>   testsuite
>>>>
>>>> That is probably fine, but! it should be done differently.
>>>>
>>>> instead of duplicating whole testsuite (and adding extra hour to
>>>   execution
>>>> and extra headaches with intermittent problems and duplication of
>>>> maintenance)
>>>> I would suggest that all security related tests get extracted to new
>>>> "security" testsuite module and than only that part is duplicated.
>>>>
>>>> This way we will have all security related stuff in one place.
>>>>
>>>>
>>>>
>>>> On Fri, Dec 2, 2016 at 12:09 PM, Darran Lofthouse <
>>>> [hidden email] <mailto:[hidden email]> >
>>>   wrote:
>>>>
>>>>
>>>> Probably should add - any duplication should only be for security
>>>   tests
>>>> - not everything else in there!
>>>>
>>>> On 02/12/16 11:08, Darran Lofthouse wrote:
>>>>> On 02/12/16 11:03, Tomaž Cerar wrote:
>>>>>>
>>>>>> On Fri, Dec 2, 2016 at 9:11 AM, Josef Cacek < [hidden email]
>>>   <mailto:[hidden email]>
>>>>>> <mailto: [hidden email] <mailto:[hidden email]> >> wrote:
>>>>>>
>>>>>> The modules would just live side by side - basic would use Elytron
>>>>>> configuration, basic-legacy-security would use configuration
>>>   similar
>>>>>> to (or same as) the current server configuration.
>>>>>>
>>>>>>
>>>>>>
>>>>>> What would this actually mean?
>>>>>> we will have two copies basic tests suites one running with elytron
>>>>>> another with legacy security subsystem?
>>>>>>
>>>>>> Do I read that right? Please say I am not.
>>>>>
>>>>> That is correct - we have two security implementations they both
>>>   need
>>>>> testing.
>>>>>
>>>>> One needs testing for backwards compatibility and regressions,
>>>   the other
>>>>> for equivalent behaviour and then new features and bugs.
>>>>>
>>>>> Needing to test both was discussed previously so this is more
>>>   about how
>>>>> to separate both and also give the Elytron testing a good
>>>   foundation to
>>>>> start from.
>>>>>
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> wildfly-dev mailing list
>>>>>> [hidden email] <mailto:[hidden email]>
>>>>>> https://lists.jboss.org/mailman/listinfo/wildfly-dev
>>>   <https://lists.jboss.org/mailman/listinfo/wildfly-dev>
>>>>>>
>>>>>
>>>>
>>>> _______________________________________________
>>>> wildfly-dev mailing list
>>>> [hidden email] <mailto:[hidden email]>
>>>> https://lists.jboss.org/mailman/listinfo/wildfly-dev
>>>   <https://lists.jboss.org/mailman/listinfo/wildfly-dev>
>>>>
>>>>
>>>> _______________________________________________
>>>> wildfly-dev mailing list
>>>> [hidden email] <mailto:[hidden email]>
>>>> https://lists.jboss.org/mailman/listinfo/wildfly-dev
>>>   <https://lists.jboss.org/mailman/listinfo/wildfly-dev>
>>>
>>>
>>
>> _______________________________________________
>> wildfly-dev mailing list
>> [hidden email]
>> https://lists.jboss.org/mailman/listinfo/wildfly-dev
>
> --
> Brian Stansberry
> Manager, Senior Principal Software Engineer
> JBoss by Red Hat
>
>
>
>
> _______________________________________________
> wildfly-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/wildfly-dev

--
Jason T. Greene
WildFly Lead / JBoss EAP Platform Architect
JBoss, a division of Red Hat


_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Elytron integration tests in WildFly testsuite

Stuart Douglas
In reply to this post by Josef Cacek

On 2 Dec 2016 11:00 PM, "Josef Cacek" <[hidden email]> wrote:
>
> We prefer duplication of all tests in the testsuite to keep test coverage.
>
> To be sure we don't have regressions with introducing Elytron and at the same time we are able to cover current features with Elytron, we must run all tests with both security subsystems.
> Even if a test is not security related on the first glance, the tested component may use a security feature internally. By switching to the new security subsystem the feature may stop work.
>

I think this is massive overkill, and would severely reduce the quality and usability of the testsuite (more tests != a better test suite, they need to be useful test otherwise they just add technical debt for no gain).

I just did a quick search through the code base, and as far as I can tell there is maybe ~50 test cases that test security related functionality (i.e. use the SecurityDomainSetup functionality), out of ~950 test cases, so this would amount to adding around 900 useless test cases.

The argument that Elytron may cause regressions in these cases does really work either, because these tests are not linked to a security domain Elytron should never actually be activated, and even if this was a concern the correct approach would be to write some smoke tests to ensure that non Elytron deployments still work correctly.

The expediency argument also does not really work, as the tests that actually use security will need to be modified anyway so they use Elytron instead of the old security subsystem, so if the whole test suite was copied the only tests that worked 'out of the box' would be the tests that don't actually test anything security related.

IMHO a better way to test this would be to move all security related tests into their own module, and then create subclasses of these tests in Elytron and legacy modules. Elytron should be added to the default config, so that even non security related tests run with Elytron present (although we should still have at least one test module that runs without it, just as a smoke test to make sure that legacy configs without it will still work).

Stuart


> Let's keep the current (legacy security based) tests alive until the legacy security subsystem is fully removed.
> Then we'll simply remove the *-legacy-security testsuite modules.
>
> -- Josef
>
> ----- Original Message -----
> > From: "Tomaž Cerar" <[hidden email]>
> > To: "Darran Lofthouse" <[hidden email]>
> > Cc: [hidden email]
> > Sent: Friday, December 2, 2016 12:37:22 PM
> > Subject: Re: [wildfly-dev] Elytron integration tests in WildFly testsuite
> >
> > That is probably fine, but! it should be done differently.
> >
> > instead of duplicating whole testsuite (and adding extra hour to execution
> > and extra headaches with intermittent problems and duplication of
> > maintenance)
> > I would suggest that all security related tests get extracted to new
> > "security" testsuite module and than only that part is duplicated.
> >
> > This way we will have all security related stuff in one place.
> >
> >
> >
> > On Fri, Dec 2, 2016 at 12:09 PM, Darran Lofthouse <
> > [hidden email] > wrote:
> >
> >
> > Probably should add - any duplication should only be for security tests
> > - not everything else in there!
> >
> > On 02/12/16 11:08, Darran Lofthouse wrote:
> > > On 02/12/16 11:03, Tomaž Cerar wrote:
> > >>
> > >> On Fri, Dec 2, 2016 at 9:11 AM, Josef Cacek < [hidden email]
> > >> <mailto: [hidden email] >> wrote:
> > >>
> > >> The modules would just live side by side - basic would use Elytron
> > >> configuration, basic-legacy-security would use configuration similar
> > >> to (or same as) the current server configuration.
> > >>
> > >>
> > >>
> > >> What would this actually mean?
> > >> we will have two copies basic tests suites one running with elytron
> > >> another with legacy security subsystem?
> > >>
> > >> Do I read that right? Please say I am not.
> > >
> > > That is correct - we have two security implementations they both need
> > > testing.
> > >
> > > One needs testing for backwards compatibility and regressions, the other
> > > for equivalent behaviour and then new features and bugs.
> > >
> > > Needing to test both was discussed previously so this is more about how
> > > to separate both and also give the Elytron testing a good foundation to
> > > start from.
> > >
> > >>
> > >>
> > >> _______________________________________________
> > >> wildfly-dev mailing list
> > >> [hidden email]
> > >> https://lists.jboss.org/mailman/listinfo/wildfly-dev
> > >>
> > >
> >
> > _______________________________________________
> > wildfly-dev mailing list
> > [hidden email]
> > https://lists.jboss.org/mailman/listinfo/wildfly-dev
> >
> >
> > _______________________________________________
> > wildfly-dev mailing list
> > [hidden email]
> > https://lists.jboss.org/mailman/listinfo/wildfly-dev
>
> _______________________________________________
> wildfly-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/wildfly-dev


_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Elytron integration tests in WildFly testsuite

Darran Lofthouse
I am not a fan of the subclassing as at some point in the future we do
need to be able to delete the legacy tests so I would like to keep them
independent enough so we can do that without another testsuite refactoring.

But I do like the moving the 50 tests out, that fits more closely with
the three modules I described earlier.

So a new legacy-security module where we move the existing security
tests to.  Clone this to a new security module and those tests will be
enhanced to test Elytron specifically, this new module will also be
where all new tests get added.

So that leaves 900 tests behind - I would suggest that remains by
default using the default configuration of the server only.

Then if we want I think as a side task the 900 tests could have some
alternative profiles to be defined so we can ensure the following
permutations are covered: -

  1 - security extension installed.
  2 - elytron extenstion installed.
  3 - security and elytron extensions installed.
  4 - neither extension installed.

I think the default config will either be #2 or #3 so the testing will
be more about verifying we don't have hidden dependencies on specific
extensions.

We probably don't want the overhead of those permutations in either
local builds or our current CI builds but maybe a separate CI build to
run those to watch for regressions in those areas.

I expect the *-elytron.xml configurations will be removed entirely from
the distribution.

Regards,
Darran Lofthouse.


On 04/12/16 02:35, Stuart Douglas wrote:

> On 2 Dec 2016 11:00 PM, "Josef Cacek" <[hidden email]
> <mailto:[hidden email]>> wrote:
>>
>> We prefer duplication of all tests in the testsuite to keep test coverage.
>>
>> To be sure we don't have regressions with introducing Elytron and at
> the same time we are able to cover current features with Elytron, we
> must run all tests with both security subsystems.
>> Even if a test is not security related on the first glance, the tested
> component may use a security feature internally. By switching to the new
> security subsystem the feature may stop work.
>>
>
> I think this is massive overkill, and would severely reduce the quality
> and usability of the testsuite (more tests != a better test suite, they
> need to be useful test otherwise they just add technical debt for no gain).
>
> I just did a quick search through the code base, and as far as I can
> tell there is maybe ~50 test cases that test security related
> functionality (i.e. use the SecurityDomainSetup functionality), out of
> ~950 test cases, so this would amount to adding around 900 useless test
> cases.
>
> The argument that Elytron may cause regressions in these cases does
> really work either, because these tests are not linked to a security
> domain Elytron should never actually be activated, and even if this was
> a concern the correct approach would be to write some smoke tests to
> ensure that non Elytron deployments still work correctly.
>
> The expediency argument also does not really work, as the tests that
> actually use security will need to be modified anyway so they use
> Elytron instead of the old security subsystem, so if the whole test
> suite was copied the only tests that worked 'out of the box' would be
> the tests that don't actually test anything security related.
>
> IMHO a better way to test this would be to move all security related
> tests into their own module, and then create subclasses of these tests
> in Elytron and legacy modules. Elytron should be added to the default
> config, so that even non security related tests run with Elytron present
> (although we should still have at least one test module that runs
> without it, just as a smoke test to make sure that legacy configs
> without it will still work).
>
> Stuart
>
>
>> Let's keep the current (legacy security based) tests alive until the
> legacy security subsystem is fully removed.
>> Then we'll simply remove the *-legacy-security testsuite modules.
>>
>> -- Josef
>>
>> ----- Original Message -----
>> > From: "Tomaž Cerar" <[hidden email]
> <mailto:[hidden email]>>
>> > To: "Darran Lofthouse" <[hidden email]
> <mailto:[hidden email]>>
>> > Cc: [hidden email] <mailto:[hidden email]>
>> > Sent: Friday, December 2, 2016 12:37:22 PM
>> > Subject: Re: [wildfly-dev] Elytron integration tests in WildFly
> testsuite
>> >
>> > That is probably fine, but! it should be done differently.
>> >
>> > instead of duplicating whole testsuite (and adding extra hour to
> execution
>> > and extra headaches with intermittent problems and duplication of
>> > maintenance)
>> > I would suggest that all security related tests get extracted to new
>> > "security" testsuite module and than only that part is duplicated.
>> >
>> > This way we will have all security related stuff in one place.
>> >
>> >
>> >
>> > On Fri, Dec 2, 2016 at 12:09 PM, Darran Lofthouse <
>> > [hidden email] <mailto:[hidden email]> > wrote:
>> >
>> >
>> > Probably should add - any duplication should only be for security tests
>> > - not everything else in there!
>> >
>> > On 02/12/16 11:08, Darran Lofthouse wrote:
>> > > On 02/12/16 11:03, Tomaž Cerar wrote:
>> > >>
>> > >> On Fri, Dec 2, 2016 at 9:11 AM, Josef Cacek < [hidden email]
> <mailto:[hidden email]>
>> > >> <mailto: [hidden email] <mailto:[hidden email]> >> wrote:
>> > >>
>> > >> The modules would just live side by side - basic would use Elytron
>> > >> configuration, basic-legacy-security would use configuration similar
>> > >> to (or same as) the current server configuration.
>> > >>
>> > >>
>> > >>
>> > >> What would this actually mean?
>> > >> we will have two copies basic tests suites one running with elytron
>> > >> another with legacy security subsystem?
>> > >>
>> > >> Do I read that right? Please say I am not.
>> > >
>> > > That is correct - we have two security implementations they both need
>> > > testing.
>> > >
>> > > One needs testing for backwards compatibility and regressions, the
> other
>> > > for equivalent behaviour and then new features and bugs.
>> > >
>> > > Needing to test both was discussed previously so this is more
> about how
>> > > to separate both and also give the Elytron testing a good
> foundation to
>> > > start from.
>> > >
>> > >>
>> > >>
>> > >> _______________________________________________
>> > >> wildfly-dev mailing list
>> > >> [hidden email] <mailto:[hidden email]>
>> > >> https://lists.jboss.org/mailman/listinfo/wildfly-dev
> <https://lists.jboss.org/mailman/listinfo/wildfly-dev>
>> > >>
>> > >
>> >
>> > _______________________________________________
>> > wildfly-dev mailing list
>> > [hidden email] <mailto:[hidden email]>
>> > https://lists.jboss.org/mailman/listinfo/wildfly-dev
> <https://lists.jboss.org/mailman/listinfo/wildfly-dev>
>> >
>> >
>> > _______________________________________________
>> > wildfly-dev mailing list
>> > [hidden email] <mailto:[hidden email]>
>> > https://lists.jboss.org/mailman/listinfo/wildfly-dev
> <https://lists.jboss.org/mailman/listinfo/wildfly-dev>
>>
>> _______________________________________________
>> wildfly-dev mailing list
>> [hidden email] <mailto:[hidden email]>
>> https://lists.jboss.org/mailman/listinfo/wildfly-dev
> <https://lists.jboss.org/mailman/listinfo/wildfly-dev>
>
>
>
> _______________________________________________
> wildfly-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/wildfly-dev
>
_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Elytron integration tests in WildFly testsuite

Josef Cacek
In reply to this post by kkhan
Hi,

We still see the copying whole modules as the best way. Let's recap our view.

Tested components may use a security feature internally and without test coverage we would not realize that the feature stopped working. Running all tests will help us to discover regressions before they occur in application server.

Pros:
- we don't lose testcoverage, we can simply detect regressions. It also provides possible early failure detection for dev.
- it would be useful for migration, it shows migration path - user can take a look on tests with legacy solution and compare their configuration with tests which use Elytron configuration
- by each testsuite run we see how many tests are skipped (i.e. TODO list for Elytron integration)
- it should be simple to reconfigure the project (copy module; update metadata in pom.xml; ignore failing tests)
- it's simple to remove legacy testsuite modules once the legacy security subsystem is removed from WildFly

Cons and their solutions:
- double fixing of tests - we don't expect many such cases, the legacy should be handled as frozen (rewriting/adding only on need-to-have basis). Only custormer ticket related tests could be backported, it means only unit of tests per months.
- testsuite execution time - we don't need to run the legacy modules with each PR, it can be resolved by using profiles

As a smoke test we tried to run testsuite/integration-tests/basic module with standalone-elytron.xml and standalone-full-elytron.xml (merged standalone-full.xml and standalone-elytron.xml). There is around 25% test failures/errors. Also some testcases, which seem that are not directly related to security, failed.

Thanks,

-- Josef


----- Original Message -----

> From: "Kabir Khan" <[hidden email]>
> To: "Brian Stansberry" <[hidden email]>
> Cc: [hidden email]
> Sent: Friday, December 2, 2016 4:28:07 PM
> Subject: Re: [wildfly-dev] Elytron integration tests in WildFly testsuite
>
> +1 doubling the testsuite size/time is not practical. Also, there is the
> issue that if all tests are duplicated, when people come to enhance an
> existing test they will need to do so in two places.
>
> Could we not do what was suggested earlier and pull out tests which obviously
> target security into two legacy/elytron modules, and leave the rest where
> they are? Then if something later turns out to have 'hidden' implications,
> that test gets pulled out into the two modules.
>
> Regarding the deprecated legacy stuff, I think we will have to keep that
> around for many years still due to it being used by the product (and in any
> case in that branch, although removing it from WF might be possible sooner).
_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Elytron integration tests in WildFly testsuite

Darran Lofthouse
Regarding that 25% failure - I would expect a lot of that to be picked
up anyway as we merge the *-elytron.xml configurations into the default
configurations and then drop the *-elytron configurations.

How about my reply to Stuart's e-mail and a 3-way split,
legacy-security, security, and basic where basic can have optional
profiles added to test different extension availability?

We could then do as Kabir suggests and pull out additional tests later
if we do find a reason to justify duplicating the test and in the
meantime have the profiles available to run the tests.

Regards,
Darran Lofthouse.

On 05/12/16 15:03, Josef Cacek wrote:

> Hi,
>
> We still see the copying whole modules as the best way. Let's recap our view.
>
> Tested components may use a security feature internally and without test coverage we would not realize that the feature stopped working. Running all tests will help us to discover regressions before they occur in application server.
>
> Pros:
> - we don't lose testcoverage, we can simply detect regressions. It also provides possible early failure detection for dev.
> - it would be useful for migration, it shows migration path - user can take a look on tests with legacy solution and compare their configuration with tests which use Elytron configuration
> - by each testsuite run we see how many tests are skipped (i.e. TODO list for Elytron integration)
> - it should be simple to reconfigure the project (copy module; update metadata in pom.xml; ignore failing tests)
> - it's simple to remove legacy testsuite modules once the legacy security subsystem is removed from WildFly
>
> Cons and their solutions:
> - double fixing of tests - we don't expect many such cases, the legacy should be handled as frozen (rewriting/adding only on need-to-have basis). Only custormer ticket related tests could be backported, it means only unit of tests per months.
> - testsuite execution time - we don't need to run the legacy modules with each PR, it can be resolved by using profiles
>
> As a smoke test we tried to run testsuite/integration-tests/basic module with standalone-elytron.xml and standalone-full-elytron.xml (merged standalone-full.xml and standalone-elytron.xml). There is around 25% test failures/errors. Also some testcases, which seem that are not directly related to security, failed.
>
> Thanks,
>
> -- Josef
>
>
> ----- Original Message -----
>> From: "Kabir Khan" <[hidden email]>
>> To: "Brian Stansberry" <[hidden email]>
>> Cc: [hidden email]
>> Sent: Friday, December 2, 2016 4:28:07 PM
>> Subject: Re: [wildfly-dev] Elytron integration tests in WildFly testsuite
>>
>> +1 doubling the testsuite size/time is not practical. Also, there is the
>> issue that if all tests are duplicated, when people come to enhance an
>> existing test they will need to do so in two places.
>>
>> Could we not do what was suggested earlier and pull out tests which obviously
>> target security into two legacy/elytron modules, and leave the rest where
>> they are? Then if something later turns out to have 'hidden' implications,
>> that test gets pulled out into the two modules.
>>
>> Regarding the deprecated legacy stuff, I think we will have to keep that
>> around for many years still due to it being used by the product (and in any
>> case in that branch, although removing it from WF might be possible sooner).
> _______________________________________________
> wildfly-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/wildfly-dev
>
_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev
12
Loading...