AS7 testsuite extension

classic Classic list List threaded Threaded
14 messages Options
Reply | Threaded
Open this post in threaded view
|

AS7 testsuite extension

Rostislav Svoboda
Hi.

I would like to discuss how to extend AS7 testsuite, my suggestion is to create 2 new modules:

-- management
Tests for management APIs, probably separate submodules for cli, http api and java api tests.

-- multi-node
Tests with several instances (managed by DC or several standalone servers) on one machine.
Cluster, failover, session replication tests should be present in this module.
ARQ with DC support is needed, start/stop/kill support for instances managed by DC is needed too (ARQ-336 is designed for standalone), maybe start/stop/kill for DC will be required too.

Module structure should be similar to DOC-69049 [1] with submodules to share tests, prepare servers and to execute tests.


What do you think about suggested changes?

Rosta

[1] https://docspace.corp.redhat.com/docs/DOC-69049
_______________________________________________
jboss-as7-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
Reply | Threaded
Open this post in threaded view
|

Re: AS7 testsuite extension

Heiko Braun


+1


On Jul 19, 2011, at 2:12 PM, Rostislav Svoboda wrote:

> Hi.
>
> I would like to discuss how to extend AS7 testsuite, my suggestion is to create 2 new modules:
>
> -- management
> Tests for management APIs, probably separate submodules for cli, http api and java api tests.
>
> -- multi-node
> Tests with several instances (managed by DC or several standalone servers) on one machine.
> Cluster, failover, session replication tests should be present in this module.
> ARQ with DC support is needed, start/stop/kill support for instances managed by DC is needed too (ARQ-336 is designed for standalone), maybe start/stop/kill for DC will be required too.
>
> Module structure should be similar to DOC-69049 [1] with submodules to share tests, prepare servers and to execute tests.
>
>
> What do you think about suggested changes?
>
> Rosta
>
> [1] https://docspace.corp.redhat.com/docs/DOC-69049
> _______________________________________________
> jboss-as7-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev


_______________________________________________
jboss-as7-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
Reply | Threaded
Open this post in threaded view
|

Re: AS7 testsuite extension

Heiko Braun
In reply to this post by Rostislav Svoboda


Btw, the console test suite contains all the bits and pieces to test the HTTP interface already:

https://github.com/heiko-braun/as7-console/tree/master/gui/src/test/java/org/jboss/as/console/client

It pretty similar to the default tests that rely on the java API and wire format.
Just that this one uses the base64 encoded protocol and the HTTP endpoint.

Maybe you want to build upon that.

Ike

On Jul 19, 2011, at 2:12 PM, Rostislav Svoboda wrote:

> Hi.
>
> I would like to discuss how to extend AS7 testsuite, my suggestion is to create 2 new modules:
>
> -- management
> Tests for management APIs, probably separate submodules for cli, http api and java api tests.
>
> -- multi-node
> Tests with several instances (managed by DC or several standalone servers) on one machine.
> Cluster, failover, session replication tests should be present in this module.
> ARQ with DC support is needed, start/stop/kill support for instances managed by DC is needed too (ARQ-336 is designed for standalone), maybe start/stop/kill for DC will be required too.
>
> Module structure should be similar to DOC-69049 [1] with submodules to share tests, prepare servers and to execute tests.
>
>
> What do you think about suggested changes?
>
> Rosta
>
> [1] https://docspace.corp.redhat.com/docs/DOC-69049
> _______________________________________________
> jboss-as7-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev


_______________________________________________
jboss-as7-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
Reply | Threaded
Open this post in threaded view
|

Re: AS7 testsuite extension

Heiko Braun


Here's an API usage example:


https://github.com/heiko-braun/as7-console/blob/master/gui/src/test/java/org/jboss/dmr/client/DispatchAPITest.java




On Jul 20, 2011, at 10:40 AM, Heiko Braun wrote:

>
>
> Btw, the console test suite contains all the bits and pieces to test the HTTP interface already:
>
> https://github.com/heiko-braun/as7-console/tree/master/gui/src/test/java/org/jboss/as/console/client
>
> It pretty similar to the default tests that rely on the java API and wire format.
> Just that this one uses the base64 encoded protocol and the HTTP endpoint.
>
> Maybe you want to build upon that.
>
> Ike
>
> On Jul 19, 2011, at 2:12 PM, Rostislav Svoboda wrote:
>
>> Hi.
>>
>> I would like to discuss how to extend AS7 testsuite, my suggestion is to create 2 new modules:
>>
>> -- management
>> Tests for management APIs, probably separate submodules for cli, http api and java api tests.
>>
>> -- multi-node
>> Tests with several instances (managed by DC or several standalone servers) on one machine.
>> Cluster, failover, session replication tests should be present in this module.
>> ARQ with DC support is needed, start/stop/kill support for instances managed by DC is needed too (ARQ-336 is designed for standalone), maybe start/stop/kill for DC will be required too.
>>
>> Module structure should be similar to DOC-69049 [1] with submodules to share tests, prepare servers and to execute tests.
>>
>>
>> What do you think about suggested changes?
>>
>> Rosta
>>
>> [1] https://docspace.corp.redhat.com/docs/DOC-69049
>> _______________________________________________
>> jboss-as7-dev mailing list
>> [hidden email]
>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
>
>
> _______________________________________________
> jboss-as7-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev


_______________________________________________
jboss-as7-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
Reply | Threaded
Open this post in threaded view
|

Re: AS7 testsuite extension

Brian Stansberry
In reply to this post by Rostislav Svoboda
On 7/19/11 7:12 AM, Rostislav Svoboda wrote:
> Hi.
>
> I would like to discuss how to extend AS7 testsuite, my suggestion is to create 2 new modules:
>
> -- management
> Tests for management APIs, probably separate submodules for cli, http api and java api tests.
>

Makes sense. The existing testsuite/domain module could be moved into
here. It's essentially testing management of a domain via the java api.

IMO management tests of standalone servers should be segregated from
tests of a managed domain.

> -- multi-node
> Tests with several instances (managed by DC or several standalone servers) on one machine.
> Cluster, failover, session replication tests should be present in this module.

The testsuite/clustering module is for this purpose.

> ARQ with DC support is needed, start/stop/kill support for instances managed by DC is needed too (ARQ-336 is designed for standalone), maybe start/stop/kill for DC will be required too.
>

I regard that as a nice to have, not a requirement. HA functionality is
orthogonal to how the servers are managed; a testsuite of Paul's HA
features based on launching multiple standalone servers is ok. The
actual HA services inside the servers have no clue whether they are
running in a standalone server or a managed domain.

Note also that ARQ with DC support is not needed for management tests of
domain.

> Module structure should be similar to DOC-69049 [1] with submodules to share tests, prepare servers and to execute tests.
>

Hopefully in 95%+ of cases "prepare servers" can consist of altering the
server/domain launch command to point at the appropriate writable
directory, config file and module path.

>
> What do you think about suggested changes?
>
> Rosta
>
> [1] https://docspace.corp.redhat.com/docs/DOC-69049
> _______________________________________________
> jboss-as7-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev


--
Brian Stansberry
Principal Software Engineer
JBoss by Red Hat
_______________________________________________
jboss-as7-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
Reply | Threaded
Open this post in threaded view
|

Re: AS7 testsuite extension

Rostislav Svoboda
In reply to this post by Heiko Braun
Thank you Heiko for link and API usage example.

You are right, we plan to start building management tests upon your tests.

Rosta

----- Original Message -----

> Btw, the console test suite contains all the bits and pieces to test
> the HTTP interface already:
>
> https://github.com/heiko-braun/as7-console/tree/master/gui/src/test/java/org/jboss/as/console/client
>
> It pretty similar to the default tests that rely on the java API and
> wire format.
> Just that this one uses the base64 encoded protocol and the HTTP
> endpoint.
>
> Maybe you want to build upon that.
>
> Ike
>
> On Jul 19, 2011, at 2:12 PM, Rostislav Svoboda wrote:
>
> > Hi.
> >
> > I would like to discuss how to extend AS7 testsuite, my suggestion
> > is to create 2 new modules:
> >
> > -- management
> > Tests for management APIs, probably separate submodules for cli,
> > http api and java api tests.
> >
> > -- multi-node
> > Tests with several instances (managed by DC or several standalone
> > servers) on one machine.
> > Cluster, failover, session replication tests should be present in
> > this module.
> > ARQ with DC support is needed, start/stop/kill support for instances
> > managed by DC is needed too (ARQ-336 is designed for standalone),
> > maybe start/stop/kill for DC will be required too.
> >
> > Module structure should be similar to DOC-69049 [1] with submodules
> > to share tests, prepare servers and to execute tests.
> >
> >
> > What do you think about suggested changes?
> >
> > Rosta
> >
> > [1] https://docspace.corp.redhat.com/docs/DOC-69049
> > _______________________________________________
> > jboss-as7-dev mailing list
> > [hidden email]
> > https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
_______________________________________________
jboss-as7-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
Reply | Threaded
Open this post in threaded view
|

Re: AS7 testsuite extension

Rostislav Svoboda
In reply to this post by Brian Stansberry
Hi Brian, thank you for your reply, comments are inline.

> On 7/19/11 7:12 AM, Rostislav Svoboda wrote:
> > Hi.
> >
> > I would like to discuss how to extend AS7 testsuite, my suggestion
> > is to create 2 new modules:
> >
> > -- management
> > Tests for management APIs, probably separate submodules for cli,
> > http api and java api tests.
> >
>
> Makes sense. The existing testsuite/domain module could be moved into
> here. It's essentially testing management of a domain via the java
> api.
>
> IMO management tests of standalone servers should be segregated from
> tests of a managed domain.

Agree, submodules should be named domain-http, domain-cli, domain-java, standalone-http, standalone-cli and standalone-java.
Maybe one submodules for shared tests and resources.

>
> > -- multi-node
> > Tests with several instances (managed by DC or several standalone
> > servers) on one machine.
> > Cluster, failover, session replication tests should be present in
> > this module.
>
> The testsuite/clustering module is for this purpose.

So we should propose some changes for this module, it's currently running against one standalone server and contains only one test.
I think testsuite/clustering will have several submodules to cover different scenarios, positive test can have server execution automatically  managed by ARQ. Negative tests (failovers) must have server execution managed by ARQ but controlled from testscases (ARQ-336 [1], FailoverTestCase.java [2]).

>
> > ARQ with DC support is needed, start/stop/kill support for instances
> > managed by DC is needed too (ARQ-336 is designed for standalone),
> > maybe start/stop/kill for DC will be required too.
> >
>
> I regard that as a nice to have, not a requirement. HA functionality
> is
> orthogonal to how the servers are managed; a testsuite of Paul's HA
> features based on launching multiple standalone servers is ok. The
> actual HA services inside the servers have no clue whether they are
> running in a standalone server or a managed domain.

OK, I agree with that, DC is only about server management, no effect on AS7 functionality. My point about ARQ with DC support is more about message to users, we really recommend to use DC, but we don't use it in our tests ...
Several standalone server in cluster are sufficient to verify that clustering functionality is working properly.

> Note also that ARQ with DC support is not needed for management tests of domain.

But I steel need ARQ to start DC (using managed mode). Could you please explain how do you mean your note?
 
> > Module structure should be similar to DOC-69049 [1] with submodules
> > to share tests, prepare servers and to execute tests.
> >
>
> Hopefully in 95%+ of cases "prepare servers" can consist of altering
> the
> server/domain launch command to point at the appropriate writable
> directory, config file and module path.

We don't want to store configuration xml files in repository, we would like to prepare necessary configuration using management API. We have quite bad experience with configuration stored in xml files for testing, at least when development is still going on.
Plan is to keep "prepare servers" as simple as possible. But still I want to have appropriate prepare module for each test module to keep it straightforward when searching for possible problem - in one module is configured server and in second is test which should work.

If I'm not wrong, pointing ARQ to custom configuration xml file can be done in arquillian.xml by defining element property named serverConfig under configuration element.

> >
> > What do you think about suggested changes?
> >
> > Rosta
> >
> > [1] https://docspace.corp.redhat.com/docs/DOC-69049
> > _______________________________________________
> > jboss-as7-dev mailing list
> > [hidden email]
> > https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
>
>
> --
> Brian Stansberry
> Principal Software Engineer
> JBoss by Red Hat
> _______________________________________________
> jboss-as7-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev

--
Rostislav Svoboda
JBoss QA, Brno

[1] https://issues.jboss.org/browse/ARQ-336
[2] https://github.com/mgencur/edg-tests/blob/master/failover/src/test/java/org/arquillian/edg/failover/FailoverTestCase.java
_______________________________________________
jboss-as7-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
Reply | Threaded
Open this post in threaded view
|

Re: AS7 testsuite extension

Brian Stansberry
On 7/21/11 4:08 AM, Rostislav Svoboda wrote:

> Hi Brian, thank you for your reply, comments are inline.
>
>> On 7/19/11 7:12 AM, Rostislav Svoboda wrote:
>>> Hi.
>>>
>>> I would like to discuss how to extend AS7 testsuite, my suggestion
>>> is to create 2 new modules:
>>>
>>> -- management
>>> Tests for management APIs, probably separate submodules for cli,
>>> http api and java api tests.
>>>
>>
>> Makes sense. The existing testsuite/domain module could be moved into
>> here. It's essentially testing management of a domain via the java
>> api.
>>
>> IMO management tests of standalone servers should be segregated from
>> tests of a managed domain.
>
> Agree, submodules should be named domain-http, domain-cli, domain-java, standalone-http, standalone-cli and standalone-java.
> Maybe one submodules for shared tests and resources.
>
>>
>>> -- multi-node
>>> Tests with several instances (managed by DC or several standalone
>>> servers) on one machine.
>>> Cluster, failover, session replication tests should be present in
>>> this module.
>>
>> The testsuite/clustering module is for this purpose.
>
> So we should propose some changes for this module, it's currently running against one standalone server and contains only one test.

No argument. And I may have spoken out of turn. Perhaps Paul Ferraro had
some other intent for this module than the kind of multi-server testing
you're talking about (but I don't think so. ;) )

> I think testsuite/clustering will have several submodules to cover different scenarios, positive test can have server execution automatically  managed by ARQ. Negative tests (failovers) must have server execution managed by ARQ but controlled from testscases (ARQ-336 [1], FailoverTestCase.java [2]).
>

Paul's your man for input on this. I just wanted to point out there's an
existing module.

>>
>>> ARQ with DC support is needed, start/stop/kill support for instances
>>> managed by DC is needed too (ARQ-336 is designed for standalone),
>>> maybe start/stop/kill for DC will be required too.
>>>
>>
>> I regard that as a nice to have, not a requirement. HA functionality
>> is
>> orthogonal to how the servers are managed; a testsuite of Paul's HA
>> features based on launching multiple standalone servers is ok. The
>> actual HA services inside the servers have no clue whether they are
>> running in a standalone server or a managed domain.
>
> OK, I agree with that, DC is only about server management, no effect on AS7 functionality. My point about ARQ with DC support is more about message to users, we really recommend to use DC, but we don't use it in our tests ...

I agree with the message to users point, but I just want to emphasize we
have limited time and a large number of hard requirements and AFAIK this
is not one of them. So always be cautious about adding new requirements,
particularly ones that become blockers earlier in the process.

We have a nice testing story already with AS 7 and Arquillian. And it's
improving rapidly. So if support for using a managed domain is on the
roadmap but doesn't get in AS 7.1 or comes in too late to be the basis
of our HA testsuite, I don't think that's too bad of a message.

> Several standalone server in cluster are sufficient to verify that clustering functionality is working properly.
>
>> Note also that ARQ with DC support is not needed for management tests of domain.
>
> But I steel need ARQ to start DC (using managed mode). Could you please explain how do you mean your note?
>

No, you don't. :) See the testsuite/domain module. It uses a couple
classes to control the DC (and a slave HostController.) They are in the
arquillian/container-managed-domain module but putting them there was a
mistake on my part. They don't actually have anything really to do with
ARQ at this point. I put them there because I dreamed I'd have time to
also use them as utility classes in a real ARQ container for a managed
domain.

A bit more on domain management tests and ARQ. I see ARQ as being
oriented toward testing deployments. Most domain management testing is
not about testing deployments. Like the HA point above, how the
deployments behave inside a server is orthogonal to managed domain vs
standalone. All the deep testing of how deployments behave can be done
with a standalone server.

When testing the domain management, I don't want to have to deal with
deployments unless there is a real need. I want to start/stop domain
processes and interact with them via the management API. For sure I
don't want to have to deploy fake deployments to a domain just to
satisfy an ARQ requirement. (We actually do that in some standalone
tests of the management API.)

>>> Module structure should be similar to DOC-69049 [1] with submodules
>>> to share tests, prepare servers and to execute tests.
>>>
>>
>> Hopefully in 95%+ of cases "prepare servers" can consist of altering
>> the
>> server/domain launch command to point at the appropriate writable
>> directory, config file and module path.
>
> We don't want to store configuration xml files in repository, we would like to prepare necessary configuration using management API. We have quite bad experience with configuration stored in xml files for testing, at least when development is still going on.
> Plan is to keep "prepare servers" as simple as possible. But still I want to have appropriate prepare module for each test module to keep it straightforward when searching for possible problem - in one module is configured server and in second is test which should work.
>

Sounds good. I didn't mean at all to imply that the "prepare" module
idea is wrong.

The biggest thing to avoid is copying the build/target/jboss-xxx/modules
multiple times into different testsuite dirs. We were doing that for a
couple weeks before CR1 and it wasted a LOT of time (and disk.)

Re: preparing necessary config using the management API, keep an eye on
https://issues.jboss.org/browse/AS7-346 and subtasks.

> If I'm not wrong, pointing ARQ to custom configuration xml file can be done in arquillian.xml by defining element property named serverConfig under configuration element.
>
>>>
>>> What do you think about suggested changes?
>>>
>>> Rosta
>>>
>>> [1] https://docspace.corp.redhat.com/docs/DOC-69049
>>> _______________________________________________
>>> jboss-as7-dev mailing list
>>> [hidden email]
>>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
>>
>>
>> --
>> Brian Stansberry
>> Principal Software Engineer
>> JBoss by Red Hat
>> _______________________________________________
>> jboss-as7-dev mailing list
>> [hidden email]
>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
>
> --
> Rostislav Svoboda
> JBoss QA, Brno
>
> [1] https://issues.jboss.org/browse/ARQ-336
> [2] https://github.com/mgencur/edg-tests/blob/master/failover/src/test/java/org/arquillian/edg/failover/FailoverTestCase.java


--
Brian Stansberry
Principal Software Engineer
JBoss by Red Hat
_______________________________________________
jboss-as7-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
Reply | Threaded
Open this post in threaded view
|

Re: AS7 testsuite extension

Rostislav Svoboda
> On 7/21/11 4:08 AM, Rostislav Svoboda wrote:
> > Hi Brian, thank you for your reply, comments are inline.
> >
> >> On 7/19/11 7:12 AM, Rostislav Svoboda wrote:
> >>> Hi.
> >>>
> >>> I would like to discuss how to extend AS7 testsuite, my suggestion
> >>> is to create 2 new modules:
> >>>
> >>> -- management
> >>> Tests for management APIs, probably separate submodules for cli,
> >>> http api and java api tests.
> >>>
> >>
> >> Makes sense. The existing testsuite/domain module could be moved
> >> into
> >> here. It's essentially testing management of a domain via the java
> >> api.
> >>
> >> IMO management tests of standalone servers should be segregated
> >> from
> >> tests of a managed domain.
> >
> > Agree, submodules should be named domain-http, domain-cli,
> > domain-java, standalone-http, standalone-cli and standalone-java.
> > Maybe one submodules for shared tests and resources.
> >
> >>
> >>> -- multi-node
> >>> Tests with several instances (managed by DC or several standalone
> >>> servers) on one machine.
> >>> Cluster, failover, session replication tests should be present in
> >>> this module.
> >>
> >> The testsuite/clustering module is for this purpose.
> >
> > So we should propose some changes for this module, it's currently
> > running against one standalone server and contains only one test.
>
> No argument. And I may have spoken out of turn. Perhaps Paul Ferraro
> had
> some other intent for this module than the kind of multi-server
> testing
> you're talking about (but I don't think so. ;) )
>
> > I think testsuite/clustering will have several submodules to cover
> > different scenarios, positive test can have server execution
> > automatically managed by ARQ. Negative tests (failovers) must have
> > server execution managed by ARQ but controlled from testscases
> > (ARQ-336 [1], FailoverTestCase.java [2]).
> >
>
> Paul's your man for input on this. I just wanted to point out there's
> an existing module.

Let's wait for Paul's reply.

> >>
> >>> ARQ with DC support is needed, start/stop/kill support for
> >>> instances
> >>> managed by DC is needed too (ARQ-336 is designed for standalone),
> >>> maybe start/stop/kill for DC will be required too.
> >>>
> >>
> >> I regard that as a nice to have, not a requirement. HA
> >> functionality
> >> is
> >> orthogonal to how the servers are managed; a testsuite of Paul's HA
> >> features based on launching multiple standalone servers is ok. The
> >> actual HA services inside the servers have no clue whether they are
> >> running in a standalone server or a managed domain.
> >
> > OK, I agree with that, DC is only about server management, no effect
> > on AS7 functionality. My point about ARQ with DC support is more
> > about message to users, we really recommend to use DC, but we don't
> > use it in our tests ...
>
> I agree with the message to users point, but I just want to emphasize
> we
> have limited time and a large number of hard requirements and AFAIK
> this
> is not one of them. So always be cautious about adding new
> requirements,
> particularly ones that become blockers earlier in the process.
>
> We have a nice testing story already with AS 7 and Arquillian. And
> it's
> improving rapidly. So if support for using a managed domain is on the
> roadmap but doesn't get in AS 7.1 or comes in too late to be the basis
> of our HA testsuite, I don't think that's too bad of a message.

ARQ without DC support is not blocker for us in this case, we can do majority of clustering/HA testing with several standalone servers.
For complicated tests we wanted to use DC for simpler control of the cluster. There is motivation to keep everything Java based and located in one testsuite. We have other tools like SmartFrog (SF) which we use for HA testing but we planned to drop SF usage for AS7 tests. It's not easy to develop components and test templates, debugging is really painful. Maintenance of SF tests and components is resource consuming, not speaking about learning curve. That's our main motivation for ARQ with DC support.

> > Several standalone server in cluster are sufficient to verify that
> > clustering functionality is working properly.
> >
> >> Note also that ARQ with DC support is not needed for management
> >> tests of domain.
> >
> > But I steel need ARQ to start DC (using managed mode). Could you
> > please explain how do you mean your note?
> >
>
> No, you don't. :) See the testsuite/domain module. It uses a couple
> classes to control the DC (and a slave HostController.) They are in
> the
> arquillian/container-managed-domain module but putting them there was
> a
> mistake on my part. They don't actually have anything really to do
> with
> ARQ at this point. I put them there because I dreamed I'd have time to
> also use them as utility classes in a real ARQ container for a managed
> domain.

OK, see your point.

> A bit more on domain management tests and ARQ. I see ARQ as being
> oriented toward testing deployments. Most domain management testing is
> not about testing deployments. Like the HA point above, how the
> deployments behave inside a server is orthogonal to managed domain vs
> standalone. All the deep testing of how deployments behave can be done
> with a standalone server.
>
> When testing the domain management, I don't want to have to deal with
> deployments unless there is a real need. I want to start/stop domain
> processes and interact with them via the management API. For sure I
> don't want to have to deploy fake deployments to a domain just to
> satisfy an ARQ requirement. (We actually do that in some standalone
> tests of the management API.)

Testing against standalone server is sufficient in first phase, we need to make sure that management for DC is working too. So we need a way how to start DC before our tests, ARQ support this task seems to be the proper way. I found filled JIRA for this usage (https://issues.jboss.org/browse/ARQ-498), maybe it should be cloned to AS7.

> >>> Module structure should be similar to DOC-69049 [1] with
> >>> submodules
> >>> to share tests, prepare servers and to execute tests.
> >>>
> >>
> >> Hopefully in 95%+ of cases "prepare servers" can consist of
> >> altering
> >> the
> >> server/domain launch command to point at the appropriate writable
> >> directory, config file and module path.
> >
> > We don't want to store configuration xml files in repository, we
> > would like to prepare necessary configuration using management API.
> > We have quite bad experience with configuration stored in xml files
> > for testing, at least when development is still going on.
> > Plan is to keep "prepare servers" as simple as possible. But still I
> > want to have appropriate prepare module for each test module to keep
> > it straightforward when searching for possible problem - in one
> > module is configured server and in second is test which should work.
> >
>
> Sounds good. I didn't mean at all to imply that the "prepare" module
> idea is wrong.
>
> The biggest thing to avoid is copying the
> build/target/jboss-xxx/modules
> multiple times into different testsuite dirs. We were doing that for a
> couple weeks before CR1 and it wasted a LOT of time (and disk.)

Good point, that will speed up testsuite.

> Re: preparing necessary config using the management API, keep an eye
> on https://issues.jboss.org/browse/AS7-346 and subtasks.

Added as watcher :)

>
> > If I'm not wrong, pointing ARQ to custom configuration xml file can
> > be done in arquillian.xml by defining element property named
> > serverConfig under configuration element.
> >
> >>>
> >>> What do you think about suggested changes?
> >>>
> >>> Rosta
> >>>
> >>> [1] https://docspace.corp.redhat.com/docs/DOC-69049
> >>> _______________________________________________
> >>> jboss-as7-dev mailing list
> >>> [hidden email]
> >>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
> >>
> >>
> >> --
> >> Brian Stansberry
> >> Principal Software Engineer
> >> JBoss by Red Hat
> >> _______________________________________________
> >> jboss-as7-dev mailing list
> >> [hidden email]
> >> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
> >
> > --
> > Rostislav Svoboda
> > JBoss QA, Brno
> >
> > [1] https://issues.jboss.org/browse/ARQ-336
> > [2]
> > https://github.com/mgencur/edg-tests/blob/master/failover/src/test/java/org/arquillian/edg/failover/FailoverTestCase.java
>
> --
> Brian Stansberry
> Principal Software Engineer
> JBoss by Red Hat

Thanks for all comments.
--
Rostislav Svoboda
JBoss QA, Brno
_______________________________________________
jboss-as7-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
Reply | Threaded
Open this post in threaded view
|

Re: AS7 testsuite extension

Rostislav Svoboda
In reply to this post by Rostislav Svoboda
----- Original Message -----

> On 07/21/2011 10:08 AM, Rostislav Svoboda wrote:
> > Hi Brian, thank you for your reply, comments are inline.
> >> IMO management tests of standalone servers should be segregated
> >> from
> >> tests of a managed domain.
> >
> > Agree, submodules should be named domain-http, domain-cli,
> > domain-java, standalone-http, standalone-cli and standalone-java.
> > Maybe one submodules for shared tests and resources.
>
> One set of tests that I need to write is to exercise different
> security
> realm configurations for the management interfaces.
>
> Initially this would be in the form of one realm definition and a set
> of
> tests against each entry point to verify the correct behaviour - does
> this make sense to be repeated across each submodule or should
> something
> like this have it's own submodule?


Good question. I would personally introduce new module for management security tests.

This question implies another question. What about AS7 security testing?

CCing AS7 mailing list for more ideas.

>
> Regards,
> Darran Lofthouse.
_______________________________________________
jboss-as7-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
Reply | Threaded
Open this post in threaded view
|

Re: AS7 testsuite extension

Darran Lofthouse


On 07/22/2011 08:59 AM, Rostislav Svoboda wrote:

> ----- Original Message -----
>> On 07/21/2011 10:08 AM, Rostislav Svoboda wrote:
>>> Hi Brian, thank you for your reply, comments are inline.
>>>> IMO management tests of standalone servers should be segregated
>>>> from
>>>> tests of a managed domain.
>>>
>>> Agree, submodules should be named domain-http, domain-cli,
>>> domain-java, standalone-http, standalone-cli and standalone-java.
>>> Maybe one submodules for shared tests and resources.
>>
>> One set of tests that I need to write is to exercise different
>> security
>> realm configurations for the management interfaces.
>>
>> Initially this would be in the form of one realm definition and a set
>> of
>> tests against each entry point to verify the correct behaviour - does
>> this make sense to be repeated across each submodule or should
>> something
>> like this have it's own submodule?
>
>
> Good question. I would personally introduce new module for management security tests.
>
> This question implies another question. What about AS7 security testing?

The management security is completely independent of the application
server security as we run with no application server so the application
server security fits with the remaining EE testing.

> CCing AS7 mailing list for more ideas.
>
>>
>> Regards,
>> Darran Lofthouse.
_______________________________________________
jboss-as7-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
Reply | Threaded
Open this post in threaded view
|

Re: AS7 testsuite extension

Paul Ferraro
In reply to this post by Rostislav Svoboda
On Fri, 2011-07-22 at 03:34 -0400, Rostislav Svoboda wrote:

> > On 7/21/11 4:08 AM, Rostislav Svoboda wrote:
> > > Hi Brian, thank you for your reply, comments are inline.
> > >
> > >> On 7/19/11 7:12 AM, Rostislav Svoboda wrote:
> > >>> Hi.
> > >>>
> > >>> I would like to discuss how to extend AS7 testsuite, my suggestion
> > >>> is to create 2 new modules:
> > >>>
> > >>> -- management
> > >>> Tests for management APIs, probably separate submodules for cli,
> > >>> http api and java api tests.
> > >>>
> > >>
> > >> Makes sense. The existing testsuite/domain module could be moved
> > >> into
> > >> here. It's essentially testing management of a domain via the java
> > >> api.
> > >>
> > >> IMO management tests of standalone servers should be segregated
> > >> from
> > >> tests of a managed domain.
> > >
> > > Agree, submodules should be named domain-http, domain-cli,
> > > domain-java, standalone-http, standalone-cli and standalone-java.
> > > Maybe one submodules for shared tests and resources.
> > >
> > >>
> > >>> -- multi-node
> > >>> Tests with several instances (managed by DC or several standalone
> > >>> servers) on one machine.
> > >>> Cluster, failover, session replication tests should be present in
> > >>> this module.
> > >>
> > >> The testsuite/clustering module is for this purpose.
> > >
> > > So we should propose some changes for this module, it's currently
> > > running against one standalone server and contains only one test.
> >
> > No argument. And I may have spoken out of turn. Perhaps Paul Ferraro
> > had
> > some other intent for this module than the kind of multi-server
> > testing
> > you're talking about (but I don't think so. ;) )
> >
> > > I think testsuite/clustering will have several submodules to cover
> > > different scenarios, positive test can have server execution
> > > automatically managed by ARQ. Negative tests (failovers) must have
> > > server execution managed by ARQ but controlled from testscases
> > > (ARQ-336 [1], FailoverTestCase.java [2]).
> > >
> >
> > Paul's your man for input on this. I just wanted to point out there's
> > an existing module.
>
> Let's wait for Paul's reply.

The clustering testsuite is very much a work in progress (the current
content was just a crude setup to validate classloading issues).  I'm
still in the process of establishing the domain mode infrastructure, so
it can actually validate clustering behavior.  It's really only
individual server deployment that needs to be controllable via the test
- since from the perspective of AS7 clustering with a single deployed
application, an undeployed app is indistinguishable from a stopped
server.

> > >>
> > >>> ARQ with DC support is needed, start/stop/kill support for
> > >>> instances
> > >>> managed by DC is needed too (ARQ-336 is designed for standalone),
> > >>> maybe start/stop/kill for DC will be required too.
> > >>>
> > >>
> > >> I regard that as a nice to have, not a requirement. HA
> > >> functionality
> > >> is
> > >> orthogonal to how the servers are managed; a testsuite of Paul's HA
> > >> features based on launching multiple standalone servers is ok. The
> > >> actual HA services inside the servers have no clue whether they are
> > >> running in a standalone server or a managed domain.
> > >
> > > OK, I agree with that, DC is only about server management, no effect
> > > on AS7 functionality. My point about ARQ with DC support is more
> > > about message to users, we really recommend to use DC, but we don't
> > > use it in our tests ...
> >
> > I agree with the message to users point, but I just want to emphasize
> > we
> > have limited time and a large number of hard requirements and AFAIK
> > this
> > is not one of them. So always be cautious about adding new
> > requirements,
> > particularly ones that become blockers earlier in the process.
> >
> > We have a nice testing story already with AS 7 and Arquillian. And
> > it's
> > improving rapidly. So if support for using a managed domain is on the
> > roadmap but doesn't get in AS 7.1 or comes in too late to be the basis
> > of our HA testsuite, I don't think that's too bad of a message.
>
> ARQ without DC support is not blocker for us in this case, we can do majority of clustering/HA testing with several standalone servers.
> For complicated tests we wanted to use DC for simpler control of the cluster. There is motivation to keep everything Java based and located in one testsuite. We have other tools like SmartFrog (SF) which we use for HA testing but we planned to drop SF usage for AS7 tests. It's not easy to develop components and test templates, debugging is really painful. Maintenance of SF tests and components is resource consuming, not speaking about learning curve. That's our main motivation for ARQ with DC support.
>
> > > Several standalone server in cluster are sufficient to verify that
> > > clustering functionality is working properly.
> > >
> > >> Note also that ARQ with DC support is not needed for management
> > >> tests of domain.
> > >
> > > But I steel need ARQ to start DC (using managed mode). Could you
> > > please explain how do you mean your note?
> > >
> >
> > No, you don't. :) See the testsuite/domain module. It uses a couple
> > classes to control the DC (and a slave HostController.) They are in
> > the
> > arquillian/container-managed-domain module but putting them there was
> > a
> > mistake on my part. They don't actually have anything really to do
> > with
> > ARQ at this point. I put them there because I dreamed I'd have time to
> > also use them as utility classes in a real ARQ container for a managed
> > domain.
>
> OK, see your point.
>
> > A bit more on domain management tests and ARQ. I see ARQ as being
> > oriented toward testing deployments. Most domain management testing is
> > not about testing deployments. Like the HA point above, how the
> > deployments behave inside a server is orthogonal to managed domain vs
> > standalone. All the deep testing of how deployments behave can be done
> > with a standalone server.
> >
> > When testing the domain management, I don't want to have to deal with
> > deployments unless there is a real need. I want to start/stop domain
> > processes and interact with them via the management API. For sure I
> > don't want to have to deploy fake deployments to a domain just to
> > satisfy an ARQ requirement. (We actually do that in some standalone
> > tests of the management API.)
>
> Testing against standalone server is sufficient in first phase, we need to make sure that management for DC is working too. So we need a way how to start DC before our tests, ARQ support this task seems to be the proper way. I found filled JIRA for this usage (https://issues.jboss.org/browse/ARQ-498), maybe it should be cloned to AS7.
>
> > >>> Module structure should be similar to DOC-69049 [1] with
> > >>> submodules
> > >>> to share tests, prepare servers and to execute tests.
> > >>>
> > >>
> > >> Hopefully in 95%+ of cases "prepare servers" can consist of
> > >> altering
> > >> the
> > >> server/domain launch command to point at the appropriate writable
> > >> directory, config file and module path.
> > >
> > > We don't want to store configuration xml files in repository, we
> > > would like to prepare necessary configuration using management API.
> > > We have quite bad experience with configuration stored in xml files
> > > for testing, at least when development is still going on.
> > > Plan is to keep "prepare servers" as simple as possible. But still I
> > > want to have appropriate prepare module for each test module to keep
> > > it straightforward when searching for possible problem - in one
> > > module is configured server and in second is test which should work.
> > >
> >
> > Sounds good. I didn't mean at all to imply that the "prepare" module
> > idea is wrong.
> >
> > The biggest thing to avoid is copying the
> > build/target/jboss-xxx/modules
> > multiple times into different testsuite dirs. We were doing that for a
> > couple weeks before CR1 and it wasted a LOT of time (and disk.)
>
> Good point, that will speed up testsuite.
>
> > Re: preparing necessary config using the management API, keep an eye
> > on https://issues.jboss.org/browse/AS7-346 and subtasks.
>
> Added as watcher :)
>
> >
> > > If I'm not wrong, pointing ARQ to custom configuration xml file can
> > > be done in arquillian.xml by defining element property named
> > > serverConfig under configuration element.
> > >
> > >>>
> > >>> What do you think about suggested changes?
> > >>>
> > >>> Rosta
> > >>>
> > >>> [1] https://docspace.corp.redhat.com/docs/DOC-69049
> > >>> _______________________________________________
> > >>> jboss-as7-dev mailing list
> > >>> [hidden email]
> > >>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
> > >>
> > >>
> > >> --
> > >> Brian Stansberry
> > >> Principal Software Engineer
> > >> JBoss by Red Hat
> > >> _______________________________________________
> > >> jboss-as7-dev mailing list
> > >> [hidden email]
> > >> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
> > >
> > > --
> > > Rostislav Svoboda
> > > JBoss QA, Brno
> > >
> > > [1] https://issues.jboss.org/browse/ARQ-336
> > > [2]
> > > https://github.com/mgencur/edg-tests/blob/master/failover/src/test/java/org/arquillian/edg/failover/FailoverTestCase.java
> >
> > --
> > Brian Stansberry
> > Principal Software Engineer
> > JBoss by Red Hat
>
> Thanks for all comments.
> --
> Rostislav Svoboda
> JBoss QA, Brno


_______________________________________________
jboss-as7-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
Reply | Threaded
Open this post in threaded view
|

Re: AS7 testsuite extension

Brian Stansberry
In reply to this post by Rostislav Svoboda
On 7/22/11 2:34 AM, Rostislav Svoboda wrote:

<snip/>

>>>>
>>>>> ARQ with DC support is needed, start/stop/kill support for
>>>>> instances
>>>>> managed by DC is needed too (ARQ-336 is designed for standalone),
>>>>> maybe start/stop/kill for DC will be required too.
>>>>>
>>>>
>>>> I regard that as a nice to have, not a requirement. HA
>>>> functionality
>>>> is
>>>> orthogonal to how the servers are managed; a testsuite of Paul's HA
>>>> features based on launching multiple standalone servers is ok. The
>>>> actual HA services inside the servers have no clue whether they are
>>>> running in a standalone server or a managed domain.
>>>
>>> OK, I agree with that, DC is only about server management, no effect
>>> on AS7 functionality. My point about ARQ with DC support is more
>>> about message to users, we really recommend to use DC, but we don't
>>> use it in our tests ...
>>
>> I agree with the message to users point, but I just want to emphasize
>> we
>> have limited time and a large number of hard requirements and AFAIK
>> this
>> is not one of them. So always be cautious about adding new
>> requirements,
>> particularly ones that become blockers earlier in the process.
>>
>> We have a nice testing story already with AS 7 and Arquillian. And
>> it's
>> improving rapidly. So if support for using a managed domain is on the
>> roadmap but doesn't get in AS 7.1 or comes in too late to be the basis
>> of our HA testsuite, I don't think that's too bad of a message.
>
> ARQ without DC support is not blocker for us in this case, we can do majority of clustering/HA testing with several standalone servers.
> For complicated tests we wanted to use DC for simpler control of the cluster. There is motivation to keep everything Java based and located in one testsuite. We have other tools like SmartFrog (SF) which we use for HA testing but we planned to drop SF usage for AS7 tests. It's not easy to develop components and test templates, debugging is really painful. Maintenance of SF tests and components is resource consuming, not speaking about learning curve. That's our main motivation for ARQ with DC support.
>

I certainly don't object to using the DC for this kind of stuff. :-)
That is what it's for.

Keep in mind that the DC can't be used to hard kill servers; it only
exposes operations for controlled shutdown. I know that one of the
things the QE smartfrog stuff does in some HA tests is hard kill servers.


--
Brian Stansberry
Principal Software Engineer
JBoss by Red Hat
_______________________________________________
jboss-as7-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
Reply | Threaded
Open this post in threaded view
|

Re: AS7 testsuite extension

Anil Saldhana
I would also like opportunities to enable Java Security Manager for AS7
and certify the behavior on an ongoing basis for AS7.1.

On 07/22/2011 04:33 PM, Brian Stansberry wrote:

> On 7/22/11 2:34 AM, Rostislav Svoboda wrote:
>
> <snip/>
>
>>>>>> ARQ with DC support is needed, start/stop/kill support for
>>>>>> instances
>>>>>> managed by DC is needed too (ARQ-336 is designed for standalone),
>>>>>> maybe start/stop/kill for DC will be required too.
>>>>>>
>>>>> I regard that as a nice to have, not a requirement. HA
>>>>> functionality
>>>>> is
>>>>> orthogonal to how the servers are managed; a testsuite of Paul's HA
>>>>> features based on launching multiple standalone servers is ok. The
>>>>> actual HA services inside the servers have no clue whether they are
>>>>> running in a standalone server or a managed domain.
>>>> OK, I agree with that, DC is only about server management, no effect
>>>> on AS7 functionality. My point about ARQ with DC support is more
>>>> about message to users, we really recommend to use DC, but we don't
>>>> use it in our tests ...
>>> I agree with the message to users point, but I just want to emphasize
>>> we
>>> have limited time and a large number of hard requirements and AFAIK
>>> this
>>> is not one of them. So always be cautious about adding new
>>> requirements,
>>> particularly ones that become blockers earlier in the process.
>>>
>>> We have a nice testing story already with AS 7 and Arquillian. And
>>> it's
>>> improving rapidly. So if support for using a managed domain is on the
>>> roadmap but doesn't get in AS 7.1 or comes in too late to be the basis
>>> of our HA testsuite, I don't think that's too bad of a message.
>> ARQ without DC support is not blocker for us in this case, we can do majority of clustering/HA testing with several standalone servers.
>> For complicated tests we wanted to use DC for simpler control of the cluster. There is motivation to keep everything Java based and located in one testsuite. We have other tools like SmartFrog (SF) which we use for HA testing but we planned to drop SF usage for AS7 tests. It's not easy to develop components and test templates, debugging is really painful. Maintenance of SF tests and components is resource consuming, not speaking about learning curve. That's our main motivation for ARQ with DC support.
>>
> I certainly don't object to using the DC for this kind of stuff. :-)
> That is what it's for.
>
> Keep in mind that the DC can't be used to hard kill servers; it only
> exposes operations for controlled shutdown. I know that one of the
> things the QE smartfrog stuff does in some HA tests is hard kill servers.
>
>

_______________________________________________
jboss-as7-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/jboss-as7-dev