Add pull request CI coverage of Galleon layers, drop Windows JDK 8

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

Add pull request CI coverage of Galleon layers, drop Windows JDK 8

Brian Stansberry
I'd like to add jobs to the automatic testing of PRs to run the testsuite using slimmed installations provisioned by Galleon. But, we already run a lot of jobs for each PR, enough so that our CI can overburdened during busy times around deadlines. So I don't want to just add jobs. Instead I also propose to drop the Windows + JDK 8 jobs.

Galleon Testing

During our work on WildFly 18 I added the ability to run portions of the WildFly and WildFly Core testsuites with the tests executing against slimmed server installations provisioned by Galleon instead of against the complete installations normally used. The point of that was to get test coverage of those slimmed installations.

Currently we have nightly jobs that run the testsuite this way.[1] As we continue to evolve our set of Galleon layers, e.g. adding layers for MicroProfile specs, I want to be sure we catch problems before PRs get merged.

To run tests locally this way you pass -Dts.layers as an arg to maven.

Dropping Windows JDK 8 Jobs

If we'd drop something in order to free up resources for these Galleon jobs, the Windows JDK 8 ones seem a good choice. We'd still run PRs against Windows JDK 11, and we'd still run PRs against Linux JDK 8. I can't recall any situation where CI found a regression that was specific to the Windows + JDK 8 combination.

When CI gets overloaded during rush times, it's the Windows jobs that are most problematic. The Windows jobs take longer because the storage drives they use are slower. Plus we have fewer Windows agents. The effect is during a rush, overall CI for PRs ends up taking hours longer while we wait for Windows agents to come free and then run the job.

We'd still run nightly jobs with Windows + JDK 8 so in the off chance there's a problem it would get noticed that way.

Any thoughts on this?


Best regards,
Brian

[1] See

_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev
Reply | Threaded
Open this post in threaded view
|

Re: Add pull request CI coverage of Galleon layers, drop Windows JDK 8

Ken Wills-2


On Mon, Oct 28, 2019 at 8:47 PM Brian Stansberry <[hidden email]> wrote:
I'd like to add jobs to the automatic testing of PRs to run the testsuite using slimmed installations provisioned by Galleon. But, we already run a lot of jobs for each PR, enough so that our CI can overburdened during busy times around deadlines. So I don't want to just add jobs. Instead I also propose to drop the Windows + JDK 8 jobs.

Galleon Testing

During our work on WildFly 18 I added the ability to run portions of the WildFly and WildFly Core testsuites with the tests executing against slimmed server installations provisioned by Galleon instead of against the complete installations normally used. The point of that was to get test coverage of those slimmed installations.

Currently we have nightly jobs that run the testsuite this way.[1] As we continue to evolve our set of Galleon layers, e.g. adding layers for MicroProfile specs, I want to be sure we catch problems before PRs get merged.

To run tests locally this way you pass -Dts.layers as an arg to maven.

Dropping Windows JDK 8 Jobs

If we'd drop something in order to free up resources for these Galleon jobs, the Windows JDK 8 ones seem a good choice. We'd still run PRs against Windows JDK 11, and we'd still run PRs against Linux JDK 8. I can't recall any situation where CI found a regression that was specific to the Windows + JDK 8 combination.

When CI gets overloaded during rush times, it's the Windows jobs that are most problematic. The Windows jobs take longer because the storage drives they use are slower. Plus we have fewer Windows agents. The effect is during a rush, overall CI for PRs ends up taking hours longer while we wait for Windows agents to come free and then run the job.

+1 from me, I think this is a great idea, especially during busy times!
 

We'd still run nightly jobs with Windows + JDK 8 so in the off chance there's a problem it would get noticed that way.

Any thoughts on this?'

This looks good to me.

Ken
 
_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev

_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev
Reply | Threaded
Open this post in threaded view
|

Re: Add pull request CI coverage of Galleon layers, drop Windows JDK 8

James Perkins
In reply to this post by Brian Stansberry


On Mon, Oct 28, 2019 at 6:47 PM Brian Stansberry <[hidden email]> wrote:
I'd like to add jobs to the automatic testing of PRs to run the testsuite using slimmed installations provisioned by Galleon. But, we already run a lot of jobs for each PR, enough so that our CI can overburdened during busy times around deadlines. So I don't want to just add jobs. Instead I also propose to drop the Windows + JDK 8 jobs.

Galleon Testing

During our work on WildFly 18 I added the ability to run portions of the WildFly and WildFly Core testsuites with the tests executing against slimmed server installations provisioned by Galleon instead of against the complete installations normally used. The point of that was to get test coverage of those slimmed installations.

Currently we have nightly jobs that run the testsuite this way.[1] As we continue to evolve our set of Galleon layers, e.g. adding layers for MicroProfile specs, I want to be sure we catch problems before PRs get merged.

To run tests locally this way you pass -Dts.layers as an arg to maven.

Dropping Windows JDK 8 Jobs

If we'd drop something in order to free up resources for these Galleon jobs, the Windows JDK 8 ones seem a good choice. We'd still run PRs against Windows JDK 11, and we'd still run PRs against Linux JDK 8. I can't recall any situation where CI found a regression that was specific to the Windows + JDK 8 combination.

When CI gets overloaded during rush times, it's the Windows jobs that are most problematic. The Windows jobs take longer because the storage drives they use are slower. Plus we have fewer Windows agents. The effect is during a rush, overall CI for PRs ends up taking hours longer while we wait for Windows agents to come free and then run the job.

We'd still run nightly jobs with Windows + JDK 8 so in the off chance there's a problem it would get noticed that way.

This seems reasonable to me. One more option to add would be that before bulk PR's are merged maybe we run Windows JDK8 jobs against those. Though those jobs are slow so...

One other option too is maybe Windows JDK8 jobs just don't run the full test suite. The main place I could possibly see issues would be if scripts change since there are some decisions made based on the JVM. Though kicking off a manual job there isn't a huge deal as it's likely not common.

Somewhat related if we do remove Windows JDK8 jobs I think we should just remove them from the aggreator job, but keep them under the Pull Request. The reason for this is if we do want to run a custom one we could use the PR number from "Changes" tab when running a custom job and it would report back to the PR.
 

No objection from me.
 
_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev


--
James R. Perkins
JBoss by Red Hat

_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev
Reply | Threaded
Open this post in threaded view
|

Re: Add pull request CI coverage of Galleon layers, drop Windows JDK 8

Brian Stansberry


On Thu, Oct 31, 2019 at 7:10 PM James Perkins <[hidden email]> wrote:


On Mon, Oct 28, 2019 at 6:47 PM Brian Stansberry <[hidden email]> wrote:
I'd like to add jobs to the automatic testing of PRs to run the testsuite using slimmed installations provisioned by Galleon. But, we already run a lot of jobs for each PR, enough so that our CI can overburdened during busy times around deadlines. So I don't want to just add jobs. Instead I also propose to drop the Windows + JDK 8 jobs.

Galleon Testing

During our work on WildFly 18 I added the ability to run portions of the WildFly and WildFly Core testsuites with the tests executing against slimmed server installations provisioned by Galleon instead of against the complete installations normally used. The point of that was to get test coverage of those slimmed installations.

Currently we have nightly jobs that run the testsuite this way.[1] As we continue to evolve our set of Galleon layers, e.g. adding layers for MicroProfile specs, I want to be sure we catch problems before PRs get merged.

To run tests locally this way you pass -Dts.layers as an arg to maven.

Dropping Windows JDK 8 Jobs

If we'd drop something in order to free up resources for these Galleon jobs, the Windows JDK 8 ones seem a good choice. We'd still run PRs against Windows JDK 11, and we'd still run PRs against Linux JDK 8. I can't recall any situation where CI found a regression that was specific to the Windows + JDK 8 combination.

When CI gets overloaded during rush times, it's the Windows jobs that are most problematic. The Windows jobs take longer because the storage drives they use are slower. Plus we have fewer Windows agents. The effect is during a rush, overall CI for PRs ends up taking hours longer while we wait for Windows agents to come free and then run the job.

We'd still run nightly jobs with Windows + JDK 8 so in the off chance there's a problem it would get noticed that way.

This seems reasonable to me. One more option to add would be that before bulk PR's are merged maybe we run Windows JDK8 jobs against those. Though those jobs are slow so...

I'm not enthused about more steps in the merge process. :) Of course it's fine if people want to do it and it's good to do when dealing with a PR that has the look of possibly being different, e.g. windows script stuff.
 
One other option too is maybe Windows JDK8 jobs just don't run the full test suite. The main place I could possibly see issues would be if scripts change since there are some decisions made based on the JVM. Though kicking off a manual job there isn't a huge deal as it's likely not common.

If you have some ideas on how the ts + ci jobs could be configured to get that result, that would be good.

What I did with ts.layers is the root pom has a profile that turns off the default surefire execution. And then in parts of the ts that I wanted to run, there's a profile that turns things on. You have to deal with a few maven modules that have more than the default surefire execution though.


Somewhat related if we do remove Windows JDK8 jobs I think we should just remove them from the aggreator job, but keep them under the Pull Request. The reason for this is if we do want to run a custom one we could use the PR number from "Changes" tab when running a custom job and it would report back to the PR.

+1; I'll do it that way.


--
James R. Perkins
JBoss by Red Hat


--
Brian Stansberry
Manager, Senior Principal Software Engineer
Red Hat

_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev
Reply | Threaded
Open this post in threaded view
|

Re: Add pull request CI coverage of Galleon layers, drop Windows JDK 8

Darran Lofthouse
In reply to this post by Brian Stansberry
We also have one small race condition in the PRs that would be good to plug at some point.

When WildFly Core PRs are submitted we build these with the latest WildFly so that the changes will hopefully not prevent a subsequent Wildfly Core upgrade.

However when we test the WildFly PRs we don't test them against the latest WildFly Core.  Some redundant code could be moved from WildFly Core after it's CI passes, code could then be added to WildFly that depends on this removed code and will also pass as it only tests against the last Wildfly Core tag.

Regards,
Darran Lofthouse.


On Tue, Oct 29, 2019 at 1:46 AM Brian Stansberry <[hidden email]> wrote:
I'd like to add jobs to the automatic testing of PRs to run the testsuite using slimmed installations provisioned by Galleon. But, we already run a lot of jobs for each PR, enough so that our CI can overburdened during busy times around deadlines. So I don't want to just add jobs. Instead I also propose to drop the Windows + JDK 8 jobs.

Galleon Testing

During our work on WildFly 18 I added the ability to run portions of the WildFly and WildFly Core testsuites with the tests executing against slimmed server installations provisioned by Galleon instead of against the complete installations normally used. The point of that was to get test coverage of those slimmed installations.

Currently we have nightly jobs that run the testsuite this way.[1] As we continue to evolve our set of Galleon layers, e.g. adding layers for MicroProfile specs, I want to be sure we catch problems before PRs get merged.

To run tests locally this way you pass -Dts.layers as an arg to maven.

Dropping Windows JDK 8 Jobs

If we'd drop something in order to free up resources for these Galleon jobs, the Windows JDK 8 ones seem a good choice. We'd still run PRs against Windows JDK 11, and we'd still run PRs against Linux JDK 8. I can't recall any situation where CI found a regression that was specific to the Windows + JDK 8 combination.

When CI gets overloaded during rush times, it's the Windows jobs that are most problematic. The Windows jobs take longer because the storage drives they use are slower. Plus we have fewer Windows agents. The effect is during a rush, overall CI for PRs ends up taking hours longer while we wait for Windows agents to come free and then run the job.

We'd still run nightly jobs with Windows + JDK 8 so in the off chance there's a problem it would get noticed that way.

Any thoughts on this?


Best regards,
Brian

[1] See
_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev

_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev
Reply | Threaded
Open this post in threaded view
|

Re: Add pull request CI coverage of Galleon layers, drop Windows JDK 8

James Perkins
In reply to this post by Brian Stansberry


On Fri, Nov 1, 2019 at 12:08 PM Brian Stansberry <[hidden email]> wrote:


On Thu, Oct 31, 2019 at 7:10 PM James Perkins <[hidden email]> wrote:


On Mon, Oct 28, 2019 at 6:47 PM Brian Stansberry <[hidden email]> wrote:
I'd like to add jobs to the automatic testing of PRs to run the testsuite using slimmed installations provisioned by Galleon. But, we already run a lot of jobs for each PR, enough so that our CI can overburdened during busy times around deadlines. So I don't want to just add jobs. Instead I also propose to drop the Windows + JDK 8 jobs.

Galleon Testing

During our work on WildFly 18 I added the ability to run portions of the WildFly and WildFly Core testsuites with the tests executing against slimmed server installations provisioned by Galleon instead of against the complete installations normally used. The point of that was to get test coverage of those slimmed installations.

Currently we have nightly jobs that run the testsuite this way.[1] As we continue to evolve our set of Galleon layers, e.g. adding layers for MicroProfile specs, I want to be sure we catch problems before PRs get merged.

To run tests locally this way you pass -Dts.layers as an arg to maven.

Dropping Windows JDK 8 Jobs

If we'd drop something in order to free up resources for these Galleon jobs, the Windows JDK 8 ones seem a good choice. We'd still run PRs against Windows JDK 11, and we'd still run PRs against Linux JDK 8. I can't recall any situation where CI found a regression that was specific to the Windows + JDK 8 combination.

When CI gets overloaded during rush times, it's the Windows jobs that are most problematic. The Windows jobs take longer because the storage drives they use are slower. Plus we have fewer Windows agents. The effect is during a rush, overall CI for PRs ends up taking hours longer while we wait for Windows agents to come free and then run the job.

We'd still run nightly jobs with Windows + JDK 8 so in the off chance there's a problem it would get noticed that way.

This seems reasonable to me. One more option to add would be that before bulk PR's are merged maybe we run Windows JDK8 jobs against those. Though those jobs are slow so...

I'm not enthused about more steps in the merge process. :) Of course it's fine if people want to do it and it's good to do when dealing with a PR that has the look of possibly being different, e.g. windows script stuff.

Yeah I almost removed this line before sending, but figured it didn't hurt to keep it open for discussion at least.
 
 
One other option too is maybe Windows JDK8 jobs just don't run the full test suite. The main place I could possibly see issues would be if scripts change since there are some decisions made based on the JVM. Though kicking off a manual job there isn't a huge deal as it's likely not common.

If you have some ideas on how the ts + ci jobs could be configured to get that result, that would be good.

What I did with ts.layers is the root pom has a profile that turns off the default surefire execution. And then in parts of the ts that I wanted to run, there's a profile that turns things on. You have to deal with a few maven modules that have more than the default surefire execution though.

My initial thought was just to run without -DallTests, but I suppose using a more fine grained approach with profiles could work as well.
 


Somewhat related if we do remove Windows JDK8 jobs I think we should just remove them from the aggreator job, but keep them under the Pull Request. The reason for this is if we do want to run a custom one we could use the PR number from "Changes" tab when running a custom job and it would report back to the PR.

+1; I'll do it that way.


--
James R. Perkins
JBoss by Red Hat


--
Brian Stansberry
Manager, Senior Principal Software Engineer
Red Hat


--
James R. Perkins
JBoss by Red Hat

_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev
Reply | Threaded
Open this post in threaded view
|

Re: Add pull request CI coverage of Galleon layers, drop Windows JDK 8

Brian Stansberry


On Fri, Nov 1, 2019 at 3:32 PM James Perkins <[hidden email]> wrote:


On Fri, Nov 1, 2019 at 12:08 PM Brian Stansberry <[hidden email]> wrote:


On Thu, Oct 31, 2019 at 7:10 PM James Perkins <[hidden email]> wrote:


On Mon, Oct 28, 2019 at 6:47 PM Brian Stansberry <[hidden email]> wrote:
I'd like to add jobs to the automatic testing of PRs to run the testsuite using slimmed installations provisioned by Galleon. But, we already run a lot of jobs for each PR, enough so that our CI can overburdened during busy times around deadlines. So I don't want to just add jobs. Instead I also propose to drop the Windows + JDK 8 jobs.

Galleon Testing

During our work on WildFly 18 I added the ability to run portions of the WildFly and WildFly Core testsuites with the tests executing against slimmed server installations provisioned by Galleon instead of against the complete installations normally used. The point of that was to get test coverage of those slimmed installations.

Currently we have nightly jobs that run the testsuite this way.[1] As we continue to evolve our set of Galleon layers, e.g. adding layers for MicroProfile specs, I want to be sure we catch problems before PRs get merged.

To run tests locally this way you pass -Dts.layers as an arg to maven.

Dropping Windows JDK 8 Jobs

If we'd drop something in order to free up resources for these Galleon jobs, the Windows JDK 8 ones seem a good choice. We'd still run PRs against Windows JDK 11, and we'd still run PRs against Linux JDK 8. I can't recall any situation where CI found a regression that was specific to the Windows + JDK 8 combination.

When CI gets overloaded during rush times, it's the Windows jobs that are most problematic. The Windows jobs take longer because the storage drives they use are slower. Plus we have fewer Windows agents. The effect is during a rush, overall CI for PRs ends up taking hours longer while we wait for Windows agents to come free and then run the job.

We'd still run nightly jobs with Windows + JDK 8 so in the off chance there's a problem it would get noticed that way.

This seems reasonable to me. One more option to add would be that before bulk PR's are merged maybe we run Windows JDK8 jobs against those. Though those jobs are slow so...

I'm not enthused about more steps in the merge process. :) Of course it's fine if people want to do it and it's good to do when dealing with a PR that has the look of possibly being different, e.g. windows script stuff.

Yeah I almost removed this line before sending, but figured it didn't hurt to keep it open for discussion at least.
 
 
One other option too is maybe Windows JDK8 jobs just don't run the full test suite. The main place I could possibly see issues would be if scripts change since there are some decisions made based on the JVM. Though kicking off a manual job there isn't a huge deal as it's likely not common.

If you have some ideas on how the ts + ci jobs could be configured to get that result, that would be good.

What I did with ts.layers is the root pom has a profile that turns off the default surefire execution. And then in parts of the ts that I wanted to run, there's a profile that turns things on. You have to deal with a few maven modules that have more than the default surefire execution though.

My initial thought was just to run without -DallTests, but I suppose using a more fine grained approach with profiles could work as well.

Or a mix. It's true that without -DallTests the bulk of the ts doesn't run, which probably eliminates the maven modules I mentioned that have more than the default surefire execution.

The -Dts.layers profile doesn't just turn off the testsuite/* stuff. It turns off surefire execution for all the other maven modules.

Is it really just testsuite/scripts in WildFly Core that tests scripts?

 


Somewhat related if we do remove Windows JDK8 jobs I think we should just remove them from the aggreator job, but keep them under the Pull Request. The reason for this is if we do want to run a custom one we could use the PR number from "Changes" tab when running a custom job and it would report back to the PR.

+1; I'll do it that way.


--
James R. Perkins
JBoss by Red Hat


--
Brian Stansberry
Manager, Senior Principal Software Engineer
Red Hat


--
James R. Perkins
JBoss by Red Hat


--
Brian Stansberry
Manager, Senior Principal Software Engineer
Red Hat

_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev
Reply | Threaded
Open this post in threaded view
|

Re: Add pull request CI coverage of Galleon layers, drop Windows JDK 8

James Perkins


On Fri, Nov 1, 2019 at 3:04 PM Brian Stansberry <[hidden email]> wrote:


On Fri, Nov 1, 2019 at 3:32 PM James Perkins <[hidden email]> wrote:


On Fri, Nov 1, 2019 at 12:08 PM Brian Stansberry <[hidden email]> wrote:


On Thu, Oct 31, 2019 at 7:10 PM James Perkins <[hidden email]> wrote:


On Mon, Oct 28, 2019 at 6:47 PM Brian Stansberry <[hidden email]> wrote:
I'd like to add jobs to the automatic testing of PRs to run the testsuite using slimmed installations provisioned by Galleon. But, we already run a lot of jobs for each PR, enough so that our CI can overburdened during busy times around deadlines. So I don't want to just add jobs. Instead I also propose to drop the Windows + JDK 8 jobs.

Galleon Testing

During our work on WildFly 18 I added the ability to run portions of the WildFly and WildFly Core testsuites with the tests executing against slimmed server installations provisioned by Galleon instead of against the complete installations normally used. The point of that was to get test coverage of those slimmed installations.

Currently we have nightly jobs that run the testsuite this way.[1] As we continue to evolve our set of Galleon layers, e.g. adding layers for MicroProfile specs, I want to be sure we catch problems before PRs get merged.

To run tests locally this way you pass -Dts.layers as an arg to maven.

Dropping Windows JDK 8 Jobs

If we'd drop something in order to free up resources for these Galleon jobs, the Windows JDK 8 ones seem a good choice. We'd still run PRs against Windows JDK 11, and we'd still run PRs against Linux JDK 8. I can't recall any situation where CI found a regression that was specific to the Windows + JDK 8 combination.

When CI gets overloaded during rush times, it's the Windows jobs that are most problematic. The Windows jobs take longer because the storage drives they use are slower. Plus we have fewer Windows agents. The effect is during a rush, overall CI for PRs ends up taking hours longer while we wait for Windows agents to come free and then run the job.

We'd still run nightly jobs with Windows + JDK 8 so in the off chance there's a problem it would get noticed that way.

This seems reasonable to me. One more option to add would be that before bulk PR's are merged maybe we run Windows JDK8 jobs against those. Though those jobs are slow so...

I'm not enthused about more steps in the merge process. :) Of course it's fine if people want to do it and it's good to do when dealing with a PR that has the look of possibly being different, e.g. windows script stuff.

Yeah I almost removed this line before sending, but figured it didn't hurt to keep it open for discussion at least.
 
 
One other option too is maybe Windows JDK8 jobs just don't run the full test suite. The main place I could possibly see issues would be if scripts change since there are some decisions made based on the JVM. Though kicking off a manual job there isn't a huge deal as it's likely not common.

If you have some ideas on how the ts + ci jobs could be configured to get that result, that would be good.

What I did with ts.layers is the root pom has a profile that turns off the default surefire execution. And then in parts of the ts that I wanted to run, there's a profile that turns things on. You have to deal with a few maven modules that have more than the default surefire execution though.

My initial thought was just to run without -DallTests, but I suppose using a more fine grained approach with profiles could work as well.

Or a mix. It's true that without -DallTests the bulk of the ts doesn't run, which probably eliminates the maven modules I mentioned that have more than the default surefire execution.

The -Dts.layers profile doesn't just turn off the testsuite/* stuff. It turns off surefire execution for all the other maven modules.

Is it really just testsuite/scripts in WildFly Core that tests scripts?

Yeah that's all it is. It honestly might not be a big deal as the scripts don't change that often. It was just the one place I had of concern.
 

 


Somewhat related if we do remove Windows JDK8 jobs I think we should just remove them from the aggreator job, but keep them under the Pull Request. The reason for this is if we do want to run a custom one we could use the PR number from "Changes" tab when running a custom job and it would report back to the PR.

+1; I'll do it that way.


--
James R. Perkins
JBoss by Red Hat


--
Brian Stansberry
Manager, Senior Principal Software Engineer
Red Hat


--
James R. Perkins
JBoss by Red Hat


--
Brian Stansberry
Manager, Senior Principal Software Engineer
Red Hat


--
James R. Perkins
JBoss by Red Hat

_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev