Tests for incomplete bugs

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Tests for incomplete bugs

Stuart Douglas
Hi Everyone,

Historically we have been ok with pull requests being submitted that add tests for things that are not yet complete.

In practice this has been kinda problematic for reviewers, as these pull requests can hang around for a long time, and this ends up wasting a lot of reviewer time as we have to continually check the status of the PR. You basically end up with multiple reviewers continually polling the issue to check if it is ready, instead of a single person just submitting it when it is complete.

I think that there is no real benefit to having the tests as a seperate PR that gets opened before the fix. Instead I propose that tests that do not pass yet are linked/co-ordinated under the relevant JIRA, and when the fix is submitted the tests can be submitted as part of the same PR.

Does anyone have any objections to this?

Stuart

_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev
Reply | Threaded
Open this post in threaded view
|

Re: Tests for incomplete bugs

Darran Lofthouse-2
+1

If multiple engineers are working on the same problem they should set up their own topic branch (can just be in one of their personal repos) somewhere and collaborate together then submit a combined PR from that branch that contains both the fix and the test case.

Sometimes I think PRs are used as an easy way to get branches into CI, we should probably look again at how easy it is to get topic branches into CI without relying on the PR process to get jobs running.

Regards,
Darran Lofthouse.


On Wed, 18 Apr 2018 at 02:12 Stuart Douglas <[hidden email]> wrote:
Hi Everyone,

Historically we have been ok with pull requests being submitted that add tests for things that are not yet complete.

In practice this has been kinda problematic for reviewers, as these pull requests can hang around for a long time, and this ends up wasting a lot of reviewer time as we have to continually check the status of the PR. You basically end up with multiple reviewers continually polling the issue to check if it is ready, instead of a single person just submitting it when it is complete.

I think that there is no real benefit to having the tests as a seperate PR that gets opened before the fix. Instead I propose that tests that do not pass yet are linked/co-ordinated under the relevant JIRA, and when the fix is submitted the tests can be submitted as part of the same PR.

Does anyone have any objections to this?

Stuart
_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev

_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev
Reply | Threaded
Open this post in threaded view
|

Re: Tests for incomplete bugs

Stuart Douglas
In the cases I am taking about the is no point even testing on CI, as it is already known that the underling bug is not fixed yet.

Stuart

On Wed, 18 Apr. 2018, 9:05 pm Darran Lofthouse, <[hidden email]> wrote:
+1

If multiple engineers are working on the same problem they should set up their own topic branch (can just be in one of their personal repos) somewhere and collaborate together then submit a combined PR from that branch that contains both the fix and the test case.

Sometimes I think PRs are used as an easy way to get branches into CI, we should probably look again at how easy it is to get topic branches into CI without relying on the PR process to get jobs running.

Regards,
Darran Lofthouse.


On Wed, 18 Apr 2018 at 02:12 Stuart Douglas <[hidden email]> wrote:
Hi Everyone,

Historically we have been ok with pull requests being submitted that add tests for things that are not yet complete.

In practice this has been kinda problematic for reviewers, as these pull requests can hang around for a long time, and this ends up wasting a lot of reviewer time as we have to continually check the status of the PR. You basically end up with multiple reviewers continually polling the issue to check if it is ready, instead of a single person just submitting it when it is complete.

I think that there is no real benefit to having the tests as a seperate PR that gets opened before the fix. Instead I propose that tests that do not pass yet are linked/co-ordinated under the relevant JIRA, and when the fix is submitted the tests can be submitted as part of the same PR.

Does anyone have any objections to this?

Stuart
_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev

_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev
Reply | Threaded
Open this post in threaded view
|

Re: Tests for incomplete bugs

Brian Stansberry
In reply to this post by Stuart Douglas
I agree.

Separating things also means that the eventual fix PR probably isn't going to have the test. So it can either get merged without the test (bad) or the people doing merging are misusing their time coordinating combined testing of two PRs.

On Tue, Apr 17, 2018 at 8:11 PM, Stuart Douglas <[hidden email]> wrote:
Hi Everyone,

Historically we have been ok with pull requests being submitted that add tests for things that are not yet complete.

In practice this has been kinda problematic for reviewers, as these pull requests can hang around for a long time, and this ends up wasting a lot of reviewer time as we have to continually check the status of the PR. You basically end up with multiple reviewers continually polling the issue to check if it is ready, instead of a single person just submitting it when it is complete.

I think that there is no real benefit to having the tests as a seperate PR that gets opened before the fix. Instead I propose that tests that do not pass yet are linked/co-ordinated under the relevant JIRA, and when the fix is submitted the tests can be submitted as part of the same PR.

Does anyone have any objections to this?

Stuart

_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev



--
Brian Stansberry
Manager, Senior Principal Software Engineer
Red Hat

_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev
Reply | Threaded
Open this post in threaded view
|

Re: Tests for incomplete bugs

Carlo de Wolf
In reply to this post by Stuart Douglas
It would prove the test code is valid. I have seen some false positives. :)

So having an independent run of the test is nice, but it must end up in the same PR as the fix.

Filing a test-only PR isn't very pretty, because it creates smoke for the mergers.

Carlo

On 18-04-18 14:54, Stuart Douglas wrote:
In the cases I am taking about the is no point even testing on CI, as it is already known that the underling bug is not fixed yet.

Stuart

On Wed, 18 Apr. 2018, 9:05 pm Darran Lofthouse, <[hidden email]> wrote:
+1

If multiple engineers are working on the same problem they should set up their own topic branch (can just be in one of their personal repos) somewhere and collaborate together then submit a combined PR from that branch that contains both the fix and the test case.

Sometimes I think PRs are used as an easy way to get branches into CI, we should probably look again at how easy it is to get topic branches into CI without relying on the PR process to get jobs running.

Regards,
Darran Lofthouse.


On Wed, 18 Apr 2018 at 02:12 Stuart Douglas <[hidden email]> wrote:
Hi Everyone,

Historically we have been ok with pull requests being submitted that add tests for things that are not yet complete.

In practice this has been kinda problematic for reviewers, as these pull requests can hang around for a long time, and this ends up wasting a lot of reviewer time as we have to continually check the status of the PR. You basically end up with multiple reviewers continually polling the issue to check if it is ready, instead of a single person just submitting it when it is complete.

I think that there is no real benefit to having the tests as a seperate PR that gets opened before the fix. Instead I propose that tests that do not pass yet are linked/co-ordinated under the relevant JIRA, and when the fix is submitted the tests can be submitted as part of the same PR.

Does anyone have any objections to this?

Stuart
_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev


_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev


_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev
Reply | Threaded
Open this post in threaded view
|

Re: Tests for incomplete bugs

Panagiotis Sotiropoulos
+1 (In cases of 1 to 1 server code and testcases)

On Thu, Apr 19, 2018 at 9:52 AM, Carlo de Wolf <[hidden email]> wrote:
It would prove the test code is valid. I have seen some false positives. :)

So having an independent run of the test is nice, but it must end up in the same PR as the fix.

Filing a test-only PR isn't very pretty, because it creates smoke for the mergers.

Carlo


On 18-04-18 14:54, Stuart Douglas wrote:
In the cases I am taking about the is no point even testing on CI, as it is already known that the underling bug is not fixed yet.

Stuart

On Wed, 18 Apr. 2018, 9:05 pm Darran Lofthouse, <[hidden email]> wrote:
+1

If multiple engineers are working on the same problem they should set up their own topic branch (can just be in one of their personal repos) somewhere and collaborate together then submit a combined PR from that branch that contains both the fix and the test case.

Sometimes I think PRs are used as an easy way to get branches into CI, we should probably look again at how easy it is to get topic branches into CI without relying on the PR process to get jobs running.

Regards,
Darran Lofthouse.


On Wed, 18 Apr 2018 at 02:12 Stuart Douglas <[hidden email]> wrote:
Hi Everyone,

Historically we have been ok with pull requests being submitted that add tests for things that are not yet complete.

In practice this has been kinda problematic for reviewers, as these pull requests can hang around for a long time, and this ends up wasting a lot of reviewer time as we have to continually check the status of the PR. You basically end up with multiple reviewers continually polling the issue to check if it is ready, instead of a single person just submitting it when it is complete.

I think that there is no real benefit to having the tests as a seperate PR that gets opened before the fix. Instead I propose that tests that do not pass yet are linked/co-ordinated under the relevant JIRA, and when the fix is submitted the tests can be submitted as part of the same PR.

Does anyone have any objections to this?

Stuart
_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev


_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev


_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev


_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev