Srcdeps in WildFly and WildFly Core

classic Classic list List threaded Threaded
26 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Srcdeps in WildFly and WildFly Core

Peter Palaga
Hi *,

this is not new to those of you who attended my talk on the F2F 2016 in
Brno. Let me explain the idea here again for all others who did not have
a chance to be there.

Srcdeps [1] is a tool to build Maven dependencies from their sources.
With srcdeps, wildfly-core can depend on a specific commit of, e.g.,
undertow:

<version.io.undertow>1.4.8.Final-SRC-revision-aabbccd</version.io.undertow>

where aabbccd is the git commit id to build when any undertow artifact
is requested during the build of wildfly-core.

[1] describes in detail, how it works.

The main advantage of srcdeps is that changes in components can be
integrated and tested in wildfly-core immediately after they are
committed to a public component branch. There is no need to wait for the
component release.

Here in the WildFly family of projects, it is often the case that
something needs to be fixed in a component, but the verification (using
bug reproducer, or integration test) is possible only at the level of
wildfly or wildfly-core. Engineers typically work with snapshots
locally, but when their changes need to get shared (CI, reviews) in a
reproducible manner, snapshots cannot be used anymore.
In such situations a source dependency come in handy: it is very easy to
share and it is as reproducible as a Maven build from a specific commit
can be. All CIs and reviewers can work with it, because all source
dependency compilation is done under the hood by Maven.

Developers working on changes that span over multiple interdependent git
repos can thus get feedback (i-tests, reviews) quickly without waiting
for releases of components.

Srcdeps emerged in the Hawkular family of projects to solve exactly this
kind of situation and is in use there since around October 2015.

When I said there is no need to wait for releases of components, I did
not mean that we can get rid of component releases altogether. Clearly,
we cannot, because i.a. for any tooling uninformed about how srcdeps
work, those source dependencies would simply be non-resolvable from
public Maven repositories. So, before releasing the dependent component
(such as wildfly-core) all its dependencies need to be released. To
enforce this, srcdeps is by default configured to make the release fail,
as long as there are source dependencies.

I have sent a PR introducing srcdeps to wildfly-core:
https://github.com/wildfly/wildfly-core/pull/2122
To get a feeling how it works, checkout the branch, switch to e.g.
<version.io.undertow>1.4.8.Final-SRC-revision-1bff8c32f0eee986e83a7589ae95ebbc1d67d6bd</version.io.undertow>
(that happens to be the commit id of the 1.4.8.Final tag)
and build wildfly-core as usual with "mvn clean install". You'll see in
the build log that undertow is being cloned to ~/.m2/srcdeps/io/undertow
and that it is built there. After the build, check that the
1.4.8.Final-SRC-revision-1bff8c32f0eee986e83a7589ae95ebbc1d67d6bd
version of Undertow got installed to your local Maven repo (usually
~/m2/repository/io/undertow/undertow-core )

Are there any questions or comments?

[1] https://github.com/srcdeps/srcdeps-maven#srcdeps-maven

Thanks,

Peter

P.S.: I will be talking about srcdeps on Saturday 2017-01-28 at 14:30 at
DevConf Brno.
_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev
Reply | Threaded
Open this post in threaded view
|

Re: Srcdeps in WildFly and WildFly Core

Brian Stansberry
My only concerns with this would relate to comitting this kind of src dependency to the poms in the main branches in the widlfly/wildfly and wildfly/wildfly-core repos. We’ve managed to survive up to now with little or no need for that kind of thing, so until we get used to using this in other ways IMHO we should follow the KISS principle and forbid that.

A trick is avoiding doing that by mistake; i.e. a PR is sent up with a SRC dependency to get CI or review and accidentally gets merged. But I suppose that’s not the end of the world, so long as the release process will eventually detect it and fail.

Can making srcdeps fail (or just disabling it) be turned on via a maven profile? With that we could set up such a profile and turn it on in CI jobs that are testing branches where it’s forbidden (e.g. the nightly builds of master.)

Oh, one other concern — how robust is this in the face of poor maintenance? I see a lot of boilerplate in that .mvn/srcdeps.yaml. If that gets out of date or something is the only effect that using a src dependency for the affected item doesn't work?

> On Jan 25, 2017, at 3:45 PM, Peter Palaga <[hidden email]> wrote:
>
> Hi *,
>
> this is not new to those of you who attended my talk on the F2F 2016 in
> Brno. Let me explain the idea here again for all others who did not have
> a chance to be there.
>
> Srcdeps [1] is a tool to build Maven dependencies from their sources.
> With srcdeps, wildfly-core can depend on a specific commit of, e.g.,
> undertow:
>
> <version.io.undertow>1.4.8.Final-SRC-revision-aabbccd</version.io.undertow>
>
> where aabbccd is the git commit id to build when any undertow artifact
> is requested during the build of wildfly-core.
>
> [1] describes in detail, how it works.
>
> The main advantage of srcdeps is that changes in components can be
> integrated and tested in wildfly-core immediately after they are
> committed to a public component branch. There is no need to wait for the
> component release.
>
> Here in the WildFly family of projects, it is often the case that
> something needs to be fixed in a component, but the verification (using
> bug reproducer, or integration test) is possible only at the level of
> wildfly or wildfly-core. Engineers typically work with snapshots
> locally, but when their changes need to get shared (CI, reviews) in a
> reproducible manner, snapshots cannot be used anymore.
> In such situations a source dependency come in handy: it is very easy to
> share and it is as reproducible as a Maven build from a specific commit
> can be. All CIs and reviewers can work with it, because all source
> dependency compilation is done under the hood by Maven.
>
> Developers working on changes that span over multiple interdependent git
> repos can thus get feedback (i-tests, reviews) quickly without waiting
> for releases of components.
>
> Srcdeps emerged in the Hawkular family of projects to solve exactly this
> kind of situation and is in use there since around October 2015.
>
> When I said there is no need to wait for releases of components, I did
> not mean that we can get rid of component releases altogether. Clearly,
> we cannot, because i.a. for any tooling uninformed about how srcdeps
> work, those source dependencies would simply be non-resolvable from
> public Maven repositories. So, before releasing the dependent component
> (such as wildfly-core) all its dependencies need to be released. To
> enforce this, srcdeps is by default configured to make the release fail,
> as long as there are source dependencies.
>
> I have sent a PR introducing srcdeps to wildfly-core:
> https://github.com/wildfly/wildfly-core/pull/2122
> To get a feeling how it works, checkout the branch, switch to e.g.
> <version.io.undertow>1.4.8.Final-SRC-revision-1bff8c32f0eee986e83a7589ae95ebbc1d67d6bd</version.io.undertow>
> (that happens to be the commit id of the 1.4.8.Final tag)
> and build wildfly-core as usual with "mvn clean install". You'll see in
> the build log that undertow is being cloned to ~/.m2/srcdeps/io/undertow
> and that it is built there. After the build, check that the
> 1.4.8.Final-SRC-revision-1bff8c32f0eee986e83a7589ae95ebbc1d67d6bd
> version of Undertow got installed to your local Maven repo (usually
> ~/m2/repository/io/undertow/undertow-core )
>
> Are there any questions or comments?
>
> [1] https://github.com/srcdeps/srcdeps-maven#srcdeps-maven
>
> Thanks,
>
> Peter
>
> P.S.: I will be talking about srcdeps on Saturday 2017-01-28 at 14:30 at
> DevConf Brno.
> _______________________________________________
> wildfly-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/wildfly-dev

--
Brian Stansberry
Manager, Senior Principal Software Engineer
JBoss by Red Hat




_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev
Reply | Threaded
Open this post in threaded view
|

Re: Srcdeps in WildFly and WildFly Core

Peter Palaga
Hi Brian, thanks for your comments, more inline...

On 2017-01-26 02:02, Brian Stansberry wrote:
> My only concerns with this would relate to comitting this kind of src
> dependency to the poms in the main branches in the widlfly/wildfly
> and wildfly/wildfly-core repos. We’ve managed to survive up to now
> with little or no need for that kind of thing, so until we get used
> to using this in other ways IMHO we should follow the KISS principle
> and forbid that.

Maybe I overestimate the amount of changes that span over multiple git
repos. Maybe you in the Core team do not do this often. But for us in
the Sustaining Engineering Team, this is quite a typical situation. A
substantial part of the reports from customers come with a description
how to reproduce on the whole server, but they need to be fixed in a
component. Having srcdeps would make the CP process simpler and faster,
allowing us to uncover the conflicts and regressions earlier.

> A trick is avoiding doing that by mistake; i.e. a PR is sent up with
> a SRC dependency to get CI or review and accidentally gets merged.

Oh, I am just realizing I have not said anything about merging. I
actually do want to propose that commits with source dependencies get
merged to e.g. wildfly-core master as early as possible. Those are the
key points of Continuous Integration: get feedback quickly, and merge as
soon as possible. This is exactly what Hawkular is doing since more than
a year.

> But I suppose that’s not the end of the world, so long as the release
> process will eventually detect it and fail.

Yes, source dependencies on a stable branch do not harm. They just need
to be avoided in releases (for which srcdeps offers technical means).

> Can making srcdeps fail (or just disabling it) be turned on via a
> maven profile? With that we could set up such a profile and turn it
> on in CI jobs that are testing branches where it’s forbidden (e.g.
> the nightly builds of master.)

Yes, the feature is called "failWith profiles" and can be configured in
.mvn/srcdeps.yaml, like here in this srcdeps quickstart:
https://github.com/srcdeps/srcdeps-maven/blob/master/srcdeps-maven-quickstarts/srcdeps-mvn-git-profile-quickstart/.mvn/srcdeps.yaml#L33 

There is also "failWith properties" and "failWith goals". It is
documented here:
https://github.com/srcdeps/srcdeps-core/blob/master/doc/srcdeps.yaml#L130
By default there is failWith: {goals: release:prepare, release:perform}.
Projects that do not use the release plugin can set e.g. failWith:
{goals: deploy:deploy} or whatever else distinguishes their releases.

> Oh, one other concern — how robust is this in the face of poor
> maintenance? I see a lot of boilerplate in that .mvn/srcdeps.yaml.

Which parts are boilerpate?

> If
> that gets out of date or something is the only effect that using a
> src dependency for the affected item doesn't work?

Yes, I think so. As long as the .mvn/srcdeps.yaml file is syntactically
correct, any misconfiguration there should not have any other effect
than eventually breaking an embedded build.

Generally, the things configured in .mvn/srcdeps.yaml tend to be quite
stable - it is basically just mapping from GAVs to their respective git
URLs. Git URLs do not change often. It is true that dependency artifacts
come and go, but as long as their groupIds are selected reasonably (one
groupId occurs in not more than one git repo) the mapping itself can be
quite stable over time too.

Thanks,

Peter

>
>> On Jan 25, 2017, at 3:45 PM, Peter Palaga <[hidden email]>
>> wrote:
>>
>> Hi *,
>>
>> this is not new to those of you who attended my talk on the F2F
>> 2016 in Brno. Let me explain the idea here again for all others who
>> did not have a chance to be there.
>>
>> Srcdeps [1] is a tool to build Maven dependencies from their
>> sources. With srcdeps, wildfly-core can depend on a specific commit
>> of, e.g., undertow:
>>
>> <version.io.undertow>1.4.8.Final-SRC-revision-aabbccd</version.io.undertow>
>>
>>
>>
where aabbccd is the git commit id to build when any undertow artifact

>> is requested during the build of wildfly-core.
>>
>> [1] describes in detail, how it works.
>>
>> The main advantage of srcdeps is that changes in components can be
>>  integrated and tested in wildfly-core immediately after they are
>> committed to a public component branch. There is no need to wait
>> for the component release.
>>
>> Here in the WildFly family of projects, it is often the case that
>> something needs to be fixed in a component, but the verification
>> (using bug reproducer, or integration test) is possible only at the
>> level of wildfly or wildfly-core. Engineers typically work with
>> snapshots locally, but when their changes need to get shared (CI,
>> reviews) in a reproducible manner, snapshots cannot be used
>> anymore. In such situations a source dependency come in handy: it
>> is very easy to share and it is as reproducible as a Maven build
>> from a specific commit can be. All CIs and reviewers can work with
>> it, because all source dependency compilation is done under the
>> hood by Maven.
>>
>> Developers working on changes that span over multiple
>> interdependent git repos can thus get feedback (i-tests, reviews)
>> quickly without waiting for releases of components.
>>
>> Srcdeps emerged in the Hawkular family of projects to solve exactly
>> this kind of situation and is in use there since around October
>> 2015.
>>
>> When I said there is no need to wait for releases of components, I
>> did not mean that we can get rid of component releases altogether.
>> Clearly, we cannot, because i.a. for any tooling uninformed about
>> how srcdeps work, those source dependencies would simply be
>> non-resolvable from public Maven repositories. So, before releasing
>> the dependent component (such as wildfly-core) all its dependencies
>> need to be released. To enforce this, srcdeps is by default
>> configured to make the release fail, as long as there are source
>> dependencies.
>>
>> I have sent a PR introducing srcdeps to wildfly-core:
>> https://github.com/wildfly/wildfly-core/pull/2122 To get a feeling
>> how it works, checkout the branch, switch to e.g.
>> <version.io.undertow>1.4.8.Final-SRC-revision-1bff8c32f0eee986e83a7589ae95ebbc1d67d6bd</version.io.undertow>
>>  (that happens to be the commit id of the 1.4.8.Final tag) and
>> build wildfly-core as usual with "mvn clean install". You'll see in
>>  the build log that undertow is being cloned to
>> ~/.m2/srcdeps/io/undertow and that it is built there. After the
>> build, check that the
>> 1.4.8.Final-SRC-revision-1bff8c32f0eee986e83a7589ae95ebbc1d67d6bd
>> version of Undertow got installed to your local Maven repo (usually
>>  ~/m2/repository/io/undertow/undertow-core )
>>
>> Are there any questions or comments?
>>
>> [1] https://github.com/srcdeps/srcdeps-maven#srcdeps-maven
>>
>> Thanks,
>>
>> Peter
>>
>> P.S.: I will be talking about srcdeps on Saturday 2017-01-28 at
>> 14:30 at DevConf Brno.
>> _______________________________________________ wildfly-dev mailing
>> list [hidden email]
>> https://lists.jboss.org/mailman/listinfo/wildfly-dev
>

_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev
Reply | Threaded
Open this post in threaded view
|

Re: Srcdeps in WildFly and WildFly Core

Petr Sakar


On 01/26/2017 09:54 AM, Peter Palaga wrote:

> Hi Brian, thanks for your comments, more inline...
>
> On 2017-01-26 02:02, Brian Stansberry wrote:
>> My only concerns with this would relate to comitting this kind of src
>> dependency to the poms in the main branches in the widlfly/wildfly
>> and wildfly/wildfly-core repos. We’ve managed to survive up to now
>> with little or no need for that kind of thing, so until we get used
>> to using this in other ways IMHO we should follow the KISS principle
>> and forbid that.
> Maybe I overestimate the amount of changes that span over multiple git
> repos. Maybe you in the Core team do not do this often. But for us in
> the Sustaining Engineering Team, this is quite a typical situation. A
> substantial part of the reports from customers come with a description
> how to reproduce on the whole server, but they need to be fixed in a
> component. Having srcdeps would make the CP process simpler and faster,
> allowing us to uncover the conflicts and regressions earlier.
>
>> A trick is avoiding doing that by mistake; i.e. a PR is sent up with
>> a SRC dependency to get CI or review and accidentally gets merged.
> Oh, I am just realizing I have not said anything about merging. I
> actually do want to propose that commits with source dependencies get
> merged to e.g. wildfly-core master as early as possible. Those are the
> key points of Continuous Integration: get feedback quickly, and merge as
> soon as possible. This is exactly what Hawkular is doing since more than
> a year.

I agree with Brian. PR with SRC dependency *should not* be merged to master branch, that would create problems down the road.
You can use other branch as base for such merges and related CI
Petr

>> But I suppose that’s not the end of the world, so long as the release
>> process will eventually detect it and fail.
> Yes, source dependencies on a stable branch do not harm. They just need
> to be avoided in releases (for which srcdeps offers technical means).
>
>> Can making srcdeps fail (or just disabling it) be turned on via a
>> maven profile? With that we could set up such a profile and turn it
>> on in CI jobs that are testing branches where it’s forbidden (e.g.
>> the nightly builds of master.)
> Yes, the feature is called "failWith profiles" and can be configured in
> .mvn/srcdeps.yaml, like here in this srcdeps quickstart:
> https://github.com/srcdeps/srcdeps-maven/blob/master/srcdeps-maven-quickstarts/srcdeps-mvn-git-profile-quickstart/.mvn/srcdeps.yaml#L33
>
> There is also "failWith properties" and "failWith goals". It is
> documented here:
> https://github.com/srcdeps/srcdeps-core/blob/master/doc/srcdeps.yaml#L130
> By default there is failWith: {goals: release:prepare, release:perform}.
> Projects that do not use the release plugin can set e.g. failWith:
> {goals: deploy:deploy} or whatever else distinguishes their releases.
>
>> Oh, one other concern — how robust is this in the face of poor
>> maintenance? I see a lot of boilerplate in that .mvn/srcdeps.yaml.
> Which parts are boilerpate?
>
>> If
>> that gets out of date or something is the only effect that using a
>> src dependency for the affected item doesn't work?
> Yes, I think so. As long as the .mvn/srcdeps.yaml file is syntactically
> correct, any misconfiguration there should not have any other effect
> than eventually breaking an embedded build.
>
> Generally, the things configured in .mvn/srcdeps.yaml tend to be quite
> stable - it is basically just mapping from GAVs to their respective git
> URLs. Git URLs do not change often. It is true that dependency artifacts
> come and go, but as long as their groupIds are selected reasonably (one
> groupId occurs in not more than one git repo) the mapping itself can be
> quite stable over time too.
>
> Thanks,
>
> Peter
>
>>> On Jan 25, 2017, at 3:45 PM, Peter Palaga <[hidden email]>
>>> wrote:
>>>
>>> Hi *,
>>>
>>> this is not new to those of you who attended my talk on the F2F
>>> 2016 in Brno. Let me explain the idea here again for all others who
>>> did not have a chance to be there.
>>>
>>> Srcdeps [1] is a tool to build Maven dependencies from their
>>> sources. With srcdeps, wildfly-core can depend on a specific commit
>>> of, e.g., undertow:
>>>
>>> <version.io.undertow>1.4.8.Final-SRC-revision-aabbccd</version.io.undertow>
>>>
>>>
>>>
> where aabbccd is the git commit id to build when any undertow artifact
>>> is requested during the build of wildfly-core.
>>>
>>> [1] describes in detail, how it works.
>>>
>>> The main advantage of srcdeps is that changes in components can be
>>>   integrated and tested in wildfly-core immediately after they are
>>> committed to a public component branch. There is no need to wait
>>> for the component release.
>>>
>>> Here in the WildFly family of projects, it is often the case that
>>> something needs to be fixed in a component, but the verification
>>> (using bug reproducer, or integration test) is possible only at the
>>> level of wildfly or wildfly-core. Engineers typically work with
>>> snapshots locally, but when their changes need to get shared (CI,
>>> reviews) in a reproducible manner, snapshots cannot be used
>>> anymore. In such situations a source dependency come in handy: it
>>> is very easy to share and it is as reproducible as a Maven build
>>> from a specific commit can be. All CIs and reviewers can work with
>>> it, because all source dependency compilation is done under the
>>> hood by Maven.
>>>
>>> Developers working on changes that span over multiple
>>> interdependent git repos can thus get feedback (i-tests, reviews)
>>> quickly without waiting for releases of components.
>>>
>>> Srcdeps emerged in the Hawkular family of projects to solve exactly
>>> this kind of situation and is in use there since around October
>>> 2015.
>>>
>>> When I said there is no need to wait for releases of components, I
>>> did not mean that we can get rid of component releases altogether.
>>> Clearly, we cannot, because i.a. for any tooling uninformed about
>>> how srcdeps work, those source dependencies would simply be
>>> non-resolvable from public Maven repositories. So, before releasing
>>> the dependent component (such as wildfly-core) all its dependencies
>>> need to be released. To enforce this, srcdeps is by default
>>> configured to make the release fail, as long as there are source
>>> dependencies.
>>>
>>> I have sent a PR introducing srcdeps to wildfly-core:
>>> https://github.com/wildfly/wildfly-core/pull/2122 To get a feeling
>>> how it works, checkout the branch, switch to e.g.
>>> <version.io.undertow>1.4.8.Final-SRC-revision-1bff8c32f0eee986e83a7589ae95ebbc1d67d6bd</version.io.undertow>
>>>   (that happens to be the commit id of the 1.4.8.Final tag) and
>>> build wildfly-core as usual with "mvn clean install". You'll see in
>>>   the build log that undertow is being cloned to
>>> ~/.m2/srcdeps/io/undertow and that it is built there. After the
>>> build, check that the
>>> 1.4.8.Final-SRC-revision-1bff8c32f0eee986e83a7589ae95ebbc1d67d6bd
>>> version of Undertow got installed to your local Maven repo (usually
>>>   ~/m2/repository/io/undertow/undertow-core )
>>>
>>> Are there any questions or comments?
>>>
>>> [1] https://github.com/srcdeps/srcdeps-maven#srcdeps-maven
>>>
>>> Thanks,
>>>
>>> Peter
>>>
>>> P.S.: I will be talking about srcdeps on Saturday 2017-01-28 at
>>> 14:30 at DevConf Brno.
>>> _______________________________________________ wildfly-dev mailing
>>> list [hidden email]
>>> https://lists.jboss.org/mailman/listinfo/wildfly-dev
> _______________________________________________
> wildfly-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/wildfly-dev

_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev
Reply | Threaded
Open this post in threaded view
|

Re: Srcdeps in WildFly and WildFly Core

Peter Palaga
Ahoj Petře, which particular problems would be caused by a SRC
dependency merged to a master branch? -- P

On 2017-01-26 10:52, Petr Sakar wrote:

>
>
> On 01/26/2017 09:54 AM, Peter Palaga wrote:
>> Hi Brian, thanks for your comments, more inline...
>>
>> On 2017-01-26 02:02, Brian Stansberry wrote:
>>> My only concerns with this would relate to comitting this kind of src
>>> dependency to the poms in the main branches in the widlfly/wildfly
>>> and wildfly/wildfly-core repos. We’ve managed to survive up to now
>>> with little or no need for that kind of thing, so until we get used
>>> to using this in other ways IMHO we should follow the KISS principle
>>> and forbid that.
>> Maybe I overestimate the amount of changes that span over multiple git
>> repos. Maybe you in the Core team do not do this often. But for us in
>> the Sustaining Engineering Team, this is quite a typical situation. A
>> substantial part of the reports from customers come with a description
>> how to reproduce on the whole server, but they need to be fixed in a
>> component. Having srcdeps would make the CP process simpler and faster,
>> allowing us to uncover the conflicts and regressions earlier.
>>
>>> A trick is avoiding doing that by mistake; i.e. a PR is sent up with
>>> a SRC dependency to get CI or review and accidentally gets merged.
>> Oh, I am just realizing I have not said anything about merging. I
>> actually do want to propose that commits with source dependencies get
>> merged to e.g. wildfly-core master as early as possible. Those are the
>> key points of Continuous Integration: get feedback quickly, and merge as
>> soon as possible. This is exactly what Hawkular is doing since more than
>> a year.
>
> I agree with Brian. PR with SRC dependency *should not* be merged to master branch, that would create problems down the road.
> You can use other branch as base for such merges and related CI
> Petr
>
>>> But I suppose that’s not the end of the world, so long as the release
>>> process will eventually detect it and fail.
>> Yes, source dependencies on a stable branch do not harm. They just need
>> to be avoided in releases (for which srcdeps offers technical means).
>>
>>> Can making srcdeps fail (or just disabling it) be turned on via a
>>> maven profile? With that we could set up such a profile and turn it
>>> on in CI jobs that are testing branches where it’s forbidden (e.g.
>>> the nightly builds of master.)
>> Yes, the feature is called "failWith profiles" and can be configured in
>> .mvn/srcdeps.yaml, like here in this srcdeps quickstart:
>> https://github.com/srcdeps/srcdeps-maven/blob/master/srcdeps-maven-quickstarts/srcdeps-mvn-git-profile-quickstart/.mvn/srcdeps.yaml#L33
>>
>> There is also "failWith properties" and "failWith goals". It is
>> documented here:
>> https://github.com/srcdeps/srcdeps-core/blob/master/doc/srcdeps.yaml#L130
>> By default there is failWith: {goals: release:prepare, release:perform}.
>> Projects that do not use the release plugin can set e.g. failWith:
>> {goals: deploy:deploy} or whatever else distinguishes their releases.
>>
>>> Oh, one other concern — how robust is this in the face of poor
>>> maintenance? I see a lot of boilerplate in that .mvn/srcdeps.yaml.
>> Which parts are boilerpate?
>>
>>> If
>>> that gets out of date or something is the only effect that using a
>>> src dependency for the affected item doesn't work?
>> Yes, I think so. As long as the .mvn/srcdeps.yaml file is syntactically
>> correct, any misconfiguration there should not have any other effect
>> than eventually breaking an embedded build.
>>
>> Generally, the things configured in .mvn/srcdeps.yaml tend to be quite
>> stable - it is basically just mapping from GAVs to their respective git
>> URLs. Git URLs do not change often. It is true that dependency artifacts
>> come and go, but as long as their groupIds are selected reasonably (one
>> groupId occurs in not more than one git repo) the mapping itself can be
>> quite stable over time too.
>>
>> Thanks,
>>
>> Peter
>>
>>>> On Jan 25, 2017, at 3:45 PM, Peter Palaga <[hidden email]>
>>>> wrote:
>>>>
>>>> Hi *,
>>>>
>>>> this is not new to those of you who attended my talk on the F2F
>>>> 2016 in Brno. Let me explain the idea here again for all others who
>>>> did not have a chance to be there.
>>>>
>>>> Srcdeps [1] is a tool to build Maven dependencies from their
>>>> sources. With srcdeps, wildfly-core can depend on a specific commit
>>>> of, e.g., undertow:
>>>>
>>>> <version.io.undertow>1.4.8.Final-SRC-revision-aabbccd</version.io.undertow>
>>>>
>>>>
>>>>
>> where aabbccd is the git commit id to build when any undertow artifact
>>>> is requested during the build of wildfly-core.
>>>>
>>>> [1] describes in detail, how it works.
>>>>
>>>> The main advantage of srcdeps is that changes in components can be
>>>>   integrated and tested in wildfly-core immediately after they are
>>>> committed to a public component branch. There is no need to wait
>>>> for the component release.
>>>>
>>>> Here in the WildFly family of projects, it is often the case that
>>>> something needs to be fixed in a component, but the verification
>>>> (using bug reproducer, or integration test) is possible only at the
>>>> level of wildfly or wildfly-core. Engineers typically work with
>>>> snapshots locally, but when their changes need to get shared (CI,
>>>> reviews) in a reproducible manner, snapshots cannot be used
>>>> anymore. In such situations a source dependency come in handy: it
>>>> is very easy to share and it is as reproducible as a Maven build
>>>> from a specific commit can be. All CIs and reviewers can work with
>>>> it, because all source dependency compilation is done under the
>>>> hood by Maven.
>>>>
>>>> Developers working on changes that span over multiple
>>>> interdependent git repos can thus get feedback (i-tests, reviews)
>>>> quickly without waiting for releases of components.
>>>>
>>>> Srcdeps emerged in the Hawkular family of projects to solve exactly
>>>> this kind of situation and is in use there since around October
>>>> 2015.
>>>>
>>>> When I said there is no need to wait for releases of components, I
>>>> did not mean that we can get rid of component releases altogether.
>>>> Clearly, we cannot, because i.a. for any tooling uninformed about
>>>> how srcdeps work, those source dependencies would simply be
>>>> non-resolvable from public Maven repositories. So, before releasing
>>>> the dependent component (such as wildfly-core) all its dependencies
>>>> need to be released. To enforce this, srcdeps is by default
>>>> configured to make the release fail, as long as there are source
>>>> dependencies.
>>>>
>>>> I have sent a PR introducing srcdeps to wildfly-core:
>>>> https://github.com/wildfly/wildfly-core/pull/2122 To get a feeling
>>>> how it works, checkout the branch, switch to e.g.
>>>> <version.io.undertow>1.4.8.Final-SRC-revision-1bff8c32f0eee986e83a7589ae95ebbc1d67d6bd</version.io.undertow>
>>>>   (that happens to be the commit id of the 1.4.8.Final tag) and
>>>> build wildfly-core as usual with "mvn clean install". You'll see in
>>>>   the build log that undertow is being cloned to
>>>> ~/.m2/srcdeps/io/undertow and that it is built there. After the
>>>> build, check that the
>>>> 1.4.8.Final-SRC-revision-1bff8c32f0eee986e83a7589ae95ebbc1d67d6bd
>>>> version of Undertow got installed to your local Maven repo (usually
>>>>   ~/m2/repository/io/undertow/undertow-core )
>>>>
>>>> Are there any questions or comments?
>>>>
>>>> [1] https://github.com/srcdeps/srcdeps-maven#srcdeps-maven
>>>>
>>>> Thanks,
>>>>
>>>> Peter
>>>>
>>>> P.S.: I will be talking about srcdeps on Saturday 2017-01-28 at
>>>> 14:30 at DevConf Brno.
>>>> _______________________________________________ wildfly-dev mailing
>>>> list [hidden email]
>>>> https://lists.jboss.org/mailman/listinfo/wildfly-dev
>> _______________________________________________
>> wildfly-dev mailing list
>> [hidden email]
>> https://lists.jboss.org/mailman/listinfo/wildfly-dev
>
> _______________________________________________
> wildfly-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/wildfly-dev
>

_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev
Reply | Threaded
Open this post in threaded view
|

Re: Srcdeps in WildFly and WildFly Core

Juraci Paixão Kröhling
In reply to this post by Petr Sakar
On 01/26/2017 10:52 AM, Petr Sakar wrote:
> I agree with Brian. PR with SRC dependency *should not* be merged to master branch, that would create problems down the road.
> You can use other branch as base for such merges and related CI

Let me share my experience with using srcdeps in Hawkular.

We've been bitten by this situation, where a srcdeps would be listed as
a dependency and the release process would eventually fail (and it was
an ugly failure). This is why there's a feature (on by default, IIRC),
which fails fast if the release profile is being used.

A good practice in Hawkular Services is to _avoid_ having srcdeps, but
it's _acceptable_ to have it on master, as long as it's removed by the
release date. Most of the time, there's no srcdeps in our pom.xml files,
but when we need it, it allows us to keep moving without asking N
developers to release the whole dependency chain.

- Juca.
_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev
Reply | Threaded
Open this post in threaded view
|

Re: Srcdeps in WildFly and WildFly Core

kkhan
Perhaps rather than a profile in the consuming project as Brian mentions, it should be disabled in srcdeps itself by default? Something along the lines that when its dependency resolution mechanism is hit, it fails unless -Dsrcdeps.enabled is passed in. Of course there is still a risk that someone adds that property to the pom by accident.

Also how does this work for nested projects? Say we add this to WildFly full, and I want to test WildFly with to something which is brought in by wildfly-core (e.g. undertow, Elytron etc.). How would that work?


> On 26 Jan 2017, at 10:09, Juraci Paixão Kröhling <[hidden email]> wrote:
>
> On 01/26/2017 10:52 AM, Petr Sakar wrote:
>> I agree with Brian. PR with SRC dependency *should not* be merged to master branch, that would create problems down the road.
>> You can use other branch as base for such merges and related CI
>
> Let me share my experience with using srcdeps in Hawkular.
>
> We've been bitten by this situation, where a srcdeps would be listed as
> a dependency and the release process would eventually fail (and it was
> an ugly failure). This is why there's a feature (on by default, IIRC),
> which fails fast if the release profile is being used.
>
> A good practice in Hawkular Services is to _avoid_ having srcdeps, but
> it's _acceptable_ to have it on master, as long as it's removed by the
> release date. Most of the time, there's no srcdeps in our pom.xml files,
> but when we need it, it allows us to keep moving without asking N
> developers to release the whole dependency chain.
>
> - Juca.
> _______________________________________________
> wildfly-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/wildfly-dev


_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev
Reply | Threaded
Open this post in threaded view
|

Re: Srcdeps in WildFly and WildFly Core

Juraci Paixão Kröhling
On 01/26/2017 11:25 AM, Kabir Khan wrote:
> Perhaps rather than a profile in the consuming project as Brian mentions, it should be disabled in srcdeps itself by default? Something along the lines that when its dependency resolution mechanism is hit, it fails unless -Dsrcdeps.enabled is passed in. Of course there is still a risk that someone adds that property to the pom by accident.
>
> Also how does this work for nested projects? Say we add this to WildFly full, and I want to test WildFly with to something which is brought in by wildfly-core (e.g. undertow, Elytron etc.). How would that work?

It works quite nicely. I remember seeing 3 levels of nesting and I'm
sure it could have even more.

It does delay the main build, though, as srcdeps builds the dependencies
on demand. It means that the first local build of "wildfly-core" that
depends on "elytron:1.2.3-src-abc123" will also build Elytron's revision
"abc123".

- Juca.
_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev
Reply | Threaded
Open this post in threaded view
|

Re: Srcdeps in WildFly and WildFly Core

Peter Palaga
On 2017-01-26 11:46, Juraci Paixão Kröhling wrote:
> On 01/26/2017 11:25 AM, Kabir Khan wrote:
>> Perhaps rather than a profile in the consuming project as Brian mentions, it should be disabled in srcdeps itself by default? Something along the lines that when its dependency resolution mechanism is hit, it fails unless -Dsrcdeps.enabled is passed in. Of course there is still a risk that someone adds that property to the pom by accident.

Not sure what would be the purpose of making srcdeps opt-in? - To allow
CI and reviews, but to prevent merging? Well, could you tell me first
what is wrong with merging source dependencies?

>> Also how does this work for nested projects? Say we add this to WildFly full, and I want to test WildFly with to something which is brought in by wildfly-core (e.g. undertow, Elytron etc.). How would that work?
>
> It works quite nicely. I remember seeing 3 levels of nesting and I'm
> sure it could have even more.
>
> It does delay the main build, though, as srcdeps builds the dependencies
> on demand. It means that the first local build of "wildfly-core" that
> depends on "elytron:1.2.3-src-abc123" will also build Elytron's revision
> "abc123".

Yes, as Juca said, running a build of project A that has a source
dependency B, the build of B can trigger yet another build of its own
source dependency C.

Perhaps I should note that the dependency builds run with -DskipTests by
default so that they finish faster. For dependencies known to use
checkstyle, license plugin or similar, their builds can be configured to
skip those too to run even faster, as I do e.g. here
https://github.com/wildfly/wildfly-core/pull/2122/commits/65326ef4ff6abc5673a605f2394003f9a9537fdb#diff-34efcdaa51afff46bba8cd9c1387e6f5R146

Further, once a source dependency is built, it is installed to Maven
local repository and re-used from there for all subsequent builds that
require it. A source dependency thus makes just the first build slower.
The local git repositories are kept too and therefore only the first
clone takes long. All subsequent dependency builds use just fetch and reset.

-- P

_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev
Reply | Threaded
Open this post in threaded view
|

Re: Srcdeps in WildFly and WildFly Core

Darran Lofthouse
In reply to this post by Brian Stansberry
On 26/01/17 01:02, Brian Stansberry wrote:
> My only concerns with this would relate to comitting this kind of src dependency to the poms in the main branches in the widlfly/wildfly and wildfly/wildfly-core repos. We’ve managed to survive up to now with little or no need for that kind of thing, so until we get used to using this in other ways IMHO we should follow the KISS principle and forbid that.

I like the idea of this being available for teams working on a topic
branch but also think we should avoid this kind of dependency in master
- engineers are continually running builds so I don't think we should
add additional source checkouts and builds.

I know Elytron has caused huge amounts of instability but generally I
think we want these branches in a state where we could tag if we needed to.

When these downloaded builds are executed are the test cases within also
executed?

> A trick is avoiding doing that by mistake; i.e. a PR is sent up with a SRC dependency to get CI or review and accidentally gets merged. But I suppose that’s not the end of the world, so long as the release process will eventually detect it and fail.

Merging topic branches these temporary SHAs would be in the history but
maybe the PR CI runs could verify they are not present in the final merge.

> Can making srcdeps fail (or just disabling it) be turned on via a maven profile? With that we could set up such a profile and turn it on in CI jobs that are testing branches where it’s forbidden (e.g. the nightly builds of master.)
>
> Oh, one other concern — how robust is this in the face of poor maintenance? I see a lot of boilerplate in that .mvn/srcdeps.yaml. If that gets out of date or something is the only effect that using a src dependency for the affected item doesn't work?

I think that file would be better almost empty but not hidden.

Thinking about the topic branch development, within the Elytron team we
are using some SNAPSHOTs from the official repos and at times we have
forked projects into our incubator and developed against SNAPSHOTs from
those as as you select the SHA for the build you would also need to
select which repository you really want.

>> On Jan 25, 2017, at 3:45 PM, Peter Palaga <[hidden email]> wrote:
>>
>> Hi *,
>>
>> this is not new to those of you who attended my talk on the F2F 2016 in
>> Brno. Let me explain the idea here again for all others who did not have
>> a chance to be there.
>>
>> Srcdeps [1] is a tool to build Maven dependencies from their sources.
>> With srcdeps, wildfly-core can depend on a specific commit of, e.g.,
>> undertow:
>>
>> <version.io.undertow>1.4.8.Final-SRC-revision-aabbccd</version.io.undertow>
>>
>> where aabbccd is the git commit id to build when any undertow artifact
>> is requested during the build of wildfly-core.
>>
>> [1] describes in detail, how it works.
>>
>> The main advantage of srcdeps is that changes in components can be
>> integrated and tested in wildfly-core immediately after they are
>> committed to a public component branch. There is no need to wait for the
>> component release.
>>
>> Here in the WildFly family of projects, it is often the case that
>> something needs to be fixed in a component, but the verification (using
>> bug reproducer, or integration test) is possible only at the level of
>> wildfly or wildfly-core. Engineers typically work with snapshots
>> locally, but when their changes need to get shared (CI, reviews) in a
>> reproducible manner, snapshots cannot be used anymore.
>> In such situations a source dependency come in handy: it is very easy to
>> share and it is as reproducible as a Maven build from a specific commit
>> can be. All CIs and reviewers can work with it, because all source
>> dependency compilation is done under the hood by Maven.
>>
>> Developers working on changes that span over multiple interdependent git
>> repos can thus get feedback (i-tests, reviews) quickly without waiting
>> for releases of components.
>>
>> Srcdeps emerged in the Hawkular family of projects to solve exactly this
>> kind of situation and is in use there since around October 2015.
>>
>> When I said there is no need to wait for releases of components, I did
>> not mean that we can get rid of component releases altogether. Clearly,
>> we cannot, because i.a. for any tooling uninformed about how srcdeps
>> work, those source dependencies would simply be non-resolvable from
>> public Maven repositories. So, before releasing the dependent component
>> (such as wildfly-core) all its dependencies need to be released. To
>> enforce this, srcdeps is by default configured to make the release fail,
>> as long as there are source dependencies.
>>
>> I have sent a PR introducing srcdeps to wildfly-core:
>> https://github.com/wildfly/wildfly-core/pull/2122
>> To get a feeling how it works, checkout the branch, switch to e.g.
>> <version.io.undertow>1.4.8.Final-SRC-revision-1bff8c32f0eee986e83a7589ae95ebbc1d67d6bd</version.io.undertow>
>> (that happens to be the commit id of the 1.4.8.Final tag)
>> and build wildfly-core as usual with "mvn clean install". You'll see in
>> the build log that undertow is being cloned to ~/.m2/srcdeps/io/undertow
>> and that it is built there. After the build, check that the
>> 1.4.8.Final-SRC-revision-1bff8c32f0eee986e83a7589ae95ebbc1d67d6bd
>> version of Undertow got installed to your local Maven repo (usually
>> ~/m2/repository/io/undertow/undertow-core )
>>
>> Are there any questions or comments?
>>
>> [1] https://github.com/srcdeps/srcdeps-maven#srcdeps-maven
>>
>> Thanks,
>>
>> Peter
>>
>> P.S.: I will be talking about srcdeps on Saturday 2017-01-28 at 14:30 at
>> DevConf Brno.
>> _______________________________________________
>> wildfly-dev mailing list
>> [hidden email]
>> https://lists.jboss.org/mailman/listinfo/wildfly-dev
>
_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev
Reply | Threaded
Open this post in threaded view
|

Re: Srcdeps in WildFly and WildFly Core

kkhan
In reply to this post by Peter Palaga

> On 26 Jan 2017, at 11:37, Peter Palaga <[hidden email]> wrote:
>
> On 2017-01-26 11:46, Juraci Paixão Kröhling wrote:
>> On 01/26/2017 11:25 AM, Kabir Khan wrote:
>>> Perhaps rather than a profile in the consuming project as Brian mentions, it should be disabled in srcdeps itself by default? Something along the lines that when its dependency resolution mechanism is hit, it fails unless -Dsrcdeps.enabled is passed in. Of course there is still a risk that someone adds that property to the pom by accident.
>
> Not sure what would be the purpose of making srcdeps opt-in? - To allow CI and reviews, but to prevent merging?
Yes, or rather it would fail our default CI, apart from runs where the opt in happens explicitly, which will help with accidentally merging.
> Well, could you tell me first what is wrong with merging source dependencies?
Perhaps that is fine for some projects/branches, but for WildFly(-core) master we want proper released maven artifacts. I've had a quick look at http://maven.apache.org/xsd/core-extensions-1.0.0.xsd but there doesn't seem to be a way to pass in any config properties to extensions.xml, but perhaps something could be added to the yaml so you can set a policy there. It still runs the risk of getting merged though :)

>
>>> Also how does this work for nested projects? Say we add this to WildFly full, and I want to test WildFly with to something which is brought in by wildfly-core (e.g. undertow, Elytron etc.). How would that work?
>>
>> It works quite nicely. I remember seeing 3 levels of nesting and I'm
>> sure it could have even more.
>>
>> It does delay the main build, though, as srcdeps builds the dependencies
>> on demand. It means that the first local build of "wildfly-core" that
>> depends on "elytron:1.2.3-src-abc123" will also build Elytron's revision
>> "abc123".
>
> Yes, as Juca said, running a build of project A that has a source dependency B, the build of B can trigger yet another build of its own source dependency C.
>
> Perhaps I should note that the dependency builds run with -DskipTests by default so that they finish faster. For dependencies known to use checkstyle, license plugin or similar, their builds can be configured to skip those too to run even faster, as I do e.g. here https://github.com/wildfly/wildfly-core/pull/2122/commits/65326ef4ff6abc5673a605f2394003f9a9537fdb#diff-34efcdaa51afff46bba8cd9c1387e6f5R146
>
> Further, once a source dependency is built, it is installed to Maven local repository and re-used from there for all subsequent builds that require it. A source dependency thus makes just the first build slower.
> The local git repositories are kept too and therefore only the first clone takes long. All subsequent dependency builds use just fetch and reset.
>
> -- P
>


_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev
Reply | Threaded
Open this post in threaded view
|

Re: Srcdeps in WildFly and WildFly Core

Peter Palaga
On 2017-01-26 12:51, Kabir Khan wrote:

>
>> On 26 Jan 2017, at 11:37, Peter Palaga <[hidden email]> wrote:
>>
>> On 2017-01-26 11:46, Juraci Paixão Kröhling wrote:
>>> On 01/26/2017 11:25 AM, Kabir Khan wrote:
>>>> Perhaps rather than a profile in the consuming project as Brian mentions, it should be disabled in srcdeps itself by default? Something along the lines that when its dependency resolution mechanism is hit, it fails unless -Dsrcdeps.enabled is passed in. Of course there is still a risk that someone adds that property to the pom by accident.
>>
>> Not sure what would be the purpose of making srcdeps opt-in? - To allow CI and reviews, but to prevent merging?
> Yes, or rather it would fail our default CI, apart from runs where the opt in happens explicitly, which will help with accidentally merging.
>> Well, could you tell me first what is wrong with merging source dependencies?
> Perhaps that is fine for some projects/branches, but for WildFly(-core) master we want proper released maven artifacts.

Yes, I see your preference, but I do not see clearly why you want to ban
source dependencies from master.

 > I've had a quick look at
http://maven.apache.org/xsd/core-extensions-1.0.0.xsd but there doesn't
seem to be a way to pass in any config properties to extensions.xml, but
perhaps something could be added to the yaml so you can set a policy
there. It still runs the risk of getting merged though :)

Yes, extensions.xml does not allow for passing any config params to
extensions. Srcdeps.yaml would be a better place for such an option. I
can add it once there is enough consensus to go that way. -- P

>>
>>>> Also how does this work for nested projects? Say we add this to WildFly full, and I want to test WildFly with to something which is brought in by wildfly-core (e.g. undertow, Elytron etc.). How would that work?
>>>
>>> It works quite nicely. I remember seeing 3 levels of nesting and I'm
>>> sure it could have even more.
>>>
>>> It does delay the main build, though, as srcdeps builds the dependencies
>>> on demand. It means that the first local build of "wildfly-core" that
>>> depends on "elytron:1.2.3-src-abc123" will also build Elytron's revision
>>> "abc123".
>>
>> Yes, as Juca said, running a build of project A that has a source dependency B, the build of B can trigger yet another build of its own source dependency C.
>>
>> Perhaps I should note that the dependency builds run with -DskipTests by default so that they finish faster. For dependencies known to use checkstyle, license plugin or similar, their builds can be configured to skip those too to run even faster, as I do e.g. here https://github.com/wildfly/wildfly-core/pull/2122/commits/65326ef4ff6abc5673a605f2394003f9a9537fdb#diff-34efcdaa51afff46bba8cd9c1387e6f5R146
>>
>> Further, once a source dependency is built, it is installed to Maven local repository and re-used from there for all subsequent builds that require it. A source dependency thus makes just the first build slower.
>> The local git repositories are kept too and therefore only the first clone takes long. All subsequent dependency builds use just fetch and reset.
>>
>> -- P
>>
>

_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev
Reply | Threaded
Open this post in threaded view
|

Re: Srcdeps in WildFly and WildFly Core

Peter Palaga
In reply to this post by Darran Lofthouse
Thanks for the comments, Darran, more inline...

On 2017-01-26 12:40, Darran Lofthouse wrote:
> On 26/01/17 01:02, Brian Stansberry wrote:
>> My only concerns with this would relate to comitting this kind of src dependency to the poms in the main branches in the widlfly/wildfly and wildfly/wildfly-core repos. We’ve managed to survive up to now with little or no need for that kind of thing, so until we get used to using this in other ways IMHO we should follow the KISS principle and forbid that.
>
> I like the idea of this being available for teams working on a topic
> branch but also think we should avoid this kind of dependency in master
> - engineers are continually running builds so I don't think we should
> add additional source checkouts and builds.

So you also vote for having the support for source dependencies when the
e.g. the wildfly-core CI builds a PR that integrates a change in Elytron
or when a reviewer does the same, but such PRs should wait for a proper
release of the component, right?

> I know Elytron has caused huge amounts of instability but generally I
> think we want these branches in a state where we could tag if we needed to.

"these branches" - you mean topic branches in a component (such as
Elytron) or in a consuming project (such as WF Core) or both?

> When these downloaded builds are executed are the test cases within also
> executed?

The builds of source dependencies run with -DskipTests by default so
that they finish faster. An explicit per git repository setting in
srcdeps.yaml is needed if somebody wants a dependency build to run
without -DskipTests.

>> A trick is avoiding doing that by mistake; i.e. a PR is sent up with a SRC dependency to get CI or review and accidentally gets merged. But I suppose that’s not the end of the world, so long as the release process will eventually detect it and fail.
>
> Merging topic branches these temporary SHAs would be in the history but
> maybe the PR CI runs could verify they are not present in the final merge.
>
>> Can making srcdeps fail (or just disabling it) be turned on via a maven profile? With that we could set up such a profile and turn it on in CI jobs that are testing branches where it’s forbidden (e.g. the nightly builds of master.)
>>
>> Oh, one other concern — how robust is this in the face of poor maintenance? I see a lot of boilerplate in that .mvn/srcdeps.yaml. If that gets out of date or something is the only effect that using a src dependency for the affected item doesn't work?
>
> I think that file would be better almost empty

The size of the srcdeps.yaml file depends on how many dependencies the
given project has and which of those are selected for the srcdeps
support. In the PR https://git.io/vMjUC , I have included only the
projects owned by us, excluding non-Maven and non-git projects. I am
open to discussion whether we should include or exclude more projects.

> but not hidden.

The .mvn directory was introduced by Maven 3.3.1. Maven reads
extensions.xml file from there and I found it rather natural to have
srcdeps.yaml file there too.

> Thinking about the topic branch development, within the Elytron team we
> are using some SNAPSHOTs from the official repos and at times we have
> forked projects into our incubator and developed against SNAPSHOTs from
> those

Allowing SNAPSHOTs on any remote Maven repository leads to loosing
reproducibility of the builds of the dependent project. Srcdeps offers
much more reproducibility and portalbility over environments in such
situations.

> as as you select the SHA for the build you would also need to
> select which repository you really want.

Yes, the mapping which dependencies are built from which git
repositories is defined in srcdeps.yaml. I see no problem in having one
git URL in one branch and having different git URL in another branch.

Thanks,

Peter

>>> On Jan 25, 2017, at 3:45 PM, Peter Palaga <[hidden email]> wrote:
>>>
>>> Hi *,
>>>
>>> this is not new to those of you who attended my talk on the F2F 2016 in
>>> Brno. Let me explain the idea here again for all others who did not have
>>> a chance to be there.
>>>
>>> Srcdeps [1] is a tool to build Maven dependencies from their sources.
>>> With srcdeps, wildfly-core can depend on a specific commit of, e.g.,
>>> undertow:
>>>
>>> <version.io.undertow>1.4.8.Final-SRC-revision-aabbccd</version.io.undertow>
>>>
>>> where aabbccd is the git commit id to build when any undertow artifact
>>> is requested during the build of wildfly-core.
>>>
>>> [1] describes in detail, how it works.
>>>
>>> The main advantage of srcdeps is that changes in components can be
>>> integrated and tested in wildfly-core immediately after they are
>>> committed to a public component branch. There is no need to wait for the
>>> component release.
>>>
>>> Here in the WildFly family of projects, it is often the case that
>>> something needs to be fixed in a component, but the verification (using
>>> bug reproducer, or integration test) is possible only at the level of
>>> wildfly or wildfly-core. Engineers typically work with snapshots
>>> locally, but when their changes need to get shared (CI, reviews) in a
>>> reproducible manner, snapshots cannot be used anymore.
>>> In such situations a source dependency come in handy: it is very easy to
>>> share and it is as reproducible as a Maven build from a specific commit
>>> can be. All CIs and reviewers can work with it, because all source
>>> dependency compilation is done under the hood by Maven.
>>>
>>> Developers working on changes that span over multiple interdependent git
>>> repos can thus get feedback (i-tests, reviews) quickly without waiting
>>> for releases of components.
>>>
>>> Srcdeps emerged in the Hawkular family of projects to solve exactly this
>>> kind of situation and is in use there since around October 2015.
>>>
>>> When I said there is no need to wait for releases of components, I did
>>> not mean that we can get rid of component releases altogether. Clearly,
>>> we cannot, because i.a. for any tooling uninformed about how srcdeps
>>> work, those source dependencies would simply be non-resolvable from
>>> public Maven repositories. So, before releasing the dependent component
>>> (such as wildfly-core) all its dependencies need to be released. To
>>> enforce this, srcdeps is by default configured to make the release fail,
>>> as long as there are source dependencies.
>>>
>>> I have sent a PR introducing srcdeps to wildfly-core:
>>> https://github.com/wildfly/wildfly-core/pull/2122
>>> To get a feeling how it works, checkout the branch, switch to e.g.
>>> <version.io.undertow>1.4.8.Final-SRC-revision-1bff8c32f0eee986e83a7589ae95ebbc1d67d6bd</version.io.undertow>
>>> (that happens to be the commit id of the 1.4.8.Final tag)
>>> and build wildfly-core as usual with "mvn clean install". You'll see in
>>> the build log that undertow is being cloned to ~/.m2/srcdeps/io/undertow
>>> and that it is built there. After the build, check that the
>>> 1.4.8.Final-SRC-revision-1bff8c32f0eee986e83a7589ae95ebbc1d67d6bd
>>> version of Undertow got installed to your local Maven repo (usually
>>> ~/m2/repository/io/undertow/undertow-core )
>>>
>>> Are there any questions or comments?
>>>
>>> [1] https://github.com/srcdeps/srcdeps-maven#srcdeps-maven
>>>
>>> Thanks,
>>>
>>> Peter
>>>
>>> P.S.: I will be talking about srcdeps on Saturday 2017-01-28 at 14:30 at
>>> DevConf Brno.
>>> _______________________________________________
>>> wildfly-dev mailing list
>>> [hidden email]
>>> https://lists.jboss.org/mailman/listinfo/wildfly-dev
>>
> _______________________________________________
> wildfly-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/wildfly-dev
>

_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev
Reply | Threaded
Open this post in threaded view
|

Re: Srcdeps in WildFly and WildFly Core

Rostislav Svoboda
In reply to this post by Peter Palaga
> On 2017-01-26 12:51, Kabir Khan wrote:
> >
> >> On 26 Jan 2017, at 11:37, Peter Palaga <[hidden email]> wrote:
> >>
> >> On 2017-01-26 11:46, Juraci Paixão Kröhling wrote:
> >>> On 01/26/2017 11:25 AM, Kabir Khan wrote:
> >>>> Perhaps rather than a profile in the consuming project as Brian
> >>>> mentions, it should be disabled in srcdeps itself by default? Something
> >>>> along the lines that when its dependency resolution mechanism is hit,
> >>>> it fails unless -Dsrcdeps.enabled is passed in. Of course there is
> >>>> still a risk that someone adds that property to the pom by accident.
> >>
> >> Not sure what would be the purpose of making srcdeps opt-in? - To allow CI
> >> and reviews, but to prevent merging?
> > Yes, or rather it would fail our default CI, apart from runs where the opt
> > in happens explicitly, which will help with accidentally merging.
> >> Well, could you tell me first what is wrong with merging source
> >> dependencies?
> > Perhaps that is fine for some projects/branches, but for WildFly(-core)
> > master we want proper released maven artifacts.
>
> Yes, I see your preference, but I do not see clearly why you want to ban
> source dependencies from master.


I think the question is different:

Which of current problems is srcdeps solving ?

Rostislav

>  > I've had a quick look at
> http://maven.apache.org/xsd/core-extensions-1.0.0.xsd but there doesn't
> seem to be a way to pass in any config properties to extensions.xml, but
> perhaps something could be added to the yaml so you can set a policy
> there. It still runs the risk of getting merged though :)
>
> Yes, extensions.xml does not allow for passing any config params to
> extensions. Srcdeps.yaml would be a better place for such an option. I
> can add it once there is enough consensus to go that way. -- P
>
> >>
> >>>> Also how does this work for nested projects? Say we add this to WildFly
> >>>> full, and I want to test WildFly with to something which is brought in
> >>>> by wildfly-core (e.g. undertow, Elytron etc.). How would that work?
> >>>
> >>> It works quite nicely. I remember seeing 3 levels of nesting and I'm
> >>> sure it could have even more.
> >>>
> >>> It does delay the main build, though, as srcdeps builds the dependencies
> >>> on demand. It means that the first local build of "wildfly-core" that
> >>> depends on "elytron:1.2.3-src-abc123" will also build Elytron's revision
> >>> "abc123".
> >>
> >> Yes, as Juca said, running a build of project A that has a source
> >> dependency B, the build of B can trigger yet another build of its own
> >> source dependency C.
> >>
> >> Perhaps I should note that the dependency builds run with -DskipTests by
> >> default so that they finish faster. For dependencies known to use
> >> checkstyle, license plugin or similar, their builds can be configured to
> >> skip those too to run even faster, as I do e.g. here
> >> https://github.com/wildfly/wildfly-core/pull/2122/commits/65326ef4ff6abc5673a605f2394003f9a9537fdb#diff-34efcdaa51afff46bba8cd9c1387e6f5R146
> >>
> >> Further, once a source dependency is built, it is installed to Maven local
> >> repository and re-used from there for all subsequent builds that require
> >> it. A source dependency thus makes just the first build slower.
> >> The local git repositories are kept too and therefore only the first clone
> >> takes long. All subsequent dependency builds use just fetch and reset.
> >>
> >> -- P
> >>
> >
>
> _______________________________________________
> wildfly-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/wildfly-dev

_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev
Reply | Threaded
Open this post in threaded view
|

Re: Srcdeps in WildFly and WildFly Core

Darran Lofthouse
In reply to this post by Peter Palaga


On 26/01/17 12:42, Peter Palaga wrote:

> Thanks for the comments, Darran, more inline...
>
> On 2017-01-26 12:40, Darran Lofthouse wrote:
>> On 26/01/17 01:02, Brian Stansberry wrote:
>>> My only concerns with this would relate to comitting this kind of src
>>> dependency to the poms in the main branches in the widlfly/wildfly
>>> and wildfly/wildfly-core repos. We’ve managed to survive up to now
>>> with little or no need for that kind of thing, so until we get used
>>> to using this in other ways IMHO we should follow the KISS principle
>>> and forbid that.
>>
>> I like the idea of this being available for teams working on a topic
>> branch but also think we should avoid this kind of dependency in master
>> - engineers are continually running builds so I don't think we should
>> add additional source checkouts and builds.
>
> So you also vote for having the support for source dependencies when the
> e.g. the wildfly-core CI builds a PR that integrates a change in Elytron
> or when a reviewer does the same, but such PRs should wait for a proper
> release of the component, right?

I think by the time we get to the PR stage engineers should have had an
opportunity to discuss the proposed solution sufficiently that we
shouldn't need a srcdep and should have been able to tag Elytron.

>> I know Elytron has caused huge amounts of instability but generally I
>> think we want these branches in a state where we could tag if we
>> needed to.
>
> "these branches" - you mean topic branches in a component (such as
> Elytron) or in a consuming project (such as WF Core) or both?

I think WildFly Core / master and WildFly / master are generally not
supposed to be as unstable as we have made them recently.

>> When these downloaded builds are executed are the test cases within also
>> executed?
>
> The builds of source dependencies run with -DskipTests by default so
> that they finish faster. An explicit per git repository setting in
> srcdeps.yaml is needed if somebody wants a dependency build to run
> without -DskipTests.
>
>>> A trick is avoiding doing that by mistake; i.e. a PR is sent up with
>>> a SRC dependency to get CI or review and accidentally gets merged.
>>> But I suppose that’s not the end of the world, so long as the release
>>> process will eventually detect it and fail.
>>
>> Merging topic branches these temporary SHAs would be in the history but
>> maybe the PR CI runs could verify they are not present in the final
>> merge.
>>
>>> Can making srcdeps fail (or just disabling it) be turned on via a
>>> maven profile? With that we could set up such a profile and turn it
>>> on in CI jobs that are testing branches where it’s forbidden (e.g.
>>> the nightly builds of master.)
>>>
>>> Oh, one other concern — how robust is this in the face of poor
>>> maintenance? I see a lot of boilerplate in that .mvn/srcdeps.yaml. If
>>> that gets out of date or something is the only effect that using a
>>> src dependency for the affected item doesn't work?
>>
>> I think that file would be better almost empty
>
> The size of the srcdeps.yaml file depends on how many dependencies the
> given project has and which of those are selected for the srcdeps
> support. In the PR https://git.io/vMjUC , I have included only the
> projects owned by us, excluding non-Maven and non-git projects. I am
> open to discussion whether we should include or exclude more projects.
>
>> but not hidden.
>
> The .mvn directory was introduced by Maven 3.3.1. Maven reads
> extensions.xml file from there and I found it rather natural to have
> srcdeps.yaml file there too.

What that means is that a file that can directly influence the outcome
of a local build is hidden from the engineer.  There are already quite a
few things in out builds where when they go wrong we have to spend time
tracking them down, having a hidden configuration may not help this.

>> Thinking about the topic branch development, within the Elytron team we
>> are using some SNAPSHOTs from the official repos and at times we have
>> forked projects into our incubator and developed against SNAPSHOTs from
>> those
>
> Allowing SNAPSHOTs on any remote Maven repository leads to loosing
> reproducibility of the builds of the dependent project. Srcdeps offers
> much more reproducibility and portalbility over environments in such
> situations.

Yes I am saying in the development of the Elytron topic branches this
could help a lot - but the assumptions as to which git repos we are
using would be wrong.

>> as as you select the SHA for the build you would also need to
>> select which repository you really want.
>
> Yes, the mapping which dependencies are built from which git
> repositories is defined in srcdeps.yaml. I see no problem in having one
> git URL in one branch and having different git URL in another branch.

Which I think as this is a file engineers could be editing to tweak
their builds means it should not be hidden.

> Thanks,
>
> Peter
>
>>>> On Jan 25, 2017, at 3:45 PM, Peter Palaga <[hidden email]> wrote:
>>>>
>>>> Hi *,
>>>>
>>>> this is not new to those of you who attended my talk on the F2F 2016 in
>>>> Brno. Let me explain the idea here again for all others who did not
>>>> have
>>>> a chance to be there.
>>>>
>>>> Srcdeps [1] is a tool to build Maven dependencies from their sources.
>>>> With srcdeps, wildfly-core can depend on a specific commit of, e.g.,
>>>> undertow:
>>>>
>>>> <version.io.undertow>1.4.8.Final-SRC-revision-aabbccd</version.io.undertow>
>>>>
>>>>
>>>> where aabbccd is the git commit id to build when any undertow artifact
>>>> is requested during the build of wildfly-core.
>>>>
>>>> [1] describes in detail, how it works.
>>>>
>>>> The main advantage of srcdeps is that changes in components can be
>>>> integrated and tested in wildfly-core immediately after they are
>>>> committed to a public component branch. There is no need to wait for
>>>> the
>>>> component release.
>>>>
>>>> Here in the WildFly family of projects, it is often the case that
>>>> something needs to be fixed in a component, but the verification (using
>>>> bug reproducer, or integration test) is possible only at the level of
>>>> wildfly or wildfly-core. Engineers typically work with snapshots
>>>> locally, but when their changes need to get shared (CI, reviews) in a
>>>> reproducible manner, snapshots cannot be used anymore.
>>>> In such situations a source dependency come in handy: it is very
>>>> easy to
>>>> share and it is as reproducible as a Maven build from a specific commit
>>>> can be. All CIs and reviewers can work with it, because all source
>>>> dependency compilation is done under the hood by Maven.
>>>>
>>>> Developers working on changes that span over multiple interdependent
>>>> git
>>>> repos can thus get feedback (i-tests, reviews) quickly without waiting
>>>> for releases of components.
>>>>
>>>> Srcdeps emerged in the Hawkular family of projects to solve exactly
>>>> this
>>>> kind of situation and is in use there since around October 2015.
>>>>
>>>> When I said there is no need to wait for releases of components, I did
>>>> not mean that we can get rid of component releases altogether. Clearly,
>>>> we cannot, because i.a. for any tooling uninformed about how srcdeps
>>>> work, those source dependencies would simply be non-resolvable from
>>>> public Maven repositories. So, before releasing the dependent component
>>>> (such as wildfly-core) all its dependencies need to be released. To
>>>> enforce this, srcdeps is by default configured to make the release
>>>> fail,
>>>> as long as there are source dependencies.
>>>>
>>>> I have sent a PR introducing srcdeps to wildfly-core:
>>>> https://github.com/wildfly/wildfly-core/pull/2122
>>>> To get a feeling how it works, checkout the branch, switch to e.g.
>>>> <version.io.undertow>1.4.8.Final-SRC-revision-1bff8c32f0eee986e83a7589ae95ebbc1d67d6bd</version.io.undertow>
>>>>
>>>> (that happens to be the commit id of the 1.4.8.Final tag)
>>>> and build wildfly-core as usual with "mvn clean install". You'll see in
>>>> the build log that undertow is being cloned to
>>>> ~/.m2/srcdeps/io/undertow
>>>> and that it is built there. After the build, check that the
>>>> 1.4.8.Final-SRC-revision-1bff8c32f0eee986e83a7589ae95ebbc1d67d6bd
>>>> version of Undertow got installed to your local Maven repo (usually
>>>> ~/m2/repository/io/undertow/undertow-core )
>>>>
>>>> Are there any questions or comments?
>>>>
>>>> [1] https://github.com/srcdeps/srcdeps-maven#srcdeps-maven
>>>>
>>>> Thanks,
>>>>
>>>> Peter
>>>>
>>>> P.S.: I will be talking about srcdeps on Saturday 2017-01-28 at
>>>> 14:30 at
>>>> DevConf Brno.
>>>> _______________________________________________
>>>> wildfly-dev mailing list
>>>> [hidden email]
>>>> https://lists.jboss.org/mailman/listinfo/wildfly-dev
>>>
>> _______________________________________________
>> wildfly-dev mailing list
>> [hidden email]
>> https://lists.jboss.org/mailman/listinfo/wildfly-dev
>>
>
_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev
Reply | Threaded
Open this post in threaded view
|

Re: Srcdeps in WildFly and WildFly Core

Peter Palaga
In reply to this post by Rostislav Svoboda
On 2017-01-26 13:46, Rostislav Svoboda wrote:

>> On 2017-01-26 12:51, Kabir Khan wrote:
>>>
>>>> On 26 Jan 2017, at 11:37, Peter Palaga <[hidden email]> wrote:
>>>>
>>>> On 2017-01-26 11:46, Juraci Paixão Kröhling wrote:
>>>>> On 01/26/2017 11:25 AM, Kabir Khan wrote:
>>>>>> Perhaps rather than a profile in the consuming project as Brian
>>>>>> mentions, it should be disabled in srcdeps itself by default? Something
>>>>>> along the lines that when its dependency resolution mechanism is hit,
>>>>>> it fails unless -Dsrcdeps.enabled is passed in. Of course there is
>>>>>> still a risk that someone adds that property to the pom by accident.
>>>>
>>>> Not sure what would be the purpose of making srcdeps opt-in? - To allow CI
>>>> and reviews, but to prevent merging?
>>> Yes, or rather it would fail our default CI, apart from runs where the opt
>>> in happens explicitly, which will help with accidentally merging.
>>>> Well, could you tell me first what is wrong with merging source
>>>> dependencies?
>>> Perhaps that is fine for some projects/branches, but for WildFly(-core)
>>> master we want proper released maven artifacts.
>>
>> Yes, I see your preference, but I do not see clearly why you want to ban
>> source dependencies from master.
>
>
> I think the question is different:
>
> Which of current problems is srcdeps solving ?

It is an improvement of the development process through simplifying it
and making it faster.
As I said in the initial message, the main advantage of srcdeps is that
changes in components can be integrated, tested and reviewed in wildfly
and wildfly-core immediately after they are committed to a public
component branch. The developer does not need to wait for the component
release. He gets the feedback from CI and reviewers faster. I do not
think anything changes for quality engineers, given that the current
proposal assumes srcdeps-free releases. -- P

> Rostislav
>
>>  > I've had a quick look at
>> http://maven.apache.org/xsd/core-extensions-1.0.0.xsd but there doesn't
>> seem to be a way to pass in any config properties to extensions.xml, but
>> perhaps something could be added to the yaml so you can set a policy
>> there. It still runs the risk of getting merged though :)
>>
>> Yes, extensions.xml does not allow for passing any config params to
>> extensions. Srcdeps.yaml would be a better place for such an option. I
>> can add it once there is enough consensus to go that way. -- P
>>
>>>>
>>>>>> Also how does this work for nested projects? Say we add this to WildFly
>>>>>> full, and I want to test WildFly with to something which is brought in
>>>>>> by wildfly-core (e.g. undertow, Elytron etc.). How would that work?
>>>>>
>>>>> It works quite nicely. I remember seeing 3 levels of nesting and I'm
>>>>> sure it could have even more.
>>>>>
>>>>> It does delay the main build, though, as srcdeps builds the dependencies
>>>>> on demand. It means that the first local build of "wildfly-core" that
>>>>> depends on "elytron:1.2.3-src-abc123" will also build Elytron's revision
>>>>> "abc123".
>>>>
>>>> Yes, as Juca said, running a build of project A that has a source
>>>> dependency B, the build of B can trigger yet another build of its own
>>>> source dependency C.
>>>>
>>>> Perhaps I should note that the dependency builds run with -DskipTests by
>>>> default so that they finish faster. For dependencies known to use
>>>> checkstyle, license plugin or similar, their builds can be configured to
>>>> skip those too to run even faster, as I do e.g. here
>>>> https://github.com/wildfly/wildfly-core/pull/2122/commits/65326ef4ff6abc5673a605f2394003f9a9537fdb#diff-34efcdaa51afff46bba8cd9c1387e6f5R146
>>>>
>>>> Further, once a source dependency is built, it is installed to Maven local
>>>> repository and re-used from there for all subsequent builds that require
>>>> it. A source dependency thus makes just the first build slower.
>>>> The local git repositories are kept too and therefore only the first clone
>>>> takes long. All subsequent dependency builds use just fetch and reset.
>>>>
>>>> -- P
>>>>
>>>
>>
>> _______________________________________________
>> wildfly-dev mailing list
>> [hidden email]
>> https://lists.jboss.org/mailman/listinfo/wildfly-dev

_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev
Reply | Threaded
Open this post in threaded view
|

Re: Srcdeps in WildFly and WildFly Core

Brian Stansberry
In reply to this post by Peter Palaga
There’s been a lot of discussion overnight, but I’ll reply to this one directly since my answers better align with your questions here. :)

> On Jan 26, 2017, at 2:54 AM, Peter Palaga <[hidden email]> wrote:
>
> Hi Brian, thanks for your comments, more inline...
>
> On 2017-01-26 02:02, Brian Stansberry wrote:
>> My only concerns with this would relate to comitting this kind of src
>> dependency to the poms in the main branches in the widlfly/wildfly
>> and wildfly/wildfly-core repos. We’ve managed to survive up to now
>> with little or no need for that kind of thing, so until we get used
>> to using this in other ways IMHO we should follow the KISS principle
>> and forbid that.
>
> Maybe I overestimate the amount of changes that span over multiple git repos. Maybe you in the Core team do not do this often. But for us in the Sustaining Engineering Team, this is quite a typical situation. A substantial part of the reports from customers come with a description how to reproduce on the whole server, but they need to be fixed in a component. Having srcdeps would make the CP process simpler and faster, allowing us to uncover the conflicts and regressions earlier.

I don’t see how merging to the main branches is required to get this benefit. Git topic branches are fully sharable and CI jobs against them are easily done. All CI tests of pull requests are tests of topic branches.

But, in any case perhaps you’ve seen clear need for merging to the main branches with the EAP CP branches. I haven’t seen it in WildFly / WildFly Core. I deliberately used specific repo names in my last comment to try and scope it. ;)

Note I’m not saying we should disallow PRs with src deps in the pom. We should just disallow merging until those are replaced.

>
>> A trick is avoiding doing that by mistake; i.e. a PR is sent up with
>> a SRC dependency to get CI or review and accidentally gets merged.
>
> Oh, I am just realizing I have not said anything about merging. I actually do want to propose that commits with source dependencies get merged to e.g. wildfly-core master as early as possible. Those are the key points of Continuous Integration: get feedback quickly, and merge as soon as possible. This is exactly what Hawkular is doing since more than a year.

We regularly produce releases (ideally weekly for WildFly Core), often at short notice under pressure.  Allowing merging of changes that are not acceptable for release increases the risk and effort required to do that, since now we have to scan for src deps and figure out how to get them out of the build. Perhaps needing assistance from whoever added the src dep and the lead of relevant component, both of whom are on the other side of the world asleep. (This is a real issue since we often do releases on Friday afternoon US time or Monday morning European time.) We already have too much risk and effort doing releases so adding more will need a really strong justification.

>
>> But I suppose that’s not the end of the world, so long as the release
>> process will eventually detect it and fail.
>
> Yes, source dependencies on a stable branch do not harm. They just need to be avoided in releases (for which srcdeps offers technical means).

They do do harm as they mean the branch is no longer releasable. It’s not end-of-the-world harm but it’s harm.

>
>> Can making srcdeps fail (or just disabling it) be turned on via a
>> maven profile? With that we could set up such a profile and turn it
>> on in CI jobs that are testing branches where it’s forbidden (e.g.
>> the nightly builds of master.)
>
> Yes, the feature is called "failWith profiles" and can be configured in .mvn/srcdeps.yaml, like here in this srcdeps quickstart: https://github.com/srcdeps/srcdeps-maven/blob/master/srcdeps-maven-quickstarts/srcdeps-mvn-git-profile-quickstart/.mvn/srcdeps.yaml#L33 
> There is also "failWith properties" and "failWith goals". It is documented here: https://github.com/srcdeps/srcdeps-core/blob/master/doc/srcdeps.yaml#L130
> By default there is failWith: {goals: release:prepare, release:perform}. Projects that do not use the release plugin can set e.g. failWith: {goals: deploy:deploy} or whatever else distinguishes their releases.
>

Thanks.

>> Oh, one other concern — how robust is this in the face of poor
>> maintenance? I see a lot of boilerplate in that .mvn/srcdeps.yaml.
>
> Which parts are boilerpate?

All of it. :)

I’m not using that word as an attack. I’m just saying it’s extra text that needs to be maintained, and since it’s separate from the usual place similar text occurs (the poms) it is more likely to diverge.

>
>> If
>> that gets out of date or something is the only effect that using a
>> src dependency for the affected item doesn't work?
>
> Yes, I think so. As long as the .mvn/srcdeps.yaml file is syntactically correct, any misconfiguration there should not have any other effect than eventually breaking an embedded build.
>
> Generally, the things configured in .mvn/srcdeps.yaml tend to be quite stable - it is basically just mapping from GAVs to their respective git URLs. Git URLs do not change often. It is true that dependency artifacts come and go, but as long as their groupIds are selected reasonably (one groupId occurs in not more than one git repo) the mapping itself can be quite stable over time too.

Yeah, that’s true. Where this file would be more likely to go unmaintained is adding new entries or cleaning out old ones. But the latter is just noise and if the only harm of the former is a srcdep can’t be used for that lib, then that will naturally get handled by whoever wants to use the srcdep.

>
> Thanks,
>
> Peter
>
>>
>>> On Jan 25, 2017, at 3:45 PM, Peter Palaga <[hidden email]>
>>> wrote:
>>>
>>> Hi *,
>>>
>>> this is not new to those of you who attended my talk on the F2F
>>> 2016 in Brno. Let me explain the idea here again for all others who
>>> did not have a chance to be there.
>>>
>>> Srcdeps [1] is a tool to build Maven dependencies from their
>>> sources. With srcdeps, wildfly-core can depend on a specific commit
>>> of, e.g., undertow:
>>>
>>> <version.io.undertow>1.4.8.Final-SRC-revision-aabbccd</version.io.undertow>
>>>
>>>
>>>
> where aabbccd is the git commit id to build when any undertow artifact
>>> is requested during the build of wildfly-core.
>>>
>>> [1] describes in detail, how it works.
>>>
>>> The main advantage of srcdeps is that changes in components can be
>>> integrated and tested in wildfly-core immediately after they are
>>> committed to a public component branch. There is no need to wait
>>> for the component release.
>>>
>>> Here in the WildFly family of projects, it is often the case that
>>> something needs to be fixed in a component, but the verification
>>> (using bug reproducer, or integration test) is possible only at the
>>> level of wildfly or wildfly-core. Engineers typically work with
>>> snapshots locally, but when their changes need to get shared (CI,
>>> reviews) in a reproducible manner, snapshots cannot be used
>>> anymore. In such situations a source dependency come in handy: it
>>> is very easy to share and it is as reproducible as a Maven build
>>> from a specific commit can be. All CIs and reviewers can work with
>>> it, because all source dependency compilation is done under the
>>> hood by Maven.
>>>
>>> Developers working on changes that span over multiple
>>> interdependent git repos can thus get feedback (i-tests, reviews)
>>> quickly without waiting for releases of components.
>>>
>>> Srcdeps emerged in the Hawkular family of projects to solve exactly
>>> this kind of situation and is in use there since around October
>>> 2015.
>>>
>>> When I said there is no need to wait for releases of components, I
>>> did not mean that we can get rid of component releases altogether.
>>> Clearly, we cannot, because i.a. for any tooling uninformed about
>>> how srcdeps work, those source dependencies would simply be
>>> non-resolvable from public Maven repositories. So, before releasing
>>> the dependent component (such as wildfly-core) all its dependencies
>>> need to be released. To enforce this, srcdeps is by default
>>> configured to make the release fail, as long as there are source
>>> dependencies.
>>>
>>> I have sent a PR introducing srcdeps to wildfly-core:
>>> https://github.com/wildfly/wildfly-core/pull/2122 To get a feeling
>>> how it works, checkout the branch, switch to e.g.
>>> <version.io.undertow>1.4.8.Final-SRC-revision-1bff8c32f0eee986e83a7589ae95ebbc1d67d6bd</version.io.undertow>
>>> (that happens to be the commit id of the 1.4.8.Final tag) and
>>> build wildfly-core as usual with "mvn clean install". You'll see in
>>> the build log that undertow is being cloned to
>>> ~/.m2/srcdeps/io/undertow and that it is built there. After the
>>> build, check that the
>>> 1.4.8.Final-SRC-revision-1bff8c32f0eee986e83a7589ae95ebbc1d67d6bd
>>> version of Undertow got installed to your local Maven repo (usually
>>> ~/m2/repository/io/undertow/undertow-core )
>>>
>>> Are there any questions or comments?
>>>
>>> [1] https://github.com/srcdeps/srcdeps-maven#srcdeps-maven
>>>
>>> Thanks,
>>>
>>> Peter
>>>
>>> P.S.: I will be talking about srcdeps on Saturday 2017-01-28 at
>>> 14:30 at DevConf Brno.
>>> _______________________________________________ wildfly-dev mailing
>>> list [hidden email]
>>> https://lists.jboss.org/mailman/listinfo/wildfly-dev
>>
>

--
Brian Stansberry
Manager, Senior Principal Software Engineer
JBoss by Red Hat




_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev
Reply | Threaded
Open this post in threaded view
|

Re: Srcdeps in WildFly and WildFly Core

Peter Palaga
On 2017-01-26 15:44, Brian Stansberry wrote:

> There’s been a lot of discussion overnight, but I’ll reply to this one directly since my answers better align with your questions here. :)
>
>> On Jan 26, 2017, at 2:54 AM, Peter Palaga <[hidden email]> wrote:
>>
>> Hi Brian, thanks for your comments, more inline...
>>
>> On 2017-01-26 02:02, Brian Stansberry wrote:
>>> My only concerns with this would relate to comitting this kind of src
>>> dependency to the poms in the main branches in the widlfly/wildfly
>>> and wildfly/wildfly-core repos. We’ve managed to survive up to now
>>> with little or no need for that kind of thing, so until we get used
>>> to using this in other ways IMHO we should follow the KISS principle
>>> and forbid that.
>>
>> Maybe I overestimate the amount of changes that span over multiple git repos. Maybe you in the Core team do not do this often. But for us in the Sustaining Engineering Team, this is quite a typical situation. A substantial part of the reports from customers come with a description how to reproduce on the whole server, but they need to be fixed in a component. Having srcdeps would make the CP process simpler and faster, allowing us to uncover the conflicts and regressions earlier.
>
> I don’t see how merging to the main branches is required to get this benefit. Git topic branches are fully sharable and CI jobs against them are easily done. All CI tests of pull requests are tests of topic branches.

Yes, for me as the submitter of the PR, it is nice to get the feedback
from the CI and a review early, even before the component is released,
but it is quite bothersome to have to revisit the PR again once the
component gets released and rebase (in case there there are conflicts)
and either upgrade to the released component version or remove the
upgrade change (if the upgrade was merged separately).

As long as my PR is not merged, my changes are not binding for the rest
of the team. I want my PR to get merged as fast as possible and make
others care that their changes are compatible with mine. I want to
happily forget about the PR as soon as possible and pick a new task :)

> But, in any case perhaps you’ve seen clear need for merging to the main branches with the EAP CP branches. I haven’t seen it in WildFly / WildFly Core. I deliberately used specific repo names in my last comment to try and scope it. ;)

My reasons for merging there in EAP CP branches are the same as here in
the community branches: it is better for PR submitters to merge as early
as possible to avoid conflicts, subsequent PR edits and to keep the list
of open tasks short.

> Note I’m not saying we should disallow PRs with src deps in the pom. We should just disallow merging until those are replaced.

Yes, I understand that and I appreciate that. That would be a progress too.

>>> A trick is avoiding doing that by mistake; i.e. a PR is sent up with
>>> a SRC dependency to get CI or review and accidentally gets merged.
>>
>> Oh, I am just realizing I have not said anything about merging. I actually do want to propose that commits with source dependencies get merged to e.g. wildfly-core master as early as possible. Those are the key points of Continuous Integration: get feedback quickly, and merge as soon as possible. This is exactly what Hawkular is doing since more than a year.
>
> We regularly produce releases (ideally weekly for WildFly Core), often at short notice under pressure.  Allowing merging of changes that are not acceptable for release increases the risk and effort required to do that, since now we have to scan for src deps and figure out how to get them out of the build. Perhaps needing assistance from whoever added the src dep and the lead of relevant component, both of whom are on the other side of the world asleep. (This is a real issue since we often do releases on Friday afternoon US time or Monday morning European time.) We already have too much risk and effort doing releases so adding more will need a really strong justification.

This sounds as a valid concern. I must admit I know little about how you
plan and perform the releases of wildfly-core, wildfly and of the
components in the community. Knowing how complex the graph of WF
components is, I am far from underestimating any manual release efforts
or efforts to setup a CI jobs to do that automagically. I'll have to
gather more info about how you work.

>>> But I suppose that’s not the end of the world, so long as the release
>>> process will eventually detect it and fail.
>>
>> Yes, source dependencies on a stable branch do not harm. They just need to be avoided in releases (for which srcdeps offers technical means).
>
> They do do harm as they mean the branch is no longer releasable. It’s not end-of-the-world harm but it’s harm.

Well, I naivelly thought, that the components are obligated to provide a
release, say, one day before a planned wildfly-core release and send a
PRs that would then sweep out all source dependencies. And TBH, I did
not think "releasable at any time" is important in wildfly-core.
"Releasable once a week" still sounds good enough to me :)

>>> Can making srcdeps fail (or just disabling it) be turned on via a
>>> maven profile? With that we could set up such a profile and turn it
>>> on in CI jobs that are testing branches where it’s forbidden (e.g.
>>> the nightly builds of master.)
>>
>> Yes, the feature is called "failWith profiles" and can be configured in .mvn/srcdeps.yaml, like here in this srcdeps quickstart: https://github.com/srcdeps/srcdeps-maven/blob/master/srcdeps-maven-quickstarts/srcdeps-mvn-git-profile-quickstart/.mvn/srcdeps.yaml#L33
>> There is also "failWith properties" and "failWith goals". It is documented here: https://github.com/srcdeps/srcdeps-core/blob/master/doc/srcdeps.yaml#L130
>> By default there is failWith: {goals: release:prepare, release:perform}. Projects that do not use the release plugin can set e.g. failWith: {goals: deploy:deploy} or whatever else distinguishes their releases.
>>
>
> Thanks.
>
>>> Oh, one other concern — how robust is this in the face of poor
>>> maintenance? I see a lot of boilerplate in that .mvn/srcdeps.yaml.
>>
>> Which parts are boilerpate?
>
> All of it. :)
>
> I’m not using that word as an attack. I’m just saying it’s extra text that needs to be maintained, and since it’s separate from the usual place similar text occurs (the poms) it is more likely to diverge.

OK, now I know what you mean :) You are right that poms can diverge from
srcdeps.yaml.

>>> If
>>> that gets out of date or something is the only effect that using a
>>> src dependency for the affected item doesn't work?
>>
>> Yes, I think so. As long as the .mvn/srcdeps.yaml file is syntactically correct, any misconfiguration there should not have any other effect than eventually breaking an embedded build.
>>
>> Generally, the things configured in .mvn/srcdeps.yaml tend to be quite stable - it is basically just mapping from GAVs to their respective git URLs. Git URLs do not change often. It is true that dependency artifacts come and go, but as long as their groupIds are selected reasonably (one groupId occurs in not more than one git repo) the mapping itself can be quite stable over time too.
>
> Yeah, that’s true. Where this file would be more likely to go unmaintained is adding new entries or cleaning out old ones. But the latter is just noise and if the only harm of the former is a srcdep can’t be used for that lib, then that will naturally get handled by whoever wants to use the srcdep.

Yes, exactly.

Thanks,

Peter

>>
>> Thanks,
>>
>> Peter
>>
>>>
>>>> On Jan 25, 2017, at 3:45 PM, Peter Palaga <[hidden email]>
>>>> wrote:
>>>>
>>>> Hi *,
>>>>
>>>> this is not new to those of you who attended my talk on the F2F
>>>> 2016 in Brno. Let me explain the idea here again for all others who
>>>> did not have a chance to be there.
>>>>
>>>> Srcdeps [1] is a tool to build Maven dependencies from their
>>>> sources. With srcdeps, wildfly-core can depend on a specific commit
>>>> of, e.g., undertow:
>>>>
>>>> <version.io.undertow>1.4.8.Final-SRC-revision-aabbccd</version.io.undertow>
>>>>
>>>>
>>>>
>> where aabbccd is the git commit id to build when any undertow artifact
>>>> is requested during the build of wildfly-core.
>>>>
>>>> [1] describes in detail, how it works.
>>>>
>>>> The main advantage of srcdeps is that changes in components can be
>>>> integrated and tested in wildfly-core immediately after they are
>>>> committed to a public component branch. There is no need to wait
>>>> for the component release.
>>>>
>>>> Here in the WildFly family of projects, it is often the case that
>>>> something needs to be fixed in a component, but the verification
>>>> (using bug reproducer, or integration test) is possible only at the
>>>> level of wildfly or wildfly-core. Engineers typically work with
>>>> snapshots locally, but when their changes need to get shared (CI,
>>>> reviews) in a reproducible manner, snapshots cannot be used
>>>> anymore. In such situations a source dependency come in handy: it
>>>> is very easy to share and it is as reproducible as a Maven build
>>>> from a specific commit can be. All CIs and reviewers can work with
>>>> it, because all source dependency compilation is done under the
>>>> hood by Maven.
>>>>
>>>> Developers working on changes that span over multiple
>>>> interdependent git repos can thus get feedback (i-tests, reviews)
>>>> quickly without waiting for releases of components.
>>>>
>>>> Srcdeps emerged in the Hawkular family of projects to solve exactly
>>>> this kind of situation and is in use there since around October
>>>> 2015.
>>>>
>>>> When I said there is no need to wait for releases of components, I
>>>> did not mean that we can get rid of component releases altogether.
>>>> Clearly, we cannot, because i.a. for any tooling uninformed about
>>>> how srcdeps work, those source dependencies would simply be
>>>> non-resolvable from public Maven repositories. So, before releasing
>>>> the dependent component (such as wildfly-core) all its dependencies
>>>> need to be released. To enforce this, srcdeps is by default
>>>> configured to make the release fail, as long as there are source
>>>> dependencies.
>>>>
>>>> I have sent a PR introducing srcdeps to wildfly-core:
>>>> https://github.com/wildfly/wildfly-core/pull/2122 To get a feeling
>>>> how it works, checkout the branch, switch to e.g.
>>>> <version.io.undertow>1.4.8.Final-SRC-revision-1bff8c32f0eee986e83a7589ae95ebbc1d67d6bd</version.io.undertow>
>>>> (that happens to be the commit id of the 1.4.8.Final tag) and
>>>> build wildfly-core as usual with "mvn clean install". You'll see in
>>>> the build log that undertow is being cloned to
>>>> ~/.m2/srcdeps/io/undertow and that it is built there. After the
>>>> build, check that the
>>>> 1.4.8.Final-SRC-revision-1bff8c32f0eee986e83a7589ae95ebbc1d67d6bd
>>>> version of Undertow got installed to your local Maven repo (usually
>>>> ~/m2/repository/io/undertow/undertow-core )
>>>>
>>>> Are there any questions or comments?
>>>>
>>>> [1] https://github.com/srcdeps/srcdeps-maven#srcdeps-maven
>>>>
>>>> Thanks,
>>>>
>>>> Peter
>>>>
>>>> P.S.: I will be talking about srcdeps on Saturday 2017-01-28 at
>>>> 14:30 at DevConf Brno.
>>>> _______________________________________________ wildfly-dev mailing
>>>> list [hidden email]
>>>> https://lists.jboss.org/mailman/listinfo/wildfly-dev
>>>
>>
>

_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev
Reply | Threaded
Open this post in threaded view
|

Re: Srcdeps in WildFly and WildFly Core

Brian Stansberry

> On Jan 26, 2017, at 10:28 AM, Peter Palaga <[hidden email]> wrote:
>
> On 2017-01-26 15:44, Brian Stansberry wrote:
>> There’s been a lot of discussion overnight, but I’ll reply to this one directly since my answers better align with your questions here. :)
>>
>>> On Jan 26, 2017, at 2:54 AM, Peter Palaga <[hidden email]> wrote:
>>>
>>> Hi Brian, thanks for your comments, more inline...
>>>
>>> On 2017-01-26 02:02, Brian Stansberry wrote:
>>>> My only concerns with this would relate to comitting this kind of src
>>>> dependency to the poms in the main branches in the widlfly/wildfly
>>>> and wildfly/wildfly-core repos. We’ve managed to survive up to now
>>>> with little or no need for that kind of thing, so until we get used
>>>> to using this in other ways IMHO we should follow the KISS principle
>>>> and forbid that.
>>>
>>> Maybe I overestimate the amount of changes that span over multiple git repos. Maybe you in the Core team do not do this often. But for us in the Sustaining Engineering Team, this is quite a typical situation. A substantial part of the reports from customers come with a description how to reproduce on the whole server, but they need to be fixed in a component. Having srcdeps would make the CP process simpler and faster, allowing us to uncover the conflicts and regressions earlier.
>>
>> I don’t see how merging to the main branches is required to get this benefit. Git topic branches are fully sharable and CI jobs against them are easily done. All CI tests of pull requests are tests of topic branches.
>
> Yes, for me as the submitter of the PR, it is nice to get the feedback from the CI and a review early, even before the component is released, but it is quite bothersome to have to revisit the PR again once the component gets released and rebase (in case there there are conflicts) and either upgrade to the released component version or remove the upgrade change (if the upgrade was merged separately).
>
> As long as my PR is not merged, my changes are not binding for the rest of the team. I want my PR to get merged as fast as possible and make others care that their changes are compatible with mine. I want to happily forget about the PR as soon as possible and pick a new task :)
>

You’ve convinced me! Convinced me that we shouldn’t allow this. :D

We don’t have a role analogous to the “release coordinator” used with EAP CPs, i.e. someone whose primary responsibility is coordinating to make sure that the untidy pieces get tidied. Most of our non-CR/Final releases are done as side tasks by people who are stealing time from other tasks. They need to be simple and mechanical. We also have a far greater volume of changes to manage than EAP CPs do. A process based on merging half the necessary change and then letting the issue owner walk away and assume someone else is going to come tidy up is a recipe for disaster.

>> But, in any case perhaps you’ve seen clear need for merging to the main branches with the EAP CP branches. I haven’t seen it in WildFly / WildFly Core. I deliberately used specific repo names in my last comment to try and scope it. ;)
>
> My reasons for merging there in EAP CP branches are the same as here in the community branches: it is better for PR submitters to merge as early as possible to avoid conflicts, subsequent PR edits and to keep the list of open tasks short.
>
>> Note I’m not saying we should disallow PRs with src deps in the pom. We should just disallow merging until those are replaced.
>
> Yes, I understand that and I appreciate that. That would be a progress too.
>
>>>> A trick is avoiding doing that by mistake; i.e. a PR is sent up with
>>>> a SRC dependency to get CI or review and accidentally gets merged.
>>>
>>> Oh, I am just realizing I have not said anything about merging. I actually do want to propose that commits with source dependencies get merged to e.g. wildfly-core master as early as possible. Those are the key points of Continuous Integration: get feedback quickly, and merge as soon as possible. This is exactly what Hawkular is doing since more than a year.
>>
>> We regularly produce releases (ideally weekly for WildFly Core), often at short notice under pressure.  Allowing merging of changes that are not acceptable for release increases the risk and effort required to do that, since now we have to scan for src deps and figure out how to get them out of the build. Perhaps needing assistance from whoever added the src dep and the lead of relevant component, both of whom are on the other side of the world asleep. (This is a real issue since we often do releases on Friday afternoon US time or Monday morning European time.) We already have too much risk and effort doing releases so adding more will need a really strong justification.
>
> This sounds as a valid concern. I must admit I know little about how you plan and perform the releases of wildfly-core, wildfly and of the components in the community. Knowing how complex the graph of WF components is, I am far from underestimating any manual release efforts or efforts to setup a CI jobs to do that automagically. I'll have to gather more info about how you work.
>
>>>> But I suppose that’s not the end of the world, so long as the release
>>>> process will eventually detect it and fail.
>>>
>>> Yes, source dependencies on a stable branch do not harm. They just need to be avoided in releases (for which srcdeps offers technical means).
>>
>> They do do harm as they mean the branch is no longer releasable. It’s not end-of-the-world harm but it’s harm.
>
> Well, I naivelly thought, that the components are obligated to provide a release, say, one day before a planned wildfly-core release and send a PRs that would then sweep out all source dependencies. And TBH, I did not think "releasable at any time" is important in wildfly-core. "Releasable once a week" still sounds good enough to me :)

Unfortunately, it’s not.

>
>>>> Can making srcdeps fail (or just disabling it) be turned on via a
>>>> maven profile? With that we could set up such a profile and turn it
>>>> on in CI jobs that are testing branches where it’s forbidden (e.g.
>>>> the nightly builds of master.)
>>>
>>> Yes, the feature is called "failWith profiles" and can be configured in .mvn/srcdeps.yaml, like here in this srcdeps quickstart: https://github.com/srcdeps/srcdeps-maven/blob/master/srcdeps-maven-quickstarts/srcdeps-mvn-git-profile-quickstart/.mvn/srcdeps.yaml#L33
>>> There is also "failWith properties" and "failWith goals". It is documented here: https://github.com/srcdeps/srcdeps-core/blob/master/doc/srcdeps.yaml#L130
>>> By default there is failWith: {goals: release:prepare, release:perform}. Projects that do not use the release plugin can set e.g. failWith: {goals: deploy:deploy} or whatever else distinguishes their releases.
>>>
>>
>> Thanks.
>>
>>>> Oh, one other concern — how robust is this in the face of poor
>>>> maintenance? I see a lot of boilerplate in that .mvn/srcdeps.yaml.
>>>
>>> Which parts are boilerpate?
>>
>> All of it. :)
>>
>> I’m not using that word as an attack. I’m just saying it’s extra text that needs to be maintained, and since it’s separate from the usual place similar text occurs (the poms) it is more likely to diverge.
>
> OK, now I know what you mean :) You are right that poms can diverge from srcdeps.yaml.
>
>>>> If
>>>> that gets out of date or something is the only effect that using a
>>>> src dependency for the affected item doesn't work?
>>>
>>> Yes, I think so. As long as the .mvn/srcdeps.yaml file is syntactically correct, any misconfiguration there should not have any other effect than eventually breaking an embedded build.
>>>
>>> Generally, the things configured in .mvn/srcdeps.yaml tend to be quite stable - it is basically just mapping from GAVs to their respective git URLs. Git URLs do not change often. It is true that dependency artifacts come and go, but as long as their groupIds are selected reasonably (one groupId occurs in not more than one git repo) the mapping itself can be quite stable over time too.
>>
>> Yeah, that’s true. Where this file would be more likely to go unmaintained is adding new entries or cleaning out old ones. But the latter is just noise and if the only harm of the former is a srcdep can’t be used for that lib, then that will naturally get handled by whoever wants to use the srcdep.
>
> Yes, exactly.
>
> Thanks,
>
> Peter
>
>>>
>>> Thanks,
>>>
>>> Peter
>>>
>>>>
>>>>> On Jan 25, 2017, at 3:45 PM, Peter Palaga <[hidden email]>
>>>>> wrote:
>>>>>
>>>>> Hi *,
>>>>>
>>>>> this is not new to those of you who attended my talk on the F2F
>>>>> 2016 in Brno. Let me explain the idea here again for all others who
>>>>> did not have a chance to be there.
>>>>>
>>>>> Srcdeps [1] is a tool to build Maven dependencies from their
>>>>> sources. With srcdeps, wildfly-core can depend on a specific commit
>>>>> of, e.g., undertow:
>>>>>
>>>>> <version.io.undertow>1.4.8.Final-SRC-revision-aabbccd</version.io.undertow>
>>>>>
>>>>>
>>>>>
>>> where aabbccd is the git commit id to build when any undertow artifact
>>>>> is requested during the build of wildfly-core.
>>>>>
>>>>> [1] describes in detail, how it works.
>>>>>
>>>>> The main advantage of srcdeps is that changes in components can be
>>>>> integrated and tested in wildfly-core immediately after they are
>>>>> committed to a public component branch. There is no need to wait
>>>>> for the component release.
>>>>>
>>>>> Here in the WildFly family of projects, it is often the case that
>>>>> something needs to be fixed in a component, but the verification
>>>>> (using bug reproducer, or integration test) is possible only at the
>>>>> level of wildfly or wildfly-core. Engineers typically work with
>>>>> snapshots locally, but when their changes need to get shared (CI,
>>>>> reviews) in a reproducible manner, snapshots cannot be used
>>>>> anymore. In such situations a source dependency come in handy: it
>>>>> is very easy to share and it is as reproducible as a Maven build
>>>>> from a specific commit can be. All CIs and reviewers can work with
>>>>> it, because all source dependency compilation is done under the
>>>>> hood by Maven.
>>>>>
>>>>> Developers working on changes that span over multiple
>>>>> interdependent git repos can thus get feedback (i-tests, reviews)
>>>>> quickly without waiting for releases of components.
>>>>>
>>>>> Srcdeps emerged in the Hawkular family of projects to solve exactly
>>>>> this kind of situation and is in use there since around October
>>>>> 2015.
>>>>>
>>>>> When I said there is no need to wait for releases of components, I
>>>>> did not mean that we can get rid of component releases altogether.
>>>>> Clearly, we cannot, because i.a. for any tooling uninformed about
>>>>> how srcdeps work, those source dependencies would simply be
>>>>> non-resolvable from public Maven repositories. So, before releasing
>>>>> the dependent component (such as wildfly-core) all its dependencies
>>>>> need to be released. To enforce this, srcdeps is by default
>>>>> configured to make the release fail, as long as there are source
>>>>> dependencies.
>>>>>
>>>>> I have sent a PR introducing srcdeps to wildfly-core:
>>>>> https://github.com/wildfly/wildfly-core/pull/2122 To get a feeling
>>>>> how it works, checkout the branch, switch to e.g.
>>>>> <version.io.undertow>1.4.8.Final-SRC-revision-1bff8c32f0eee986e83a7589ae95ebbc1d67d6bd</version.io.undertow>
>>>>> (that happens to be the commit id of the 1.4.8.Final tag) and
>>>>> build wildfly-core as usual with "mvn clean install". You'll see in
>>>>> the build log that undertow is being cloned to
>>>>> ~/.m2/srcdeps/io/undertow and that it is built there. After the
>>>>> build, check that the
>>>>> 1.4.8.Final-SRC-revision-1bff8c32f0eee986e83a7589ae95ebbc1d67d6bd
>>>>> version of Undertow got installed to your local Maven repo (usually
>>>>> ~/m2/repository/io/undertow/undertow-core )
>>>>>
>>>>> Are there any questions or comments?
>>>>>
>>>>> [1] https://github.com/srcdeps/srcdeps-maven#srcdeps-maven
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Peter
>>>>>
>>>>> P.S.: I will be talking about srcdeps on Saturday 2017-01-28 at
>>>>> 14:30 at DevConf Brno.
>>>>> _______________________________________________ wildfly-dev mailing
>>>>> list [hidden email]
>>>>> https://lists.jboss.org/mailman/listinfo/wildfly-dev
>>>>
>>>
>>
>

--
Brian Stansberry
Manager, Senior Principal Software Engineer
JBoss by Red Hat




_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev
Reply | Threaded
Open this post in threaded view
|

Re: Srcdeps in WildFly and WildFly Core

Peter Palaga
In reply to this post by Darran Lofthouse
Hi Darran, inline...

On 2017-01-26 13:56, Darran Lofthouse wrote:

>
>
> On 26/01/17 12:42, Peter Palaga wrote:
>> Thanks for the comments, Darran, more inline...
>>
>> On 2017-01-26 12:40, Darran Lofthouse wrote:
>>> On 26/01/17 01:02, Brian Stansberry wrote:
>>>> My only concerns with this would relate to comitting this kind of src
>>>> dependency to the poms in the main branches in the widlfly/wildfly
>>>> and wildfly/wildfly-core repos. We’ve managed to survive up to now
>>>> with little or no need for that kind of thing, so until we get used
>>>> to using this in other ways IMHO we should follow the KISS principle
>>>> and forbid that.
>>>
>>> I like the idea of this being available for teams working on a topic
>>> branch but also think we should avoid this kind of dependency in master
>>> - engineers are continually running builds so I don't think we should
>>> add additional source checkouts and builds.
>>
>> So you also vote for having the support for source dependencies when the
>> e.g. the wildfly-core CI builds a PR that integrates a change in Elytron
>> or when a reviewer does the same, but such PRs should wait for a proper
>> release of the component, right?
>
> I think by the time we get to the PR stage engineers should have had an
> opportunity to discuss the proposed solution sufficiently that we
> shouldn't need a srcdep and should have been able to tag Elytron.

OK, thanks for explaining!

>>> I know Elytron has caused huge amounts of instability but generally I
>>> think we want these branches in a state where we could tag if we
>>> needed to.
>>
>> "these branches" - you mean topic branches in a component (such as
>> Elytron) or in a consuming project (such as WF Core) or both?
>
> I think WildFly Core / master and WildFly / master are generally not
> supposed to be as unstable as we have made them recently.
>
>>> When these downloaded builds are executed are the test cases within also
>>> executed?
>>
>> The builds of source dependencies run with -DskipTests by default so
>> that they finish faster. An explicit per git repository setting in
>> srcdeps.yaml is needed if somebody wants a dependency build to run
>> without -DskipTests.
>>
>>>> A trick is avoiding doing that by mistake; i.e. a PR is sent up with
>>>> a SRC dependency to get CI or review and accidentally gets merged.
>>>> But I suppose that’s not the end of the world, so long as the release
>>>> process will eventually detect it and fail.
>>>
>>> Merging topic branches these temporary SHAs would be in the history but
>>> maybe the PR CI runs could verify they are not present in the final
>>> merge.
>>>
>>>> Can making srcdeps fail (or just disabling it) be turned on via a
>>>> maven profile? With that we could set up such a profile and turn it
>>>> on in CI jobs that are testing branches where it’s forbidden (e.g.
>>>> the nightly builds of master.)
>>>>
>>>> Oh, one other concern — how robust is this in the face of poor
>>>> maintenance? I see a lot of boilerplate in that .mvn/srcdeps.yaml. If
>>>> that gets out of date or something is the only effect that using a
>>>> src dependency for the affected item doesn't work?
>>>
>>> I think that file would be better almost empty
>>
>> The size of the srcdeps.yaml file depends on how many dependencies the
>> given project has and which of those are selected for the srcdeps
>> support. In the PR https://git.io/vMjUC , I have included only the
>> projects owned by us, excluding non-Maven and non-git projects. I am
>> open to discussion whether we should include or exclude more projects.
>>
>>> but not hidden.
>>
>> The .mvn directory was introduced by Maven 3.3.1. Maven reads
>> extensions.xml file from there and I found it rather natural to have
>> srcdeps.yaml file there too.
>
> What that means is that a file that can directly influence the outcome
> of a local build is hidden from the engineer.  There are already quite a
> few things in out builds where when they go wrong we have to spend time
> tracking them down, having a hidden configuration may not help this.

If I heart this earlier, I'd most probably choose a non-hidden location.
I perhaps do not see this as a problem, because I configure each of my
Eclipse workspaces to show all .* resources because .mvn is not the only
important file that is hidden: there is also .gitignore, .gitattributes,
.travis.yml, .appveyor.yml and even .git and others that one wants to
open from time to time.
But now that srcdeps has some users, sorry, I do not think I find this
important enough to break backwards compatibility.

-- P

>>> Thinking about the topic branch development, within the Elytron team we
>>> are using some SNAPSHOTs from the official repos and at times we have
>>> forked projects into our incubator and developed against SNAPSHOTs from
>>> those
>>
>> Allowing SNAPSHOTs on any remote Maven repository leads to loosing
>> reproducibility of the builds of the dependent project. Srcdeps offers
>> much more reproducibility and portalbility over environments in such
>> situations.
>
> Yes I am saying in the development of the Elytron topic branches this
> could help a lot - but the assumptions as to which git repos we are
> using would be wrong.
>
>>> as as you select the SHA for the build you would also need to
>>> select which repository you really want.
>>
>> Yes, the mapping which dependencies are built from which git
>> repositories is defined in srcdeps.yaml. I see no problem in having one
>> git URL in one branch and having different git URL in another branch.
>
> Which I think as this is a file engineers could be editing to tweak
> their builds means it should not be hidden.
 >

>> Thanks,
>>
>> Peter
>>
>>>>> On Jan 25, 2017, at 3:45 PM, Peter Palaga <[hidden email]> wrote:
>>>>>
>>>>> Hi *,
>>>>>
>>>>> this is not new to those of you who attended my talk on the F2F
>>>>> 2016 in
>>>>> Brno. Let me explain the idea here again for all others who did not
>>>>> have
>>>>> a chance to be there.
>>>>>
>>>>> Srcdeps [1] is a tool to build Maven dependencies from their sources.
>>>>> With srcdeps, wildfly-core can depend on a specific commit of, e.g.,
>>>>> undertow:
>>>>>
>>>>> <version.io.undertow>1.4.8.Final-SRC-revision-aabbccd</version.io.undertow>
>>>>>
>>>>>
>>>>>
>>>>> where aabbccd is the git commit id to build when any undertow artifact
>>>>> is requested during the build of wildfly-core.
>>>>>
>>>>> [1] describes in detail, how it works.
>>>>>
>>>>> The main advantage of srcdeps is that changes in components can be
>>>>> integrated and tested in wildfly-core immediately after they are
>>>>> committed to a public component branch. There is no need to wait for
>>>>> the
>>>>> component release.
>>>>>
>>>>> Here in the WildFly family of projects, it is often the case that
>>>>> something needs to be fixed in a component, but the verification
>>>>> (using
>>>>> bug reproducer, or integration test) is possible only at the level of
>>>>> wildfly or wildfly-core. Engineers typically work with snapshots
>>>>> locally, but when their changes need to get shared (CI, reviews) in a
>>>>> reproducible manner, snapshots cannot be used anymore.
>>>>> In such situations a source dependency come in handy: it is very
>>>>> easy to
>>>>> share and it is as reproducible as a Maven build from a specific
>>>>> commit
>>>>> can be. All CIs and reviewers can work with it, because all source
>>>>> dependency compilation is done under the hood by Maven.
>>>>>
>>>>> Developers working on changes that span over multiple interdependent
>>>>> git
>>>>> repos can thus get feedback (i-tests, reviews) quickly without waiting
>>>>> for releases of components.
>>>>>
>>>>> Srcdeps emerged in the Hawkular family of projects to solve exactly
>>>>> this
>>>>> kind of situation and is in use there since around October 2015.
>>>>>
>>>>> When I said there is no need to wait for releases of components, I did
>>>>> not mean that we can get rid of component releases altogether.
>>>>> Clearly,
>>>>> we cannot, because i.a. for any tooling uninformed about how srcdeps
>>>>> work, those source dependencies would simply be non-resolvable from
>>>>> public Maven repositories. So, before releasing the dependent
>>>>> component
>>>>> (such as wildfly-core) all its dependencies need to be released. To
>>>>> enforce this, srcdeps is by default configured to make the release
>>>>> fail,
>>>>> as long as there are source dependencies.
>>>>>
>>>>> I have sent a PR introducing srcdeps to wildfly-core:
>>>>> https://github.com/wildfly/wildfly-core/pull/2122
>>>>> To get a feeling how it works, checkout the branch, switch to e.g.
>>>>> <version.io.undertow>1.4.8.Final-SRC-revision-1bff8c32f0eee986e83a7589ae95ebbc1d67d6bd</version.io.undertow>
>>>>>
>>>>>
>>>>> (that happens to be the commit id of the 1.4.8.Final tag)
>>>>> and build wildfly-core as usual with "mvn clean install". You'll
>>>>> see in
>>>>> the build log that undertow is being cloned to
>>>>> ~/.m2/srcdeps/io/undertow
>>>>> and that it is built there. After the build, check that the
>>>>> 1.4.8.Final-SRC-revision-1bff8c32f0eee986e83a7589ae95ebbc1d67d6bd
>>>>> version of Undertow got installed to your local Maven repo (usually
>>>>> ~/m2/repository/io/undertow/undertow-core )
>>>>>
>>>>> Are there any questions or comments?
>>>>>
>>>>> [1] https://github.com/srcdeps/srcdeps-maven#srcdeps-maven
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Peter
>>>>>
>>>>> P.S.: I will be talking about srcdeps on Saturday 2017-01-28 at
>>>>> 14:30 at
>>>>> DevConf Brno.
>>>>> _______________________________________________
>>>>> wildfly-dev mailing list
>>>>> [hidden email]
>>>>> https://lists.jboss.org/mailman/listinfo/wildfly-dev
>>>>
>>> _______________________________________________
>>> wildfly-dev mailing list
>>> [hidden email]
>>> https://lists.jboss.org/mailman/listinfo/wildfly-dev
>>>
>>

_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev
12