Bill of Materials

classic Classic list List threaded Threaded
24 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Bill of Materials

Carlos Sanchez
Hi there, 

I have filed a new JEP to address the concept of Bill of Materials (BoM).


The BoM idea came up after different conversations with Tyler/KK/Oleg and several more people - thanks for the feedback! - talking about how can we define a Jenkins package ("classic" jenkins, essentials, custom) to ensure that latest master of core and plugins can be easily used and tested, as well as the too typical use case of PRs in progress across multiple modules.

The main goal is to increase velocity by providing developers automatic artifacts and the result of tests against those artifacts before these combinations of PRs are merged. In order to create some safety nets that encourage contributions and facilitate reviews.

Please share your feedback

Thanks!

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CALHFn6PNnYTpF-0mnr0_pee9yw0tYaX%3D%2B6akgBNp%3DXnMgToOrQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Bill of Materials

R. Tyler Croy
(replies inline)

On Fri, 20 Apr 2018, Carlos Sanchez wrote:

> Hi there,
>
> I have filed a new JEP to address the concept of Bill of Materials (BoM).
>
> *https://github.com/jenkinsci/jep/pull/92
> <https://github.com/jenkinsci/jep/pull/92>*
>
> The BoM idea came up after different conversations with Tyler/KK/Oleg and
> several more people - thanks for the feedback! - talking about how can we
> define a Jenkins package ("classic" jenkins, essentials, custom) to ensure
> that latest master of core and plugins can be easily used and tested, as
> well as the too typical use case of PRs in progress across multiple modules.
>
> The main goal is to increase velocity by providing developers automatic
> artifacts and the result of tests against those artifacts before these
> combinations of PRs are merged. In order to create some safety nets that
> encourage contributions and facilitate reviews.

There have been a lot of comments on that pull request, but I wanted to bring
one aspect of the discussion back to the development mailing list.

As I have been working this past week with jglick on the incrementals
publishing, I'm recognizing that the yaml we sketched out previously requires
some logic to be re-implemented in a few different places.

Take the following snippet for example:

  plugins:
    - groupId: org.jenkins-ci.plugins
      artifactId: git
      ref: PR-1663 # or how are pull requests handled here?


Whether it's the Jenkins Essentials tooling, custom-war-packager, or any
test/maven tooling, something is going to have to translate those GAV
coordinates into an actual artifact which is either downloaded, or locally
referenced.

In an incrementals build, a pull-request built Git plugin artifact would actually be
located:
    https://repo.jenkins-ci.org/incrementals/org/jenkins-ci/plugins/git/3.8.1-rc1663.454c4a5e4d30/git-3.8.1-rc1663.454c4a5e4d30.hpi

I'm not sure how I would build the tooling to use the above YAML to resolve to
the URL, which makes me wonder: why not just have the full artifact URL listed
in the Bill of Materials? For example:

  plugins:
    - groupId: org.jenkins-ci.plugins
      artifactId: git
      url: https://repo.jenkins-ci.org/incrementals/org/jenkins-ci/plugins/git/3.8.1-rc1663.454c4a5e4d30/git-3.8.1-rc1663.454c4a5e4d30.hpi


From the Jenkins Essentials perspective, all I really need is an artifact ID
and URL for the artifact, but I'm not sure if that's all that's needed for the
other use-cases here.


Just thought I would raise the question since I don't feel like I understand
how to go from the Bill of Materials to binaries right now.



Cheers
-R Tyler Croy




--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/20180427165458.GG2935%40grape.lasagna.io.
For more options, visit https://groups.google.com/d/optout.

signature.asc (201 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Bill of Materials

Jesse Glick-4
On Fri, Apr 27, 2018 at 12:54 PM, R. Tyler Croy <[hidden email]> wrote:

> why not just have the full artifact URL listed
> in the Bill of Materials? For example:
>
>   plugins:
>     - groupId: org.jenkins-ci.plugins
>       artifactId: git
>       url: https://repo.jenkins-ci.org/incrementals/org/jenkins-ci/plugins/git/3.8.1-rc1663.454c4a5e4d30/git-3.8.1-rc1663.454c4a5e4d30.hpi
>
> From the Jenkins Essentials perspective, all I really need is an artifact ID
> and URL for the artifact, but I'm not sure if that's all that's needed for the
> other use-cases here.

You could do that, though a plain `version` attribute would suffice,
so long as the tool pulling binaries includes the Incrementals repo in
its list of locations. _If_ you wish to permit branch references or
local disk paths or other options in this format, then you would
anyway need some alternative ways of referring to binaries.

The more interesting question to my mind is how you make edits to the
manifest like the above. After you have more than a handful of plugins
with versions like these, manually copying and pasting such
URLs/versions would be cumbersome. I think JENKINS-50953 can be used
to track this. For example, I might run something like

$ update-incrementals configuration/essentials.yaml

or

$ update-incrementals configuration/essentials.yaml git workflow-api

which for each selected component (or all) would check the current
version; gather a list of all deployed versions (from
`maven_metadata.xml`) that compare as strictly “newer” than that;
filter those by commits which are ancestors of current `master`; pick
the newest of the resulting versions; and edit the file to select that
version.

You could then commit the result and push to a pull request, which
would get validated by a PR builder somehow TBD, and if all automated
checks pass it would be eligible for merging, meaning deployment to
production.

WDYT? Admittedly the above is a very rough sketch, which I need to
flesh out quite a bit.

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CANfRfr1YDTcmNQmgp1yb7nE3MmO6uFiEY1%3DTaAkFcf_gMBqtRw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Bill of Materials

R. Tyler Croy
(replies inline)

On Mon, 30 Apr 2018, Jesse Glick wrote:

> On Fri, Apr 27, 2018 at 12:54 PM, R. Tyler Croy <[hidden email]> wrote:
> > why not just have the full artifact URL listed
> > in the Bill of Materials? For example:
> >
> >   plugins:
> >     - groupId: org.jenkins-ci.plugins
> >       artifactId: git
> >       url: https://repo.jenkins-ci.org/incrementals/org/jenkins-ci/plugins/git/3.8.1-rc1663.454c4a5e4d30/git-3.8.1-rc1663.454c4a5e4d30.hpi
> >
> > From the Jenkins Essentials perspective, all I really need is an artifact ID
> > and URL for the artifact, but I'm not sure if that's all that's needed for the
> > other use-cases here.
>
> You could do that, though a plain `version` attribute would suffice,
> so long as the tool pulling binaries includes the Incrementals repo in
> its list of locations. _If_ you wish to permit branch references or
> local disk paths or other options in this format, then you would
> anyway need some alternative ways of referring to binaries.
>
> The more interesting question to my mind is how you make edits to the
> manifest like the above. After you have more than a handful of plugins
> with versions like these, manually copying and pasting such
> URLs/versions would be cumbersome. I think JENKINS-50953 can be used
> to track this. For example, I might run something like
>
> $ update-incrementals configuration/essentials.yaml
>
> or
>
> $ update-incrementals configuration/essentials.yaml git workflow-api
>
> which for each selected component (or all) would check the current
> version; gather a list of all deployed versions (from
> `maven_metadata.xml`) that compare as strictly ???newer??? than that;
> filter those by commits which are ancestors of current `master`; pick
> the newest of the resulting versions; and edit the file to select that
> version.
>
> You could then commit the result and push to a pull request, which
> would get validated by a PR builder somehow TBD, and if all automated
> checks pass it would be eligible for merging, meaning deployment to
> production.
>
> WDYT? Admittedly the above is a very rough sketch, which I need to
> flesh out quite a bit.


If an update-incrementals tool existed, then yes, that would address my
concerns here. So long as something can automatically update my yaml file, i'm
happy to use it, I more wanted to make sure that we weren't going to have two
or three implementations of what `update-incrementals` effectively should be
providing.


--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/20180430204749.GL2935%40grape.lasagna.io.
For more options, visit https://groups.google.com/d/optout.

signature.asc (201 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Bill of Materials

Jesse Glick-4
On Mon, Apr 30, 2018 at 4:47 PM, R. Tyler Croy <[hidden email]> wrote:
> If an update-incrementals tool existed, then yes, that would address my
> concerns here.

OK, good to know I am not completely off base here.

> I more wanted to make sure that we weren't going to have two
> or three implementations of what `update-incrementals` effectively should be
> providing.

There might be two or three scripts to deal with different syntaxes;
for example, it would be useful to have an equivalent to `mvn
versions:use-latest-releases` which grokked Incrementals and updated a
`pom.xml`. But yes they should share an underlying library of some
kind.

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CANfRfr3wTjrORgYGbhMgUJZ1REqEub82sJZvkZNRN_DN9V4r1Q%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Bill of Materials

Oleg Nenashev
Hi all,

Just to provide an update on this topic, I have added support of Bill of Materials (in the current draft version) to Custom WAR Packager and used it in several automation flows. E.g. see Integration Testing for Artifact Manager S3. I can also rework Remoting (PR) and Stapler (PR) flows to use BOM there if it is needed for a reference implementation.

Do I gravely need BOM in these pull requests? The answer is "No" as Jesse pointed out. Custom WAR packager uses its own YAML format which can be integrated into essentials.yml format created by Raul Arabaolaza for integration tests. This format is generally more flexible for packaging WARs for testing purposes (build settings, relative FS paths, support of embedding System properties and Groovy scripts, etc.). In this PR and JEP comments I have pointed out several weaknesses of the current BOM proposal:
  • The provided justification for BOM is true, but actually it is already achieved by Custom WAR Packager.
    • BOM does not make much difference there as described above
    • "We want to unify the description format across projects within Jenkins (e.g. Jenkins Essentials, Jenkins X, Docker Packaging, etc.)." is the real thing we should focus on IMHO, but it needs to be expanded (see proposals below)
  • BOM format is not specified. The JEP proposal just provides an example without specifying fields. It refers the K8s notation, but that's it. There is no link to K8s specification and the version, and this specification may drift over time. It leads to all my further comments here
  • "Metadata/labels" are not specified. What is a "must", what is "optional"?
    • E.g. what is field 'name' and how should it be used?
  • Purpose of "Metadata/annotations" section is not clear.
    • There is neither examples nor documentation
    • I would expect to be able to put any custom data there. E.g. for Custom WAR packager I would expect to put build and packaging settings there. OTOH for Jenkins Essentials I would expect this section to have some deployment rules and so
    • We had a private discussion with Carlos, and his explanation was that "annotations" in K8s notation is just a key-value mapping without object support. It does not allow passing complex configurations via them and hence does not allow using BOM as a sole input for test automation flows (like ATH which is used for justification)
  • "Core" section needs specification
    • I'd guess it defines version of WAR file, but it's not clear from documentation. Note that version of WAR and version of the bundled Jenkins Core may differ (timestamped releases, custom WAR builds, etc.)
    • "Core" does not define "artifactId" or "groupId". It means that only standard Jenkins WAR "org.jenkins-ci.main:jenkins-war" may be used. What if I want to define BOM for a custom product based on Jenkins? It is a real use-case for me
  • Vague "components" section. It is not clear what can be defined there and how it should be handled.
    • Is it only JARs? What about Jenkins Modules, Resources, System Groovy Scripts, etc.
    • There is no "type" field or whatever which would allow to distinguish it
  • Plugins and Components support only "version" and "ref" fields
    • In test automation flows we need to test unreleased components in repositories, and it would be really useful to just reference components by path to sources and by path to HPI/JAR like it's done in custom WAR packager
  • It is not clear how "environment" should be used.
    • Can we define multiple environments? Or just a single one?
    • Can environments override plugins/components? Or do they just append new ones?
  • "status" section requires all components/plugins to have artifactId/groupId/versions
    • What if I add a "Groovy init gook" script from a repository as a component? This component has no real version since it's not released separately. Am I expected to mock them somehow (e.g. commitId)?
    • Does BOM expect all listed components to be available in Maven repo?

IMHO BOM specification should be created and documented properly, maybe using one of standard YAML specification languages.


Even with the comments above, I think that the BOM is actually useful and should be finalized. Why?

  • We really need an inter-exchange format. As Jesse said somewhere, Maven POM/BOM is not a silver-bullet in this area since it does not allow passing extra metadata easily
  • Examples of inter-exchange:
    • Custom Packager generates a specification of bundled components which is then used by runPCT/runATH steps to define defaults for running steps (INFRA-1623)
    • WAR bundles ship BOMs internally so that tools like PCT can easily determine what's inside without parsing files
    • BOM can be used as a source for repackaging third-party WARs. E.g. somebody may be building a custom Docker image on the top of jenkins/blueocean, and BOM could help to resolve dependencies there.
    • Jenkins Essentials could be shipping BOM so that the same set of "Verified plugins" could be used by users who are not ready to switch to AWS but still want to get extra insurance about the plugin cross-compatibility

So I am +1 regarding finalizing BOM as JEP.


Bes regards,

Oleg


On Tuesday, May 1, 2018 at 3:17:44 PM UTC+2, Jesse Glick wrote:
On Mon, Apr 30, 2018 at 4:47 PM, R. Tyler Croy <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="j-f1jCi8AQAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">ty...@...> wrote:
> If an update-incrementals tool existed, then yes, that would address my
> concerns here.

OK, good to know I am not completely off base here.

> I more wanted to make sure that we weren't going to have two
> or three implementations of what `update-incrementals` effectively should be
> providing.

There might be two or three scripts to deal with different syntaxes;
for example, it would be useful to have an equivalent to `mvn
versions:use-latest-releases` which grokked Incrementals and updated a
`pom.xml`. But yes they should share an underlying library of some
kind.

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/542d0afb-86bd-4f00-a714-96a2bec639b4%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Bill of Materials

Carlos Sanchez



On Fri, May 11, 2018 at 11:44 AM Oleg Nenashev <[hidden email]> wrote:
Hi all,

Just to provide an update on this topic, I have added support of Bill of Materials (in the current draft version) to Custom WAR Packager and used it in several automation flows. E.g. see Integration Testing for Artifact Manager S3. I can also rework Remoting (PR) and Stapler (PR) flows to use BOM there if it is needed for a reference implementation.

Do I gravely need BOM in these pull requests? The answer is "No" as Jesse pointed out. Custom WAR packager uses its own YAML format which can be integrated into essentials.yml format created by Raul Arabaolaza for integration tests. This format is generally more flexible for packaging WARs for testing purposes (build settings, relative FS paths, support of embedding System properties and Groovy scripts, etc.). In this PR and JEP comments I have pointed out several weaknesses of the current BOM proposal:
  • The provided justification for BOM is true, but actually it is already achieved by Custom WAR Packager.
    • BOM does not make much difference there as described above
    • "We want to unify the description format across projects within Jenkins (e.g. Jenkins Essentials, Jenkins X, Docker Packaging, etc.)." is the real thing we should focus on IMHO, but it needs to be expanded (see proposals below)
  • BOM format is not specified. The JEP proposal just provides an example without specifying fields. It refers the K8s notation, but that's it. There is no link to K8s specification and the version, and this specification may drift over time. It leads to all my further comments here
  • "Metadata/labels" are not specified. What is a "must", what is "optional"?
    • E.g. what is field 'name' and how should it be used?
  • Purpose of "Metadata/annotations" section is not clear.
    • There is neither examples nor documentation
    • I would expect to be able to put any custom data there. E.g. for Custom WAR packager I would expect to put build and packaging settings there. OTOH for Jenkins Essentials I would expect this section to have some deployment rules and so
    • We had a private discussion with Carlos, and his explanation was that "annotations" in K8s notation is just a key-value mapping without object support. It does not allow passing complex configurations via them and hence does not allow using BOM as a sole input for test automation flows (like ATH which is used for justification)
  • "Core" section needs specification
    • I'd guess it defines version of WAR file, but it's not clear from documentation. Note that version of WAR and version of the bundled Jenkins Core may differ (timestamped releases, custom WAR builds, etc.)
    • "Core" does not define "artifactId" or "groupId". It means that only standard Jenkins WAR "org.jenkins-ci.main:jenkins-war" may be used. What if I want to define BOM for a custom product based on Jenkins? It is a real use-case for me
  • Vague "components" section. It is not clear what can be defined there and how it should be handled.
    • Is it only JARs? What about Jenkins Modules, Resources, System Groovy Scripts, etc.
    • There is no "type" field or whatever which would allow to distinguish it
  • Plugins and Components support only "version" and "ref" fields
    • In test automation flows we need to test unreleased components in repositories, and it would be really useful to just reference components by path to sources and by path to HPI/JAR like it's done in custom WAR packager
  • It is not clear how "environment" should be used.
    • Can we define multiple environments? Or just a single one?
    • Can environments override plugins/components? Or do they just append new ones?
  • "status" section requires all components/plugins to have artifactId/groupId/versions
    • What if I add a "Groovy init gook" script from a repository as a component? This component has no real version since it's not released separately. Am I expected to mock them somehow (e.g. commitId)?
    • Does BOM expect all listed components to be available in Maven repo?

IMHO BOM specification should be created and documented properly, maybe using one of standard YAML specification languages.


Even with the comments above, I think that the BOM is actually useful and should be finalized. Why?

  • We really need an inter-exchange format. As Jesse said somewhere, Maven POM/BOM is not a silver-bullet in this area since it does not allow passing extra metadata easily
  • Examples of inter-exchange:
    • Custom Packager generates a specification of bundled components which is then used by runPCT/runATH steps to define defaults for running steps (INFRA-1623)
    • WAR bundles ship BOMs internally so that tools like PCT can easily determine what's inside without parsing files
    • BOM can be used as a source for repackaging third-party WARs. E.g. somebody may be building a custom Docker image on the top of jenkins/blueocean, and BOM could help to resolve dependencies there.
    • Jenkins Essentials could be shipping BOM so that the same set of "Verified plugins" could be used by users who are not ready to switch to AWS but still want to get extra insurance about the plugin cross-compatibility

So I am +1 regarding finalizing BOM as JEP.


Thanks for the input. Yes there are missing bits and that was the point of circulating the definition. The format is the least important of things, more need to focus on the interoperability and common definition for war packager, essentials and any future usage
 


Bes regards,

Oleg


On Tuesday, May 1, 2018 at 3:17:44 PM UTC+2, Jesse Glick wrote:
On Mon, Apr 30, 2018 at 4:47 PM, R. Tyler Croy <[hidden email]> wrote:
> If an update-incrementals tool existed, then yes, that would address my
> concerns here.

OK, good to know I am not completely off base here.

> I more wanted to make sure that we weren't going to have two
> or three implementations of what `update-incrementals` effectively should be
> providing.

There might be two or three scripts to deal with different syntaxes;
for example, it would be useful to have an equivalent to `mvn
versions:use-latest-releases` which grokked Incrementals and updated a
`pom.xml`. But yes they should share an underlying library of some
kind.

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/542d0afb-86bd-4f00-a714-96a2bec639b4%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CALHFn6NYkni%3Ds8vct-FOqrAoFWhQ01M0yci3dO1Q4-UBDbAfLg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Bill of Materials

Jesse Glick-4
In reply to this post by Oleg Nenashev
On Fri, May 11, 2018 at 6:43 AM, Oleg Nenashev <[hidden email]> wrote:
> We really need an inter-exchange format. As Jesse said somewhere, Maven
> POM/BOM is not a silver-bullet in this area since it does not allow passing
> extra metadata easily

Indeed the v4 POM format does not permit custom metadata inline in
`<dependency>`s. On the other hand you can always use a POM to define
component versions and then mix it with other metadata for particular
purposes. This is certainly friendlier to existing developer workflows
than attempting to generate or update POMs from another source. We
anyway need a Maven BOM (i.e., `pom.xml` with
`<dependencyManagement>`) just to slim down per-plugin POMs and make
it easy to develop against the “latest” versions of other stuff.

I think the trouble here is that we have a bunch of different tools
and workflows with different needs, and it is not that clear to me how
much overlap there really is, or what problems we are trying to solve
with a unified format. For example, for Evergreen deployment, we just
need a list of components with versions (all assumed to come from
`master`), which could easily fit in a POM if you also differentiate
“environments” like AWS somewhere. For `buildPlugin` integration
testing, it makes the most sense to pick up versions of other
components from the Maven `test` scope, which is already used for
functional tests. (You also need an optional branch override for
`incrementals:update`, but this can easily be handled in mojo
configuration.)

> Custom Packager generates a specification of bundled components which is
> then used by runPCT/runATH steps to define defaults for running steps

Details of these test suites are configured via the `essentials.yaml`.
I think you do not also need a BOM, if you are picking up `test`
scope.

> WAR bundles ship BOMs internally so that tools like PCT can easily determine
> what's inside without parsing files

The code to look up component versions in PCT already exists. Are
there some subtleties not captured by the traditional logic?

> BOM can be used as a source for repackaging third-party WARs.

This is Custom WAR Packager IIUC?

> Jenkins Essentials could be shipping BOM

Well, Evergreen will be “shipping” a flat list of core & plugin URLs,
if I gather Tyler’s intention correctly—the minimum information needed
by the client. It is an open question what build process generates
this list and from what source.

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CANfRfr1Nq%3DSzsWQJ0WzVBZbBhh4HGX1Cr0XRiD-%3DPxPX_6gN_A%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Bill of Materials

Liam Newman
I've approved JEP-309 as Draft.  
https://github.com/jenkinsci/jep/tree/master/jep/309


On Friday, May 11, 2018 at 9:20:36 AM UTC-7, Jesse Glick wrote:
On Fri, May 11, 2018 at 6:43 AM, Oleg Nenashev <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="01x8IFuOBgAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">o.v.ne...@...> wrote:
> We really need an inter-exchange format. As Jesse said somewhere, Maven
> POM/BOM is not a silver-bullet in this area since it does not allow passing
> extra metadata easily

Indeed the v4 POM format does not permit custom metadata inline in
`<dependency>`s. On the other hand you can always use a POM to define
component versions and then mix it with other metadata for particular
purposes. This is certainly friendlier to existing developer workflows
than attempting to generate or update POMs from another source. We
anyway need a Maven BOM (i.e., `pom.xml` with
`<dependencyManagement>`) just to slim down per-plugin POMs and make
it easy to develop against the “latest” versions of other stuff.

I think the trouble here is that we have a bunch of different tools
and workflows with different needs, and it is not that clear to me how
much overlap there really is, or what problems we are trying to solve
with a unified format. For example, for Evergreen deployment, we just
need a list of components with versions (all assumed to come from
`master`), which could easily fit in a POM if you also differentiate
“environments” like AWS somewhere. For `buildPlugin` integration
testing, it makes the most sense to pick up versions of other
components from the Maven `test` scope, which is already used for
functional tests. (You also need an optional branch override for
`incrementals:update`, but this can easily be handled in mojo
configuration.)

> Custom Packager generates a specification of bundled components which is
> then used by runPCT/runATH steps to define defaults for running steps

Details of these test suites are configured via the `essentials.yaml`.
I think you do not also need a BOM, if you are picking up `test`
scope.

> WAR bundles ship BOMs internally so that tools like PCT can easily determine
> what's inside without parsing files

The code to look up component versions in PCT already exists. Are
there some subtleties not captured by the traditional logic?

> BOM can be used as a source for repackaging third-party WARs.

This is Custom WAR Packager IIUC?

> Jenkins Essentials could be shipping BOM

Well, Evergreen will be “shipping” a flat list of core & plugin URLs,
if I gather Tyler’s intention correctly—the minimum information needed
by the client. It is an open question what build process generates
this list and from what source.

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/b8b09e81-92cf-486d-99c2-3f0b67292d40%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Bill of Materials

Oleg Nenashev
Hi all,

Status update: By now Custom War Packager has been released in 1.0, and there are also many updates in Evergreen. IMHO it is a good time to get this story over the fence.

Last week I have submitted a patch to the design, which does the following:
  • Full format specification has been added to the JEP
  • New "version" field has been added to support changing the format in the future. It addresses my comment about extensibility in the future
  • "components" section has been documented, so it is no longer an empty blob
  • "type" has been added to "components". It addresses my comment about component type packaging
  • Dependency resolution rules have been documented explicitly
  • Reference implementation links are updated
  • Tooling has been explicitly excluded out of the scope (though Custom War Packager offers a lib)
With the current changes, I believe that the JEP can be reviewed by the BDFL Delegate. The only outstanding comment is "YAGNI" from Jesse, but I believe that the reference implementations justify it a bit. I do not expect this Bill of Materials to be widely used, but accepting this JEP at least allows to stabilize and specify the current implementations which we already use in some cases.

Best regards,
Oleg

On Saturday, June 16, 2018 at 2:00:22 AM UTC+2, Liam Newman wrote:
I've approved JEP-309 as Draft.  
<a href="https://github.com/jenkinsci/jep/tree/master/jep/309" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Fjenkinsci%2Fjep%2Ftree%2Fmaster%2Fjep%2F309\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFMuy5kN8Hc2-C70w-1E5ne8ujMlg&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Fjenkinsci%2Fjep%2Ftree%2Fmaster%2Fjep%2F309\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFMuy5kN8Hc2-C70w-1E5ne8ujMlg&#39;;return true;">https://github.com/jenkinsci/jep/tree/master/jep/309


On Friday, May 11, 2018 at 9:20:36 AM UTC-7, Jesse Glick wrote:
On Fri, May 11, 2018 at 6:43 AM, Oleg Nenashev <[hidden email]> wrote:
> We really need an inter-exchange format. As Jesse said somewhere, Maven
> POM/BOM is not a silver-bullet in this area since it does not allow passing
> extra metadata easily

Indeed the v4 POM format does not permit custom metadata inline in
`<dependency>`s. On the other hand you can always use a POM to define
component versions and then mix it with other metadata for particular
purposes. This is certainly friendlier to existing developer workflows
than attempting to generate or update POMs from another source. We
anyway need a Maven BOM (i.e., `pom.xml` with
`<dependencyManagement>`) just to slim down per-plugin POMs and make
it easy to develop against the “latest” versions of other stuff.

I think the trouble here is that we have a bunch of different tools
and workflows with different needs, and it is not that clear to me how
much overlap there really is, or what problems we are trying to solve
with a unified format. For example, for Evergreen deployment, we just
need a list of components with versions (all assumed to come from
`master`), which could easily fit in a POM if you also differentiate
“environments” like AWS somewhere. For `buildPlugin` integration
testing, it makes the most sense to pick up versions of other
components from the Maven `test` scope, which is already used for
functional tests. (You also need an optional branch override for
`incrementals:update`, but this can easily be handled in mojo
configuration.)

> Custom Packager generates a specification of bundled components which is
> then used by runPCT/runATH steps to define defaults for running steps

Details of these test suites are configured via the `essentials.yaml`.
I think you do not also need a BOM, if you are picking up `test`
scope.

> WAR bundles ship BOMs internally so that tools like PCT can easily determine
> what's inside without parsing files

The code to look up component versions in PCT already exists. Are
there some subtleties not captured by the traditional logic?

> BOM can be used as a source for repackaging third-party WARs.

This is Custom WAR Packager IIUC?

> Jenkins Essentials could be shipping BOM

Well, Evergreen will be “shipping” a flat list of core & plugin URLs,
if I gather Tyler’s intention correctly—the minimum information needed
by the client. It is an open question what build process generates
this list and from what source.

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/fe094a4d-053a-4b91-a37e-bdd94c17c33e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Bill of Materials

Jesse Glick-4
On Mon, Aug 6, 2018 at 6:11 AM Oleg Nenashev <[hidden email]> wrote:
> The only outstanding comment is "YAGNI" from Jesse, but I believe that the reference implementations justify it a bit.

So I see that

https://github.com/jenkinsci/jep/blob/master/jep/309/README.adoc#prototype-implementation

lists Evergreen as a consumer, yet

https://github.com/jenkins-infra/evergreen/blob/master/services/essentials.yaml

is not really ready for use; together with the `ingest-update-center`
script that processes it, it looks like a throwaway implementation,
and as I found in PR #144, it does not support JEP-305 adequately and
does not properly manage transitive dependencies. I have proposed ways
to fix `essentials.yaml` so that it would actually serve the needs of
Evergreen, but it is not clear the resulting format would still match
whatever is being proposed in JEP-309.

As to `essentialsTest`, IIUC this is a usage of `custom-war-packager`,
rather than a separate client. So if Evergreen is indeed fixed at some
point by reworking the format of this YAML file, that leaves CWP as
the only direct consumer of the BOM, making it not be an intetrchange
format. Even if Evergreen can productively use the same format as the
JEP currently specifies, it is not apparent to me why we care: there
seems to be no overlap in use case between the YAML file stored in the
Evergreen repository, to define what is pushed to production, and the
YAML files used now in a few plugins to request integration tests
during PR builds.

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CANfRfr09_dhT%3D6ataL2iMF%3D7sGex49a%3DR6hbNn50-x3wGCrJ%2BA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Bill of Materials

Oleg Nenashev
I would prefer to get JEP-309 accepted, maybe as "Bill of Materials 1.0" specification. We define cross-dependencies between CWP formats and Jenkins Infrastructure (essentialsTest() on ci.jenkins.io), so I would prefer to have this format accepted even if it is the only reference implementation.

I am fine with renaming JEP if we want to narrow down the scope.

BR, Oleg


On Mon, Aug 6, 2018 at 5:12 PM, Jesse Glick <[hidden email]> wrote:
On Mon, Aug 6, 2018 at 6:11 AM Oleg Nenashev <[hidden email]> wrote:
> The only outstanding comment is "YAGNI" from Jesse, but I believe that the reference implementations justify it a bit.

So I see that

https://github.com/jenkinsci/jep/blob/master/jep/309/README.adoc#prototype-implementation

lists Evergreen as a consumer, yet

https://github.com/jenkins-infra/evergreen/blob/master/services/essentials.yaml

is not really ready for use; together with the `ingest-update-center`
script that processes it, it looks like a throwaway implementation,
and as I found in PR #144, it does not support JEP-305 adequately and
does not properly manage transitive dependencies. I have proposed ways
to fix `essentials.yaml` so that it would actually serve the needs of
Evergreen, but it is not clear the resulting format would still match
whatever is being proposed in JEP-309.

As to `essentialsTest`, IIUC this is a usage of `custom-war-packager`,
rather than a separate client. So if Evergreen is indeed fixed at some
point by reworking the format of this YAML file, that leaves CWP as
the only direct consumer of the BOM, making it not be an intetrchange
format. Even if Evergreen can productively use the same format as the
JEP currently specifies, it is not apparent to me why we care: there
seems to be no overlap in use case between the YAML file stored in the
Evergreen repository, to define what is pushed to production, and the
YAML files used now in a few plugins to request integration tests
during PR builds.

--
You received this message because you are subscribed to a topic in the Google Groups "Jenkins Developers" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jenkinsci-dev/pR2ZQMj95Zc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CANfRfr09_dhT%3D6ataL2iMF%3D7sGex49a%3DR6hbNn50-x3wGCrJ%2BA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CAPfivLAFH9OOpHLFMLGY_yAaoV1hH0vRNctq-iSeOm4qAQBiDA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Bill of Materials

Oleg Nenashev
Anyway, let's get some feedback from Jenkins Essentials folks before we proceed.


On Mon, Aug 6, 2018 at 5:25 PM, Oleg Nenashev <[hidden email]> wrote:
I would prefer to get JEP-309 accepted, maybe as "Bill of Materials 1.0" specification. We define cross-dependencies between CWP formats and Jenkins Infrastructure (essentialsTest() on ci.jenkins.io), so I would prefer to have this format accepted even if it is the only reference implementation.

I am fine with renaming JEP if we want to narrow down the scope.

BR, Oleg


On Mon, Aug 6, 2018 at 5:12 PM, Jesse Glick <[hidden email]> wrote:
On Mon, Aug 6, 2018 at 6:11 AM Oleg Nenashev <[hidden email]> wrote:
> The only outstanding comment is "YAGNI" from Jesse, but I believe that the reference implementations justify it a bit.

So I see that

https://github.com/jenkinsci/jep/blob/master/jep/309/README.adoc#prototype-implementation

lists Evergreen as a consumer, yet

https://github.com/jenkins-infra/evergreen/blob/master/services/essentials.yaml

is not really ready for use; together with the `ingest-update-center`
script that processes it, it looks like a throwaway implementation,
and as I found in PR #144, it does not support JEP-305 adequately and
does not properly manage transitive dependencies. I have proposed ways
to fix `essentials.yaml` so that it would actually serve the needs of
Evergreen, but it is not clear the resulting format would still match
whatever is being proposed in JEP-309.

As to `essentialsTest`, IIUC this is a usage of `custom-war-packager`,
rather than a separate client. So if Evergreen is indeed fixed at some
point by reworking the format of this YAML file, that leaves CWP as
the only direct consumer of the BOM, making it not be an intetrchange
format. Even if Evergreen can productively use the same format as the
JEP currently specifies, it is not apparent to me why we care: there
seems to be no overlap in use case between the YAML file stored in the
Evergreen repository, to define what is pushed to production, and the
YAML files used now in a few plugins to request integration tests
during PR builds.

--
You received this message because you are subscribed to a topic in the Google Groups "Jenkins Developers" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jenkinsci-dev/pR2ZQMj95Zc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CANfRfr09_dhT%3D6ataL2iMF%3D7sGex49a%3DR6hbNn50-x3wGCrJ%2BA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CAPfivLBq14K39Q_J0HyuX13T8UT5Vc8XHMC1AXc90-N2mU-0rg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Bill of Materials

Jesse Glick-4
In reply to this post by Oleg Nenashev
On Mon, Aug 6, 2018 at 11:25 AM Oleg Nenashev <[hidden email]> wrote:
> We define cross-dependencies between CWP formats and Jenkins Infrastructure (essentialsTest() on ci.jenkins.io)

Yes, this is a cross-dependency, since the `pipeline-library` method
calls do not seem to specify a version of CWP.

Where is this method being used BTW?

https://github.com/search?ref=simplesearch&type=Code&q=user%3Ajenkinsci+essentialstest

does not turn up any hits. I searched for

https://github.com/search?q=user%3Ajenkinsci+user%3Ajenkins-infra+filename%3Aessentials.yml+filename%3Aessentials.yaml&type=Code

(note the extension mismatch!) and saw that only Jenkins core and the
Git plugin were currently using this. Neither `Jenkinsfile` calls the
library function; they seem to use `runATH` and `runPCT` directly.
Jenkins core runs only ATH, no PCT (even though it would be very
useful to check compatibility against some plugins like `workflow-job`
which could easily be broken by subtle core changes). The Git plugin
runs some ATH tests and also claims to run PCT but only on itself,
which seems to make no sense (it should be running tests of _other_
plugins which would likely be broken by mistakes in `git-plugin`). Or
rather it _would_ be running ATH & PCT—currently this is all disabled.

So not a very large sample size here. Are there some plans to clean
this up and get `essentialsTest` widely used?

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CANfRfr27AogB1URmHPF8sh%2B4ACS4u0qCG9RVOVnTazKg6edimg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Bill of Materials

Oleg Nenashev
Hi Jesse,

Yes, essentialsTest() didn't get as much progress as I would have expected. Not my decision, the story has been handed over to other contributors. But there is a number of stories related to it, e.g. are pending patches for essentialsTest():
These stories will be completed by somebody at some point (sorry for being sooo specific), and that's why I would rather prefer to have JEP-309 finalized instead of staying in flux. essentialTest() will also eventually use BOM as consumer in order to automatically determine defaults for PCT and ATH. So far it just produces BOM and publishes it as an artifact.

Hopefully it answers the question,
Oleg



On Mon, Aug 6, 2018 at 6:10 PM, Jesse Glick <[hidden email]> wrote:
On Mon, Aug 6, 2018 at 11:25 AM Oleg Nenashev <[hidden email]> wrote:
> We define cross-dependencies between CWP formats and Jenkins Infrastructure (essentialsTest() on ci.jenkins.io)

Yes, this is a cross-dependency, since the `pipeline-library` method
calls do not seem to specify a version of CWP.

Where is this method being used BTW?

https://github.com/search?ref=simplesearch&type=Code&q=user%3Ajenkinsci+essentialstest

does not turn up any hits. I searched for

https://github.com/search?q=user%3Ajenkinsci+user%3Ajenkins-infra+filename%3Aessentials.yml+filename%3Aessentials.yaml&type=Code

(note the extension mismatch!) and saw that only Jenkins core and the
Git plugin were currently using this. Neither `Jenkinsfile` calls the
library function; they seem to use `runATH` and `runPCT` directly.
Jenkins core runs only ATH, no PCT (even though it would be very
useful to check compatibility against some plugins like `workflow-job`
which could easily be broken by subtle core changes). The Git plugin
runs some ATH tests and also claims to run PCT but only on itself,
which seems to make no sense (it should be running tests of _other_
plugins which would likely be broken by mistakes in `git-plugin`). Or
rather it _would_ be running ATH & PCT—currently this is all disabled.

So not a very large sample size here. Are there some plans to clean
this up and get `essentialsTest` widely used?

--
You received this message because you are subscribed to a topic in the Google Groups "Jenkins Developers" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jenkinsci-dev/pR2ZQMj95Zc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CANfRfr27AogB1URmHPF8sh%2B4ACS4u0qCG9RVOVnTazKg6edimg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CAPfivLBCP7Uma9zOwFePVCefKam_0-Lz04%3D_c3rpn9SWc7oAcA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Bill of Materials

Jesse Glick-4
On Mon, Aug 6, 2018 at 12:23 PM Oleg Nenashev <[hidden email]> wrote:
> there is a number of stories related to it, e.g. are pending patches for essentialsTest()

Interesting, I will take a look at these.

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CANfRfr3KxMhkEFFN-CGwrHMhNweyHK9OaUKU1R-LbkVA4vxnrQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Bill of Materials

R. Tyler Croy
In reply to this post by Oleg Nenashev
(replies inline)

On Mon, 06 Aug 2018, Oleg Nenashev wrote:

> Hi all,
>
> Status update: By now Custom War Packager has been released in 1.0, and
> there are also many updates in Evergreen. IMHO it is a good time to get
> this story over the fence.


Thanks for working to drive this to completion Oleg! Overall I think the
document looks great, and as of last Friday I had actually gotten the
implementation prepared for the Evergreen backend system (see:
https://github.com/jenkins-infra/evergreen/pull/169) which processes the
essentials.yaml and prepares the update records needed.


I believe the only think which needs to be resolved which is likely just an
obsolete part of the example YAML.  The root `status` key in the YAML for a
"realized" BOM I don't believe we've ever actually used and is worth removing.

I wanted to sanity check that with you first though.


If you remove that, I think overall this JEP looks good and is just about ready
to go.


Cheers

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/20180806224950.GC17800%40grape.lasagna.io.
For more options, visit https://groups.google.com/d/optout.

signature.asc (201 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Bill of Materials

Oleg Nenashev
Hi Tyler,

Thanks for the feedback!
 
I believe the only think which needs to be resolved which is likely just an
obsolete part of the example YAML.  The root `status` key in the YAML for a
"realized" BOM I don't believe we've ever actually used and is worth removing.

Actually I use it in some cases in order to implement custom packaging Pipelines after customWARPackager().
  • BOM's specification lists explicit dependencies
  • BOM's specification does not require all dependencies to be explicit
    • Some dependencies may have "dir" references
    • Some dependencies may be transitive. JEP-309 permits that though does not recommend for production use (dependency resolution in the spec)
  • "status" key returns the full list of resolved dependencies
    • In addition to transitive deps, CWP uses "status" to squash the "environment" definitions into a single list in order to show what was actually packaged into the WAR file
I would rather prefer the "status" section to stay in the specification. It is helpful for CWP at least (though it may be possible to just generate a new output BOM). If we do that, it would be nice to get feedback from Raul who is also experimenting with processing of BOMs.

In order to address your comment, we could explicitly say that the "status" section is optional so that you do not need to implement it in Evergreen if not needed. WDYT?

BR, Oleg


On Tue, Aug 7, 2018 at 12:49 AM, R. Tyler Croy <[hidden email]> wrote:
(replies inline)

On Mon, 06 Aug 2018, Oleg Nenashev wrote:

> Hi all,
>
> Status update: By now Custom War Packager has been released in 1.0, and
> there are also many updates in Evergreen. IMHO it is a good time to get
> this story over the fence.


Thanks for working to drive this to completion Oleg! Overall I think the
document looks great, and as of last Friday I had actually gotten the
implementation prepared for the Evergreen backend system (see:
https://github.com/jenkins-infra/evergreen/pull/169) which processes the
essentials.yaml and prepares the update records needed.


I believe the only think which needs to be resolved which is likely just an
obsolete part of the example YAML.  The root `status` key in the YAML for a
"realized" BOM I don't believe we've ever actually used and is worth removing.

I wanted to sanity check that with you first though.


If you remove that, I think overall this JEP looks good and is just about ready
to go.


Cheers

--
You received this message because you are subscribed to a topic in the Google Groups "Jenkins Developers" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jenkinsci-dev/pR2ZQMj95Zc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/20180806224950.GC17800%40grape.lasagna.io.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CAPfivLD-V-Njwiae83FUyoySKdvMyMAee3QFVpqqrkSJiFO_%2BA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Bill of Materials

R. Tyler Croy
(replies inline)

On Tue, 07 Aug 2018, Oleg Nenashev wrote:

> Hi Tyler,
>
> Thanks for the feedback!
>
>
> > I believe the only think which needs to be resolved which is likely just an
> > obsolete part of the example YAML.  The root `status` key in the YAML for a
> > "realized" BOM I don't believe we've ever actually used and is worth
> > removing.
>
>
> Actually I use it in some cases in order to implement custom packaging
> Pipelines after customWARPackager()
> <https://github.com/jenkins-infra/pipeline-library/blob/master/vars/customWARPackager.groovy>.
>
>
>    - BOM's specification lists explicit dependencies
>    - BOM's specification does not require all dependencies to be explicit
>       - Some dependencies may have "dir" references
>       - Some dependencies may be transitive. JEP-309 permits that though
>       does not recommend for production use (dependency resolution
>       <https://github.com/jenkinsci/jep/tree/master/jep/309#dependency-resolution>
>       in the spec)
>       - "status" key returns the full list of resolved dependencies
>       - In addition to transitive deps, CWP uses "status" to squash the
>       "environment" definitions into a single list in order to show what was
>       actually packaged into the WAR file
>
> I would rather prefer the "status" section to stay in the specification. It
> is helpful for CWP at least (though it may be possible to just generate a
> new output BOM). If we do that, it would be nice to get feedback from Raul
> who is also experimenting with processing of BOMs.
>
> In order to address your comment, we could explicitly say that the "status"
> section is optional so that you do not need to implement it in Evergreen if
> not needed. WDYT?


I mentioned in a video call with Oleg this morning that I've gone ahead and
implemented the `status` section for the  Bill of Materials being used in the
jenkins-infra/evergreen repository.


Overall I'm quite happy with this work by Oleg and Carlos, and I will be
submitting a PR (with my BDFL-Delegate hat on) to mark JEP-309 as 'Accepted'
later today.


Thanks for the  hard work everybody!

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/20180814153826.GH17800%40grape.lasagna.io.
For more options, visit https://groups.google.com/d/optout.

signature.asc (201 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Bill of Materials

Oleg Nenashev
Thanks for the update, Tyler!


On Tue, Aug 14, 2018 at 5:38 PM R. Tyler Croy <[hidden email]> wrote:
(replies inline)

On Tue, 07 Aug 2018, Oleg Nenashev wrote:

> Hi Tyler,
>
> Thanks for the feedback!
>
>
> > I believe the only think which needs to be resolved which is likely just an
> > obsolete part of the example YAML.  The root `status` key in the YAML for a
> > "realized" BOM I don't believe we've ever actually used and is worth
> > removing.
>
>
> Actually I use it in some cases in order to implement custom packaging
> Pipelines after customWARPackager()
> <https://github.com/jenkins-infra/pipeline-library/blob/master/vars/customWARPackager.groovy>.
>
>
>    - BOM's specification lists explicit dependencies
>    - BOM's specification does not require all dependencies to be explicit
>       - Some dependencies may have "dir" references
>       - Some dependencies may be transitive. JEP-309 permits that though
>       does not recommend for production use (dependency resolution
>       <https://github.com/jenkinsci/jep/tree/master/jep/309#dependency-resolution>
>       in the spec)
>       - "status" key returns the full list of resolved dependencies
>       - In addition to transitive deps, CWP uses "status" to squash the
>       "environment" definitions into a single list in order to show what was
>       actually packaged into the WAR file
>
> I would rather prefer the "status" section to stay in the specification. It
> is helpful for CWP at least (though it may be possible to just generate a
> new output BOM). If we do that, it would be nice to get feedback from Raul
> who is also experimenting with processing of BOMs.
>
> In order to address your comment, we could explicitly say that the "status"
> section is optional so that you do not need to implement it in Evergreen if
> not needed. WDYT?



I mentioned in a video call with Oleg this morning that I've gone ahead and
implemented the `status` section for the  Bill of Materials being used in the
jenkins-infra/evergreen repository.


Overall I'm quite happy with this work by Oleg and Carlos, and I will be
submitting a PR (with my BDFL-Delegate hat on) to mark JEP-309 as 'Accepted'
later today.


Thanks for the  hard work everybody!

--
You received this message because you are subscribed to a topic in the Google Groups "Jenkins Developers" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jenkinsci-dev/pR2ZQMj95Zc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/20180814153826.GH17800%40grape.lasagna.io.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CAPfivLBdGOuR0vCUd06ggkfAfV%2BUMEPWaOk_9%2B7TdKEZPNVz8A%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
12