Incrementals and ATH tests (inside a plugin CI)

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

Incrementals and ATH tests (inside a plugin CI)

Ulli Hafner
My warnings plugin is using in our Jenkins CI build also UI tests (using the ATH container). Is there a way to start these UI tests (using the ATH container) so that depending plugins that have an incremental version are also resolved? Incrementals are resolved fine in integration tests but in UI tests the official update center is used which does not contain the incremental versions.


java.io.IOException: Failed to load: Warnings Next Generation Plugin (8.5.0-SNAPSHOT (private-2e85864e-ubuntu))
master49529| - Plugin is missing: forensics-api (0.8.0-rc587.65fc61fe9be2)

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/10362FBD-68D4-4ADD-BEA9-267CB38E6EED%40gmail.com.
Reply | Threaded
Open this post in threaded view
|

Re: Incrementals and ATH tests (inside a plugin CI)

Oliver Gondža-2
I am not aware that any work have been done to support this.

On 27/08/2020 21.06, Ullrich Hafner wrote:

> My warnings plugin is using in our Jenkins CI build also UI tests (using
> the ATH container). Is there a way to start these UI tests (using the
> ATH container) so that depending plugins that have an incremental
> version are also resolved? Incrementals are resolved fine in integration
> tests but in UI tests the official update center is used which does not
> contain the incremental versions.
>
> See:
> https://ci.jenkins.io/job/Plugins/job/warnings-ng-plugin/job/PR-595/5/testReport/io.jenkins.plugins.analysis.warnings/SmokeTests/shouldRecordIssuesInPipelineAndExpandTokens/
>
> java.io.IOException: Failed to load: Warnings Next Generation Plugin (8.5.0-SNAPSHOT (private-2e85864e-ubuntu))
> master49529| - Plugin is missing: forensics-api (0.8.0-rc587.65fc61fe9be2)
>
>
> --
> You received this message because you are subscribed to the Google
> Groups "Jenkins Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to [hidden email]
> <mailto:[hidden email]>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/jenkinsci-dev/10362FBD-68D4-4ADD-BEA9-267CB38E6EED%40gmail.com 
> <https://groups.google.com/d/msgid/jenkinsci-dev/10362FBD-68D4-4ADD-BEA9-267CB38E6EED%40gmail.com?utm_medium=email&utm_source=footer>.


--
oliver

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/aca6a70f-861b-9723-bdf2-d034b4dfc452%40gmail.com.
Reply | Threaded
Open this post in threaded view
|

Re: Incrementals and ATH tests (inside a plugin CI)

Jesse Glick-4
In reply to this post by Ulli Hafner
On Thu, Aug 27, 2020 at 3:07 PM Ullrich Hafner <[hidden email]> wrote:
> Incrementals are resolved fine in integration tests but in UI tests the official update center is used

I suspect this is just the most visible special case of a general
problem that the ATH tests are using the latest release at the time of
the test run, regardless of what versions your POM asks to test
against. This is a bad thing IMO: as one example, it means tests can
start failing in a PR which did not change sources, leading to
confusion. Verifying compatibility with recent upstream changes is of
course valuable but this can be accomplished more deterministically
with Dependabot.

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CANfRfr3qetGKx9g1Rz6xLWL%2BB9SSeWeZm-8SR79HgGMXTyUJ-w%40mail.gmail.com.
Reply | Threaded
Open this post in threaded view
|

Re: Incrementals and ATH tests (inside a plugin CI)

Ulli Hafner
>
> I suspect this is just the most visible special case of a general
> problem that the ATH tests are using the latest release at the time of
> the test run, regardless of what versions your POM asks to test
> against. This is a bad thing IMO: as one example, it means tests can
> start failing in a PR which did not change sources, leading to
> confusion.

Yes, I had that problem a couple of weeks ago. One of my dependencies upgraded its Jenkins version to a higher version than the version I was using in my UI tests: that broke my build.

A possible workaround would be to use our plugin-installation-manager to download all dependencies in the correct version before the ATH container starts the Jenkins instance (the ATH already can load a set of pre-configured plugins). That would also have the benefit, that the startup time when running several ATH tests will be reduced as well.

Do we have something existing that can extract the list of dependencies for a plugin so that the plugin-installation-manager can be feeded with that input?

> Verifying compatibility with recent upstream changes is of
> course valuable but this can be accomplished more deterministically
> with Dependabot.
>
> --
> You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
> To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CANfRfr3qetGKx9g1Rz6xLWL%2BB9SSeWeZm-8SR79HgGMXTyUJ-w%40mail.gmail.com.

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/D7972A1D-2AC9-44C3-91F3-545FE9A7DCFA%40gmail.com.
Reply | Threaded
Open this post in threaded view
|

Re: Incrementals and ATH tests (inside a plugin CI)

Daniel Beck-2

On Thu, Aug 27, 2020 at 11:22 PM Ullrich Hafner <[hidden email]> wrote:

Yes, I had that problem a couple of weeks ago. One of my dependencies upgraded its Jenkins version to a higher version than the version I was using in my UI tests: that broke my build.

Do you still have the failing build log, or know when exactly this happened?

There was a bug with the update center tiers I introduced a few months ago that made the tiers not work quite correctly ( https://github.com/jenkins-infra/update-center2/pull/406 ) but I fixed that four weeks ago, so for that time it should have been working without problems.

Otherwise, the ATH (WithPlugins) should download plugins using the plugin manager, and that should only install compatible dependencies.

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CAMo7PtLr_H3Hv%3DFHHka4qLErackYfZ27yRdLKKa7msg4QxKyyg%40mail.gmail.com.
Reply | Threaded
Open this post in threaded view
|

Re: Incrementals and ATH tests (inside a plugin CI)

Ulli Hafner

Am 28.08.2020 um 07:52 schrieb Daniel Beck <[hidden email]>:


On Thu, Aug 27, 2020 at 11:22 PM Ullrich Hafner <[hidden email]> wrote:

Yes, I had that problem a couple of weeks ago. One of my dependencies upgraded its Jenkins version to a higher version than the version I was using in my UI tests: that broke my build.

Do you still have the failing build log, or know when exactly this happened?

There was a bug with the update center tiers I introduced a few months ago that made the tiers not work quite correctly ( https://github.com/jenkins-infra/update-center2/pull/406 ) but I fixed that four weeks ago, so for that time it should have been working without problems.


Hmm, I don’t know the exact time and those builds are already deleted. But it was around 4 weeks ago as well. So maybe this was actually caused by the bug you are describing… 

Otherwise, the ATH (WithPlugins) should download plugins using the plugin manager, and that should only install compatible dependencies.

Since no builds fail anymore now I think ATH picks the right versions anyway. 

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/BFB991AC-3294-4107-B453-4400559F2C8A%40gmail.com.
Reply | Threaded
Open this post in threaded view
|

Re: Incrementals and ATH tests (inside a plugin CI)

Jesse Glick-4
In reply to this post by Ulli Hafner
On Thu, Aug 27, 2020 at 5:22 PM Ullrich Hafner <[hidden email]> wrote:
> Do we have something existing that can extract the list of dependencies for a plugin…

I think it would suffice to inject a new implementation of
`UpdateCenterMetadataDecorator` using
`PluginMetadata.LocalOverride.create` based on
`target/{,test-classes/test-dependencies/}*.hpi`. Or simply use the
existing implementation and define the environment variable
`LOCAL_JARS` in your launcher script.

Plugins not mentioned in this list will still be retrieved according
to versions in the online UC (though by default using Maven to
retrieve local copies of the actual binaries). You could inject an
alternate `UpdateCenterMetadataProvider` to completely override that,
so that only dependencies in the test classpath are offered; this
would match the behavior of `JenkinsRule` (other than when annotated
by `@WithPlugin`).

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CANfRfr32vCiP5eJdqpq%2BUc1dATxmf%3DzBRBZkGLx%3Dt3%3DwvdpAKg%40mail.gmail.com.
Reply | Threaded
Open this post in threaded view
|

Re: Incrementals and ATH tests (inside a plugin CI)

Ulli Hafner
Am 28.08.2020 um 17:21 schrieb Jesse Glick <[hidden email]>:

On Thu, Aug 27, 2020 at 5:22 PM Ullrich Hafner <[hidden email]> wrote:
Do we have something existing that can extract the list of dependencies for a plugin…

I think it would suffice to inject a new implementation of
`UpdateCenterMetadataDecorator` using
`PluginMetadata.LocalOverride.create` based on
`target/{,test-classes/test-dependencies/}*.hpi`. Or simply use the
existing implementation and define the environment variable
`LOCAL_JARS` in your launcher script.


This is a beautiful solution! Thank you for the tip: I now use the following properties:
<environmentVariables>
<PLUGINS_DIR>../plugin/target/test-classes/test-dependencies</PLUGINS_DIR>
<JENKINS_VERSION>${jenkins.version}</JENKINS_VERSION>
<BROWSER>firefox-container</BROWSER>
</environmentVariables>
(And before the UI tests start it is required to copy the hpi under test into the test-dependencies folder as well) 

Now everything is green (or blue) again! This also improves the runtime of the tests (now about 2min per test):

Old variant (about 4min per test):


Plugins not mentioned in this list will still be retrieved according
to versions in the online UC (though by default using Maven to
retrieve local copies of the actual binaries). You could inject an
alternate `UpdateCenterMetadataProvider` to completely override that,
so that only dependencies in the test classpath are offered; this
would match the behavior of `JenkinsRule` (other than when annotated
by `@WithPlugin`).

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CANfRfr32vCiP5eJdqpq%2BUc1dATxmf%3DzBRBZkGLx%3Dt3%3DwvdpAKg%40mail.gmail.com.

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/34A7CDC5-0967-44F1-8A62-10F01D18DF50%40gmail.com.