[PROPOSAL] Split of `jenkins-test-harness` out of `jenkinsci/jenkins`

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
18 messages Options
Reply | Threaded
Open this post in threaded view
|

[PROPOSAL] Split of `jenkins-test-harness` out of `jenkinsci/jenkins`

Jesse Glick-4
Today the Jenkins functional test harness, containing `JenkinsRule`
and its ilk¹, is built from the Jenkins core repository, in
`test/src/main/`. That is fine for functional tests of core itself
(`test/src/test/`). But this scheme is undesirable for functional
tests in plugins, for two reasons.

One, whenever a fix is made to the test harness itself, such as my
recent PR 1967², it does not become available to plugins until they
switch their core baseline to the version including the fix. That
could be months away, even if the fix is backported to an LTS branch
as is occasionally done. Sometimes it is possible to work around the
lack of the fix in plugin tests³, but even when possible this is
awkward.

Second, incompatible changes to the test harness are sometimes
necessary or desirable. We try hard to avoid incompatible changes to
core itself—these could break user installations—but incompatibilities
of tests is a price paid only by plugin developers, usually
manageable. In particular, PR 1774⁴ changed some details of HtmlUnit
usage. Now if you upgrade your plugin’s core baseline to a version
after that point (generally 1.625.x), you update your tests to
match—fine. The problem comes as we try to expand usage of
`plugin-compat-tester`⁵ to hunt for core regressions. To verify that a
plugin’s tests still pass against a newer core, this tool updates the
baseline in a working copy of the POM. But if the tests are still
written against the old forked HtmlUnit, they may no longer compile;
and a single version of a test cannot always be made to work against
either version of HtmlUnit. You can see an example of this dilemma in
the `copyartifact` plugin⁶.

My proposal is to start treating the harness as an independently
versioned test library, in its own GitHub repository.
`jenkinsci/jenkins/test/src/main/` would be extracted there (with
history intact). `test/src/test/` core functional tests would pick it
up as a binary dependency, as plugin tests already do. The advantage
would be that we could iterate quickly on new features or fixes to the
harness, cutting a Maven release at any time. Core and each plugin
would decide which version to pick up; whenever you wish to update to
the latest version, you can edit your tests in the same commit to
reflect any required changes, or to take advantage of new features.
`plugin-compat-tester` would leave this version untouched.

In principle there could be a change to core which requires a
synchronized matching change to the test harness, which would become
more difficult to handle in such a system, but I am not aware of any
examples.⁷

After a split it should be possible to do further refactorings that
would make for smaller artifacts and better modularity. Currently most
of the size of the test harness artifact is consumed by bundling four
(!) copies of Maven, plus one copy of Ant, which are only relevant to
plugins actually testing builds using these tools. The harness also
has fixed dependencies on a number of plugins which were long since
split out of core—`subversion`, `maven-plugin`, `matrix-project`,
etc.—which thickens the `test`-scope dependency tree for every plugin
and can cause weird version resolution problems that complicate test
development and maintenance.

I am sending this proposal verbally rather than in a PR since
cross-repository refactorings like this are not easily handled with
PRs, especially since a new artifact release is required to get
everything working. There are some details that will only fall out of
an actual attempt to do it, but I wanted to get general agreement that
this is a good direction before starting on things like the slow Git
command to extract a subdirectory with history.

For context, there is also some work being prepared to create a new
plugin POM which, unlike the current one coversioned with core⁸, would
be in its own repository and independently versioned, with the core
version simply being a Maven property. Stephen Connolly and I have
long discussed the possibility—CloudBees proprietary plugins do
exactly this, and it is much more comfortable to work with as a plugin
developer and from `plugin-compat-tester`—but there is finally some
movement. The idea would be that with a refactored test harness, your
plugin POM would specify both the versions of core and the test
harness independently.

WDYT?


¹ https://github.com/jenkinsci/jenkins/blob/6306952779acc311719c175201e736685349e507/test/src/main/java/org/jvnet/hudson/test/JenkinsRule.java#L225-L234
² https://github.com/jenkinsci/jenkins/pull/1967
³ https://github.com/jenkinsci/workflow-plugin/pull/277
https://github.com/jenkinsci/jenkins/pull/1774
https://wiki.jenkins-ci.org/display/JENKINS/Plugin+Compatibility+Tester
https://github.com/jenkinsci/copyartifact-plugin/pull/76
https://github.com/jenkinsci/jenkins/commits/6306952779acc311719c175201e736685349e507/test/src/main
includes https://github.com/jenkinsci/jenkins/commit/6aabbbd282083716d631be7f2fa07ce93b93ee23#diff-8e0ebb909d001d948b257dbbc1450cfd
that would imply a newer core dep; this part of the patch was very low
priority. https://github.com/jenkinsci/jenkins/commit/b6ae4f5b798068fad48d292dee860fe97bdc7d55
and https://github.com/jenkinsci/jenkins/commit/fe1d4472015d978f45f219369fc7123bbdd558dd
handle core incompatibilities but are harmless to use with older
cores.
https://github.com/jenkinsci/jenkins/blob/6306952779acc311719c175201e736685349e507/plugins/pom.xml#L15

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CANfRfr1w18BohqOEpcnf0r_2PYvV-wmHzDEjLj%2BNst6WNtaOPA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: [PROPOSAL] Split of `jenkins-test-harness` out of `jenkinsci/jenkins`

Mark Waite-2
I like the idea of moving JenkinsRule outside jenkins core very much.  The git plugin and git client plugin intentionally support older Jenkins versions, but that means their test infrastructure is also limited to those older Jenkins versions.  I'd much rather be able to use the latest Jenkins test infrastructure to create better tests.  

That would also provide me more incentive to attempt fixes to test infrastructure.  Today, if the test classes have a problem, there is very little chance that I'll investigate the problem in the test classes, because I can't access updates to those test classes anyway.

Mark Waite

On Fri, Jan 8, 2016 at 10:16 AM Jesse Glick <[hidden email]> wrote:
Today the Jenkins functional test harness, containing `JenkinsRule`
and its ilk¹, is built from the Jenkins core repository, in
`test/src/main/`. That is fine for functional tests of core itself
(`test/src/test/`). But this scheme is undesirable for functional
tests in plugins, for two reasons.

One, whenever a fix is made to the test harness itself, such as my
recent PR 1967², it does not become available to plugins until they
switch their core baseline to the version including the fix. That
could be months away, even if the fix is backported to an LTS branch
as is occasionally done. Sometimes it is possible to work around the
lack of the fix in plugin tests³, but even when possible this is
awkward.

Second, incompatible changes to the test harness are sometimes
necessary or desirable. We try hard to avoid incompatible changes to
core itself—these could break user installations—but incompatibilities
of tests is a price paid only by plugin developers, usually
manageable. In particular, PR 1774⁴ changed some details of HtmlUnit
usage. Now if you upgrade your plugin’s core baseline to a version
after that point (generally 1.625.x), you update your tests to
match—fine. The problem comes as we try to expand usage of
`plugin-compat-tester`⁵ to hunt for core regressions. To verify that a
plugin’s tests still pass against a newer core, this tool updates the
baseline in a working copy of the POM. But if the tests are still
written against the old forked HtmlUnit, they may no longer compile;
and a single version of a test cannot always be made to work against
either version of HtmlUnit. You can see an example of this dilemma in
the `copyartifact` plugin⁶.

My proposal is to start treating the harness as an independently
versioned test library, in its own GitHub repository.
`jenkinsci/jenkins/test/src/main/` would be extracted there (with
history intact). `test/src/test/` core functional tests would pick it
up as a binary dependency, as plugin tests already do. The advantage
would be that we could iterate quickly on new features or fixes to the
harness, cutting a Maven release at any time. Core and each plugin
would decide which version to pick up; whenever you wish to update to
the latest version, you can edit your tests in the same commit to
reflect any required changes, or to take advantage of new features.
`plugin-compat-tester` would leave this version untouched.

In principle there could be a change to core which requires a
synchronized matching change to the test harness, which would become
more difficult to handle in such a system, but I am not aware of any
examples.⁷

After a split it should be possible to do further refactorings that
would make for smaller artifacts and better modularity. Currently most
of the size of the test harness artifact is consumed by bundling four
(!) copies of Maven, plus one copy of Ant, which are only relevant to
plugins actually testing builds using these tools. The harness also
has fixed dependencies on a number of plugins which were long since
split out of core—`subversion`, `maven-plugin`, `matrix-project`,
etc.—which thickens the `test`-scope dependency tree for every plugin
and can cause weird version resolution problems that complicate test
development and maintenance.

I am sending this proposal verbally rather than in a PR since
cross-repository refactorings like this are not easily handled with
PRs, especially since a new artifact release is required to get
everything working. There are some details that will only fall out of
an actual attempt to do it, but I wanted to get general agreement that
this is a good direction before starting on things like the slow Git
command to extract a subdirectory with history.

For context, there is also some work being prepared to create a new
plugin POM which, unlike the current one coversioned with core⁸, would
be in its own repository and independently versioned, with the core
version simply being a Maven property. Stephen Connolly and I have
long discussed the possibility—CloudBees proprietary plugins do
exactly this, and it is much more comfortable to work with as a plugin
developer and from `plugin-compat-tester`—but there is finally some
movement. The idea would be that with a refactored test harness, your
plugin POM would specify both the versions of core and the test
harness independently.

WDYT?


¹ https://github.com/jenkinsci/jenkins/blob/6306952779acc311719c175201e736685349e507/test/src/main/java/org/jvnet/hudson/test/JenkinsRule.java#L225-L234
² https://github.com/jenkinsci/jenkins/pull/1967
³ https://github.com/jenkinsci/workflow-plugin/pull/277
https://github.com/jenkinsci/jenkins/pull/1774
https://wiki.jenkins-ci.org/display/JENKINS/Plugin+Compatibility+Tester
https://github.com/jenkinsci/copyartifact-plugin/pull/76
https://github.com/jenkinsci/jenkins/commits/6306952779acc311719c175201e736685349e507/test/src/main
includes https://github.com/jenkinsci/jenkins/commit/6aabbbd282083716d631be7f2fa07ce93b93ee23#diff-8e0ebb909d001d948b257dbbc1450cfd
that would imply a newer core dep; this part of the patch was very low
priority. https://github.com/jenkinsci/jenkins/commit/b6ae4f5b798068fad48d292dee860fe97bdc7d55
and https://github.com/jenkinsci/jenkins/commit/fe1d4472015d978f45f219369fc7123bbdd558dd
handle core incompatibilities but are harmless to use with older
cores.
https://github.com/jenkinsci/jenkins/blob/6306952779acc311719c175201e736685349e507/plugins/pom.xml#L15

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CANfRfr1w18BohqOEpcnf0r_2PYvV-wmHzDEjLj%2BNst6WNtaOPA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CAO49JtGrzf5%3DtD3w8%2Bo7qSGjHBON5sG0cuxbRYwrVrrq9xwo2g%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: [PROPOSAL] Split of `jenkins-test-harness` out of `jenkinsci/jenkins`

Andrew Bayer
I'd be a big +1 on this as well. It makes sense to have the harness as a separate project consumed by Jenkins core and plugins for their testing. A nice side effect is that we'd be able to actually move the core tests currently in jenkins.git/test/src/test/... into jenkins.git/core/src/test/..., I think, which would just make things feel much more organic.

A.

On Fri, Jan 8, 2016 at 11:01 AM, Mark Waite <[hidden email]> wrote:
I like the idea of moving JenkinsRule outside jenkins core very much.  The git plugin and git client plugin intentionally support older Jenkins versions, but that means their test infrastructure is also limited to those older Jenkins versions.  I'd much rather be able to use the latest Jenkins test infrastructure to create better tests.  

That would also provide me more incentive to attempt fixes to test infrastructure.  Today, if the test classes have a problem, there is very little chance that I'll investigate the problem in the test classes, because I can't access updates to those test classes anyway.

Mark Waite

On Fri, Jan 8, 2016 at 10:16 AM Jesse Glick <[hidden email]> wrote:
Today the Jenkins functional test harness, containing `JenkinsRule`
and its ilk¹, is built from the Jenkins core repository, in
`test/src/main/`. That is fine for functional tests of core itself
(`test/src/test/`). But this scheme is undesirable for functional
tests in plugins, for two reasons.

One, whenever a fix is made to the test harness itself, such as my
recent PR 1967², it does not become available to plugins until they
switch their core baseline to the version including the fix. That
could be months away, even if the fix is backported to an LTS branch
as is occasionally done. Sometimes it is possible to work around the
lack of the fix in plugin tests³, but even when possible this is
awkward.

Second, incompatible changes to the test harness are sometimes
necessary or desirable. We try hard to avoid incompatible changes to
core itself—these could break user installations—but incompatibilities
of tests is a price paid only by plugin developers, usually
manageable. In particular, PR 1774⁴ changed some details of HtmlUnit
usage. Now if you upgrade your plugin’s core baseline to a version
after that point (generally 1.625.x), you update your tests to
match—fine. The problem comes as we try to expand usage of
`plugin-compat-tester`⁵ to hunt for core regressions. To verify that a
plugin’s tests still pass against a newer core, this tool updates the
baseline in a working copy of the POM. But if the tests are still
written against the old forked HtmlUnit, they may no longer compile;
and a single version of a test cannot always be made to work against
either version of HtmlUnit. You can see an example of this dilemma in
the `copyartifact` plugin⁶.

My proposal is to start treating the harness as an independently
versioned test library, in its own GitHub repository.
`jenkinsci/jenkins/test/src/main/` would be extracted there (with
history intact). `test/src/test/` core functional tests would pick it
up as a binary dependency, as plugin tests already do. The advantage
would be that we could iterate quickly on new features or fixes to the
harness, cutting a Maven release at any time. Core and each plugin
would decide which version to pick up; whenever you wish to update to
the latest version, you can edit your tests in the same commit to
reflect any required changes, or to take advantage of new features.
`plugin-compat-tester` would leave this version untouched.

In principle there could be a change to core which requires a
synchronized matching change to the test harness, which would become
more difficult to handle in such a system, but I am not aware of any
examples.⁷

After a split it should be possible to do further refactorings that
would make for smaller artifacts and better modularity. Currently most
of the size of the test harness artifact is consumed by bundling four
(!) copies of Maven, plus one copy of Ant, which are only relevant to
plugins actually testing builds using these tools. The harness also
has fixed dependencies on a number of plugins which were long since
split out of core—`subversion`, `maven-plugin`, `matrix-project`,
etc.—which thickens the `test`-scope dependency tree for every plugin
and can cause weird version resolution problems that complicate test
development and maintenance.

I am sending this proposal verbally rather than in a PR since
cross-repository refactorings like this are not easily handled with
PRs, especially since a new artifact release is required to get
everything working. There are some details that will only fall out of
an actual attempt to do it, but I wanted to get general agreement that
this is a good direction before starting on things like the slow Git
command to extract a subdirectory with history.

For context, there is also some work being prepared to create a new
plugin POM which, unlike the current one coversioned with core⁸, would
be in its own repository and independently versioned, with the core
version simply being a Maven property. Stephen Connolly and I have
long discussed the possibility—CloudBees proprietary plugins do
exactly this, and it is much more comfortable to work with as a plugin
developer and from `plugin-compat-tester`—but there is finally some
movement. The idea would be that with a refactored test harness, your
plugin POM would specify both the versions of core and the test
harness independently.

WDYT?


¹ https://github.com/jenkinsci/jenkins/blob/6306952779acc311719c175201e736685349e507/test/src/main/java/org/jvnet/hudson/test/JenkinsRule.java#L225-L234
² https://github.com/jenkinsci/jenkins/pull/1967
³ https://github.com/jenkinsci/workflow-plugin/pull/277
https://github.com/jenkinsci/jenkins/pull/1774
https://wiki.jenkins-ci.org/display/JENKINS/Plugin+Compatibility+Tester
https://github.com/jenkinsci/copyartifact-plugin/pull/76
https://github.com/jenkinsci/jenkins/commits/6306952779acc311719c175201e736685349e507/test/src/main
includes https://github.com/jenkinsci/jenkins/commit/6aabbbd282083716d631be7f2fa07ce93b93ee23#diff-8e0ebb909d001d948b257dbbc1450cfd
that would imply a newer core dep; this part of the patch was very low
priority. https://github.com/jenkinsci/jenkins/commit/b6ae4f5b798068fad48d292dee860fe97bdc7d55
and https://github.com/jenkinsci/jenkins/commit/fe1d4472015d978f45f219369fc7123bbdd558dd
handle core incompatibilities but are harmless to use with older
cores.
https://github.com/jenkinsci/jenkins/blob/6306952779acc311719c175201e736685349e507/plugins/pom.xml#L15

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CANfRfr1w18BohqOEpcnf0r_2PYvV-wmHzDEjLj%2BNst6WNtaOPA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CAO49JtGrzf5%3DtD3w8%2Bo7qSGjHBON5sG0cuxbRYwrVrrq9xwo2g%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CAPbPdOacnvj1wJBEKCkzt_bzr5U8arkjo7Kgvi_2-XuTbWdhLA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: [PROPOSAL] Split of `jenkins-test-harness` out of `jenkinsci/jenkins`

stephenconnolly
+1

On Friday 8 January 2016, Andrew Bayer <[hidden email]> wrote:
I'd be a big +1 on this as well. It makes sense to have the harness as a separate project consumed by Jenkins core and plugins for their testing. A nice side effect is that we'd be able to actually move the core tests currently in jenkins.git/test/src/test/... into jenkins.git/core/src/test/..., I think, which would just make things feel much more organic.

A.

On Fri, Jan 8, 2016 at 11:01 AM, Mark Waite <<a href="javascript:_e(%7B%7D,&#39;cvml&#39;,&#39;mark.earl.waite@gmail.com&#39;);" target="_blank">mark.earl.waite@...> wrote:
I like the idea of moving JenkinsRule outside jenkins core very much.  The git plugin and git client plugin intentionally support older Jenkins versions, but that means their test infrastructure is also limited to those older Jenkins versions.  I'd much rather be able to use the latest Jenkins test infrastructure to create better tests.  

That would also provide me more incentive to attempt fixes to test infrastructure.  Today, if the test classes have a problem, there is very little chance that I'll investigate the problem in the test classes, because I can't access updates to those test classes anyway.

Mark Waite

On Fri, Jan 8, 2016 at 10:16 AM Jesse Glick <<a href="javascript:_e(%7B%7D,&#39;cvml&#39;,&#39;jglick@cloudbees.com&#39;);" target="_blank">jglick@...> wrote:
Today the Jenkins functional test harness, containing `JenkinsRule`
and its ilk¹, is built from the Jenkins core repository, in
`test/src/main/`. That is fine for functional tests of core itself
(`test/src/test/`). But this scheme is undesirable for functional
tests in plugins, for two reasons.

One, whenever a fix is made to the test harness itself, such as my
recent PR 1967², it does not become available to plugins until they
switch their core baseline to the version including the fix. That
could be months away, even if the fix is backported to an LTS branch
as is occasionally done. Sometimes it is possible to work around the
lack of the fix in plugin tests³, but even when possible this is
awkward.

Second, incompatible changes to the test harness are sometimes
necessary or desirable. We try hard to avoid incompatible changes to
core itself—these could break user installations—but incompatibilities
of tests is a price paid only by plugin developers, usually
manageable. In particular, PR 1774⁴ changed some details of HtmlUnit
usage. Now if you upgrade your plugin’s core baseline to a version
after that point (generally 1.625.x), you update your tests to
match—fine. The problem comes as we try to expand usage of
`plugin-compat-tester`⁵ to hunt for core regressions. To verify that a
plugin’s tests still pass against a newer core, this tool updates the
baseline in a working copy of the POM. But if the tests are still
written against the old forked HtmlUnit, they may no longer compile;
and a single version of a test cannot always be made to work against
either version of HtmlUnit. You can see an example of this dilemma in
the `copyartifact` plugin⁶.

My proposal is to start treating the harness as an independently
versioned test library, in its own GitHub repository.
`jenkinsci/jenkins/test/src/main/` would be extracted there (with
history intact). `test/src/test/` core functional tests would pick it
up as a binary dependency, as plugin tests already do. The advantage
would be that we could iterate quickly on new features or fixes to the
harness, cutting a Maven release at any time. Core and each plugin
would decide which version to pick up; whenever you wish to update to
the latest version, you can edit your tests in the same commit to
reflect any required changes, or to take advantage of new features.
`plugin-compat-tester` would leave this version untouched.

In principle there could be a change to core which requires a
synchronized matching change to the test harness, which would become
more difficult to handle in such a system, but I am not aware of any
examples.⁷

After a split it should be possible to do further refactorings that
would make for smaller artifacts and better modularity. Currently most
of the size of the test harness artifact is consumed by bundling four
(!) copies of Maven, plus one copy of Ant, which are only relevant to
plugins actually testing builds using these tools. The harness also
has fixed dependencies on a number of plugins which were long since
split out of core—`subversion`, `maven-plugin`, `matrix-project`,
etc.—which thickens the `test`-scope dependency tree for every plugin
and can cause weird version resolution problems that complicate test
development and maintenance.

I am sending this proposal verbally rather than in a PR since
cross-repository refactorings like this are not easily handled with
PRs, especially since a new artifact release is required to get
everything working. There are some details that will only fall out of
an actual attempt to do it, but I wanted to get general agreement that
this is a good direction before starting on things like the slow Git
command to extract a subdirectory with history.

For context, there is also some work being prepared to create a new
plugin POM which, unlike the current one coversioned with core⁸, would
be in its own repository and independently versioned, with the core
version simply being a Maven property. Stephen Connolly and I have
long discussed the possibility—CloudBees proprietary plugins do
exactly this, and it is much more comfortable to work with as a plugin
developer and from `plugin-compat-tester`—but there is finally some
movement. The idea would be that with a refactored test harness, your
plugin POM would specify both the versions of core and the test
harness independently.

WDYT?


¹ https://github.com/jenkinsci/jenkins/blob/6306952779acc311719c175201e736685349e507/test/src/main/java/org/jvnet/hudson/test/JenkinsRule.java#L225-L234
² https://github.com/jenkinsci/jenkins/pull/1967
³ https://github.com/jenkinsci/workflow-plugin/pull/277
https://github.com/jenkinsci/jenkins/pull/1774
https://wiki.jenkins-ci.org/display/JENKINS/Plugin+Compatibility+Tester
https://github.com/jenkinsci/copyartifact-plugin/pull/76
https://github.com/jenkinsci/jenkins/commits/6306952779acc311719c175201e736685349e507/test/src/main
includes https://github.com/jenkinsci/jenkins/commit/6aabbbd282083716d631be7f2fa07ce93b93ee23#diff-8e0ebb909d001d948b257dbbc1450cfd
that would imply a newer core dep; this part of the patch was very low
priority. https://github.com/jenkinsci/jenkins/commit/b6ae4f5b798068fad48d292dee860fe97bdc7d55
and https://github.com/jenkinsci/jenkins/commit/fe1d4472015d978f45f219369fc7123bbdd558dd
handle core incompatibilities but are harmless to use with older
cores.
https://github.com/jenkinsci/jenkins/blob/6306952779acc311719c175201e736685349e507/plugins/pom.xml#L15

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to <a href="javascript:_e(%7B%7D,&#39;cvml&#39;,&#39;jenkinsci-dev%2Bunsubscribe@googlegroups.com&#39;);" target="_blank">jenkinsci-dev+unsubscribe@....
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CANfRfr1w18BohqOEpcnf0r_2PYvV-wmHzDEjLj%2BNst6WNtaOPA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to <a href="javascript:_e(%7B%7D,&#39;cvml&#39;,&#39;jenkinsci-dev%2Bunsubscribe@googlegroups.com&#39;);" target="_blank">jenkinsci-dev+unsubscribe@....
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CAO49JtGrzf5%3DtD3w8%2Bo7qSGjHBON5sG0cuxbRYwrVrrq9xwo2g%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to <a href="javascript:_e(%7B%7D,&#39;cvml&#39;,&#39;jenkinsci-dev%2Bunsubscribe@googlegroups.com&#39;);" target="_blank">jenkinsci-dev+unsubscribe@....
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CAPbPdOacnvj1wJBEKCkzt_bzr5U8arkjo7Kgvi_2-XuTbWdhLA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


--
Sent from my phone

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CA%2BnPnMy7haSLnQba1Y1dAzMPnpYQ7CPoB2i7S_kWs-HQw62XzA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: [PROPOSAL] Split of `jenkins-test-harness` out of `jenkinsci/jenkins`

Jesse Glick-4
In reply to this post by Andrew Bayer

Moving core tests into the same module would be great, but I am not going to promise that it works until I try it.

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CANfRfr2nvOBLB1LejpqmVVgB3Cpb6Y5myUESNDLad9R2MmYTwQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: [PROPOSAL] Split of `jenkins-test-harness` out of `jenkinsci/jenkins`

Arnaud Héritier
In reply to this post by stephenconnolly
+1 we did a similar on maven core and maven integration tests a long time ago
It is now very useful and help to apply all these tests to all maven core versions (with few conditions to exclude some tests for some versions)

On Fri, Jan 8, 2016 at 9:32 PM, Stephen Connolly <[hidden email]> wrote:
+1


On Friday 8 January 2016, Andrew Bayer <[hidden email]> wrote:
I'd be a big +1 on this as well. It makes sense to have the harness as a separate project consumed by Jenkins core and plugins for their testing. A nice side effect is that we'd be able to actually move the core tests currently in jenkins.git/test/src/test/... into jenkins.git/core/src/test/..., I think, which would just make things feel much more organic.

A.

On Fri, Jan 8, 2016 at 11:01 AM, Mark Waite <[hidden email]> wrote:
I like the idea of moving JenkinsRule outside jenkins core very much.  The git plugin and git client plugin intentionally support older Jenkins versions, but that means their test infrastructure is also limited to those older Jenkins versions.  I'd much rather be able to use the latest Jenkins test infrastructure to create better tests.  

That would also provide me more incentive to attempt fixes to test infrastructure.  Today, if the test classes have a problem, there is very little chance that I'll investigate the problem in the test classes, because I can't access updates to those test classes anyway.

Mark Waite

On Fri, Jan 8, 2016 at 10:16 AM Jesse Glick <[hidden email]> wrote:
Today the Jenkins functional test harness, containing `JenkinsRule`
and its ilk¹, is built from the Jenkins core repository, in
`test/src/main/`. That is fine for functional tests of core itself
(`test/src/test/`). But this scheme is undesirable for functional
tests in plugins, for two reasons.

One, whenever a fix is made to the test harness itself, such as my
recent PR 1967², it does not become available to plugins until they
switch their core baseline to the version including the fix. That
could be months away, even if the fix is backported to an LTS branch
as is occasionally done. Sometimes it is possible to work around the
lack of the fix in plugin tests³, but even when possible this is
awkward.

Second, incompatible changes to the test harness are sometimes
necessary or desirable. We try hard to avoid incompatible changes to
core itself—these could break user installations—but incompatibilities
of tests is a price paid only by plugin developers, usually
manageable. In particular, PR 1774⁴ changed some details of HtmlUnit
usage. Now if you upgrade your plugin’s core baseline to a version
after that point (generally 1.625.x), you update your tests to
match—fine. The problem comes as we try to expand usage of
`plugin-compat-tester`⁵ to hunt for core regressions. To verify that a
plugin’s tests still pass against a newer core, this tool updates the
baseline in a working copy of the POM. But if the tests are still
written against the old forked HtmlUnit, they may no longer compile;
and a single version of a test cannot always be made to work against
either version of HtmlUnit. You can see an example of this dilemma in
the `copyartifact` plugin⁶.

My proposal is to start treating the harness as an independently
versioned test library, in its own GitHub repository.
`jenkinsci/jenkins/test/src/main/` would be extracted there (with
history intact). `test/src/test/` core functional tests would pick it
up as a binary dependency, as plugin tests already do. The advantage
would be that we could iterate quickly on new features or fixes to the
harness, cutting a Maven release at any time. Core and each plugin
would decide which version to pick up; whenever you wish to update to
the latest version, you can edit your tests in the same commit to
reflect any required changes, or to take advantage of new features.
`plugin-compat-tester` would leave this version untouched.

In principle there could be a change to core which requires a
synchronized matching change to the test harness, which would become
more difficult to handle in such a system, but I am not aware of any
examples.⁷

After a split it should be possible to do further refactorings that
would make for smaller artifacts and better modularity. Currently most
of the size of the test harness artifact is consumed by bundling four
(!) copies of Maven, plus one copy of Ant, which are only relevant to
plugins actually testing builds using these tools. The harness also
has fixed dependencies on a number of plugins which were long since
split out of core—`subversion`, `maven-plugin`, `matrix-project`,
etc.—which thickens the `test`-scope dependency tree for every plugin
and can cause weird version resolution problems that complicate test
development and maintenance.

I am sending this proposal verbally rather than in a PR since
cross-repository refactorings like this are not easily handled with
PRs, especially since a new artifact release is required to get
everything working. There are some details that will only fall out of
an actual attempt to do it, but I wanted to get general agreement that
this is a good direction before starting on things like the slow Git
command to extract a subdirectory with history.

For context, there is also some work being prepared to create a new
plugin POM which, unlike the current one coversioned with core⁸, would
be in its own repository and independently versioned, with the core
version simply being a Maven property. Stephen Connolly and I have
long discussed the possibility—CloudBees proprietary plugins do
exactly this, and it is much more comfortable to work with as a plugin
developer and from `plugin-compat-tester`—but there is finally some
movement. The idea would be that with a refactored test harness, your
plugin POM would specify both the versions of core and the test
harness independently.

WDYT?


¹ https://github.com/jenkinsci/jenkins/blob/6306952779acc311719c175201e736685349e507/test/src/main/java/org/jvnet/hudson/test/JenkinsRule.java#L225-L234
² https://github.com/jenkinsci/jenkins/pull/1967
³ https://github.com/jenkinsci/workflow-plugin/pull/277
https://github.com/jenkinsci/jenkins/pull/1774
https://wiki.jenkins-ci.org/display/JENKINS/Plugin+Compatibility+Tester
https://github.com/jenkinsci/copyartifact-plugin/pull/76
https://github.com/jenkinsci/jenkins/commits/6306952779acc311719c175201e736685349e507/test/src/main
includes https://github.com/jenkinsci/jenkins/commit/6aabbbd282083716d631be7f2fa07ce93b93ee23#diff-8e0ebb909d001d948b257dbbc1450cfd
that would imply a newer core dep; this part of the patch was very low
priority. https://github.com/jenkinsci/jenkins/commit/b6ae4f5b798068fad48d292dee860fe97bdc7d55
and https://github.com/jenkinsci/jenkins/commit/fe1d4472015d978f45f219369fc7123bbdd558dd
handle core incompatibilities but are harmless to use with older
cores.
https://github.com/jenkinsci/jenkins/blob/6306952779acc311719c175201e736685349e507/plugins/pom.xml#L15

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CANfRfr1w18BohqOEpcnf0r_2PYvV-wmHzDEjLj%2BNst6WNtaOPA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CAO49JtGrzf5%3DtD3w8%2Bo7qSGjHBON5sG0cuxbRYwrVrrq9xwo2g%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CAPbPdOacnvj1wJBEKCkzt_bzr5U8arkjo7Kgvi_2-XuTbWdhLA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


--
Sent from my phone

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CA%2BnPnMy7haSLnQba1Y1dAzMPnpYQ7CPoB2i7S_kWs-HQw62XzA%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.



--
-----
Arnaud Héritier
Mail/GTalk: aheritier AT gmail DOT com
Twitter/Skype : aheritier

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CAFNCU-_oHTeKJvZkkTa9WahzafC0h9qG-Y2FwnZPyy2Sb1iuRQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: [PROPOSAL] Split of `jenkins-test-harness` out of `jenkinsci/jenkins`

Jesse Glick-4

Well I was not proposing to move the core tests themselves. Just the harness part. The tests are routinely edited in conjunction with code changes.

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CANfRfr2ky-trCRJoRh4Azw2kDAh-RzTLxQs315yL_fVcTXFayw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: [PROPOSAL] Split of `jenkins-test-harness` out of `jenkinsci/jenkins`

Oleg Nenashev
+1 for decoupling.
I suppose we want to have multiple independent libs instead of a single multi-module Maven monster. BTW the test harness is simple enough to be kept as a single library.

суббота, 9 января 2016 г., 4:06:24 UTC+3 пользователь Jesse Glick написал:

Well I was not proposing to move the core tests themselves. Just the harness part. The tests are routinely edited in conjunction with code changes.

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/c77dcc68-6395-4e1f-93c9-18c19b58bd8c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: [PROPOSAL] Split of `jenkins-test-harness` out of `jenkinsci/jenkins`

Kanstantsin Shautsou-2
In reply to this post by Jesse Glick-4
Jesse, would it include issues that i described on IRC yesterday?
- plugin extraction happens only from hpi context. Allow using JenkinsRule from modules.
- unhardcode JenkinsRule from been localhost (have working test example with docker + CLI, but it re-implements parts of rule)

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/c6521d12-31e5-4baa-9238-46d87334c40d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: [PROPOSAL] Split of `jenkins-test-harness` out of `jenkinsci/jenkins`

Jesse Glick-4

No, my proposal was only to move the code, not change its behavior.

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CANfRfr1OmTwyhV%3DbmJKAewtcnPdiU1mNfDv%3DPbCOOin7JSGkZQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: [PROPOSAL] Split of `jenkins-test-harness` out of `jenkinsci/jenkins`

Robert Sandell-2
+1



On Sat, Jan 9, 2016 at 3:17 PM, Jesse Glick <[hidden email]> wrote:

No, my proposal was only to move the code, not change its behavior.

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CANfRfr1OmTwyhV%3DbmJKAewtcnPdiU1mNfDv%3DPbCOOin7JSGkZQ%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.



--
Robert Sandell
Software Engineer
CloudBees Inc.

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CALzHZS3xj2PbYYD4BqS920jZyvSmd2%3DhL7V6JQqzS0g4_3%2B44w%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: [PROPOSAL] Split of `jenkins-test-harness` out of `jenkinsci/jenkins`

Kohsuke Kawaguchi
Administrator
In reply to this post by Jesse Glick-4
+1, sounds good to me.

2016-01-08 9:16 GMT-08:00 Jesse Glick <[hidden email]>:
Today the Jenkins functional test harness, containing `JenkinsRule`
and its ilk¹, is built from the Jenkins core repository, in
`test/src/main/`. That is fine for functional tests of core itself
(`test/src/test/`). But this scheme is undesirable for functional
tests in plugins, for two reasons.

One, whenever a fix is made to the test harness itself, such as my
recent PR 1967², it does not become available to plugins until they
switch their core baseline to the version including the fix. That
could be months away, even if the fix is backported to an LTS branch
as is occasionally done. Sometimes it is possible to work around the
lack of the fix in plugin tests³, but even when possible this is
awkward.

Second, incompatible changes to the test harness are sometimes
necessary or desirable. We try hard to avoid incompatible changes to
core itself—these could break user installations—but incompatibilities
of tests is a price paid only by plugin developers, usually
manageable. In particular, PR 1774⁴ changed some details of HtmlUnit
usage. Now if you upgrade your plugin’s core baseline to a version
after that point (generally 1.625.x), you update your tests to
match—fine. The problem comes as we try to expand usage of
`plugin-compat-tester`⁵ to hunt for core regressions. To verify that a
plugin’s tests still pass against a newer core, this tool updates the
baseline in a working copy of the POM. But if the tests are still
written against the old forked HtmlUnit, they may no longer compile;
and a single version of a test cannot always be made to work against
either version of HtmlUnit. You can see an example of this dilemma in
the `copyartifact` plugin⁶.

My proposal is to start treating the harness as an independently
versioned test library, in its own GitHub repository.
`jenkinsci/jenkins/test/src/main/` would be extracted there (with
history intact). `test/src/test/` core functional tests would pick it
up as a binary dependency, as plugin tests already do. The advantage
would be that we could iterate quickly on new features or fixes to the
harness, cutting a Maven release at any time. Core and each plugin
would decide which version to pick up; whenever you wish to update to
the latest version, you can edit your tests in the same commit to
reflect any required changes, or to take advantage of new features.
`plugin-compat-tester` would leave this version untouched.

In principle there could be a change to core which requires a
synchronized matching change to the test harness, which would become
more difficult to handle in such a system, but I am not aware of any
examples.⁷

After a split it should be possible to do further refactorings that
would make for smaller artifacts and better modularity. Currently most
of the size of the test harness artifact is consumed by bundling four
(!) copies of Maven, plus one copy of Ant, which are only relevant to
plugins actually testing builds using these tools. The harness also
has fixed dependencies on a number of plugins which were long since
split out of core—`subversion`, `maven-plugin`, `matrix-project`,
etc.—which thickens the `test`-scope dependency tree for every plugin
and can cause weird version resolution problems that complicate test
development and maintenance.

I am sending this proposal verbally rather than in a PR since
cross-repository refactorings like this are not easily handled with
PRs, especially since a new artifact release is required to get
everything working. There are some details that will only fall out of
an actual attempt to do it, but I wanted to get general agreement that
this is a good direction before starting on things like the slow Git
command to extract a subdirectory with history.

For context, there is also some work being prepared to create a new
plugin POM which, unlike the current one coversioned with core⁸, would
be in its own repository and independently versioned, with the core
version simply being a Maven property. Stephen Connolly and I have
long discussed the possibility—CloudBees proprietary plugins do
exactly this, and it is much more comfortable to work with as a plugin
developer and from `plugin-compat-tester`—but there is finally some
movement. The idea would be that with a refactored test harness, your
plugin POM would specify both the versions of core and the test
harness independently.

WDYT?


¹ https://github.com/jenkinsci/jenkins/blob/6306952779acc311719c175201e736685349e507/test/src/main/java/org/jvnet/hudson/test/JenkinsRule.java#L225-L234
² https://github.com/jenkinsci/jenkins/pull/1967
³ https://github.com/jenkinsci/workflow-plugin/pull/277
https://github.com/jenkinsci/jenkins/pull/1774
https://wiki.jenkins-ci.org/display/JENKINS/Plugin+Compatibility+Tester
https://github.com/jenkinsci/copyartifact-plugin/pull/76
https://github.com/jenkinsci/jenkins/commits/6306952779acc311719c175201e736685349e507/test/src/main
includes https://github.com/jenkinsci/jenkins/commit/6aabbbd282083716d631be7f2fa07ce93b93ee23#diff-8e0ebb909d001d948b257dbbc1450cfd
that would imply a newer core dep; this part of the patch was very low
priority. https://github.com/jenkinsci/jenkins/commit/b6ae4f5b798068fad48d292dee860fe97bdc7d55
and https://github.com/jenkinsci/jenkins/commit/fe1d4472015d978f45f219369fc7123bbdd558dd
handle core incompatibilities but are harmless to use with older
cores.
https://github.com/jenkinsci/jenkins/blob/6306952779acc311719c175201e736685349e507/plugins/pom.xml#L15

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CANfRfr1w18BohqOEpcnf0r_2PYvV-wmHzDEjLj%2BNst6WNtaOPA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.



--
Kohsuke Kawaguchi

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CAN4CQ4yByQKcLjCeSbZoC1qs%3DdhxQf_-LUyvqqWvHU-kMjcK9w%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: [PROPOSAL] Split of `jenkins-test-harness` out of `jenkinsci/jenkins`

Baptiste MATHUS
+1 for splitting. I also like the idea of being able to switch Jenkins version quickly with a prop like you or Stephen described some time ago.

2016-01-11 19:13 GMT+01:00 Kohsuke Kawaguchi <[hidden email]>:
+1, sounds good to me.

2016-01-08 9:16 GMT-08:00 Jesse Glick <[hidden email]>:
Today the Jenkins functional test harness, containing `JenkinsRule`
and its ilk¹, is built from the Jenkins core repository, in
`test/src/main/`. That is fine for functional tests of core itself
(`test/src/test/`). But this scheme is undesirable for functional
tests in plugins, for two reasons.

One, whenever a fix is made to the test harness itself, such as my
recent PR 1967², it does not become available to plugins until they
switch their core baseline to the version including the fix. That
could be months away, even if the fix is backported to an LTS branch
as is occasionally done. Sometimes it is possible to work around the
lack of the fix in plugin tests³, but even when possible this is
awkward.

Second, incompatible changes to the test harness are sometimes
necessary or desirable. We try hard to avoid incompatible changes to
core itself—these could break user installations—but incompatibilities
of tests is a price paid only by plugin developers, usually
manageable. In particular, PR 1774⁴ changed some details of HtmlUnit
usage. Now if you upgrade your plugin’s core baseline to a version
after that point (generally 1.625.x), you update your tests to
match—fine. The problem comes as we try to expand usage of
`plugin-compat-tester`⁵ to hunt for core regressions. To verify that a
plugin’s tests still pass against a newer core, this tool updates the
baseline in a working copy of the POM. But if the tests are still
written against the old forked HtmlUnit, they may no longer compile;
and a single version of a test cannot always be made to work against
either version of HtmlUnit. You can see an example of this dilemma in
the `copyartifact` plugin⁶.

My proposal is to start treating the harness as an independently
versioned test library, in its own GitHub repository.
`jenkinsci/jenkins/test/src/main/` would be extracted there (with
history intact). `test/src/test/` core functional tests would pick it
up as a binary dependency, as plugin tests already do. The advantage
would be that we could iterate quickly on new features or fixes to the
harness, cutting a Maven release at any time. Core and each plugin
would decide which version to pick up; whenever you wish to update to
the latest version, you can edit your tests in the same commit to
reflect any required changes, or to take advantage of new features.
`plugin-compat-tester` would leave this version untouched.

In principle there could be a change to core which requires a
synchronized matching change to the test harness, which would become
more difficult to handle in such a system, but I am not aware of any
examples.⁷

After a split it should be possible to do further refactorings that
would make for smaller artifacts and better modularity. Currently most
of the size of the test harness artifact is consumed by bundling four
(!) copies of Maven, plus one copy of Ant, which are only relevant to
plugins actually testing builds using these tools. The harness also
has fixed dependencies on a number of plugins which were long since
split out of core—`subversion`, `maven-plugin`, `matrix-project`,
etc.—which thickens the `test`-scope dependency tree for every plugin
and can cause weird version resolution problems that complicate test
development and maintenance.

I am sending this proposal verbally rather than in a PR since
cross-repository refactorings like this are not easily handled with
PRs, especially since a new artifact release is required to get
everything working. There are some details that will only fall out of
an actual attempt to do it, but I wanted to get general agreement that
this is a good direction before starting on things like the slow Git
command to extract a subdirectory with history.

For context, there is also some work being prepared to create a new
plugin POM which, unlike the current one coversioned with core⁸, would
be in its own repository and independently versioned, with the core
version simply being a Maven property. Stephen Connolly and I have
long discussed the possibility—CloudBees proprietary plugins do
exactly this, and it is much more comfortable to work with as a plugin
developer and from `plugin-compat-tester`—but there is finally some
movement. The idea would be that with a refactored test harness, your
plugin POM would specify both the versions of core and the test
harness independently.

WDYT?


¹ https://github.com/jenkinsci/jenkins/blob/6306952779acc311719c175201e736685349e507/test/src/main/java/org/jvnet/hudson/test/JenkinsRule.java#L225-L234
² https://github.com/jenkinsci/jenkins/pull/1967
³ https://github.com/jenkinsci/workflow-plugin/pull/277
https://github.com/jenkinsci/jenkins/pull/1774
https://wiki.jenkins-ci.org/display/JENKINS/Plugin+Compatibility+Tester
https://github.com/jenkinsci/copyartifact-plugin/pull/76
https://github.com/jenkinsci/jenkins/commits/6306952779acc311719c175201e736685349e507/test/src/main
includes https://github.com/jenkinsci/jenkins/commit/6aabbbd282083716d631be7f2fa07ce93b93ee23#diff-8e0ebb909d001d948b257dbbc1450cfd
that would imply a newer core dep; this part of the patch was very low
priority. https://github.com/jenkinsci/jenkins/commit/b6ae4f5b798068fad48d292dee860fe97bdc7d55
and https://github.com/jenkinsci/jenkins/commit/fe1d4472015d978f45f219369fc7123bbdd558dd
handle core incompatibilities but are harmless to use with older
cores.
https://github.com/jenkinsci/jenkins/blob/6306952779acc311719c175201e736685349e507/plugins/pom.xml#L15

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CANfRfr1w18BohqOEpcnf0r_2PYvV-wmHzDEjLj%2BNst6WNtaOPA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.



--
Kohsuke Kawaguchi

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CAN4CQ4yByQKcLjCeSbZoC1qs%3DdhxQf_-LUyvqqWvHU-kMjcK9w%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CANWgJS7Z7jCxr0Nn4b_jTnQRen1-zD5kRQ50%3DmRgrjjVK1PJ4A%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: [PROPOSAL] Split of `jenkins-test-harness` out of `jenkinsci/jenkins`

James Nord-2
Sounds good to me

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/b08be9cb-c39f-4794-ba35-420b12e0290a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: [PROPOSAL] Split of `jenkins-test-harness` out of `jenkinsci/jenkins`

Manuel Jesús Recena Soto-2
I always try to avoid big repositories. In this case, there are
several reason to split it.

+1

Maybe a good moment to review this page:
https://wiki.jenkins-ci.org/display/JENKINS/Unit+Test#UnitTest-Overview

Regards,

2016-01-12 22:08 GMT+01:00 James Nord <[hidden email]>:
> Sounds good to me
>
> --
> You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
> To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/b08be9cb-c39f-4794-ba35-420b12e0290a%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.



--
Manuel Recena Soto
* manuelrecena.com [/blog]
* linkedin.com/in/recena

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CABa-Uofktu0-nOFFhxKBD7DOZctps5%2BdC6OB_bC2OLtzq8dPLg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: [PROPOSAL] Split of `jenkins-test-harness` out of `jenkinsci/jenkins`

Jesse Glick-4
In reply to this post by Jesse Glick-4
https://github.com/jenkinsci/jenkins/pull/1977 as a starting point.

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CANfRfr20m9jNE0rDpvfW-2%2BMLRYBwdKLhDad1ujahOTg5A2H3Q%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: [PROPOSAL] Split of `jenkins-test-harness` out of `jenkinsci/jenkins`

Jesse Glick-4
To continue: https://github.com/jenkinsci/jenkins/pull/1982

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CANfRfr0FphaQbBCuQMmkvNe2VTGEJm_7DXtDhD%2BJ%2BTHpbXg27g%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: [PROPOSAL] Split of `jenkins-test-harness` out of `jenkinsci/jenkins`

Jesse Glick-4
Completed toward 1.645.

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CANfRfr3nZhhyGntnGXUd%3D6W7UtpX5Mc%2Bq%3D56Fh0CbbKaOEoF8g%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.