Alternative directory implementation of ArtifactManager?

classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|

Alternative directory implementation of ArtifactManager?

Staffan Forsell
Hello,
I have been looking for a way moving the artifacts out of the builds directory and have been looking at the ArtifactManager. 
Is there a reason that I can't find an implementation of this that just uses a different local directory on the master?
Would should an plugin be hard to implement and how would one start?
Is there any obvious reasons that just creating a system property that switches the artifacts to a different dir would not work?

Our use-case is that we create a lot of artifacts (zips/vms etc) that we want to save for testing. We do this by changing the whole buildsDir to a glusterfs. This gives us massive amounts of space but the less than ideal performance with many small files, so we would like to separate the rest of the jobs and build files from the artifacts.

Best regards Staffan

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/00cbdc00-ed36-4a4d-9187-4af486e44822%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Alternative directory implementation of ArtifactManager?

Markus Winter
Implementation is quite easy I would say.
You need an ArtifactManager (look at the StandardArtifactManager, you can probably just copy and replace the getArtifactsDir() method)
And an ArtifactManagerFactory where you can configure the root path for the artifacts


On 30.11.2018 10:40, Staffan Forsell wrote:
Hello,
I have been looking for a way moving the artifacts out of the builds directory and have been looking at the ArtifactManager. 
Is there a reason that I can't find an implementation of this that just uses a different local directory on the master?
Would should an plugin be hard to implement and how would one start?
Is there any obvious reasons that just creating a system property that switches the artifacts to a different dir would not work?

Our use-case is that we create a lot of artifacts (zips/vms etc) that we want to save for testing. We do this by changing the whole buildsDir to a glusterfs. This gives us massive amounts of space but the less than ideal performance with many small files, so we would like to separate the rest of the jobs and build files from the artifacts.

Best regards Staffan
--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/00cbdc00-ed36-4a4d-9187-4af486e44822%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/6852ca44-3eac-ce6e-63de-32e1ee921daf%40gmx.de.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Alternative directory implementation of ArtifactManager?

Daniel Beck
In reply to this post by Staffan Forsell


> On 30. Nov 2018, at 10:40, Staffan Forsell <[hidden email]> wrote:
>
> Is there any obvious reasons that just creating a system property that switches the artifacts to a different dir would not work?

Given how many regressions the custom builds dir introduced due to moving jobs between folders, renaming them, etc., and all those actions newly having to be mirrored in a different structure, I'm not surprised nobody bothered -- just having custom builds dir is good enough in many cases. Would not recommend attempting it with using job names for the lookup.

Would look at artifact-manager-s3 as a sort of template for custom implementation for artifact storage.

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/844502C8-816C-4FC6-9431-A17ACBC8CD8C%40beckweb.net.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Alternative directory implementation of ArtifactManager?

Staffan Forsell
artifact-manager-s3 seems to contain examples of most of the things needed. 
I did not quite understand...do you recommend against trying to do this or something else? If we would implement the same tests and use the same key as artifact-manager-s3, I would thing it would not be unreasonable work?

I figure if there is infrastructure now for this to work on s3 or azure, then just mapping it to a different directory should be pretty simple in my mind.

Another way of doing this would be to just invest in lots of SSD storage for our jenkins servers... Might be cheaper if this would prove to be very complicated.


On Friday, 30 November 2018 13:30:37 UTC+1, Daniel Beck wrote:


> On 30. Nov 2018, at 10:40, Staffan Forsell <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="pdyc3SIOBQAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">sta...@...> wrote:
>
> Is there any obvious reasons that just creating a system property that switches the artifacts to a different dir would not work?

Given how many regressions the custom builds dir introduced due to moving jobs between folders, renaming them, etc., and all those actions newly having to be mirrored in a different structure, I'm not surprised nobody bothered -- just having custom builds dir is good enough in many cases. Would not recommend attempting it with using job names for the lookup.

Would look at artifact-manager-s3 as a sort of template for custom implementation for artifact storage.

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/5b9104a7-43b9-4980-b193-6c7a6b4d0cb1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Alternative directory implementation of ArtifactManager?

Jesse Glick-4
On Fri, Nov 30, 2018 at 8:59 AM Staffan Forsell <[hidden email]> wrote:
> Another way of doing this would be to just invest in lots of SSD storage for our jenkins servers.

Or just changing your projects to deploy artifacts to a proper
repository manager. The `archiveArtifacts` step and its kin is
intended only as a simple built-in way to retain smallish files if you
have not yet bothered to set up a real system.

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CANfRfr3dFiBDEAsv%2BgeoGAFckAwfv1h1h1_deE4pAgbnokCQ0Q%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Alternative directory implementation of ArtifactManager?

Staffan Forsell
Well, we have lots of jobs, both freestyle and different styles of the new pipeline, not all for sw that's currently in active development but we still need to know we can build these.
We would rather invest our time in something that's useful for all jobs without changing all those old job definitions.
Also everything that we keep for a while on Jenkins as an artifact is not necessarily something that we would deploy to artifactory. Coupling  the artifact retention to the number of builds in Jenkins is often easier than having a "works for all cleanup policy" in artifactory.

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/429aa64d-c305-4bbe-b6a9-67a4c08a232c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Alternative directory implementation of ArtifactManager?

Jesse Glick-4
In reply to this post by Staffan Forsell
On Fri, Nov 30, 2018 at 8:59 AM Staffan Forsell <[hidden email]> wrote:
> artifact-manager-s3 seems to contain examples of most of the things needed.

Too complicated for your purpose. Creating a plugin with a lightly
edited clone of `StandardArtifactManager` would suffice. (Use the
Maven archetype for a plugin with global configuration, so the user
can pick the location in the GUI or using configuration-as-code.)

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CANfRfr0zFubqsfZeoW3uKeGESuky957ABM-4wjVGMh7tXs15BQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Alternative directory implementation of ArtifactManager?

Staffan Forsell
I started and got the basic function and configuration working.

How are moves and renames handled? E.g. if a admin moves a job to a folder? How does the ArtifactManager handle this?
I was thinking of saving the redirected artifact build dir in the build.xml in order to be able to keep the artifacts working in case of a move.
The alternative is to move the artifacts so they match the new build dir structure in the alternative location, but I don't know where I would do this...
I found StashAwareArtifactManager.copyAllArtifactsAndStashes() from the s3-plugn implementation, but this implies handling stashes also which is not in the scope for my ArtifactManager.
Is there a correct way of handling artifacts when jobs are moved?

/S

On Friday, 30 November 2018 20:06:28 UTC+1, Jesse Glick wrote:
On Fri, Nov 30, 2018 at 8:59 AM Staffan Forsell <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="4xOK2LwjBQAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">sta...@...> wrote:
> artifact-manager-s3 seems to contain examples of most of the things needed.

Too complicated for your purpose. Creating a plugin with a lightly
edited clone of `StandardArtifactManager` would suffice. (Use the
Maven archetype for a plugin with global configuration, so the user
can pick the location in the GUI or using configuration-as-code.)

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/a4d2e037-43a1-4f2c-abe3-286717cce3bf%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Alternative directory implementation of ArtifactManager?

Jesse Glick-4
On Tue, Dec 4, 2018 at 3:48 AM Staffan Forsell <[hidden email]> wrote:
> How are moves and renames handled? E.g. if a admin moves a job to a folder? How does the ArtifactManager handle this?

For `StandardArtifactManager`, that would be implicit in the job
directory itself moving. For others, including `artifact-manager-s3`,
this is not handled automatically. `JCloudsArtifactManager.key` could
perhaps be made non-`transient` to track the originally stored
location, or there could be an `ItemListener.onLocationChanged` to
physically relocate the blob(s), or the `unique-id` plugin could be
relied on for another key which persists across moves. For now we
decided to keep things simple and transparent—there is a predictable
blob location based on what you see in Jenkins—and just punt on the
use case of historical builds preceding a move/rename.

(On a related note, by default `artifact-manager-s3` declines to
remove artifacts after build deletion in Jenkins, on the grounds that
AWS retention policies and Glacier migration are likely better, and we
want the Jenkins master to require minimal IAM permissions: a
compromised master should not be able to destroy or alter cloud data.)

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CANfRfr2Wii6KnaMpkz_oAsLyH74JbkP%3Da4MEk8-twJHgWRu7tA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Alternative directory implementation of ArtifactManager?

Staffan Forsell
Well for us such retention policys does not apply and the time to keep artifacts is better handled though jenkins since it created them, than by other time based policies. We typically keep a number of artifacts per job while some builds are kept forever, typically release builds for traceability.
Well now I got most of it working but I discovered in my unit tests that deleting the job does not remove the artifacts, but removing the build does.
It seems that removing the job only recursively removed the job directory, see hudson.model.Job#delete.
To me this seems like a bug, it makes it harder to clean clean artifacts for a custom ArtifactsManager.
I managed to get around it using ItemListener.onDelete() to call run.deleteArtifacts() but I kind of think that this should be jenkins own responsibility while deleting a job.
Should I file an issue or do you have an other opinion?

On Tuesday, 4 December 2018 16:37:26 UTC+1, Jesse Glick wrote:
On Tue, Dec 4, 2018 at 3:48 AM Staffan Forsell <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="Pofx7KZSBgAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">sta...@...> wrote:
> How are moves and renames handled? E.g. if a admin moves a job to a folder? How does the ArtifactManager handle this?

For `StandardArtifactManager`, that would be implicit in the job
directory itself moving. For others, including `artifact-manager-s3`,
this is not handled automatically. `JCloudsArtifactManager.key` could
perhaps be made non-`transient` to track the originally stored
location, or there could be an `ItemListener.onLocationChanged` to
physically relocate the blob(s), or the `unique-id` plugin could be
relied on for another key which persists across moves. For now we
decided to keep things simple and transparent—there is a predictable
blob location based on what you see in Jenkins—and just punt on the
use case of historical builds preceding a move/rename.

(On a related note, by default `artifact-manager-s3` declines to
remove artifacts after build deletion in Jenkins, on the grounds that
AWS retention policies and Glacier migration are likely better, and we
want the Jenkins master to require minimal IAM permissions: a
compromised master should not be able to destroy or alter cloud data.)

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/c4e02423-e915-44b2-a5bc-0f2280b3764b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Alternative directory implementation of ArtifactManager?

Jesse Glick-4
On Wed, Dec 5, 2018 at 4:33 PM Staffan Forsell <[hidden email]> wrote:
> I managed to get around it using ItemListener.onDelete() to call run.deleteArtifacts() but I kind of think that this should be jenkins own responsibility while deleting a job.

Perhaps. `Job.delete` could call `ArtifactManager.delete` on every
`Run`. But this would be inefficient for `StandardArtifactManager`
(much slower than just deleting the whole job directory), and useless
overhead for `JCloudsArtifactManager` (which would typically not
delete anything). For now I would just be inclined to leave the
`ItemListener` workaround as is.

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CANfRfr2L7merKrDFFJiyF6NYG8H_wRtk78CQtVu_8TDm08BxeQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.