Many Freestyle jobs or many Jenkinsfiles?

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

Many Freestyle jobs or many Jenkinsfiles?

Al Silver
I've used Jenkins a little but not in the traditional method of CI/CD. I'm using Jenkins as a Web UI to run individual python scripts for networking devices... Users would log into Jenkins and then select any one of numerous Freestyle projects (with parameters) to run (there could be 100 or more...).    The job executes a shell which runs a pytest script that logs into (via ssh) multiple networking devices and performs some actions... pretty straightforward and it works ok but has some limitations.  With that said, I think I'll have more flexibility and power building and maintaining these jobs as pipelines and treating them as code.  My question is I guess I would need to convert my pipelines into Jenkinsfiles?  However, most of the docs I read show a single Jenkinsfile (named "Jenkinsfile") I would need one for each job (a 100 or so).  I must be missing something here or not understanding.  Can someone point in the right direction for what I'm trying to accomplish?

Thx
Al

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/64f8998b-67f2-48ba-a0fa-6cdbdd4113ab%40googlegroups.com.
Reply | Threaded
Open this post in threaded view
|

Re: Many Freestyle jobs or many Jenkinsfiles?

Anatoly Utkin
You can, of course, configure a pipeline and use any groovy file that you check out from SCM, Jenkinsfile is the general name of that file, and in some cases default, but see here n.5:
https://www.jenkins.io/doc/book/pipeline/getting-started/#defining-a-pipeline-in-scm

Anatoly.

On Monday, June 1, 2020 at 6:37:46 PM UTC+3, Al Silver wrote:
I've used Jenkins a little but not in the traditional method of CI/CD. I'm using Jenkins as a Web UI to run individual python scripts for networking devices... Users would log into Jenkins and then select any one of numerous Freestyle projects (with parameters) to run (there could be 100 or more...).    The job executes a shell which runs a pytest script that logs into (via ssh) multiple networking devices and performs some actions... pretty straightforward and it works ok but has some limitations.  With that said, I think I'll have more flexibility and power building and maintaining these jobs as pipelines and treating them as code.  My question is I guess I would need to convert my pipelines into Jenkinsfiles?  However, most of the docs I read show a single Jenkinsfile (named "Jenkinsfile") I would need one for each job (a 100 or so).  I must be missing something here or not understanding.  Can someone point in the right direction for what I'm trying to accomplish?

Thx
Al

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/efa19331-a38f-4660-bafa-528abf24b158%40googlegroups.com.
Reply | Threaded
Open this post in threaded view
|

RE: Many Freestyle jobs or many Jenkinsfiles?

Randy Beckwith
In reply to this post by Al Silver

As far as I know, “Jenkinsfile” is just a naming convention. I have multiple Jenkins files:

 

JenkinsFile.Parallel.groovy

JenkinsFile.Parallel.Snapshot.groovy

JenkinsFile.PatchWeekend-VmManager.groovy

 

I also add the .groovy extension so that editors can do syntax highlighting.

 

Randy

 

 

From: [hidden email] [mailto:[hidden email]] On Behalf Of Al Silver
Sent: Monday, June 1, 2020 11:21 AM
To: Jenkins Users <[hidden email]>
Subject: Many Freestyle jobs or many Jenkinsfiles?

 

I've used Jenkins a little but not in the traditional method of CI/CD. I'm using Jenkins as a Web UI to run individual python scripts for networking devices... Users would log into Jenkins and then select any one of numerous Freestyle projects (with parameters) to run (there could be 100 or more...).    The job executes a shell which runs a pytest script that logs into (via ssh) multiple networking devices and performs some actions... pretty straightforward and it works ok but has some limitations.  With that said, I think I'll have more flexibility and power building and maintaining these jobs as pipelines and treating them as code.  My question is I guess I would need to convert my pipelines into Jenkinsfiles?  However, most of the docs I read show a single Jenkinsfile (named "Jenkinsfile") I would need one for each job (a 100 or so).  I must be missing something here or not understanding.  Can someone point in the right direction for what I'm trying to accomplish?

 

Thx

Al

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/64f8998b-67f2-48ba-a0fa-6cdbdd4113ab%40googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/CY4PR12MB18007C2DFA8386B61DA29D00BC8B0%40CY4PR12MB1800.namprd12.prod.outlook.com.
Reply | Threaded
Open this post in threaded view
|

Re: Many Freestyle jobs or many Jenkinsfiles?

Giles
In reply to this post by Al Silver
I wrote a large library file full of Groovy methods.  I have a big stack of Jenkinsfiles, similar to Randy Beckworth, although mine are generally named "<purpose>/<environment>.Jenkinsfile" (as you can see, Jenkins isn't very particular about naming).  The Jenkinsfiles are generally very short, just long enough to call one of the library methods with a set of parameters.  We have 50-80 Jenkins jobs, and each of those has its own Jenkins file.  The initial set-up was tedious, but now that it's all in place it works very well.  Not sure if you could call a Python library, or if you'll end up using Groovy to call Python scripts.  My Groovy calls a lot of shell scripts - it's ugly, but works surprisingly well so you can make that work if you have to.

I think there are plugins and/or programmatic methods of generating jobs to avoid doing a lot of that tedious work yourself, but I also think you'd have to be looking at a very big setup (thousands of jobs) before it would be worth the investment of time to understand those plugins/methods.

Learning Jenkins' version of Groovy is a bit of a trip: it has a lot of its own methods, and I've found some weird edge cases where Jenkins' Groovy doesn't behave the same as standard Groovy.  All that said - it's definitely been worth the work.

On Monday, 1 June 2020 11:37:46 UTC-4, Al Silver wrote:
I've used Jenkins a little but not in the traditional method of CI/CD. I'm using Jenkins as a Web UI to run individual python scripts for networking devices... Users would log into Jenkins and then select any one of numerous Freestyle projects (with parameters) to run (there could be 100 or more...).    The job executes a shell which runs a pytest script that logs into (via ssh) multiple networking devices and performs some actions... pretty straightforward and it works ok but has some limitations.  With that said, I think I'll have more flexibility and power building and maintaining these jobs as pipelines and treating them as code.  My question is I guess I would need to convert my pipelines into Jenkinsfiles?  However, most of the docs I read show a single Jenkinsfile (named "Jenkinsfile") I would need one for each job (a 100 or so).  I must be missing something here or not understanding.  Can someone point in the right direction for what I'm trying to accomplish?

Thx
Al

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/4cae1d19-a93d-49f5-8257-96912ea49279%40googlegroups.com.
Reply | Threaded
Open this post in threaded view
|

Re: Many Freestyle jobs or many Jenkinsfiles?

David Riemens
similar to other users I also have a quite large set of jobs that use the (scripted, not declarative) Groovy pipelines. In my setup I have deliberately chosen to:
  • store all the Groovy scripts in SCM
  • have a seperate dedicated repository for all my Groovy scripts;
    alternative would be to store the JenkinsFile in the same repo as where the test data is stored.
  • use groovy 'load' to pull in generic utility code.
    alternative is using shared libraries; I prefer local load since it allows me to use SCM branches of my jenkins repository

I also started from historic Freestyle jobs. First thing I did is transform the Freestyle job into one that calls a single script with parameters from a repo.
Then create a very simple pipeline job that call that very script. Then slowly start taking more parts of the script into the groovy code.
For me doing it big-bang was a no-go. Have been hit by the Groovy diffs similar to what @Giles also mentioned. Take small steps. Debugging Groovy
in Jenkins is extremely hard. Investigate the use of "Replay" !
Once you have the first job running, the rest will be much easier.
good luck.

On Tuesday, June 2, 2020, 09:58:35 PM GMT+2, Giles <[hidden email]> wrote:


I wrote a large library file full of Groovy methods.  I have a big stack of Jenkinsfiles, similar to Randy Beckworth, although mine are generally named "<purpose>/<environment>.Jenkinsfile" (as you can see, Jenkins isn't very particular about naming).  The Jenkinsfiles are generally very short, just long enough to call one of the library methods with a set of parameters.  We have 50-80 Jenkins jobs, and each of those has its own Jenkins file.  The initial set-up was tedious, but now that it's all in place it works very well.  Not sure if you could call a Python library, or if you'll end up using Groovy to call Python scripts.  My Groovy calls a lot of shell scripts - it's ugly, but works surprisingly well so you can make that work if you have to.

I think there are plugins and/or programmatic methods of generating jobs to avoid doing a lot of that tedious work yourself, but I also think you'd have to be looking at a very big setup (thousands of jobs) before it would be worth the investment of time to understand those plugins/methods.

Learning Jenkins' version of Groovy is a bit of a trip: it has a lot of its own methods, and I've found some weird edge cases where Jenkins' Groovy doesn't behave the same as standard Groovy.  All that said - it's definitely been worth the work.

On Monday, 1 June 2020 11:37:46 UTC-4, Al Silver wrote:
I've used Jenkins a little but not in the traditional method of CI/CD. I'm using Jenkins as a Web UI to run individual python scripts for networking devices... Users would log into Jenkins and then select any one of numerous Freestyle projects (with parameters) to run (there could be 100 or more...).    The job executes a shell which runs a pytest script that logs into (via ssh) multiple networking devices and performs some actions... pretty straightforward and it works ok but has some limitations.  With that said, I think I'll have more flexibility and power building and maintaining these jobs as pipelines and treating them as code.  My question is I guess I would need to convert my pipelines into Jenkinsfiles?  However, most of the docs I read show a single Jenkinsfile (named "Jenkinsfile") I would need one for each job (a 100 or so).  I must be missing something here or not understanding.  Can someone point in the right direction for what I'm trying to accomplish?

Thx
Al

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit
https://groups.google.com/d/msgid/jenkinsci-users/4cae1d19-a93d-49f5-8257-96912ea49279%40googlegroups.com
.

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/776171713.2826360.1591268446434%40mail.yahoo.com.
Reply | Threaded
Open this post in threaded view
|

RE: Many Freestyle jobs or many Jenkinsfiles?

Jérôme Godbout

Nearly the same here:

  1. Common code into one repos
  2. Jenkinsfile for each project into his own repos
  3. Source into his own repos

 

When I run, I can checkout the proper jenkinsfile (branch, version) from the repos #2. It will checkout his needed version of the toolings repos #1. Then it will checkout the related source code and start building/testing, etc.

 

This split the way we do the build from the source code, the source code is totally agnostic of the build system. I can have different build script for different system. I can versionize my tools to get the same results every times. The slave doesn’t need anything installed, so the Jenkins only need the required plugins and credentials and tons of scripts approvals upon install.

 

 

From: 'monger_39' via Jenkins Users <[hidden email]>
Sent: June 4, 2020 7:01 AM
To: Jenkins Users <[hidden email]>
Subject: Re: Many Freestyle jobs or many Jenkinsfiles?

 

similar to other users I also have a quite large set of jobs that use the (scripted, not declarative) Groovy pipelines. In my setup I have deliberately chosen to:

  • store all the Groovy scripts in SCM
  • have a seperate dedicated repository for all my Groovy scripts;
    alternative would be to store the JenkinsFile in the same repo as where the test data is stored.
  • use groovy 'load' to pull in generic utility code.
    alternative is using shared libraries; I prefer local load since it allows me to use SCM branches of my jenkins repository

 

I also started from historic Freestyle jobs. First thing I did is transform the Freestyle job into one that calls a single script with parameters from a repo.

Then create a very simple pipeline job that call that very script. Then slowly start taking more parts of the script into the groovy code.
For me doing it big-bang was a no-go. Have been hit by the Groovy diffs similar to what @Giles also mentioned. Take small steps. Debugging Groovy
in Jenkins is extremely hard. Investigate the use of "Replay" !

Once you have the first job running, the rest will be much easier.

good luck.

 

On Tuesday, June 2, 2020, 09:58:35 PM GMT+2, Giles <[hidden email]> wrote:

 

 

I wrote a large library file full of Groovy methods.  I have a big stack of Jenkinsfiles, similar to Randy Beckworth, although mine are generally named "<purpose>/<environment>.Jenkinsfile" (as you can see, Jenkins isn't very particular about naming).  The Jenkinsfiles are generally very short, just long enough to call one of the library methods with a set of parameters.  We have 50-80 Jenkins jobs, and each of those has its own Jenkins file.  The initial set-up was tedious, but now that it's all in place it works very well.  Not sure if you could call a Python library, or if you'll end up using Groovy to call Python scripts.  My Groovy calls a lot of shell scripts - it's ugly, but works surprisingly well so you can make that work if you have to.

 

I think there are plugins and/or programmatic methods of generating jobs to avoid doing a lot of that tedious work yourself, but I also think you'd have to be looking at a very big setup (thousands of jobs) before it would be worth the investment of time to understand those plugins/methods.

 

Learning Jenkins' version of Groovy is a bit of a trip: it has a lot of its own methods, and I've found some weird edge cases where Jenkins' Groovy doesn't behave the same as standard Groovy.  All that said - it's definitely been worth the work.


On Monday, 1 June 2020 11:37:46 UTC-4, Al Silver wrote:

I've used Jenkins a little but not in the traditional method of CI/CD. I'm using Jenkins as a Web UI to run individual python scripts for networking devices... Users would log into Jenkins and then select any one of numerous Freestyle projects (with parameters) to run (there could be 100 or more...).    The job executes a shell which runs a pytest script that logs into (via ssh) multiple networking devices and performs some actions... pretty straightforward and it works ok but has some limitations.  With that said, I think I'll have more flexibility and power building and maintaining these jobs as pipelines and treating them as code.  My question is I guess I would need to convert my pipelines into Jenkinsfiles?  However, most of the docs I read show a single Jenkinsfile (named "Jenkinsfile") I would need one for each job (a 100 or so).  I must be missing something here or not understanding.  Can someone point in the right direction for what I'm trying to accomplish?

 

Thx

Al

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit

https://groups.google.com/d/msgid/jenkinsci-users/4cae1d19-a93d-49f5-8257-96912ea49279%40googlegroups.com

.

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/776171713.2826360.1591268446434%40mail.yahoo.com.

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/YQBPR0101MB1074CFF53A1ABDF0B34C0180CD890%40YQBPR0101MB1074.CANPRD01.PROD.OUTLOOK.COM.
Reply | Threaded
Open this post in threaded view
|

Re: Many Freestyle jobs or many Jenkinsfiles?

Al Silver
In reply to this post by Al Silver
Thanks for all of the ideas.  For now, I created a jenkins directory in my pytest root directory (basically the root directory of the project).  I then created name specific directories under there for organization.  Under those directories, there are multiple jenkinsfiles (not named jenkinsFile).  The pipeline files call a python script which processes all of the parameters and then generates a command line.  The entire command line is stored as a variable and then run through the shell.  Glad I got it working and seems ok for now.  One thing I don't really like about the workflow is for each pipeline job that is created, I still have to first create the job using the GUI and then I need to manually enter the "Script Path" to point to the appropriate pipeline file.  Is there anyway to create the entire Jenkins Job in code so that when the Jenkins GUI is opened up, the new jobs "magically" appear?

On Monday, June 1, 2020 at 11:37:46 AM UTC-4, Al Silver wrote:
I've used Jenkins a little but not in the traditional method of CI/CD. I'm using Jenkins as a Web UI to run individual python scripts for networking devices... Users would log into Jenkins and then select any one of numerous Freestyle projects (with parameters) to run (there could be 100 or more...).    The job executes a shell which runs a pytest script that logs into (via ssh) multiple networking devices and performs some actions... pretty straightforward and it works ok but has some limitations.  With that said, I think I'll have more flexibility and power building and maintaining these jobs as pipelines and treating them as code.  My question is I guess I would need to convert my pipelines into Jenkinsfiles?  However, most of the docs I read show a single Jenkinsfile (named "Jenkinsfile") I would need one for each job (a 100 or so).  I must be missing something here or not understanding.  Can someone point in the right direction for what I'm trying to accomplish?

Thx
Al

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/ca571ee2-c842-4603-9c34-9a5b1dbd70d8o%40googlegroups.com.
Reply | Threaded
Open this post in threaded view
|

Re: Many Freestyle jobs or many Jenkinsfiles?

David Riemens
check Jenkins Configuration as Code

After installing the plugin, use the export to generate a 'representative' piece of code for the type of jobs 
you want to use. Then you could run some script to generate a new JCASC file with similar jobs, and 
import that. I personally find the exact syntax of the JCASC yaml files still hard to hand-write, but for
simple jobs like running a pipeline from SCM it's quite easy.




On Friday, June 12, 2020, 05:29:43 PM GMT+2, Al Silver <[hidden email]> wrote:


Thanks for all of the ideas.  For now, I created a jenkins directory in my pytest root directory (basically the root directory of the project).  I then created name specific directories under there for organization.  Under those directories, there are multiple jenkinsfiles (not named jenkinsFile).  The pipeline files call a python script which processes all of the parameters and then generates a command line.  The entire command line is stored as a variable and then run through the shell.  Glad I got it working and seems ok for now.  One thing I don't really like about the workflow is for each pipeline job that is created, I still have to first create the job using the GUI and then I need to manually enter the "Script Path" to point to the appropriate pipeline file.  Is there anyway to create the entire Jenkins Job in code so that when the Jenkins GUI is opened up, the new jobs "magically" appear?

On Monday, June 1, 2020 at 11:37:46 AM UTC-4, Al Silver wrote:
I've used Jenkins a little but not in the traditional method of CI/CD. I'm using Jenkins as a Web UI to run individual python scripts for networking devices... Users would log into Jenkins and then select any one of numerous Freestyle projects (with parameters) to run (there could be 100 or more...).    The job executes a shell which runs a pytest script that logs into (via ssh) multiple networking devices and performs some actions... pretty straightforward and it works ok but has some limitations.  With that said, I think I'll have more flexibility and power building and maintaining these jobs as pipelines and treating them as code.  My question is I guess I would need to convert my pipelines into Jenkinsfiles?  However, most of the docs I read show a single Jenkinsfile (named "Jenkinsfile") I would need one for each job (a 100 or so).  I must be missing something here or not understanding.  Can someone point in the right direction for what I'm trying to accomplish?

Thx
Al

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit
https://groups.google.com/d/msgid/jenkinsci-users/ca571ee2-c842-4603-9c34-9a5b1dbd70d8o%40googlegroups.com
.

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/725072633.1201947.1592290321532%40mail.yahoo.com.