Docker agent build script wrapper

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Docker agent build script wrapper

Oleksandr Rudak
Hi,

We are building scientific model running and testing environment product. We use Jenkins as a core building, integration and model running tool. Now our users are scientist and have little technological background so we look to fold our current Jenkinsfile into something more compact and transparent. We've tried few things. SimpleBuildStep, which is too basic to handle agent/docker stuff. Step is too complicated as it involved distributed execution parallelism. Also we tried 'load' to load Jenkinsfile generated in the working directory. All ways have pros/cons but none provides in full what we try to accomplish.

Any ideas how to do such wrapping? I think about custom `agent` which would provide docker context and will imply the rest of the configuration and hide it from user. Possibly some workflow extension or something like this. Any thoughts are welcome.

Below is the anticipated Jenkinsfile how it would ideally look. Attached is the original Jenkins with detailed steps we want to cloak from user.

node {
    stage ('setup') {
        setupEnv (dockerImage:'dockerImage123')
    }
    stage ('train') {
        runNotebook (notebookPath:'notebookPath212')
        // { prepare, run, archive, legion }
        // runNotebook (notebookPath:'notebookPath213')
    }
    stage ('deploy') {
        deployModel ()
    }
    stage ('test') {
        testModel ()
    }
}



Thanks
Alex

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/5a2441d6-72ef-45f7-a415-356510c8a9cc%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Jenkinsfile.txt (1K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Docker agent build script wrapper

nicolas de loof-2
Hi,

next release of docker-plugin is going to introduce "dockerNode" keyword in pipeline, so you can create an agent from a docker image: 

dockerNode(dockerHost: 'unix:///var/run/docker.sock', image: 'jenkins/slave', remoteFs: '/home/jenkins') { sh 'echo "hello there"' }


this is still under development and will first be an experimental feature
in the meantime, you can rely on docker-pipeline-plugin and docker.inside for an equivalent.

Generally speaking, if you target multiple environments, you should not declare a top level node, but have one per stage.

2017-11-27 11:12 GMT+01:00 Oleksandr Rudak <[hidden email]>:
Hi,

We are building scientific model running and testing environment product. We use Jenkins as a core building, integration and model running tool. Now our users are scientist and have little technological background so we look to fold our current Jenkinsfile into something more compact and transparent. We've tried few things. SimpleBuildStep, which is too basic to handle agent/docker stuff. Step is too complicated as it involved distributed execution parallelism. Also we tried 'load' to load Jenkinsfile generated in the working directory. All ways have pros/cons but none provides in full what we try to accomplish.

Any ideas how to do such wrapping? I think about custom `agent` which would provide docker context and will imply the rest of the configuration and hide it from user. Possibly some workflow extension or something like this. Any thoughts are welcome.

Below is the anticipated Jenkinsfile how it would ideally look. Attached is the original Jenkins with detailed steps we want to cloak from user.

node {
    stage ('setup') {
        setupEnv (dockerImage:'dockerImage123')
    }
    stage ('train') {
        runNotebook (notebookPath:'notebookPath212')
        // { prepare, run, archive, legion }
        // runNotebook (notebookPath:'notebookPath213')
    }
    stage ('deploy') {
        deployModel ()
    }
    stage ('test') {
        testModel ()
    }
}



Thanks
Alex

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/5a2441d6-72ef-45f7-a415-356510c8a9cc%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CANMVJzmLW_c67%3DrO7CLk7k_L5FZeZFiw33SGQh2X36Ty-rovDg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Docker agent build script wrapper

Oleksandr Rudak
In reply to this post by Oleksandr Rudak
We have reached acceptable solution for the problem. It is loading groovy script with black magic around closure. Attaching code for the community benefit.

It boils down to construct in the loaded groovy module:
def container(myImageName, Closure body){
    docker.image(myImageName).inside(dockerArgs(), body)
}


 And then using it passing closure body:
    def myImage = 'php:latest'
    def modelId = drun.modelId()

    echo modelId

    drun.container(myImage) {
        stage ('setup'){
            sh 'id'
            echo 'value = ' + modelId
        }

        stage('stage 1'){
            sh '/usr/local/bin/php -version'
        }
}

There is also custom step returning map of properties from the environment implemented as Java class:
public class DrunPropertiesStep extends Step {
...
    public static class Execution extends SynchronousStepExecution<Object> {
        private static final long serialVersionUID = 1;

        Execution(StepContext context) {
            super(context);
        }

        @Override
        protected Object run() throws Exception {
            Run run = getContext().get(Run.class);
            //todo: implement

            JSONObject json = Utils.getJson(run.getRootDir().getPath(), run.getLogReader());

            Map map = new HashMap<String, String>();
            Iterator iter = json.keys();
            while( iter.hasNext() ) {
                String key = iter.next().toString();

                map.put(key, json.getString(key));
            }

            return map;
        }
    }
...
}

Hope it will help to prevent someone from banging a head against the keyboard :)

Thanks
Alex

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/3cd3005a-9a64-48a4-b67b-2cae076d0e93%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

drun.groovy (1K) Download Attachment
Jenkinsfile (542 bytes) Download Attachment