Kubernetes Plugin: Option to run pod as different user (e.g. as root)

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Kubernetes Plugin: Option to run pod as different user (e.g. as root)

Torsten Gippert

Hello everybody,


this question is about implementing an option to run a build pod (using the kubernetes plugin from Carlos Sanchez) as a different user.

@Carlos: Great work, we love the ability to define our own pod templates!


Software versions:

  • Jenkins version: 2.89.2

  • Kubernetes Plugin version: 1.1.2

  • Durable Task Plugin Version: 1.17


First, let's assume the following scenario:

One JNLP container and 2 containers with different images in a podTemplate.


  • jnlp container - image jenkins/jnlp-slave:alpine - running as user 'jenkins' with uid 10000 (ten thousand)

  • node container - image node:9.3-alpine - running as root user with uid 0

  • gradle container - image gradle:4.4-alpine - running as user 'gradle' with uid 1000 (one thousand).


podTemplate from Jenkinsfile:

podTemplate(label: nodeLabel, containers: [

  containerTemplate(name: 'node', image: 'node:9.3-alpine', ttyEnabled: true, command: 'cat'),

  containerTemplate(name: 'gradle', image: 'gradle:4.4-alpine', ttyEnabled: true, command: 'cat')

])


Switching into container 'gradle' is not possible due to permission denied errors (seems to be caused by the durable task plugin, but I am not sure about that).
Root cause seems to be that the jnlp container runs with uid 10000 (yes, ten thousand - the 'old' image jenkins/jnlp-slave:2.62 ran with uid 1000) and therefore the working directory has file/directory permissions that prevent other users except uid 10000 and root (uid 0) from accessing the working directory.



I also added a pod yaml file (see attachment pod-permission-denied.yaml) that shows the "problem":
1. You can run that pod (kubectl apply -f pod-permission-denied.yaml) and
2. start a shell in the different containers (kubectl exec -ti -c [jnlp|node|gradle] permission-denied sh) to

3. run some commands like ‘id’ or ‘whoami’ within a container to show you the different user-ids the containers are running with.


See attachments

  • Jenkinsfile-permission-denied.groovy   (contains Jenkinsfile to reproduce the “problem”)

  • Jenkinsfile-permission-denied_output.txt   (contains the stdout of the Jenkins build job)

  • pod-permission-denied.yaml (simplified k8s pod declaration)


Suggested solution:

Add options to set 'runAsUser' and 'fsGroup' at podTemplate level to be able to get rid of permission problems.


'runAsUser' and 'fsGroup' stand for the user id (uid) and match the options from podSecurityContext at pod level in kubernetes pod specification (see https://kubernetes.io/docs/tasks/configure-pod-container/security-context/)


suggested podTemplate from Jenkinsfile:

podTemplate(label: nodeLabel, runAsUser: 0, fsGroup: 0,

containers: [

  containerTemplate(name: 'node', image: 'node:9.3-alpine', ttyEnabled: true, command: 'cat'),

  containerTemplate(name: 'gradle', image: 'gradle:4.4-alpine', ttyEnabled: true, command: 'cat')

])






The pod ‘run-as-root’ (see attachment pod-run-as-root.yaml) is a minimal modified version of the pod ‘permission-denied’ (see above) with a different pod security context:


securityContext:

  runAsUser: 0

  fsGroup: 0


1. You can run that pod (kubectl apply -f pod-run-as-root.yaml) and
2. start a shell in the different containers (kubectl exec -ti -c [jnlp|node|gradle] run-as-root sh) to

3. run some commands like ‘id’ or ‘whoami’ within a container to show you that each container is running as root now.

Therefore no permission denied problems
should occur if this will be implemented in the kubernetes plugin.

See attachments

  • pod-run-as-root.yaml (simplified k8s pod declaration with podSecurityContext set to root user)


What do you think? Could this be a proper way to “override” the the container specific default users?


--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/e06a9c7b-79bd-43cc-9428-498432f0fcfd%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Jenkinsfile-permission-denied.groovy (1K) Download Attachment
Jenkinsfile-permission-denied_output.txt (3K) Download Attachment
pod-permission-denied.yaml (2K) Download Attachment
pod-run-as-root.yaml (2K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Kubernetes Plugin: Option to run pod as different user (e.g. as root)

Carlos Sanchez
Yes runAsUser is what you need but it's not yet supported as a field in the plugin

This will hopefully be supported soon, working on a way to directly import Pod yaml




On Tue, Jan 9, 2018 at 4:20 PM, Torsten Gippert <[hidden email]> wrote:

Hello everybody,


this question is about implementing an option to run a build pod (using the kubernetes plugin from Carlos Sanchez) as a different user.

@Carlos: Great work, we love the ability to define our own pod templates!


Software versions:

  • Jenkins version: 2.89.2

  • Kubernetes Plugin version: 1.1.2

  • Durable Task Plugin Version: 1.17


First, let's assume the following scenario:

One JNLP container and 2 containers with different images in a podTemplate.


  • jnlp container - image jenkins/jnlp-slave:alpine - running as user 'jenkins' with uid 10000 (ten thousand)

  • node container - image node:9.3-alpine - running as root user with uid 0

  • gradle container - image gradle:4.4-alpine - running as user 'gradle' with uid 1000 (one thousand).


podTemplate from Jenkinsfile:

podTemplate(label: nodeLabel, containers: [

  containerTemplate(name: 'node', image: 'node:9.3-alpine', ttyEnabled: true, command: 'cat'),

  containerTemplate(name: 'gradle', image: 'gradle:4.4-alpine', ttyEnabled: true, command: 'cat')

])


Switching into container 'gradle' is not possible due to permission denied errors (seems to be caused by the durable task plugin, but I am not sure about that).
Root cause seems to be that the jnlp container runs with uid 10000 (yes, ten thousand - the 'old' image jenkins/jnlp-slave:2.62 ran with uid 1000) and therefore the working directory has file/directory permissions that prevent other users except uid 10000 and root (uid 0) from accessing the working directory.



I also added a pod yaml file (see attachment pod-permission-denied.yaml) that shows the "problem":
1. You can run that pod (kubectl apply -f pod-permission-denied.yaml) and
2. start a shell in the different containers (kubectl exec -ti -c [jnlp|node|gradle] permission-denied sh) to

3. run some commands like ‘id’ or ‘whoami’ within a container to show you the different user-ids the containers are running with.


See attachments

  • Jenkinsfile-permission-denied.groovy   (contains Jenkinsfile to reproduce the “problem”)

  • Jenkinsfile-permission-denied_output.txt   (contains the stdout of the Jenkins build job)

  • pod-permission-denied.yaml (simplified k8s pod declaration)


Suggested solution:

Add options to set 'runAsUser' and 'fsGroup' at podTemplate level to be able to get rid of permission problems.


'runAsUser' and 'fsGroup' stand for the user id (uid) and match the options from podSecurityContext at pod level in kubernetes pod specification (see https://kubernetes.io/docs/tasks/configure-pod-container/security-context/)


suggested podTemplate from Jenkinsfile:

podTemplate(label: nodeLabel, runAsUser: 0, fsGroup: 0,

containers: [

  containerTemplate(name: 'node', image: 'node:9.3-alpine', ttyEnabled: true, command: 'cat'),

  containerTemplate(name: 'gradle', image: 'gradle:4.4-alpine', ttyEnabled: true, command: 'cat')

])






The pod ‘run-as-root’ (see attachment pod-run-as-root.yaml) is a minimal modified version of the pod ‘permission-denied’ (see above) with a different pod security context:


securityContext:

  runAsUser: 0

  fsGroup: 0


1. You can run that pod (kubectl apply -f pod-run-as-root.yaml) and
2. start a shell in the different containers (kubectl exec -ti -c [jnlp|node|gradle] run-as-root sh) to

3. run some commands like ‘id’ or ‘whoami’ within a container to show you that each container is running as root now.

Therefore no permission denied problems
should occur if this will be implemented in the kubernetes plugin.

See attachments

  • pod-run-as-root.yaml (simplified k8s pod declaration with podSecurityContext set to root user)


What do you think? Could this be a proper way to “override” the the container specific default users?


--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/e06a9c7b-79bd-43cc-9428-498432f0fcfd%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/CALHFn6OfDpOv_qjwuP6JtGTPWpdXvSuLUkkh5MZ4Pxyeke-2mA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Kubernetes Plugin: Option to run pod as different user (e.g. as root)

Surjit Bains
In reply to this post by Torsten Gippert
Hi @Carlos,

I have the same problem as Torsten, I'd like to use the Kubernetes plugin alongside using workflow steps. When do you plan to introduce runAs?

many thanks

Surj

def label = "jnlp-slave"

podTemplate(label: label, containers: [
containerTemplate(name: 'gradle', image: 'gradle:4.4-alpine', ttyEnabled: true, command: 'cat', workingDir: '/var/lib/jenkins')
])

{

node(label) {
stage('test') {
container('gradle') {
stage('test') {
try {
sh 'gradle clean test'
} catch (err) {
println err.message
} finally {
sleep 1
}
}
}
}
}
}




Agent specification [Kubernetes Pod Template] (jnlp-slave): 
* [gradle] gradle:4.4-alpine(resourceRequestCpu: , resourceRequestMemory: , resourceLimitCpu: , resourceLimitMemory: )

Running on jnlp-slave-hgbsp in /home/jenkins/workspace/roductcatalogue_development-DN7AGMLQXLAWWWMLAFESYWBNB4L6VPMJ3GRJRRRYC5D5ODGXGLSQ
[Pipeline] {
[Pipeline] stage
[Pipeline] { (test)
[Pipeline] container
[Pipeline] {
[Pipeline] stage
[Pipeline] { (test)
[Pipeline] sh
[roductcatalogue_development-DN7AGMLQXLAWWWMLAFESYWBNB4L6VPMJ3GRJRRRYC5D5ODGXGLSQ] Running shell script
sh: can't create /home/jenkins/workspace/roductcatalogue_development-DN7AGMLQXLAWWWMLAFESYWBNB4L6VPMJ3GRJRRRYC5D5ODGXGLSQ@tmp/durable-5bff21c2/jenkins-log.txt: Permission denied
sh: can't create /home/jenkins/workspace/roductcatalogue_development-DN7AGMLQXLAWWWMLAFESYWBNB4L6VPMJ3GRJRRRYC5D5ODGXGLSQ@tmp/durable-5bff21c2/jenkins-result.txt: Permission denied
touch: /home/jenkins/workspace/roductcatalogue_development-DN7AGMLQXLAWWWMLAFESYWBNB4L6VPMJ3GRJRRRYC5D5ODGXGLSQ@tmp/durable-5bff21c2/jenkins-log.txt: Permission denied
touch: /home/jenkins/workspace/roductcatalogue_development-DN7AGMLQXLAWWWMLAFESYWBNB4L6VPMJ3GRJRRRYC5D5ODGXGLSQ@tmp/durable-5bff21c2/jenkins-log.txt: Permission denied
touch: /home/jenkins/workspace/roductcatalogue_development-DN7AGMLQXLAWWWMLAFESYWBNB4L6VPMJ3GRJRRRYC5D5ODGXGLSQ@tmp/durable-5bff21c2/jenkins-log.txt: Permission denied
touch: /home/jenkins/workspace/roductcatalogue_development-DN7AGMLQXLAWWWMLAFESYWBNB4L6VPMJ3GRJRRRYC5D5ODGXGLSQ@tmp/durable-5bff21c2/jenkins-log.txt: Permission denied
touch: /home/jenkins/workspace/roductcatalogue_development-DN7AGMLQXLAWWWMLAFESYWBNB4L6VPMJ3GRJRRRYC5D5ODGXGLSQ@tmp/durable-5bff21c2/jenkins-log.txt: Permission denied
touch: /home/jenkins/workspace/roductcatalogue_development-DN7AGMLQXLAWWWMLAFESYWBNB4L6VPMJ3GRJRRRYC5D5ODGXGLSQ@tmp/durable-5bff21c2/jenkins-log.txt: Permission denied
touch: /home/jenkins/workspace/roductcatalogue_development-DN7AGMLQXLAWWWMLAFESYWBNB4L6VPMJ3GRJRRRYC5D5ODGXGLSQ@tmp/durable-5bff21c2/jenkins-log.txt: Permission denied
[Pipeline] echo 
script returned exit code -2 




On Tuesday, 9 January 2018 15:20:01 UTC, Torsten Gippert wrote:

Hello everybody,


this question is about implementing an option to run a build pod (using the kubernetes plugin from Carlos Sanchez) as a different user.

@Carlos: Great work, we love the ability to define our own pod templates!


Software versions:

  • Jenkins version: 2.89.2

  • Kubernetes Plugin version: 1.1.2

  • Durable Task Plugin Version: 1.17


First, let's assume the following scenario:

One JNLP container and 2 containers with different images in a podTemplate.


  • jnlp container - image jenkins/jnlp-slave:alpine - running as user 'jenkins' with uid 10000 (ten thousand)

  • node container - image node:9.3-alpine - running as root user with uid 0

  • gradle container - image gradle:4.4-alpine - running as user 'gradle' with uid 1000 (one thousand).


podTemplate from Jenkinsfile:

podTemplate(label: nodeLabel, containers: [

  containerTemplate(name: 'node', image: 'node:9.3-alpine', ttyEnabled: true, command: 'cat'),

  containerTemplate(name: 'gradle', image: 'gradle:4.4-alpine', ttyEnabled: true, command: 'cat')

])


Switching into container 'gradle' is not possible due to permission denied errors (seems to be caused by the durable task plugin, but I am not sure about that).
Root cause seems to be that the jnlp container runs with uid 10000 (yes, ten thousand - the 'old' image jenkins/jnlp-slave:2.62 ran with uid 1000) and therefore the working directory has file/directory permissions that prevent other users except uid 10000 and root (uid 0) from accessing the working directory.



I also added a pod yaml file (see attachment pod-permission-denied.yaml) that shows the "problem":
1. You can run that pod (kubectl apply -f pod-permission-denied.yaml) and
2. start a shell in the different containers (kubectl exec -ti -c [jnlp|node|gradle] permission-denied sh) to

3. run some commands like ‘id’ or ‘whoami’ within a container to show you the different user-ids the containers are running with.


See attachments

  • Jenkinsfile-permission-denied.groovy   (contains Jenkinsfile to reproduce the “problem”)

  • Jenkinsfile-permission-denied_output.txt   (contains the stdout of the Jenkins build job)

  • pod-permission-denied.yaml (simplified k8s pod declaration)


Suggested solution:

Add options to set 'runAsUser' and 'fsGroup' at podTemplate level to be able to get rid of permission problems.


'runAsUser' and 'fsGroup' stand for the user id (uid) and match the options from podSecurityContext at pod level in kubernetes pod specification (see <a href="https://kubernetes.io/docs/tasks/configure-pod-container/security-context/" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fkubernetes.io%2Fdocs%2Ftasks%2Fconfigure-pod-container%2Fsecurity-context%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEM5jZlOy5ZfAmrs0byX2Vj9HTB-Q&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fkubernetes.io%2Fdocs%2Ftasks%2Fconfigure-pod-container%2Fsecurity-context%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEM5jZlOy5ZfAmrs0byX2Vj9HTB-Q&#39;;return true;">https://kubernetes.io/docs/tasks/configure-pod-container/security-context/)


suggested podTemplate from Jenkinsfile:

podTemplate(label: nodeLabel, runAsUser: 0, fsGroup: 0,

containers: [

  containerTemplate(name: 'node', image: 'node:9.3-alpine', ttyEnabled: true, command: 'cat'),

  containerTemplate(name: 'gradle', image: 'gradle:4.4-alpine', ttyEnabled: true, command: 'cat')

])






The pod ‘run-as-root’ (see attachment pod-run-as-root.yaml) is a minimal modified version of the pod ‘permission-denied’ (see above) with a different pod security context:


securityContext:

  runAsUser: 0

  fsGroup: 0


1. You can run that pod (kubectl apply -f pod-run-as-root.yaml) and
2. start a shell in the different containers (kubectl exec -ti -c [jnlp|node|gradle] run-as-root sh) to

3. run some commands like ‘id’ or ‘whoami’ within a container to show you that each container is running as root now.

Therefore no permission denied problems
should occur if this will be implemented in the kubernetes plugin.

See attachments

  • pod-run-as-root.yaml (simplified k8s pod declaration with podSecurityContext set to root user)


What do you think? Could this be a proper way to “override” the the container specific default users?


--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/f6239d6d-97a4-4727-9798-c223b08c218e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Kubernetes Plugin: Option to run pod as different user (e.g. as root)

kuisathaverat
In reply to this post by Torsten Gippert
This example run the overall context with the UID 1000, the "jnlp" container with the UID 2000 and the "dind" container as a privileged container with UID 0

labelDind = "agent-k8s-${UUID.randomUUID().toString()}"
def yamlDinD = """
apiVersion: v1
kind: Pod
metadata:
  generateName: agent-k8s-
  labels:
    name: jnlp
    label: jnlp
spec:
  securityContext:
    runAsUser: 1000
  containers:
  - name: jnlp
    image: jenkins/jnlp-slave
    tty: true
    securityContext:
      runAsUser: 2000
      allowPrivilegeEscalation: false
  - name: dind
    image: docker:dind
    tty: true
    securityContext:
      runAsUser: 0
      privileged: true
"""
    timestamps {
      podTemplate(label: labelDind, yaml:yamlDinD) {
          node(labelDind){
            stage('Build Docker Image'){
                sh "id"
                  container('jnlp'){
                     sh "id"
                  }
                  container('dind'){
                     sh "id"
                     sh "docker version"
                  }
                }
              }
            }
          }
    }

El martes, 9 de enero de 2018, 16:20:01 (UTC+1), Torsten Gippert escribió:

Hello everybody,


this question is about implementing an option to run a build pod (using the kubernetes plugin from Carlos Sanchez) as a different user.

@Carlos: Great work, we love the ability to define our own pod templates!


Software versions:

  • Jenkins version: 2.89.2

  • Kubernetes Plugin version: 1.1.2

  • Durable Task Plugin Version: 1.17


First, let's assume the following scenario:

One JNLP container and 2 containers with different images in a podTemplate.


  • jnlp container - image jenkins/jnlp-slave:alpine - running as user 'jenkins' with uid 10000 (ten thousand)

  • node container - image node:9.3-alpine - running as root user with uid 0

  • gradle container - image gradle:4.4-alpine - running as user 'gradle' with uid 1000 (one thousand).


podTemplate from Jenkinsfile:

podTemplate(label: nodeLabel, containers: [

  containerTemplate(name: 'node', image: 'node:9.3-alpine', ttyEnabled: true, command: 'cat'),

  containerTemplate(name: 'gradle', image: 'gradle:4.4-alpine', ttyEnabled: true, command: 'cat')

])


Switching into container 'gradle' is not possible due to permission denied errors (seems to be caused by the durable task plugin, but I am not sure about that).
Root cause seems to be that the jnlp container runs with uid 10000 (yes, ten thousand - the 'old' image jenkins/jnlp-slave:2.62 ran with uid 1000) and therefore the working directory has file/directory permissions that prevent other users except uid 10000 and root (uid 0) from accessing the working directory.



I also added a pod yaml file (see attachment pod-permission-denied.yaml) that shows the "problem":
1. You can run that pod (kubectl apply -f pod-permission-denied.yaml) and
2. start a shell in the different containers (kubectl exec -ti -c [jnlp|node|gradle] permission-denied sh) to

3. run some commands like ‘id’ or ‘whoami’ within a container to show you the different user-ids the containers are running with.


See attachments

  • Jenkinsfile-permission-denied.groovy   (contains Jenkinsfile to reproduce the “problem”)

  • Jenkinsfile-permission-denied_output.txt   (contains the stdout of the Jenkins build job)

  • pod-permission-denied.yaml (simplified k8s pod declaration)


Suggested solution:

Add options to set 'runAsUser' and 'fsGroup' at podTemplate level to be able to get rid of permission problems.


'runAsUser' and 'fsGroup' stand for the user id (uid) and match the options from podSecurityContext at pod level in kubernetes pod specification (see <a href="https://kubernetes.io/docs/tasks/configure-pod-container/security-context/" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fkubernetes.io%2Fdocs%2Ftasks%2Fconfigure-pod-container%2Fsecurity-context%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEM5jZlOy5ZfAmrs0byX2Vj9HTB-Q&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fkubernetes.io%2Fdocs%2Ftasks%2Fconfigure-pod-container%2Fsecurity-context%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEM5jZlOy5ZfAmrs0byX2Vj9HTB-Q&#39;;return true;">https://kubernetes.io/docs/tasks/configure-pod-container/security-context/)


suggested podTemplate from Jenkinsfile:

podTemplate(label: nodeLabel, runAsUser: 0, fsGroup: 0,

containers: [

  containerTemplate(name: 'node', image: 'node:9.3-alpine', ttyEnabled: true, command: 'cat'),

  containerTemplate(name: 'gradle', image: 'gradle:4.4-alpine', ttyEnabled: true, command: 'cat')

])






The pod ‘run-as-root’ (see attachment pod-run-as-root.yaml) is a minimal modified version of the pod ‘permission-denied’ (see above) with a different pod security context:


securityContext:

  runAsUser: 0

  fsGroup: 0


1. You can run that pod (kubectl apply -f pod-run-as-root.yaml) and
2. start a shell in the different containers (kubectl exec -ti -c [jnlp|node|gradle] run-as-root sh) to

3. run some commands like ‘id’ or ‘whoami’ within a container to show you that each container is running as root now.

Therefore no permission denied problems
should occur if this will be implemented in the kubernetes plugin.

See attachments

  • pod-run-as-root.yaml (simplified k8s pod declaration with podSecurityContext set to root user)


What do you think? Could this be a proper way to “override” the the container specific default users?


--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/2f4b459b-e3a0-4e7f-9b89-59a96266b32f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.