ANN: Jenkins release artifacts uploads blockage on June 09, and a temporary downloads issue

classic Classic list List threaded Threaded
27 messages Options
12
Reply | Threaded
Open this post in threaded view
|

ANN: Jenkins release artifacts uploads blockage on June 09, and a temporary downloads issue

Oleg Nenashev

Dear all,


As you may have noticed, the release artifact uploads are currently blocked in the Jenkins Artifactory instances (https://repo.jenkins-ci.org/). We are doing a security investigation due to a partial user database loss on June 02. Today we blocked releases to the Jenkins artifactory, and there also was a temporary outage of the Artifactory downloads which was a collateral damage of the temporary permissions. You can find more details about it in this Jenkins Infra Thread and in this Dev List thread.


Current status:

  • Downloads are restored for all artifacts on https://repo.jenkins-ci.org/, Jenkins core historical releases, Remoting library and Windows Service Wrapper which were among ones reported by Jenkins users.

  • Uploads: Jenkins artifact uploads are blocked for the most of Jenkins plugin maintainers and contributors. It affects releases of Jenkins plugins, Jenkins core and modules, developer tools and all libraries hosted on https://repo.jenkins-ci.org/. Incremental and Snapshot deployments are not affected.


Quick summary: 

  • Jun 02 - There was a Kubernetes Cluster outage on June 02. During this outage we had to rebuild the cluster from scratch to get some services working again.

  • Jun 02 - After the recovery we lost three months of LDAP changes. It has happened due to the broken backup of the LDAP database.

  • Jun 02 - We identified a number of potential security risks which may be caused by the LDAP outage. Account overtake and malicious upload was one of the identified risks. FTR this issue is tracked as SECURITY-1895 as a follow-up to these discussions. Only the Security team members have access to it, so I am not sharing a link here.

  • Jun 09 - After the security risk was independently reported in public by a plugin maintainer in the dev list thread, we decided to block uploads of release artifacts to the Jenkins Artifactory instance.

  • Jun 09, 8:50AM UTC - All uploads of release artifacts were blocked (plugins, Jenkins core and modules, developer tools, etc.). Downloads of some binaries were also blocked as an unexpected collateral damage. Jenkins core historical releases, Remoting library and Windows Service Wrapper are among the affected binaries

  • Jun 09, 10AM UTC - We finished reviews of all artifact releases to https://repo.jenkins-ci.org/, which happened between the infra outage on June 02 and the blockage of the releases. There are no maliciously uploaded artifacts. Note that the common plugin release flow requires access to GitHub in order to push the release commits, so a malicious attacker would need to overtake both Jenkins and GitHub accounts of a single user to submit a legitimately-looking release.

  • Jun 09, ~1PM UTC - Artifact downloads are restored, alternate patch in the Repository Permission Updater was applied to prevent uploads. Artifact uploads are still blocking

  • Jun 09, 2PM UTC, based on repo.jenkins-ci.org and issues.jenkins-ci.org data, we restored maintainers accounts.


Our next steps would be to communicate the issue to all maintainers and contributors who might have been affected by the LDAP history loss. We will likely need to perform additional user verification steps for plugin maintainers to ensure that there are no contributors affected by the issues. Today at 3:30PM UTC we will also have a Jenkins Infrastructure team meeting where this issue will be discussed in more detail. This is a public meeting, and everyone is welcome to join. Calendar link


Thanks to Olivier Vernin, Daniel Beck and other Jenkins Infra and Security team members who contributed to this investigation.


Best regards,

Oleg Nenashev


--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CAPfivLD4AWGkQCh0mGTRtViyHT-UBXoE3SbaKgVe5%3DsbSjBE%3Dg%40mail.gmail.com.
Reply | Threaded
Open this post in threaded view
|

Re: ANN: Jenkins release artifacts uploads blockage on June 09, and a temporary downloads issue

Dmitry Sotnikov
Thank you, Oleg and the Security Team!

Dmitry

On Tuesday, June 9, 2020 at 8:00:25 AM UTC-7, Oleg Nenashev wrote:

Dear all,


As you may have noticed, the release artifact uploads are currently blocked in the Jenkins Artifactory instances (<a href="https://repo.jenkins-ci.org/" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Frepo.jenkins-ci.org%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNH4WVSnOUybsblDMAtolGO0Jbm_oQ&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Frepo.jenkins-ci.org%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNH4WVSnOUybsblDMAtolGO0Jbm_oQ&#39;;return true;">https://repo.jenkins-ci.org/). We are doing a security investigation due to a partial user database loss on June 02. Today we blocked releases to the Jenkins artifactory, and there also was a temporary outage of the Artifactory downloads which was a collateral damage of the temporary permissions. You can find more details about it in this<a href="https://groups.google.com/forum/#!topic/jenkins-infra/zRqdiyarLDE" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://groups.google.com/forum/#!topic/jenkins-infra/zRqdiyarLDE&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/forum/#!topic/jenkins-infra/zRqdiyarLDE&#39;;return true;"> Jenkins Infra Thread and in this<a href="https://groups.google.com/d/msg/jenkinsci-dev/juHejx8zfdg/xpySiv1_CQAJ" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://groups.google.com/d/msg/jenkinsci-dev/juHejx8zfdg/xpySiv1_CQAJ&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/msg/jenkinsci-dev/juHejx8zfdg/xpySiv1_CQAJ&#39;;return true;"> Dev List thread.


Current status:

  • Downloads are restored for all artifacts on <a href="https://repo.jenkins-ci.org/" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Frepo.jenkins-ci.org%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNH4WVSnOUybsblDMAtolGO0Jbm_oQ&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Frepo.jenkins-ci.org%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNH4WVSnOUybsblDMAtolGO0Jbm_oQ&#39;;return true;">https://repo.jenkins-ci.org/, Jenkins core historical releases, Remoting library and Windows Service Wrapper which were among ones reported by Jenkins users.

  • Uploads: Jenkins artifact uploads are blocked for the most of Jenkins plugin maintainers and contributors. It affects releases of Jenkins plugins, Jenkins core and modules, developer tools and all libraries hosted on <a href="https://repo.jenkins-ci.org/" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Frepo.jenkins-ci.org%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNH4WVSnOUybsblDMAtolGO0Jbm_oQ&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Frepo.jenkins-ci.org%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNH4WVSnOUybsblDMAtolGO0Jbm_oQ&#39;;return true;">https://repo.jenkins-ci.org/. Incremental and Snapshot deployments are not affected.


Quick summary: 

  • Jun 02 - There was a Kubernetes Cluster outage on June 02. During this outage we had to rebuild the cluster from scratch to get some services working again.

  • Jun 02 - After the recovery we lost three months of LDAP changes. It has happened due to the broken backup of the LDAP database.

  • Jun 02 - We identified a number of potential security risks which may be caused by the LDAP outage. Account overtake and malicious upload was one of the identified risks. FTR this issue is tracked as SECURITY-1895 as a follow-up to these discussions. Only the Security team members have access to it, so I am not sharing a link here.

  • Jun 09 - After the security risk was independently reported in public by a plugin maintainer in the <a href="https://groups.google.com/g/jenkinsci-dev/c/juHejx8zfdg" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://groups.google.com/g/jenkinsci-dev/c/juHejx8zfdg&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/g/jenkinsci-dev/c/juHejx8zfdg&#39;;return true;">dev list thread, we decided to block uploads of release artifacts to the Jenkins Artifactory instance.

  • Jun 09, 8:50AM UTC - All uploads of release artifacts were blocked (plugins, Jenkins core and modules, developer tools, etc.). Downloads of some binaries were also blocked as an unexpected collateral damage. Jenkins core historical releases, Remoting library and Windows Service Wrapper are among the affected binaries

  • Jun 09, 10AM UTC - We finished reviews of all artifact releases to <a href="https://repo.jenkins-ci.org/" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Frepo.jenkins-ci.org%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNH4WVSnOUybsblDMAtolGO0Jbm_oQ&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Frepo.jenkins-ci.org%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNH4WVSnOUybsblDMAtolGO0Jbm_oQ&#39;;return true;">https://repo.jenkins-ci.org/, which happened between the infra outage on June 02 and the blockage of the releases. There are no maliciously uploaded artifacts. Note that the common plugin release flow requires access to GitHub in order to push the release commits, so a malicious attacker would need to overtake both Jenkins and GitHub accounts of a single user to submit a legitimately-looking release.

  • Jun 09, ~1PM UTC - Artifact downloads are restored, <a href="https://github.com/jenkins-infra/repository-permissions-updater/pull/1569" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Fjenkins-infra%2Frepository-permissions-updater%2Fpull%2F1569\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHnXqHe8DbBsyaWsN32PYLgiFuxww&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Fjenkins-infra%2Frepository-permissions-updater%2Fpull%2F1569\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHnXqHe8DbBsyaWsN32PYLgiFuxww&#39;;return true;">alternate patch in the Repository Permission Updater was applied to prevent uploads. Artifact uploads are still blocking

  • Jun 09, 2PM UTC, based on <a href="http://repo.jenkins-ci.org" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Frepo.jenkins-ci.org\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNH-lqjQdpr_N3OJq6LbHFX-8CE0vg&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Frepo.jenkins-ci.org\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNH-lqjQdpr_N3OJq6LbHFX-8CE0vg&#39;;return true;">repo.jenkins-ci.org and <a href="http://issues.jenkins-ci.org" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fissues.jenkins-ci.org\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEt2OBDOWbsPhB3LOncdqtsX9Mk0Q&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fissues.jenkins-ci.org\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEt2OBDOWbsPhB3LOncdqtsX9Mk0Q&#39;;return true;">issues.jenkins-ci.org data, we restored maintainers accounts.


Our next steps would be to communicate the issue to all maintainers and contributors who might have been affected by the LDAP history loss. We will likely need to perform additional user verification steps for plugin maintainers to ensure that there are no contributors affected by the issues. Today at 3:30PM UTC we will also have a Jenkins Infrastructure team meeting where this issue will be discussed in more detail. This is a public meeting, and everyone is welcome to join. <a href="https://calendar.google.com/event?action=TEMPLATE&amp;tmeid=dTJsaWoxN2xjZHFkajRsbmJlcWFiaXI5b2JfMjAyMDA2MDlUMTUzMDAwWiA0c3MxMmYwbXFyM3RicDF0MmZlMzY5c2xmNEBn&amp;tmsrc=4ss12f0mqr3tbp1t2fe369slf4%40group.calendar.google.com" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://calendar.google.com/event?action\x3dTEMPLATE\x26tmeid\x3ddTJsaWoxN2xjZHFkajRsbmJlcWFiaXI5b2JfMjAyMDA2MDlUMTUzMDAwWiA0c3MxMmYwbXFyM3RicDF0MmZlMzY5c2xmNEBn\x26tmsrc\x3d4ss12f0mqr3tbp1t2fe369slf4%40group.calendar.google.com&#39;;return true;" onclick="this.href=&#39;https://calendar.google.com/event?action\x3dTEMPLATE\x26tmeid\x3ddTJsaWoxN2xjZHFkajRsbmJlcWFiaXI5b2JfMjAyMDA2MDlUMTUzMDAwWiA0c3MxMmYwbXFyM3RicDF0MmZlMzY5c2xmNEBn\x26tmsrc\x3d4ss12f0mqr3tbp1t2fe369slf4%40group.calendar.google.com&#39;;return true;">Calendar link


Thanks to Olivier Vernin, Daniel Beck and other Jenkins Infra and Security team members who contributed to this investigation.


Best regards,

Oleg Nenashev


--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/819de68b-8cd6-4125-bde7-1d2e95a1c95co%40googlegroups.com.
Reply | Threaded
Open this post in threaded view
|

Re: ANN: Jenkins release artifacts uploads blockage on June 09, and a temporary downloads issue

Olblak-2
In reply to this post by Oleg Nenashev

Dear all,


We are ready to proceed with restoration of the Jenkins account database. Today we are going to restore user LDAP accounts that were created since the First of February 2020 based on the data from Jenkins Jira and the repository Permission Manager metadata data. We will also reset passwords for all users registered in the database.


Step 1. All users who lost their account will receive an email saying that their accounts were re-created. There will be no temporary password in these emails, but there will be information pointing to this thread.


Step 2. We’ll reset every user password from the LDAP database, it is more than 100 000 users. Once done, you’ll receive an email telling you that your password was reset with a reason containing a link to this mail thread.


Step 3. We will delete accounts of users who requested such deletion between February and June 2020. These users were restored from the backup, so we have to delete them again.The list of users is based on Jira tickets and private messages to the Jenkins Infra officer. If for some reason you notice that your account still exists, feel free to raise a ticket in Jenkins Jira (project=INFRA, component=account).


Please do not hesitate to contact us using the #jenkins-infra channel on Freenode IRC or the Jenkins Infrastructure mailing list if you have any questions or suggestions. If you see a security issue related to the accounts, please follow the vulnerability reporting guidelines.


Best regards,

Olivier Vernin && Jenkins Infrastructure Team



On Tuesday, 9 June 2020 17:00:25 UTC+2, Oleg Nenashev wrote:

Dear all,


As you may have noticed, the release artifact uploads are currently blocked in the Jenkins Artifactory instances (<a href="https://repo.jenkins-ci.org/" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Frepo.jenkins-ci.org%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNH4WVSnOUybsblDMAtolGO0Jbm_oQ&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Frepo.jenkins-ci.org%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNH4WVSnOUybsblDMAtolGO0Jbm_oQ&#39;;return true;">https://repo.jenkins-ci.org/). We are doing a security investigation due to a partial user database loss on June 02. Today we blocked releases to the Jenkins artifactory, and there also was a temporary outage of the Artifactory downloads which was a collateral damage of the temporary permissions. You can find more details about it in this<a href="https://groups.google.com/forum/#!topic/jenkins-infra/zRqdiyarLDE" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://groups.google.com/forum/#!topic/jenkins-infra/zRqdiyarLDE&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/forum/#!topic/jenkins-infra/zRqdiyarLDE&#39;;return true;"> Jenkins Infra Thread and in this<a href="https://groups.google.com/d/msg/jenkinsci-dev/juHejx8zfdg/xpySiv1_CQAJ" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://groups.google.com/d/msg/jenkinsci-dev/juHejx8zfdg/xpySiv1_CQAJ&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/msg/jenkinsci-dev/juHejx8zfdg/xpySiv1_CQAJ&#39;;return true;"> Dev List thread.


Current status:

  • Downloads are restored for all artifacts on <a href="https://repo.jenkins-ci.org/" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Frepo.jenkins-ci.org%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNH4WVSnOUybsblDMAtolGO0Jbm_oQ&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Frepo.jenkins-ci.org%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNH4WVSnOUybsblDMAtolGO0Jbm_oQ&#39;;return true;">https://repo.jenkins-ci.org/, Jenkins core historical releases, Remoting library and Windows Service Wrapper which were among ones reported by Jenkins users.

  • Uploads: Jenkins artifact uploads are blocked for the most of Jenkins plugin maintainers and contributors. It affects releases of Jenkins plugins, Jenkins core and modules, developer tools and all libraries hosted on <a href="https://repo.jenkins-ci.org/" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Frepo.jenkins-ci.org%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNH4WVSnOUybsblDMAtolGO0Jbm_oQ&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Frepo.jenkins-ci.org%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNH4WVSnOUybsblDMAtolGO0Jbm_oQ&#39;;return true;">https://repo.jenkins-ci.org/. Incremental and Snapshot deployments are not affected.


Quick summary: 

  • Jun 02 - There was a Kubernetes Cluster outage on June 02. During this outage we had to rebuild the cluster from scratch to get some services working again.

  • Jun 02 - After the recovery we lost three months of LDAP changes. It has happened due to the broken backup of the LDAP database.

  • Jun 02 - We identified a number of potential security risks which may be caused by the LDAP outage. Account overtake and malicious upload was one of the identified risks. FTR this issue is tracked as SECURITY-1895 as a follow-up to these discussions. Only the Security team members have access to it, so I am not sharing a link here.

  • Jun 09 - After the security risk was independently reported in public by a plugin maintainer in the <a href="https://groups.google.com/g/jenkinsci-dev/c/juHejx8zfdg" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://groups.google.com/g/jenkinsci-dev/c/juHejx8zfdg&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/g/jenkinsci-dev/c/juHejx8zfdg&#39;;return true;">dev list thread, we decided to block uploads of release artifacts to the Jenkins Artifactory instance.

  • Jun 09, 8:50AM UTC - All uploads of release artifacts were blocked (plugins, Jenkins core and modules, developer tools, etc.). Downloads of some binaries were also blocked as an unexpected collateral damage. Jenkins core historical releases, Remoting library and Windows Service Wrapper are among the affected binaries

  • Jun 09, 10AM UTC - We finished reviews of all artifact releases to <a href="https://repo.jenkins-ci.org/" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Frepo.jenkins-ci.org%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNH4WVSnOUybsblDMAtolGO0Jbm_oQ&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Frepo.jenkins-ci.org%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNH4WVSnOUybsblDMAtolGO0Jbm_oQ&#39;;return true;">https://repo.jenkins-ci.org/, which happened between the infra outage on June 02 and the blockage of the releases. There are no maliciously uploaded artifacts. Note that the common plugin release flow requires access to GitHub in order to push the release commits, so a malicious attacker would need to overtake both Jenkins and GitHub accounts of a single user to submit a legitimately-looking release.

  • Jun 09, ~1PM UTC - Artifact downloads are restored, <a href="https://github.com/jenkins-infra/repository-permissions-updater/pull/1569" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Fjenkins-infra%2Frepository-permissions-updater%2Fpull%2F1569\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHnXqHe8DbBsyaWsN32PYLgiFuxww&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Fjenkins-infra%2Frepository-permissions-updater%2Fpull%2F1569\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHnXqHe8DbBsyaWsN32PYLgiFuxww&#39;;return true;">alternate patch in the Repository Permission Updater was applied to prevent uploads. Artifact uploads are still blocking

  • Jun 09, 2PM UTC, based on <a href="http://repo.jenkins-ci.org" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Frepo.jenkins-ci.org\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNH-lqjQdpr_N3OJq6LbHFX-8CE0vg&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Frepo.jenkins-ci.org\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNH-lqjQdpr_N3OJq6LbHFX-8CE0vg&#39;;return true;">repo.jenkins-ci.org and <a href="http://issues.jenkins-ci.org" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fissues.jenkins-ci.org\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEt2OBDOWbsPhB3LOncdqtsX9Mk0Q&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fissues.jenkins-ci.org\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEt2OBDOWbsPhB3LOncdqtsX9Mk0Q&#39;;return true;">issues.jenkins-ci.org data, we restored maintainers accounts.


Our next steps would be to communicate the issue to all maintainers and contributors who might have been affected by the LDAP history loss. We will likely need to perform additional user verification steps for plugin maintainers to ensure that there are no contributors affected by the issues. Today at 3:30PM UTC we will also have a Jenkins Infrastructure team meeting where this issue will be discussed in more detail. This is a public meeting, and everyone is welcome to join. <a href="https://calendar.google.com/event?action=TEMPLATE&amp;tmeid=dTJsaWoxN2xjZHFkajRsbmJlcWFiaXI5b2JfMjAyMDA2MDlUMTUzMDAwWiA0c3MxMmYwbXFyM3RicDF0MmZlMzY5c2xmNEBn&amp;tmsrc=4ss12f0mqr3tbp1t2fe369slf4%40group.calendar.google.com" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://calendar.google.com/event?action\x3dTEMPLATE\x26tmeid\x3ddTJsaWoxN2xjZHFkajRsbmJlcWFiaXI5b2JfMjAyMDA2MDlUMTUzMDAwWiA0c3MxMmYwbXFyM3RicDF0MmZlMzY5c2xmNEBn\x26tmsrc\x3d4ss12f0mqr3tbp1t2fe369slf4%40group.calendar.google.com&#39;;return true;" onclick="this.href=&#39;https://calendar.google.com/event?action\x3dTEMPLATE\x26tmeid\x3ddTJsaWoxN2xjZHFkajRsbmJlcWFiaXI5b2JfMjAyMDA2MDlUMTUzMDAwWiA0c3MxMmYwbXFyM3RicDF0MmZlMzY5c2xmNEBn\x26tmsrc\x3d4ss12f0mqr3tbp1t2fe369slf4%40group.calendar.google.com&#39;;return true;">Calendar link


Thanks to Olivier Vernin, Daniel Beck and other Jenkins Infra and Security team members who contributed to this investigation.


Best regards,

Oleg Nenashev


--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/0768eed7-d895-4717-9f9e-924abc803f61o%40googlegroups.com.
Reply | Threaded
Open this post in threaded view
|

Re: ANN: Jenkins release artifacts uploads blockage on June 09, and a temporary downloads issue

Dave Pedu
Hello,

I have received an email linking to this thread. However, it contains a plaintext password for my account, despite this:

There will be no temporary password in these emails, but there will be information pointing to this thread.

Is this email legitimate or am I being phished? Screenshot attached.

Thanks,
Dave

On Thursday, June 11, 2020 at 3:07:01 AM UTC-7, Olblak wrote:

Dear all,


We are ready to proceed with restoration of the Jenkins account database. Today we are going to restore user LDAP accounts that were created since the First of February 2020 based on the data from Jenkins Jira and the repository Permission Manager metadata data. We will also reset passwords for all users registered in the database.


Step 1. All users who lost their account will receive an email saying that their accounts were re-created. There will be no temporary password in these emails, but there will be information pointing to this thread.


Step 2. We’ll reset every user password from the LDAP database, it is more than 100 000 users. Once done, you’ll receive an email telling you that your password was reset with a reason containing a link to this mail thread.


Step 3. We will delete accounts of users who requested such deletion between February and June 2020. These users were restored from the backup, so we have to delete them again.The list of users is based on Jira tickets and private messages to the Jenkins Infra officer. If for some reason you notice that your account still exists, feel free to raise a ticket in <a href="https://issues.jenkins-ci.org/" style="text-decoration:none" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fissues.jenkins-ci.org%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEr7Zs9c804d7zhbfP2jk75fuaVRA&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fissues.jenkins-ci.org%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEr7Zs9c804d7zhbfP2jk75fuaVRA&#39;;return true;">Jenkins Jira (project=INFRA, component=account).


Please do not hesitate to contact us using the #jenkins-infra channel on Freenode IRC or the Jenkins Infrastructure mailing list if you have any questions or suggestions. If you see a security issue related to the accounts, please follow the <a href="https://www.jenkins.io/security/#reporting-vulnerabilities" style="text-decoration:none" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fwww.jenkins.io%2Fsecurity%2F%23reporting-vulnerabilities\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFNS8fVviacDMvDRLXWORjHGybVDQ&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fwww.jenkins.io%2Fsecurity%2F%23reporting-vulnerabilities\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFNS8fVviacDMvDRLXWORjHGybVDQ&#39;;return true;">vulnerability reporting guidelines.


Best regards,

Olivier Vernin && Jenkins Infrastructure Team



On Tuesday, 9 June 2020 17:00:25 UTC+2, Oleg Nenashev wrote:

Dear all,


As you may have noticed, the release artifact uploads are currently blocked in the Jenkins Artifactory instances (<a href="https://repo.jenkins-ci.org/" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Frepo.jenkins-ci.org%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNH4WVSnOUybsblDMAtolGO0Jbm_oQ&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Frepo.jenkins-ci.org%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNH4WVSnOUybsblDMAtolGO0Jbm_oQ&#39;;return true;">https://repo.jenkins-ci.org/). We are doing a security investigation due to a partial user database loss on June 02. Today we blocked releases to the Jenkins artifactory, and there also was a temporary outage of the Artifactory downloads which was a collateral damage of the temporary permissions. You can find more details about it in this<a href="https://groups.google.com/forum/#!topic/jenkins-infra/zRqdiyarLDE" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://groups.google.com/forum/#!topic/jenkins-infra/zRqdiyarLDE&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/forum/#!topic/jenkins-infra/zRqdiyarLDE&#39;;return true;"> Jenkins Infra Thread and in this<a href="https://groups.google.com/d/msg/jenkinsci-dev/juHejx8zfdg/xpySiv1_CQAJ" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://groups.google.com/d/msg/jenkinsci-dev/juHejx8zfdg/xpySiv1_CQAJ&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/msg/jenkinsci-dev/juHejx8zfdg/xpySiv1_CQAJ&#39;;return true;"> Dev List thread.


Current status:

  • Downloads are restored for all artifacts on <a href="https://repo.jenkins-ci.org/" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Frepo.jenkins-ci.org%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNH4WVSnOUybsblDMAtolGO0Jbm_oQ&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Frepo.jenkins-ci.org%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNH4WVSnOUybsblDMAtolGO0Jbm_oQ&#39;;return true;">https://repo.jenkins-ci.org/, Jenkins core historical releases, Remoting library and Windows Service Wrapper which were among ones reported by Jenkins users.

  • Uploads: Jenkins artifact uploads are blocked for the most of Jenkins plugin maintainers and contributors. It affects releases of Jenkins plugins, Jenkins core and modules, developer tools and all libraries hosted on <a href="https://repo.jenkins-ci.org/" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Frepo.jenkins-ci.org%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNH4WVSnOUybsblDMAtolGO0Jbm_oQ&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Frepo.jenkins-ci.org%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNH4WVSnOUybsblDMAtolGO0Jbm_oQ&#39;;return true;">https://repo.jenkins-ci.org/. Incremental and Snapshot deployments are not affected.


Quick summary: 

  • Jun 02 - There was a Kubernetes Cluster outage on June 02. During this outage we had to rebuild the cluster from scratch to get some services working again.

  • Jun 02 - After the recovery we lost three months of LDAP changes. It has happened due to the broken backup of the LDAP database.

  • Jun 02 - We identified a number of potential security risks which may be caused by the LDAP outage. Account overtake and malicious upload was one of the identified risks. FTR this issue is tracked as SECURITY-1895 as a follow-up to these discussions. Only the Security team members have access to it, so I am not sharing a link here.

  • Jun 09 - After the security risk was independently reported in public by a plugin maintainer in the <a href="https://groups.google.com/g/jenkinsci-dev/c/juHejx8zfdg" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://groups.google.com/g/jenkinsci-dev/c/juHejx8zfdg&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/g/jenkinsci-dev/c/juHejx8zfdg&#39;;return true;">dev list thread, we decided to block uploads of release artifacts to the Jenkins Artifactory instance.

  • Jun 09, 8:50AM UTC - All uploads of release artifacts were blocked (plugins, Jenkins core and modules, developer tools, etc.). Downloads of some binaries were also blocked as an unexpected collateral damage. Jenkins core historical releases, Remoting library and Windows Service Wrapper are among the affected binaries

  • Jun 09, 10AM UTC - We finished reviews of all artifact releases to <a href="https://repo.jenkins-ci.org/" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Frepo.jenkins-ci.org%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNH4WVSnOUybsblDMAtolGO0Jbm_oQ&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Frepo.jenkins-ci.org%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNH4WVSnOUybsblDMAtolGO0Jbm_oQ&#39;;return true;">https://repo.jenkins-ci.org/, which happened between the infra outage on June 02 and the blockage of the releases. There are no maliciously uploaded artifacts. Note that the common plugin release flow requires access to GitHub in order to push the release commits, so a malicious attacker would need to overtake both Jenkins and GitHub accounts of a single user to submit a legitimately-looking release.

  • Jun 09, ~1PM UTC - Artifact downloads are restored, <a href="https://github.com/jenkins-infra/repository-permissions-updater/pull/1569" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Fjenkins-infra%2Frepository-permissions-updater%2Fpull%2F1569\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHnXqHe8DbBsyaWsN32PYLgiFuxww&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Fjenkins-infra%2Frepository-permissions-updater%2Fpull%2F1569\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHnXqHe8DbBsyaWsN32PYLgiFuxww&#39;;return true;">alternate patch in the Repository Permission Updater was applied to prevent uploads. Artifact uploads are still blocking

  • Jun 09, 2PM UTC, based on <a href="http://repo.jenkins-ci.org" rel="nofollow" target="_blank" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Frepo.jenkins-ci.org\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNH-lqjQdpr_N3OJq6LbHFX-8CE0vg&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Frepo.jenkins-ci.org\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNH-lqjQdpr_N3OJq6LbHFX-8CE0vg&#39;;return true;">repo.jenkins-ci.org and <a href="http://issues.jenkins-ci.org" rel="nofollow" target="_blank" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fissues.jenkins-ci.org\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEt2OBDOWbsPhB3LOncdqtsX9Mk0Q&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fissues.jenkins-ci.org\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEt2OBDOWbsPhB3LOncdqtsX9Mk0Q&#39;;return true;">issues.jenkins-ci.org data, we restored maintainers accounts.


Our next steps would be to communicate the issue to all maintainers and contributors who might have been affected by the LDAP history loss. We will likely need to perform additional user verification steps for plugin maintainers to ensure that there are no contributors affected by the issues. Today at 3:30PM UTC we will also have a Jenkins Infrastructure team meeting where this issue will be discussed in more detail. This is a public meeting, and everyone is welcome to join. <a href="https://calendar.google.com/event?action=TEMPLATE&amp;tmeid=dTJsaWoxN2xjZHFkajRsbmJlcWFiaXI5b2JfMjAyMDA2MDlUMTUzMDAwWiA0c3MxMmYwbXFyM3RicDF0MmZlMzY5c2xmNEBn&amp;tmsrc=4ss12f0mqr3tbp1t2fe369slf4%40group.calendar.google.com" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://calendar.google.com/event?action\x3dTEMPLATE\x26tmeid\x3ddTJsaWoxN2xjZHFkajRsbmJlcWFiaXI5b2JfMjAyMDA2MDlUMTUzMDAwWiA0c3MxMmYwbXFyM3RicDF0MmZlMzY5c2xmNEBn\x26tmsrc\x3d4ss12f0mqr3tbp1t2fe369slf4%40group.calendar.google.com&#39;;return true;" onclick="this.href=&#39;https://calendar.google.com/event?action\x3dTEMPLATE\x26tmeid\x3ddTJsaWoxN2xjZHFkajRsbmJlcWFiaXI5b2JfMjAyMDA2MDlUMTUzMDAwWiA0c3MxMmYwbXFyM3RicDF0MmZlMzY5c2xmNEBn\x26tmsrc\x3d4ss12f0mqr3tbp1t2fe369slf4%40group.calendar.google.com&#39;;return true;">Calendar link


Thanks to Olivier Vernin, Daniel Beck and other Jenkins Infra and Security team members who contributed to this investigation.


Best regards,

Oleg Nenashev


--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/740722e8-454f-4cfd-994e-f5c9982d0d8bo%40googlegroups.com.

jenkinsmail.png (156K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: ANN: Jenkins release artifacts uploads blockage on June 09, and a temporary downloads issue

Oleg Nenashev
In reply to this post by Olblak-2
June 12th update:
  • We are still working on the account migration
    • Step 1 is completed, all users have been restored in the database based on the data from Jenkins Jira and repository permissions updater.
    • Step 2 is in progress. Tens of thousands users have already received the password reset notifications, we had 2 batches of password resets today. We will continue the migration tomorrow
    • Step 3 - not started
  • Plugin uploads are still blocked at the moment
    • Tomorrow we plan to double-check the account resets for plugin maintainers, and we will consider reenabling uploads after that
Best regards,
Oleg

On Thursday, June 11, 2020 at 12:07:01 PM UTC+2, Olblak wrote:

Dear all,


We are ready to proceed with restoration of the Jenkins account database. Today we are going to restore user LDAP accounts that were created since the First of February 2020 based on the data from Jenkins Jira and the repository Permission Manager metadata data. We will also reset passwords for all users registered in the database.


Step 1. All users who lost their account will receive an email saying that their accounts were re-created. There will be no temporary password in these emails, but there will be information pointing to this thread.


Step 2. We’ll reset every user password from the LDAP database, it is more than 100 000 users. Once done, you’ll receive an email telling you that your password was reset with a reason containing a link to this mail thread.


Step 3. We will delete accounts of users who requested such deletion between February and June 2020. These users were restored from the backup, so we have to delete them again.The list of users is based on Jira tickets and private messages to the Jenkins Infra officer. If for some reason you notice that your account still exists, feel free to raise a ticket in <a href="https://issues.jenkins-ci.org/" style="text-decoration:none" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fissues.jenkins-ci.org%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEr7Zs9c804d7zhbfP2jk75fuaVRA&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fissues.jenkins-ci.org%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEr7Zs9c804d7zhbfP2jk75fuaVRA&#39;;return true;">Jenkins Jira (project=INFRA, component=account).


Please do not hesitate to contact us using the #jenkins-infra channel on Freenode IRC or the Jenkins Infrastructure mailing list if you have any questions or suggestions. If you see a security issue related to the accounts, please follow the <a href="https://www.jenkins.io/security/#reporting-vulnerabilities" style="text-decoration:none" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fwww.jenkins.io%2Fsecurity%2F%23reporting-vulnerabilities\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFNS8fVviacDMvDRLXWORjHGybVDQ&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fwww.jenkins.io%2Fsecurity%2F%23reporting-vulnerabilities\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFNS8fVviacDMvDRLXWORjHGybVDQ&#39;;return true;">vulnerability reporting guidelines.


Best regards,

Olivier Vernin && Jenkins Infrastructure Team



On Tuesday, 9 June 2020 17:00:25 UTC+2, Oleg Nenashev wrote:

Dear all,


As you may have noticed, the release artifact uploads are currently blocked in the Jenkins Artifactory instances (<a href="https://repo.jenkins-ci.org/" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Frepo.jenkins-ci.org%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNH4WVSnOUybsblDMAtolGO0Jbm_oQ&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Frepo.jenkins-ci.org%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNH4WVSnOUybsblDMAtolGO0Jbm_oQ&#39;;return true;">https://repo.jenkins-ci.org/). We are doing a security investigation due to a partial user database loss on June 02. Today we blocked releases to the Jenkins artifactory, and there also was a temporary outage of the Artifactory downloads which was a collateral damage of the temporary permissions. You can find more details about it in this<a href="https://groups.google.com/forum/#!topic/jenkins-infra/zRqdiyarLDE" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://groups.google.com/forum/#!topic/jenkins-infra/zRqdiyarLDE&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/forum/#!topic/jenkins-infra/zRqdiyarLDE&#39;;return true;"> Jenkins Infra Thread and in this<a href="https://groups.google.com/d/msg/jenkinsci-dev/juHejx8zfdg/xpySiv1_CQAJ" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://groups.google.com/d/msg/jenkinsci-dev/juHejx8zfdg/xpySiv1_CQAJ&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/msg/jenkinsci-dev/juHejx8zfdg/xpySiv1_CQAJ&#39;;return true;"> Dev List thread.


Current status:

  • Downloads are restored for all artifacts on <a href="https://repo.jenkins-ci.org/" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Frepo.jenkins-ci.org%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNH4WVSnOUybsblDMAtolGO0Jbm_oQ&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Frepo.jenkins-ci.org%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNH4WVSnOUybsblDMAtolGO0Jbm_oQ&#39;;return true;">https://repo.jenkins-ci.org/, Jenkins core historical releases, Remoting library and Windows Service Wrapper which were among ones reported by Jenkins users.

  • Uploads: Jenkins artifact uploads are blocked for the most of Jenkins plugin maintainers and contributors. It affects releases of Jenkins plugins, Jenkins core and modules, developer tools and all libraries hosted on <a href="https://repo.jenkins-ci.org/" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Frepo.jenkins-ci.org%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNH4WVSnOUybsblDMAtolGO0Jbm_oQ&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Frepo.jenkins-ci.org%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNH4WVSnOUybsblDMAtolGO0Jbm_oQ&#39;;return true;">https://repo.jenkins-ci.org/. Incremental and Snapshot deployments are not affected.


Quick summary: 

  • Jun 02 - There was a Kubernetes Cluster outage on June 02. During this outage we had to rebuild the cluster from scratch to get some services working again.

  • Jun 02 - After the recovery we lost three months of LDAP changes. It has happened due to the broken backup of the LDAP database.

  • Jun 02 - We identified a number of potential security risks which may be caused by the LDAP outage. Account overtake and malicious upload was one of the identified risks. FTR this issue is tracked as SECURITY-1895 as a follow-up to these discussions. Only the Security team members have access to it, so I am not sharing a link here.

  • Jun 09 - After the security risk was independently reported in public by a plugin maintainer in the <a href="https://groups.google.com/g/jenkinsci-dev/c/juHejx8zfdg" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://groups.google.com/g/jenkinsci-dev/c/juHejx8zfdg&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/g/jenkinsci-dev/c/juHejx8zfdg&#39;;return true;">dev list thread, we decided to block uploads of release artifacts to the Jenkins Artifactory instance.

  • Jun 09, 8:50AM UTC - All uploads of release artifacts were blocked (plugins, Jenkins core and modules, developer tools, etc.). Downloads of some binaries were also blocked as an unexpected collateral damage. Jenkins core historical releases, Remoting library and Windows Service Wrapper are among the affected binaries

  • Jun 09, 10AM UTC - We finished reviews of all artifact releases to <a href="https://repo.jenkins-ci.org/" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Frepo.jenkins-ci.org%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNH4WVSnOUybsblDMAtolGO0Jbm_oQ&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Frepo.jenkins-ci.org%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNH4WVSnOUybsblDMAtolGO0Jbm_oQ&#39;;return true;">https://repo.jenkins-ci.org/, which happened between the infra outage on June 02 and the blockage of the releases. There are no maliciously uploaded artifacts. Note that the common plugin release flow requires access to GitHub in order to push the release commits, so a malicious attacker would need to overtake both Jenkins and GitHub accounts of a single user to submit a legitimately-looking release.

  • Jun 09, ~1PM UTC - Artifact downloads are restored, <a href="https://github.com/jenkins-infra/repository-permissions-updater/pull/1569" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Fjenkins-infra%2Frepository-permissions-updater%2Fpull%2F1569\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHnXqHe8DbBsyaWsN32PYLgiFuxww&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Fjenkins-infra%2Frepository-permissions-updater%2Fpull%2F1569\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHnXqHe8DbBsyaWsN32PYLgiFuxww&#39;;return true;">alternate patch in the Repository Permission Updater was applied to prevent uploads. Artifact uploads are still blocking

  • Jun 09, 2PM UTC, based on <a href="http://repo.jenkins-ci.org" rel="nofollow" target="_blank" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Frepo.jenkins-ci.org\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNH-lqjQdpr_N3OJq6LbHFX-8CE0vg&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Frepo.jenkins-ci.org\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNH-lqjQdpr_N3OJq6LbHFX-8CE0vg&#39;;return true;">repo.jenkins-ci.org and <a href="http://issues.jenkins-ci.org" rel="nofollow" target="_blank" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fissues.jenkins-ci.org\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEt2OBDOWbsPhB3LOncdqtsX9Mk0Q&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fissues.jenkins-ci.org\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEt2OBDOWbsPhB3LOncdqtsX9Mk0Q&#39;;return true;">issues.jenkins-ci.org data, we restored maintainers accounts.


Our next steps would be to communicate the issue to all maintainers and contributors who might have been affected by the LDAP history loss. We will likely need to perform additional user verification steps for plugin maintainers to ensure that there are no contributors affected by the issues. Today at 3:30PM UTC we will also have a Jenkins Infrastructure team meeting where this issue will be discussed in more detail. This is a public meeting, and everyone is welcome to join. <a href="https://calendar.google.com/event?action=TEMPLATE&amp;tmeid=dTJsaWoxN2xjZHFkajRsbmJlcWFiaXI5b2JfMjAyMDA2MDlUMTUzMDAwWiA0c3MxMmYwbXFyM3RicDF0MmZlMzY5c2xmNEBn&amp;tmsrc=4ss12f0mqr3tbp1t2fe369slf4%40group.calendar.google.com" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://calendar.google.com/event?action\x3dTEMPLATE\x26tmeid\x3ddTJsaWoxN2xjZHFkajRsbmJlcWFiaXI5b2JfMjAyMDA2MDlUMTUzMDAwWiA0c3MxMmYwbXFyM3RicDF0MmZlMzY5c2xmNEBn\x26tmsrc\x3d4ss12f0mqr3tbp1t2fe369slf4%40group.calendar.google.com&#39;;return true;" onclick="this.href=&#39;https://calendar.google.com/event?action\x3dTEMPLATE\x26tmeid\x3ddTJsaWoxN2xjZHFkajRsbmJlcWFiaXI5b2JfMjAyMDA2MDlUMTUzMDAwWiA0c3MxMmYwbXFyM3RicDF0MmZlMzY5c2xmNEBn\x26tmsrc\x3d4ss12f0mqr3tbp1t2fe369slf4%40group.calendar.google.com&#39;;return true;">Calendar link


Thanks to Olivier Vernin, Daniel Beck and other Jenkins Infra and Security team members who contributed to this investigation.


Best regards,

Oleg Nenashev


--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/9f9cf846-f5b6-4906-87a7-6f2faf969c9fo%40googlegroups.com.
Reply | Threaded
Open this post in threaded view
|

Re: ANN: Jenkins release artifacts uploads blockage on June 09, and a temporary downloads issue

Oleg Nenashev
In reply to this post by Dave Pedu
Hi Dave,

This is an email from the Step 2. We’ll reset every user password from the LDAP database. This one includes a temporary password, and we expect users to change it after they login into the system.

For those who wonder: Yes, the temporary password is sent in plain text as mentioned above. This is how our current password reset system is designed. As other projects, we have a decent amount of technical debt in our infrastructure which we gradually resolve. I have already added changing the account password reset flow to the outage retrospective list, an we will be reviewing what to do there after the outage is fully resolved. Apart from fixing it, migrating to a 3rd-party identity service is on the table for me (Linux Foundation or GitHub).  If anyone is interested to participate and to improve the project, the Jenkins infrastructure team is always looking for more contributors!

If anyone has concerns about such method and wants to use alternate channels for encrypted password transfer, please send us a message through the Jenkins Infrastructure mailing list from your email registered in Jenkins. In this email please provide your public GPG key so that we can reset a password again in a secure way.

Best regards,
Oleg

On Friday, June 12, 2020 at 5:07:27 PM UTC+2, Dave Pedu wrote:
Hello,

I have received an email linking to this thread. However, it contains a plaintext password for my account, despite this:

There will be no temporary password in these emails, but there will be information pointing to this thread.

Is this email legitimate or am I being phished? Screenshot attached.

Thanks,
Dave

On Thursday, June 11, 2020 at 3:07:01 AM UTC-7, Olblak wrote:

Dear all,


We are ready to proceed with restoration of the Jenkins account database. Today we are going to restore user LDAP accounts that were created since the First of February 2020 based on the data from Jenkins Jira and the repository Permission Manager metadata data. We will also reset passwords for all users registered in the database.


Step 1. All users who lost their account will receive an email saying that their accounts were re-created. There will be no temporary password in these emails, but there will be information pointing to this thread.


Step 2. We’ll reset every user password from the LDAP database, it is more than 100 000 users. Once done, you’ll receive an email telling you that your password was reset with a reason containing a link to this mail thread.


Step 3. We will delete accounts of users who requested such deletion between February and June 2020. These users were restored from the backup, so we have to delete them again.The list of users is based on Jira tickets and private messages to the Jenkins Infra officer. If for some reason you notice that your account still exists, feel free to raise a ticket in <a href="https://issues.jenkins-ci.org/" style="text-decoration:none" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fissues.jenkins-ci.org%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEr7Zs9c804d7zhbfP2jk75fuaVRA&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fissues.jenkins-ci.org%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEr7Zs9c804d7zhbfP2jk75fuaVRA&#39;;return true;">Jenkins Jira (project=INFRA, component=account).


Please do not hesitate to contact us using the #jenkins-infra channel on Freenode IRC or the Jenkins Infrastructure mailing list if you have any questions or suggestions. If you see a security issue related to the accounts, please follow the <a href="https://www.jenkins.io/security/#reporting-vulnerabilities" style="text-decoration:none" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fwww.jenkins.io%2Fsecurity%2F%23reporting-vulnerabilities\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFNS8fVviacDMvDRLXWORjHGybVDQ&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fwww.jenkins.io%2Fsecurity%2F%23reporting-vulnerabilities\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFNS8fVviacDMvDRLXWORjHGybVDQ&#39;;return true;">vulnerability reporting guidelines.


Best regards,

Olivier Vernin && Jenkins Infrastructure Team



On Tuesday, 9 June 2020 17:00:25 UTC+2, Oleg Nenashev wrote:

Dear all,


As you may have noticed, the release artifact uploads are currently blocked in the Jenkins Artifactory instances (<a href="https://repo.jenkins-ci.org/" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Frepo.jenkins-ci.org%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNH4WVSnOUybsblDMAtolGO0Jbm_oQ&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Frepo.jenkins-ci.org%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNH4WVSnOUybsblDMAtolGO0Jbm_oQ&#39;;return true;">https://repo.jenkins-ci.org/). We are doing a security investigation due to a partial user database loss on June 02. Today we blocked releases to the Jenkins artifactory, and there also was a temporary outage of the Artifactory downloads which was a collateral damage of the temporary permissions. You can find more details about it in this<a href="https://groups.google.com/forum/#!topic/jenkins-infra/zRqdiyarLDE" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://groups.google.com/forum/#!topic/jenkins-infra/zRqdiyarLDE&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/forum/#!topic/jenkins-infra/zRqdiyarLDE&#39;;return true;"> Jenkins Infra Thread and in this<a href="https://groups.google.com/d/msg/jenkinsci-dev/juHejx8zfdg/xpySiv1_CQAJ" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://groups.google.com/d/msg/jenkinsci-dev/juHejx8zfdg/xpySiv1_CQAJ&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/msg/jenkinsci-dev/juHejx8zfdg/xpySiv1_CQAJ&#39;;return true;"> Dev List thread.


Current status:

  • Downloads are restored for all artifacts on <a href="https://repo.jenkins-ci.org/" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Frepo.jenkins-ci.org%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNH4WVSnOUybsblDMAtolGO0Jbm_oQ&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Frepo.jenkins-ci.org%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNH4WVSnOUybsblDMAtolGO0Jbm_oQ&#39;;return true;">https://repo.jenkins-ci.org/, Jenkins core historical releases, Remoting library and Windows Service Wrapper which were among ones reported by Jenkins users.

  • Uploads: Jenkins artifact uploads are blocked for the most of Jenkins plugin maintainers and contributors. It affects releases of Jenkins plugins, Jenkins core and modules, developer tools and all libraries hosted on <a href="https://repo.jenkins-ci.org/" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Frepo.jenkins-ci.org%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNH4WVSnOUybsblDMAtolGO0Jbm_oQ&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Frepo.jenkins-ci.org%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNH4WVSnOUybsblDMAtolGO0Jbm_oQ&#39;;return true;">https://repo.jenkins-ci.org/. Incremental and Snapshot deployments are not affected.


Quick summary: 

  • Jun 02 - There was a Kubernetes Cluster outage on June 02. During this outage we had to rebuild the cluster from scratch to get some services working again.

  • Jun 02 - After the recovery we lost three months of LDAP changes. It has happened due to the broken backup of the LDAP database.

  • Jun 02 - We identified a number of potential security risks which may be caused by the LDAP outage. Account overtake and malicious upload was one of the identified risks. FTR this issue is tracked as SECURITY-1895 as a follow-up to these discussions. Only the Security team members have access to it, so I am not sharing a link here.

  • Jun 09 - After the security risk was independently reported in public by a plugin maintainer in the <a href="https://groups.google.com/g/jenkinsci-dev/c/juHejx8zfdg" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://groups.google.com/g/jenkinsci-dev/c/juHejx8zfdg&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/g/jenkinsci-dev/c/juHejx8zfdg&#39;;return true;">dev list thread, we decided to block uploads of release artifacts to the Jenkins Artifactory instance.

  • Jun 09, 8:50AM UTC - All uploads of release artifacts were blocked (plugins, Jenkins core and modules, developer tools, etc.). Downloads of some binaries were also blocked as an unexpected collateral damage. Jenkins core historical releases, Remoting library and Windows Service Wrapper are among the affected binaries

  • Jun 09, 10AM UTC - We finished reviews of all artifact releases to <a href="https://repo.jenkins-ci.org/" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Frepo.jenkins-ci.org%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNH4WVSnOUybsblDMAtolGO0Jbm_oQ&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Frepo.jenkins-ci.org%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNH4WVSnOUybsblDMAtolGO0Jbm_oQ&#39;;return true;">https://repo.jenkins-ci.org/, which happened between the infra outage on June 02 and the blockage of the releases. There are no maliciously uploaded artifacts. Note that the common plugin release flow requires access to GitHub in order to push the release commits, so a malicious attacker would need to overtake both Jenkins and GitHub accounts of a single user to submit a legitimately-looking release.

  • Jun 09, ~1PM UTC - Artifact downloads are restored, <a href="https://github.com/jenkins-infra/repository-permissions-updater/pull/1569" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Fjenkins-infra%2Frepository-permissions-updater%2Fpull%2F1569\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHnXqHe8DbBsyaWsN32PYLgiFuxww&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Fjenkins-infra%2Frepository-permissions-updater%2Fpull%2F1569\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHnXqHe8DbBsyaWsN32PYLgiFuxww&#39;;return true;">alternate patch in the Repository Permission Updater was applied to prevent uploads. Artifact uploads are still blocking

  • Jun 09, 2PM UTC, based on <a href="http://repo.jenkins-ci.org" rel="nofollow" target="_blank" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Frepo.jenkins-ci.org\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNH-lqjQdpr_N3OJq6LbHFX-8CE0vg&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Frepo.jenkins-ci.org\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNH-lqjQdpr_N3OJq6LbHFX-8CE0vg&#39;;return true;">repo.jenkins-ci.org and <a href="http://issues.jenkins-ci.org" rel="nofollow" target="_blank" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fissues.jenkins-ci.org\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEt2OBDOWbsPhB3LOncdqtsX9Mk0Q&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fissues.jenkins-ci.org\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEt2OBDOWbsPhB3LOncdqtsX9Mk0Q&#39;;return true;">issues.jenkins-ci.org data, we restored maintainers accounts.


Our next steps would be to communicate the issue to all maintainers and contributors who might have been affected by the LDAP history loss. We will likely need to perform additional user verification steps for plugin maintainers to ensure that there are no contributors affected by the issues. Today at 3:30PM UTC we will also have a Jenkins Infrastructure team meeting where this issue will be discussed in more detail. This is a public meeting, and everyone is welcome to join. <a href="https://calendar.google.com/event?action=TEMPLATE&amp;tmeid=dTJsaWoxN2xjZHFkajRsbmJlcWFiaXI5b2JfMjAyMDA2MDlUMTUzMDAwWiA0c3MxMmYwbXFyM3RicDF0MmZlMzY5c2xmNEBn&amp;tmsrc=4ss12f0mqr3tbp1t2fe369slf4%40group.calendar.google.com" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://calendar.google.com/event?action\x3dTEMPLATE\x26tmeid\x3ddTJsaWoxN2xjZHFkajRsbmJlcWFiaXI5b2JfMjAyMDA2MDlUMTUzMDAwWiA0c3MxMmYwbXFyM3RicDF0MmZlMzY5c2xmNEBn\x26tmsrc\x3d4ss12f0mqr3tbp1t2fe369slf4%40group.calendar.google.com&#39;;return true;" onclick="this.href=&#39;https://calendar.google.com/event?action\x3dTEMPLATE\x26tmeid\x3ddTJsaWoxN2xjZHFkajRsbmJlcWFiaXI5b2JfMjAyMDA2MDlUMTUzMDAwWiA0c3MxMmYwbXFyM3RicDF0MmZlMzY5c2xmNEBn\x26tmsrc\x3d4ss12f0mqr3tbp1t2fe369slf4%40group.calendar.google.com&#39;;return true;">Calendar link


Thanks to Olivier Vernin, Daniel Beck and other Jenkins Infra and Security team members who contributed to this investigation.


Best regards,

Oleg Nenashev


--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/23d37119-b14c-4197-9458-0d74ce7994f7o%40googlegroups.com.
Reply | Threaded
Open this post in threaded view
|

Re: ANN: Jenkins release artifacts uploads blockage on June 09, and a temporary downloads issue

Oleg Nenashev
Dear all,

June 12 update: 
  • We continue to work on the accounts migration and will share the next update on Monday
  • Jenkins releases are still blocked. If there are any emergency releases you need to perform, please reply in this thread.
Best regards,
Oleg Nenashev

On Friday, June 12, 2020 at 6:00:32 PM UTC+2, Oleg Nenashev wrote:
Hi Dave,

This is an email from the Step 2. We’ll reset every user password from the LDAP database. This one includes a temporary password, and we expect users to change it after they login into the system.

For those who wonder: Yes, the temporary password is sent in plain text as mentioned above. This is how our current password reset system is designed. As other projects, we have a decent amount of technical debt in our infrastructure which we gradually resolve. I have already added changing the account password reset flow to the outage retrospective list, an we will be reviewing what to do there after the outage is fully resolved. Apart from fixing it, migrating to a 3rd-party identity service is on the table for me (Linux Foundation or GitHub).  If anyone is interested to participate and to improve the project, the <a href="https://www.jenkins.io/projects/infrastructure/" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fwww.jenkins.io%2Fprojects%2Finfrastructure%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHrHsrPXFz_nsKSMm20dEjtEM6oWQ&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fwww.jenkins.io%2Fprojects%2Finfrastructure%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHrHsrPXFz_nsKSMm20dEjtEM6oWQ&#39;;return true;">Jenkins infrastructure team is always looking for more contributors!

If anyone has concerns about such method and wants to use alternate channels for encrypted password transfer, please send us a message through the Jenkins Infrastructure mailing list from your email registered in Jenkins. In this email please provide your public GPG key so that we can reset a password again in a secure way.

Best regards,
Oleg

On Friday, June 12, 2020 at 5:07:27 PM UTC+2, Dave Pedu wrote:
Hello,

I have received an email linking to this thread. However, it contains a plaintext password for my account, despite this:

There will be no temporary password in these emails, but there will be information pointing to this thread.

Is this email legitimate or am I being phished? Screenshot attached.

Thanks,
Dave

On Thursday, June 11, 2020 at 3:07:01 AM UTC-7, Olblak wrote:

Dear all,


We are ready to proceed with restoration of the Jenkins account database. Today we are going to restore user LDAP accounts that were created since the First of February 2020 based on the data from Jenkins Jira and the repository Permission Manager metadata data. We will also reset passwords for all users registered in the database.


Step 1. All users who lost their account will receive an email saying that their accounts were re-created. There will be no temporary password in these emails, but there will be information pointing to this thread.


Step 2. We’ll reset every user password from the LDAP database, it is more than 100 000 users. Once done, you’ll receive an email telling you that your password was reset with a reason containing a link to this mail thread.


Step 3. We will delete accounts of users who requested such deletion between February and June 2020. These users were restored from the backup, so we have to delete them again.The list of users is based on Jira tickets and private messages to the Jenkins Infra officer. If for some reason you notice that your account still exists, feel free to raise a ticket in <a href="https://issues.jenkins-ci.org/" style="text-decoration:none" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fissues.jenkins-ci.org%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEr7Zs9c804d7zhbfP2jk75fuaVRA&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fissues.jenkins-ci.org%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEr7Zs9c804d7zhbfP2jk75fuaVRA&#39;;return true;">Jenkins Jira (project=INFRA, component=account).


Please do not hesitate to contact us using the #jenkins-infra channel on Freenode IRC or the Jenkins Infrastructure mailing list if you have any questions or suggestions. If you see a security issue related to the accounts, please follow the <a href="https://www.jenkins.io/security/#reporting-vulnerabilities" style="text-decoration:none" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fwww.jenkins.io%2Fsecurity%2F%23reporting-vulnerabilities\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFNS8fVviacDMvDRLXWORjHGybVDQ&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fwww.jenkins.io%2Fsecurity%2F%23reporting-vulnerabilities\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFNS8fVviacDMvDRLXWORjHGybVDQ&#39;;return true;">vulnerability reporting guidelines.


Best regards,

Olivier Vernin && Jenkins Infrastructure Team



On Tuesday, 9 June 2020 17:00:25 UTC+2, Oleg Nenashev wrote:

Dear all,


As you may have noticed, the release artifact uploads are currently blocked in the Jenkins Artifactory instances (<a href="https://repo.jenkins-ci.org/" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Frepo.jenkins-ci.org%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNH4WVSnOUybsblDMAtolGO0Jbm_oQ&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Frepo.jenkins-ci.org%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNH4WVSnOUybsblDMAtolGO0Jbm_oQ&#39;;return true;">https://repo.jenkins-ci.org/). We are doing a security investigation due to a partial user database loss on June 02. Today we blocked releases to the Jenkins artifactory, and there also was a temporary outage of the Artifactory downloads which was a collateral damage of the temporary permissions. You can find more details about it in this<a href="https://groups.google.com/forum/#!topic/jenkins-infra/zRqdiyarLDE" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://groups.google.com/forum/#!topic/jenkins-infra/zRqdiyarLDE&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/forum/#!topic/jenkins-infra/zRqdiyarLDE&#39;;return true;"> Jenkins Infra Thread and in this<a href="https://groups.google.com/d/msg/jenkinsci-dev/juHejx8zfdg/xpySiv1_CQAJ" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://groups.google.com/d/msg/jenkinsci-dev/juHejx8zfdg/xpySiv1_CQAJ&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/msg/jenkinsci-dev/juHejx8zfdg/xpySiv1_CQAJ&#39;;return true;"> Dev List thread.


Current status:

  • Downloads are restored for all artifacts on <a href="https://repo.jenkins-ci.org/" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Frepo.jenkins-ci.org%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNH4WVSnOUybsblDMAtolGO0Jbm_oQ&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Frepo.jenkins-ci.org%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNH4WVSnOUybsblDMAtolGO0Jbm_oQ&#39;;return true;">https://repo.jenkins-ci.org/, Jenkins core historical releases, Remoting library and Windows Service Wrapper which were among ones reported by Jenkins users.

  • Uploads: Jenkins artifact uploads are blocked for the most of Jenkins plugin maintainers and contributors. It affects releases of Jenkins plugins, Jenkins core and modules, developer tools and all libraries hosted on <a href="https://repo.jenkins-ci.org/" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Frepo.jenkins-ci.org%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNH4WVSnOUybsblDMAtolGO0Jbm_oQ&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Frepo.jenkins-ci.org%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNH4WVSnOUybsblDMAtolGO0Jbm_oQ&#39;;return true;">https://repo.jenkins-ci.org/. Incremental and Snapshot deployments are not affected.


Quick summary: 

  • Jun 02 - There was a Kubernetes Cluster outage on June 02. During this outage we had to rebuild the cluster from scratch to get some services working again.

  • Jun 02 - After the recovery we lost three months of LDAP changes. It has happened due to the broken backup of the LDAP database.

  • Jun 02 - We identified a number of potential security risks which may be caused by the LDAP outage. Account overtake and malicious upload was one of the identified risks. FTR this issue is tracked as SECURITY-1895 as a follow-up to these discussions. Only the Security team members have access to it, so I am not sharing a link here.

  • Jun 09 - After the security risk was independently reported in public by a plugin maintainer in the <a href="https://groups.google.com/g/jenkinsci-dev/c/juHejx8zfdg" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://groups.google.com/g/jenkinsci-dev/c/juHejx8zfdg&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/g/jenkinsci-dev/c/juHejx8zfdg&#39;;return true;">dev list thread, we decided to block uploads of release artifacts to the Jenkins Artifactory instance.

  • Jun 09, 8:50AM UTC - All uploads of release artifacts were blocked (plugins, Jenkins core and modules, developer tools, etc.). Downloads of some binaries were also blocked as an unexpected collateral damage. Jenkins core historical releases, Remoting library and Windows Service Wrapper are among the affected binaries

  • Jun 09, 10AM UTC - We finished reviews of all artifact releases to <a href="https://repo.jenkins-ci.org/" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Frepo.jenkins-ci.org%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNH4WVSnOUybsblDMAtolGO0Jbm_oQ&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Frepo.jenkins-ci.org%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNH4WVSnOUybsblDMAtolGO0Jbm_oQ&#39;;return true;">https://repo.jenkins-ci.org/, which happened between the infra outage on June 02 and the blockage of the releases. There are no maliciously uploaded artifacts. Note that the common plugin release flow requires access to GitHub in order to push the release commits, so a malicious attacker would need to overtake both Jenkins and GitHub accounts of a single user to submit a legitimately-looking release.

  • Jun 09, ~1PM UTC - Artifact downloads are restored, <a href="https://github.com/jenkins-infra/repository-permissions-updater/pull/1569" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Fjenkins-infra%2Frepository-permissions-updater%2Fpull%2F1569\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHnXqHe8DbBsyaWsN32PYLgiFuxww&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Fjenkins-infra%2Frepository-permissions-updater%2Fpull%2F1569\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHnXqHe8DbBsyaWsN32PYLgiFuxww&#39;;return true;">alternate patch in the Repository Permission Updater was applied to prevent uploads. Artifact uploads are still blocking

  • Jun 09, 2PM UTC, based on <a href="http://repo.jenkins-ci.org" rel="nofollow" target="_blank" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Frepo.jenkins-ci.org\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNH-lqjQdpr_N3OJq6LbHFX-8CE0vg&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Frepo.jenkins-ci.org\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNH-lqjQdpr_N3OJq6LbHFX-8CE0vg&#39;;return true;">repo.jenkins-ci.org and <a href="http://issues.jenkins-ci.org" rel="nofollow" target="_blank" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fissues.jenkins-ci.org\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEt2OBDOWbsPhB3LOncdqtsX9Mk0Q&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fissues.jenkins-ci.org\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEt2OBDOWbsPhB3LOncdqtsX9Mk0Q&#39;;return true;">issues.jenkins-ci.org data, we restored maintainers accounts.


Our next steps would be to communicate the issue to all maintainers and contributors who might have been affected by the LDAP history loss. We will likely need to perform additional user verification steps for plugin maintainers to ensure that there are no contributors affected by the issues. Today at 3:30PM UTC we will also have a Jenkins Infrastructure team meeting where this issue will be discussed in more detail. This is a public meeting, and everyone is welcome to join. <a href="https://calendar.google.com/event?action=TEMPLATE&amp;tmeid=dTJsaWoxN2xjZHFkajRsbmJlcWFiaXI5b2JfMjAyMDA2MDlUMTUzMDAwWiA0c3MxMmYwbXFyM3RicDF0MmZlMzY5c2xmNEBn&amp;tmsrc=4ss12f0mqr3tbp1t2fe369slf4%40group.calendar.google.com" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://calendar.google.com/event?action\x3dTEMPLATE\x26tmeid\x3ddTJsaWoxN2xjZHFkajRsbmJlcWFiaXI5b2JfMjAyMDA2MDlUMTUzMDAwWiA0c3MxMmYwbXFyM3RicDF0MmZlMzY5c2xmNEBn\x26tmsrc\x3d4ss12f0mqr3tbp1t2fe369slf4%40group.calendar.google.com&#39;;return true;" onclick="this.href=&#39;https://calendar.google.com/event?action\x3dTEMPLATE\x26tmeid\x3ddTJsaWoxN2xjZHFkajRsbmJlcWFiaXI5b2JfMjAyMDA2MDlUMTUzMDAwWiA0c3MxMmYwbXFyM3RicDF0MmZlMzY5c2xmNEBn\x26tmsrc\x3d4ss12f0mqr3tbp1t2fe369slf4%40group.calendar.google.com&#39;;return true;">Calendar link


Thanks to Olivier Vernin, Daniel Beck and other Jenkins Infra and Security team members who contributed to this investigation.


Best regards,

Oleg Nenashev


--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/edc41cb8-6fa0-4c44-93e3-04a0db0eb0d7o%40googlegroups.com.
Reply | Threaded
Open this post in threaded view
|

Re: ANN: Jenkins release artifacts uploads blockage on June 09, and a temporary downloads issue

Liam Newman
I have two releases that I consider high-priority: github-branch-source and github-api .

Users have been able to rollback to the previous release to unblock themselves, but people who cannot rollback (new installations) remain blocked.  




On Friday, June 12, 2020 at 10:05:33 AM UTC-7, Oleg Nenashev wrote:
Dear all,

June 12 update: 
  • We continue to work on the accounts migration and will share the next update on Monday
  • Jenkins releases are still blocked. If there are any emergency releases you need to perform, please reply in this thread.
Best regards,
Oleg Nenashev

On Friday, June 12, 2020 at 6:00:32 PM UTC+2, Oleg Nenashev wrote:
Hi Dave,

This is an email from the Step 2. We’ll reset every user password from the LDAP database. This one includes a temporary password, and we expect users to change it after they login into the system.

For those who wonder: Yes, the temporary password is sent in plain text as mentioned above. This is how our current password reset system is designed. As other projects, we have a decent amount of technical debt in our infrastructure which we gradually resolve. I have already added changing the account password reset flow to the outage retrospective list, an we will be reviewing what to do there after the outage is fully resolved. Apart from fixing it, migrating to a 3rd-party identity service is on the table for me (Linux Foundation or GitHub).  If anyone is interested to participate and to improve the project, the <a href="https://www.jenkins.io/projects/infrastructure/" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fwww.jenkins.io%2Fprojects%2Finfrastructure%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHrHsrPXFz_nsKSMm20dEjtEM6oWQ&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fwww.jenkins.io%2Fprojects%2Finfrastructure%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHrHsrPXFz_nsKSMm20dEjtEM6oWQ&#39;;return true;">Jenkins infrastructure team is always looking for more contributors!

If anyone has concerns about such method and wants to use alternate channels for encrypted password transfer, please send us a message through the Jenkins Infrastructure mailing list from your email registered in Jenkins. In this email please provide your public GPG key so that we can reset a password again in a secure way.

Best regards,
Oleg

On Friday, June 12, 2020 at 5:07:27 PM UTC+2, Dave Pedu wrote:
Hello,

I have received an email linking to this thread. However, it contains a plaintext password for my account, despite this:

There will be no temporary password in these emails, but there will be information pointing to this thread.

Is this email legitimate or am I being phished? Screenshot attached.

Thanks,
Dave

On Thursday, June 11, 2020 at 3:07:01 AM UTC-7, Olblak wrote:

Dear all,


We are ready to proceed with restoration of the Jenkins account database. Today we are going to restore user LDAP accounts that were created since the First of February 2020 based on the data from Jenkins Jira and the repository Permission Manager metadata data. We will also reset passwords for all users registered in the database.


Step 1. All users who lost their account will receive an email saying that their accounts were re-created. There will be no temporary password in these emails, but there will be information pointing to this thread.


Step 2. We’ll reset every user password from the LDAP database, it is more than 100 000 users. Once done, you’ll receive an email telling you that your password was reset with a reason containing a link to this mail thread.


Step 3. We will delete accounts of users who requested such deletion between February and June 2020. These users were restored from the backup, so we have to delete them again.The list of users is based on Jira tickets and private messages to the Jenkins Infra officer. If for some reason you notice that your account still exists, feel free to raise a ticket in <a href="https://issues.jenkins-ci.org/" style="text-decoration:none" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fissues.jenkins-ci.org%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEr7Zs9c804d7zhbfP2jk75fuaVRA&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fissues.jenkins-ci.org%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEr7Zs9c804d7zhbfP2jk75fuaVRA&#39;;return true;">Jenkins Jira (project=INFRA, component=account).


Please do not hesitate to contact us using the #jenkins-infra channel on Freenode IRC or the Jenkins Infrastructure mailing list if you have any questions or suggestions. If you see a security issue related to the accounts, please follow the <a href="https://www.jenkins.io/security/#reporting-vulnerabilities" style="text-decoration:none" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fwww.jenkins.io%2Fsecurity%2F%23reporting-vulnerabilities\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFNS8fVviacDMvDRLXWORjHGybVDQ&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fwww.jenkins.io%2Fsecurity%2F%23reporting-vulnerabilities\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFNS8fVviacDMvDRLXWORjHGybVDQ&#39;;return true;">vulnerability reporting guidelines.


Best regards,

Olivier Vernin && Jenkins Infrastructure Team



On Tuesday, 9 June 2020 17:00:25 UTC+2, Oleg Nenashev wrote:

Dear all,


As you may have noticed, the release artifact uploads are currently blocked in the Jenkins Artifactory instances (<a href="https://repo.jenkins-ci.org/" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Frepo.jenkins-ci.org%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNH4WVSnOUybsblDMAtolGO0Jbm_oQ&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Frepo.jenkins-ci.org%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNH4WVSnOUybsblDMAtolGO0Jbm_oQ&#39;;return true;">https://repo.jenkins-ci.org/). We are doing a security investigation due to a partial user database loss on June 02. Today we blocked releases to the Jenkins artifactory, and there also was a temporary outage of the Artifactory downloads which was a collateral damage of the temporary permissions. You can find more details about it in this<a href="https://groups.google.com/forum/#!topic/jenkins-infra/zRqdiyarLDE" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://groups.google.com/forum/#!topic/jenkins-infra/zRqdiyarLDE&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/forum/#!topic/jenkins-infra/zRqdiyarLDE&#39;;return true;"> Jenkins Infra Thread and in this<a href="https://groups.google.com/d/msg/jenkinsci-dev/juHejx8zfdg/xpySiv1_CQAJ" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://groups.google.com/d/msg/jenkinsci-dev/juHejx8zfdg/xpySiv1_CQAJ&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/msg/jenkinsci-dev/juHejx8zfdg/xpySiv1_CQAJ&#39;;return true;"> Dev List thread.


Current status:

  • Downloads are restored for all artifacts on <a href="https://repo.jenkins-ci.org/" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Frepo.jenkins-ci.org%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNH4WVSnOUybsblDMAtolGO0Jbm_oQ&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Frepo.jenkins-ci.org%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNH4WVSnOUybsblDMAtolGO0Jbm_oQ&#39;;return true;">https://repo.jenkins-ci.org/, Jenkins core historical releases, Remoting library and Windows Service Wrapper which were among ones reported by Jenkins users.

  • Uploads: Jenkins artifact uploads are blocked for the most of Jenkins plugin maintainers and contributors. It affects releases of Jenkins plugins, Jenkins core and modules, developer tools and all libraries hosted on <a href="https://repo.jenkins-ci.org/" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Frepo.jenkins-ci.org%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNH4WVSnOUybsblDMAtolGO0Jbm_oQ&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Frepo.jenkins-ci.org%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNH4WVSnOUybsblDMAtolGO0Jbm_oQ&#39;;return true;">https://repo.jenkins-ci.org/. Incremental and Snapshot deployments are not affected.


Quick summary: 

  • Jun 02 - There was a Kubernetes Cluster outage on June 02. During this outage we had to rebuild the cluster from scratch to get some services working again.

  • Jun 02 - After the recovery we lost three months of LDAP changes. It has happened due to the broken backup of the LDAP database.

  • Jun 02 - We identified a number of potential security risks which may be caused by the LDAP outage. Account overtake and malicious upload was one of the identified risks. FTR this issue is tracked as SECURITY-1895 as a follow-up to these discussions. Only the Security team members have access to it, so I am not sharing a link here.

  • Jun 09 - After the security risk was independently reported in public by a plugin maintainer in the <a href="https://groups.google.com/g/jenkinsci-dev/c/juHejx8zfdg" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://groups.google.com/g/jenkinsci-dev/c/juHejx8zfdg&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/g/jenkinsci-dev/c/juHejx8zfdg&#39;;return true;">dev list thread, we decided to block uploads of release artifacts to the Jenkins Artifactory instance.

  • Jun 09, 8:50AM UTC - All uploads of release artifacts were blocked (plugins, Jenkins core and modules, developer tools, etc.). Downloads of some binaries were also blocked as an unexpected collateral damage. Jenkins core historical releases, Remoting library and Windows Service Wrapper are among the affected binaries

  • Jun 09, 10AM UTC - We finished reviews of all artifact releases to <a href="https://repo.jenkins-ci.org/" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Frepo.jenkins-ci.org%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNH4WVSnOUybsblDMAtolGO0Jbm_oQ&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Frepo.jenkins-ci.org%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNH4WVSnOUybsblDMAtolGO0Jbm_oQ&#39;;return true;">https://repo.jenkins-ci.org/, which happened between the infra outage on June 02 and the blockage of the releases. There are no maliciously uploaded artifacts. Note that the common plugin release flow requires access to GitHub in order to push the release commits, so a malicious attacker would need to overtake both Jenkins and GitHub accounts of a single user to submit a legitimately-looking release.

  • Jun 09, ~1PM UTC - Artifact downloads are restored, <a href="https://github.com/jenkins-infra/repository-permissions-updater/pull/1569" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Fjenkins-infra%2Frepository-permissions-updater%2Fpull%2F1569\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHnXqHe8DbBsyaWsN32PYLgiFuxww&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Fjenkins-infra%2Frepository-permissions-updater%2Fpull%2F1569\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHnXqHe8DbBsyaWsN32PYLgiFuxww&#39;;return true;">alternate patch in the Repository Permission Updater was applied to prevent uploads. Artifact uploads are still blocking

  • Jun 09, 2PM UTC, based on <a href="http://repo.jenkins-ci.org" rel="nofollow" target="_blank" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Frepo.jenkins-ci.org\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNH-lqjQdpr_N3OJq6LbHFX-8CE0vg&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Frepo.jenkins-ci.org\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNH-lqjQdpr_N3OJq6LbHFX-8CE0vg&#39;;return true;">repo.jenkins-ci.org and <a href="http://issues.jenkins-ci.org" rel="nofollow" target="_blank" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fissues.jenkins-ci.org\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEt2OBDOWbsPhB3LOncdqtsX9Mk0Q&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fissues.jenkins-ci.org\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEt2OBDOWbsPhB3LOncdqtsX9Mk0Q&#39;;return true;">issues.jenkins-ci.org data, we restored maintainers accounts.


Our next steps would be to communicate the issue to all maintainers and contributors who might have been affected by the LDAP history loss. We will likely need to perform additional user verification steps for plugin maintainers to ensure that there are no contributors affected by the issues. Today at 3:30PM UTC we will also have a Jenkins Infrastructure team meeting where this issue will be discussed in more detail. This is a public meeting, and everyone is welcome to join. <a href="https://calendar.google.com/event?action=TEMPLATE&amp;tmeid=dTJsaWoxN2xjZHFkajRsbmJlcWFiaXI5b2JfMjAyMDA2MDlUMTUzMDAwWiA0c3MxMmYwbXFyM3RicDF0MmZlMzY5c2xmNEBn&amp;tmsrc=4ss12f0mqr3tbp1t2fe369slf4%40group.calendar.google.com" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://calendar.google.com/event?action\x3dTEMPLATE\x26tmeid\x3ddTJsaWoxN2xjZHFkajRsbmJlcWFiaXI5b2JfMjAyMDA2MDlUMTUzMDAwWiA0c3MxMmYwbXFyM3RicDF0MmZlMzY5c2xmNEBn\x26tmsrc\x3d4ss12f0mqr3tbp1t2fe369slf4%40group.calendar.google.com&#39;;return true;" onclick="this.href=&#39;https://calendar.google.com/event?action\x3dTEMPLATE\x26tmeid\x3ddTJsaWoxN2xjZHFkajRsbmJlcWFiaXI5b2JfMjAyMDA2MDlUMTUzMDAwWiA0c3MxMmYwbXFyM3RicDF0MmZlMzY5c2xmNEBn\x26tmsrc\x3d4ss12f0mqr3tbp1t2fe369slf4%40group.calendar.google.com&#39;;return true;">Calendar link


Thanks to Olivier Vernin, Daniel Beck and other Jenkins Infra and Security team members who contributed to this investigation.


Best regards,

Oleg Nenashev


--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/ab94c691-ab74-4eaa-a166-c2e16f8e2b28o%40googlegroups.com.
Reply | Threaded
Open this post in threaded view
|

Re: ANN: Jenkins release artifacts uploads blockage on June 09, and a temporary downloads issue

Oleg Nenashev
Ack. I will make sure we have a workaround applied on Monday if the user update is not finished

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/b44b553a-65e3-4630-a97a-dec3c1884e1bo%40googlegroups.com.
Reply | Threaded
Open this post in threaded view
|

Re: ANN: Jenkins release artifacts uploads blockage on June 09, and a temporary downloads issue

Oleg Nenashev
Dear all,

We have reset all plugin maintainer accounts, and we have reenabled plugin uploads in the Repository Permission Updater. By now all upload permissions should be restored, except a few new user registrations in the Jenkins Artifactory instance over the past week. All Artifactory API tokens were revoked. If you experience any issues with plugin and component releases, please let us know in this thread.

For a list of disabled user accounts, please see this pull request: https://github.com/jenkins-infra/repository-permissions-updater/pull/1574. The disabled users need to login to https://repo.jenkins-ci.org/ again, and then to submit a pull request restoring their permissions. If you use Artifactory API tokens for uploads, you need to login to Artifactory and to reconfigure them.

We apologize for any inconvenience the restrictions caused, and we will have a retrospective to discuss what we could do better to prevent it in the future. If you want to share any feedback, please send it to this thread. If you want to share a private feedback, please send it to my email.

Best regards,
Oleg Nenashev

On Sunday, June 14, 2020 at 2:49:34 PM UTC+2 Oleg Nenashev wrote:
Ack. I will make sure we have a workaround applied on Monday if the user update is not finished

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/58a6e193-9231-408c-a783-07bddfc23029n%40googlegroups.com.
Reply | Threaded
Open this post in threaded view
|

Re: ANN: Jenkins release artifacts uploads blockage on June 09, and a temporary downloads issue

Steve Springett
"Technical debt" is not an excuse to reset plugin maintainers accounts and include a clear-text email containing their username AND password. That's insane. As a security professional I will not stand for that. I will no longer be maintaining Jenkins plugins and will attempt to find new maintainers for the ones I do. No guarantees.

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/4547a00e-e223-4075-a2a1-9162b4634b5bo%40googlegroups.com.
Reply | Threaded
Open this post in threaded view
|

Re: ANN: Jenkins release artifacts uploads blockage on June 09, and a temporary downloads issue

Oleg Nenashev
Hi Steve,

Duly noted. Note that we offered an alternate way for maintainers to get their password delivered if they are not fine with the current delivery method. In my message from Jub 12: If anyone has concerns about such a method and wants to use alternate channels for encrypted password transfer, please send us a message through the Jenkins Infrastructure mailing list from your email registered in Jenkins. In this email please provide your public GPG key so that we can reset a password again in a secure way. You did not contact us, and hence you got your password reset with the standard process. If you want to get your password reset in a secure way, please feel free to use this process.

Again, we operate with resources and tools we have. The Jenkins project and its infrastructure are driven by volunteers, and we have limited capacity when it comes to fixing urgent things due to uncoordinated disclosures. You may call it insane, but it was the solution we delivered with given circumstances. Contributors have families and other commitments, and please know that the situation has taken a high toll on them. Everybody is welcome to contribute and to contribute to the infrastructure. I am cordially grateful to those several contributors who stepped up and helped to get the issue fixed, or offered to help, or sent kind words over different channels. This is an example to follow.

Everyone is welcome to join the team and to work together on a better solution for user management so that we can prevent a similar situation in the future.

Best regards,
Oleg


On Mon, Jun 15, 2020 at 5:02 PM Steve Springett <[hidden email]> wrote:
"Technical debt" is not an excuse to reset plugin maintainers accounts and include a clear-text email containing their username AND password. That's insane. As a security professional I will not stand for that. I will no longer be maintaining Jenkins plugins and will attempt to find new maintainers for the ones I do. No guarantees.

--
You received this message because you are subscribed to a topic in the Google Groups "Jenkins Developers" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jenkinsci-dev/3UvrCTflXGk/unsubscribe.
To unsubscribe from this group and all its topics, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/4547a00e-e223-4075-a2a1-9162b4634b5bo%40googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CAPfivLCz%2BPMk9E3C6nq7%3D4fn5DbK6nBy%3DJUWG1w2bwNvoqmGkA%40mail.gmail.com.
Reply | Threaded
Open this post in threaded view
|

Re: ANN: Jenkins release artifacts uploads blockage on June 09, and a temporary downloads issue

Steve Springett
Security best practices should not be opt-in. I receive the manifest (daily) emails and did not see this topic. Many others likely did not either.

Jenkins is viewed by many as Critical Cyber Infrastructure and plays an important role in the global software supply chain. That supply chain was just weakened today, on purpose.

I understand the volunteer perspective. I lead multiple OWASP projects and spend a considerable amount of time doing open source projects. I get it. Just know that the software supply chain affects everyone including CloudBees Enterprise customers and every downstream consumer of projects built using Jenkins.


On Monday, June 15, 2020 at 10:27:01 AM UTC-5 Oleg Nenashev wrote:
Hi Steve,

Duly noted. Note that we offered an alternate way for maintainers to get their password delivered if they are not fine with the current delivery method. In my message from Jub 12: If anyone has concerns about such a method and wants to use alternate channels for encrypted password transfer, please send us a message through the Jenkins Infrastructure mailing list from your email registered in Jenkins. In this email please provide your public GPG key so that we can reset a password again in a secure way. You did not contact us, and hence you got your password reset with the standard process. If you want to get your password reset in a secure way, please feel free to use this process.

Again, we operate with resources and tools we have. The Jenkins project and its infrastructure are driven by volunteers, and we have limited capacity when it comes to fixing urgent things due to uncoordinated disclosures. You may call it insane, but it was the solution we delivered with given circumstances. Contributors have families and other commitments, and please know that the situation has taken a high toll on them. Everybody is welcome to contribute and to contribute to the infrastructure. I am cordially grateful to those several contributors who stepped up and helped to get the issue fixed, or offered to help, or sent kind words over different channels. This is an example to follow.

Everyone is welcome to join the team and to work together on a better solution for user management so that we can prevent a similar situation in the future.

Best regards,
Oleg


On Mon, Jun 15, 2020 at 5:02 PM Steve Springett <[hidden email]> wrote:
"Technical debt" is not an excuse to reset plugin maintainers accounts and include a clear-text email containing their username AND password. That's insane. As a security professional I will not stand for that. I will no longer be maintaining Jenkins plugins and will attempt to find new maintainers for the ones I do. No guarantees.

--
You received this message because you are subscribed to a topic in the Google Groups "Jenkins Developers" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jenkinsci-dev/3UvrCTflXGk/unsubscribe.
To unsubscribe from this group and all its topics, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/4547a00e-e223-4075-a2a1-9162b4634b5bo%40googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/da467d52-3ad4-41f5-8d8b-2f86a68487a1n%40googlegroups.com.
Reply | Threaded
Open this post in threaded view
|

Re: ANN: Jenkins release artifacts uploads blockage on June 09, and a temporary downloads issue

Matt Sicker
No complaints about the lack of a DNSSEC record or other ways to avoid
an initial MitM attack when connecting to jenkins.io for the first
time. Or numerous other theoretical points of failure. It might be
better to offer constructive advice rather than declaring everything
broken.

On Mon, Jun 15, 2020 at 1:20 PM Steve Springett
<[hidden email]> wrote:

>
> Security best practices should not be opt-in. I receive the manifest (daily) emails and did not see this topic. Many others likely did not either.
>
> Jenkins is viewed by many as Critical Cyber Infrastructure and plays an important role in the global software supply chain. That supply chain was just weakened today, on purpose.
>
> I understand the volunteer perspective. I lead multiple OWASP projects and spend a considerable amount of time doing open source projects. I get it. Just know that the software supply chain affects everyone including CloudBees Enterprise customers and every downstream consumer of projects built using Jenkins.
>
>
> On Monday, June 15, 2020 at 10:27:01 AM UTC-5 Oleg Nenashev wrote:
>>
>> Hi Steve,
>>
>> Duly noted. Note that we offered an alternate way for maintainers to get their password delivered if they are not fine with the current delivery method. In my message from Jub 12: If anyone has concerns about such a method and wants to use alternate channels for encrypted password transfer, please send us a message through the Jenkins Infrastructure mailing list from your email registered in Jenkins. In this email please provide your public GPG key so that we can reset a password again in a secure way. You did not contact us, and hence you got your password reset with the standard process. If you want to get your password reset in a secure way, please feel free to use this process.
>>
>> Again, we operate with resources and tools we have. The Jenkins project and its infrastructure are driven by volunteers, and we have limited capacity when it comes to fixing urgent things due to uncoordinated disclosures. You may call it insane, but it was the solution we delivered with given circumstances. Contributors have families and other commitments, and please know that the situation has taken a high toll on them. Everybody is welcome to contribute and to contribute to the infrastructure. I am cordially grateful to those several contributors who stepped up and helped to get the issue fixed, or offered to help, or sent kind words over different channels. This is an example to follow.
>>
>> Everyone is welcome to join the team and to work together on a better solution for user management so that we can prevent a similar situation in the future.
>>
>> Best regards,
>> Oleg
>>
>>
>> On Mon, Jun 15, 2020 at 5:02 PM Steve Springett <[hidden email]> wrote:
>>>
>>> "Technical debt" is not an excuse to reset plugin maintainers accounts and include a clear-text email containing their username AND password. That's insane. As a security professional I will not stand for that. I will no longer be maintaining Jenkins plugins and will attempt to find new maintainers for the ones I do. No guarantees.
>>>
>>> --
>>> You received this message because you are subscribed to a topic in the Google Groups "Jenkins Developers" group.
>>> To unsubscribe from this topic, visit https://groups.google.com/d/topic/jenkinsci-dev/3UvrCTflXGk/unsubscribe.
>>> To unsubscribe from this group and all its topics, send an email to [hidden email].
>>> To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/4547a00e-e223-4075-a2a1-9162b4634b5bo%40googlegroups.com.
>
> --
> You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
> To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/da467d52-3ad4-41f5-8d8b-2f86a68487a1n%40googlegroups.com.



--
Matt Sicker
Senior Software Engineer, CloudBees

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CAEot4owMsAT7Y%2BRHyo87A_QEPeJo5Ufbf9jtu3QGJxzhHVRiFw%40mail.gmail.com.
Reply | Threaded
Open this post in threaded view
|

Re: ANN: Jenkins release artifacts uploads blockage on June 09, and a temporary downloads issue

Olivier Lamy-2
In reply to this post by Oleg Nenashev
Awesome.
Thanks a lot for the hard work!

On Mon, 15 Jun 2020 at 23:10, Oleg Nenashev <[hidden email]> wrote:
Dear all,

We have reset all plugin maintainer accounts, and we have reenabled plugin uploads in the Repository Permission Updater. By now all upload permissions should be restored, except a few new user registrations in the Jenkins Artifactory instance over the past week. All Artifactory API tokens were revoked. If you experience any issues with plugin and component releases, please let us know in this thread.

For a list of disabled user accounts, please see this pull request: https://github.com/jenkins-infra/repository-permissions-updater/pull/1574. The disabled users need to login to https://repo.jenkins-ci.org/ again, and then to submit a pull request restoring their permissions. If you use Artifactory API tokens for uploads, you need to login to Artifactory and to reconfigure them.

We apologize for any inconvenience the restrictions caused, and we will have a retrospective to discuss what we could do better to prevent it in the future. If you want to share any feedback, please send it to this thread. If you want to share a private feedback, please send it to my email.

Best regards,
Oleg Nenashev

On Sunday, June 14, 2020 at 2:49:34 PM UTC+2 Oleg Nenashev wrote:
Ack. I will make sure we have a workaround applied on Monday if the user update is not finished

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/58a6e193-9231-408c-a783-07bddfc23029n%40googlegroups.com.


--

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CAPoyBqR25q-iE%3DpOuOBAYP83aJ1u6YUOTzZrjkXQXMxVJMKnoA%40mail.gmail.com.
Reply | Threaded
Open this post in threaded view
|

Re: ANN: Jenkins release artifacts uploads blockage on June 09, and a temporary downloads issue

kjeschkies
In reply to this post by Oleg Nenashev
Hi,

thanks for you hard work. I reset my password successfully but cannot upload a release for the Mesos plugin. Are releases still blocked?

Best.
Karsten.

On Tuesday, June 9, 2020 at 5:00:25 PM UTC+2, Oleg Nenashev wrote:

Dear all,


As you may have noticed, the release artifact uploads are currently blocked in the Jenkins Artifactory instances (<a href="https://repo.jenkins-ci.org/" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Frepo.jenkins-ci.org%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNH4WVSnOUybsblDMAtolGO0Jbm_oQ&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Frepo.jenkins-ci.org%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNH4WVSnOUybsblDMAtolGO0Jbm_oQ&#39;;return true;">https://repo.jenkins-ci.org/). We are doing a security investigation due to a partial user database loss on June 02. Today we blocked releases to the Jenkins artifactory, and there also was a temporary outage of the Artifactory downloads which was a collateral damage of the temporary permissions. You can find more details about it in this<a href="https://groups.google.com/forum/#!topic/jenkins-infra/zRqdiyarLDE" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://groups.google.com/forum/#!topic/jenkins-infra/zRqdiyarLDE&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/forum/#!topic/jenkins-infra/zRqdiyarLDE&#39;;return true;"> Jenkins Infra Thread and in this<a href="https://groups.google.com/d/msg/jenkinsci-dev/juHejx8zfdg/xpySiv1_CQAJ" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://groups.google.com/d/msg/jenkinsci-dev/juHejx8zfdg/xpySiv1_CQAJ&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/msg/jenkinsci-dev/juHejx8zfdg/xpySiv1_CQAJ&#39;;return true;"> Dev List thread.


Current status:

  • Downloads are restored for all artifacts on <a href="https://repo.jenkins-ci.org/" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Frepo.jenkins-ci.org%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNH4WVSnOUybsblDMAtolGO0Jbm_oQ&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Frepo.jenkins-ci.org%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNH4WVSnOUybsblDMAtolGO0Jbm_oQ&#39;;return true;">https://repo.jenkins-ci.org/, Jenkins core historical releases, Remoting library and Windows Service Wrapper which were among ones reported by Jenkins users.

  • Uploads: Jenkins artifact uploads are blocked for the most of Jenkins plugin maintainers and contributors. It affects releases of Jenkins plugins, Jenkins core and modules, developer tools and all libraries hosted on <a href="https://repo.jenkins-ci.org/" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Frepo.jenkins-ci.org%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNH4WVSnOUybsblDMAtolGO0Jbm_oQ&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Frepo.jenkins-ci.org%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNH4WVSnOUybsblDMAtolGO0Jbm_oQ&#39;;return true;">https://repo.jenkins-ci.org/. Incremental and Snapshot deployments are not affected.


Quick summary: 

  • Jun 02 - There was a Kubernetes Cluster outage on June 02. During this outage we had to rebuild the cluster from scratch to get some services working again.

  • Jun 02 - After the recovery we lost three months of LDAP changes. It has happened due to the broken backup of the LDAP database.

  • Jun 02 - We identified a number of potential security risks which may be caused by the LDAP outage. Account overtake and malicious upload was one of the identified risks. FTR this issue is tracked as SECURITY-1895 as a follow-up to these discussions. Only the Security team members have access to it, so I am not sharing a link here.

  • Jun 09 - After the security risk was independently reported in public by a plugin maintainer in the <a href="https://groups.google.com/g/jenkinsci-dev/c/juHejx8zfdg" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://groups.google.com/g/jenkinsci-dev/c/juHejx8zfdg&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/g/jenkinsci-dev/c/juHejx8zfdg&#39;;return true;">dev list thread, we decided to block uploads of release artifacts to the Jenkins Artifactory instance.

  • Jun 09, 8:50AM UTC - All uploads of release artifacts were blocked (plugins, Jenkins core and modules, developer tools, etc.). Downloads of some binaries were also blocked as an unexpected collateral damage. Jenkins core historical releases, Remoting library and Windows Service Wrapper are among the affected binaries

  • Jun 09, 10AM UTC - We finished reviews of all artifact releases to <a href="https://repo.jenkins-ci.org/" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Frepo.jenkins-ci.org%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNH4WVSnOUybsblDMAtolGO0Jbm_oQ&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Frepo.jenkins-ci.org%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNH4WVSnOUybsblDMAtolGO0Jbm_oQ&#39;;return true;">https://repo.jenkins-ci.org/, which happened between the infra outage on June 02 and the blockage of the releases. There are no maliciously uploaded artifacts. Note that the common plugin release flow requires access to GitHub in order to push the release commits, so a malicious attacker would need to overtake both Jenkins and GitHub accounts of a single user to submit a legitimately-looking release.

  • Jun 09, ~1PM UTC - Artifact downloads are restored, <a href="https://github.com/jenkins-infra/repository-permissions-updater/pull/1569" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Fjenkins-infra%2Frepository-permissions-updater%2Fpull%2F1569\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHnXqHe8DbBsyaWsN32PYLgiFuxww&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Fjenkins-infra%2Frepository-permissions-updater%2Fpull%2F1569\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHnXqHe8DbBsyaWsN32PYLgiFuxww&#39;;return true;">alternate patch in the Repository Permission Updater was applied to prevent uploads. Artifact uploads are still blocking

  • Jun 09, 2PM UTC, based on <a href="http://repo.jenkins-ci.org" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Frepo.jenkins-ci.org\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNH-lqjQdpr_N3OJq6LbHFX-8CE0vg&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Frepo.jenkins-ci.org\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNH-lqjQdpr_N3OJq6LbHFX-8CE0vg&#39;;return true;">repo.jenkins-ci.org and <a href="http://issues.jenkins-ci.org" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fissues.jenkins-ci.org\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEt2OBDOWbsPhB3LOncdqtsX9Mk0Q&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fissues.jenkins-ci.org\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEt2OBDOWbsPhB3LOncdqtsX9Mk0Q&#39;;return true;">issues.jenkins-ci.org data, we restored maintainers accounts.


Our next steps would be to communicate the issue to all maintainers and contributors who might have been affected by the LDAP history loss. We will likely need to perform additional user verification steps for plugin maintainers to ensure that there are no contributors affected by the issues. Today at 3:30PM UTC we will also have a Jenkins Infrastructure team meeting where this issue will be discussed in more detail. This is a public meeting, and everyone is welcome to join. <a href="https://calendar.google.com/event?action=TEMPLATE&amp;tmeid=dTJsaWoxN2xjZHFkajRsbmJlcWFiaXI5b2JfMjAyMDA2MDlUMTUzMDAwWiA0c3MxMmYwbXFyM3RicDF0MmZlMzY5c2xmNEBn&amp;tmsrc=4ss12f0mqr3tbp1t2fe369slf4%40group.calendar.google.com" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://calendar.google.com/event?action\x3dTEMPLATE\x26tmeid\x3ddTJsaWoxN2xjZHFkajRsbmJlcWFiaXI5b2JfMjAyMDA2MDlUMTUzMDAwWiA0c3MxMmYwbXFyM3RicDF0MmZlMzY5c2xmNEBn\x26tmsrc\x3d4ss12f0mqr3tbp1t2fe369slf4%40group.calendar.google.com&#39;;return true;" onclick="this.href=&#39;https://calendar.google.com/event?action\x3dTEMPLATE\x26tmeid\x3ddTJsaWoxN2xjZHFkajRsbmJlcWFiaXI5b2JfMjAyMDA2MDlUMTUzMDAwWiA0c3MxMmYwbXFyM3RicDF0MmZlMzY5c2xmNEBn\x26tmsrc\x3d4ss12f0mqr3tbp1t2fe369slf4%40group.calendar.google.com&#39;;return true;">Calendar link


Thanks to Olivier Vernin, Daniel Beck and other Jenkins Infra and Security team members who contributed to this investigation.


Best regards,

Oleg Nenashev


--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/ea5483fb-6873-41dd-a82c-d5518c7de106o%40googlegroups.com.
Reply | Threaded
Open this post in threaded view
|

Re: ANN: Jenkins release artifacts uploads blockage on June 09, and a temporary downloads issue

Mark Waite-2


On Wed, Jun 17, 2020 at 6:44 AM Karsten Jeschkies <[hidden email]> wrote:
Hi,

thanks for you hard work. I reset my password successfully but cannot upload a release for the Mesos plugin. Are releases still blocked?


Releases are not blocked but a password reset will also reset your password to the artifact repository.  If you're receiving an HTTP 401 when you try to `mvn release perform` you may need to update your password in the ~/.m2/settings.xml.

I had to do that in order to release a new version of a plugin yesterday.  I logged into the Jenkins Artifactory instance and had it generate an encrypted password from my profile page on that server.  I inserted that encrypted password into my ~/.m2/settings.xml file.  I'm not sure if that is the preferred way to do it, but it worked for me.

Mark Waite
 
Best.
Karsten.

On Tuesday, June 9, 2020 at 5:00:25 PM UTC+2, Oleg Nenashev wrote:

Dear all,


As you may have noticed, the release artifact uploads are currently blocked in the Jenkins Artifactory instances (https://repo.jenkins-ci.org/). We are doing a security investigation due to a partial user database loss on June 02. Today we blocked releases to the Jenkins artifactory, and there also was a temporary outage of the Artifactory downloads which was a collateral damage of the temporary permissions. You can find more details about it in this Jenkins Infra Thread and in this Dev List thread.


Current status:

  • Downloads are restored for all artifacts on https://repo.jenkins-ci.org/, Jenkins core historical releases, Remoting library and Windows Service Wrapper which were among ones reported by Jenkins users.

  • Uploads: Jenkins artifact uploads are blocked for the most of Jenkins plugin maintainers and contributors. It affects releases of Jenkins plugins, Jenkins core and modules, developer tools and all libraries hosted on https://repo.jenkins-ci.org/. Incremental and Snapshot deployments are not affected.


Quick summary: 

  • Jun 02 - There was a Kubernetes Cluster outage on June 02. During this outage we had to rebuild the cluster from scratch to get some services working again.

  • Jun 02 - After the recovery we lost three months of LDAP changes. It has happened due to the broken backup of the LDAP database.

  • Jun 02 - We identified a number of potential security risks which may be caused by the LDAP outage. Account overtake and malicious upload was one of the identified risks. FTR this issue is tracked as SECURITY-1895 as a follow-up to these discussions. Only the Security team members have access to it, so I am not sharing a link here.

  • Jun 09 - After the security risk was independently reported in public by a plugin maintainer in the dev list thread, we decided to block uploads of release artifacts to the Jenkins Artifactory instance.

  • Jun 09, 8:50AM UTC - All uploads of release artifacts were blocked (plugins, Jenkins core and modules, developer tools, etc.). Downloads of some binaries were also blocked as an unexpected collateral damage. Jenkins core historical releases, Remoting library and Windows Service Wrapper are among the affected binaries

  • Jun 09, 10AM UTC - We finished reviews of all artifact releases to https://repo.jenkins-ci.org/, which happened between the infra outage on June 02 and the blockage of the releases. There are no maliciously uploaded artifacts. Note that the common plugin release flow requires access to GitHub in order to push the release commits, so a malicious attacker would need to overtake both Jenkins and GitHub accounts of a single user to submit a legitimately-looking release.

  • Jun 09, ~1PM UTC - Artifact downloads are restored, alternate patch in the Repository Permission Updater was applied to prevent uploads. Artifact uploads are still blocking

  • Jun 09, 2PM UTC, based on repo.jenkins-ci.org and issues.jenkins-ci.org data, we restored maintainers accounts.


Our next steps would be to communicate the issue to all maintainers and contributors who might have been affected by the LDAP history loss. We will likely need to perform additional user verification steps for plugin maintainers to ensure that there are no contributors affected by the issues. Today at 3:30PM UTC we will also have a Jenkins Infrastructure team meeting where this issue will be discussed in more detail. This is a public meeting, and everyone is welcome to join. Calendar link


Thanks to Olivier Vernin, Daniel Beck and other Jenkins Infra and Security team members who contributed to this investigation.


Best regards,

Oleg Nenashev


--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/ea5483fb-6873-41dd-a82c-d5518c7de106o%40googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CAO49JtGqXd-FwrxzgVtVhJ0nki1BOwgGawSuE%3Dc4%2B940sh07XQ%40mail.gmail.com.
Reply | Threaded
Open this post in threaded view
|

Re: ANN: Jenkins release artifacts uploads blockage on June 09, and a temporary downloads issue

Tim Jacomb
Apparently I use an artifactory API key to release, so I had to go into my artifactory settings (https://repo.jenkins-ci.org/webapp/#/home)
and generate a new API key

Tim

On Wed, 17 Jun 2020 at 13:53, Mark Waite <[hidden email]> wrote:


On Wed, Jun 17, 2020 at 6:44 AM Karsten Jeschkies <[hidden email]> wrote:
Hi,

thanks for you hard work. I reset my password successfully but cannot upload a release for the Mesos plugin. Are releases still blocked?


Releases are not blocked but a password reset will also reset your password to the artifact repository.  If you're receiving an HTTP 401 when you try to `mvn release perform` you may need to update your password in the ~/.m2/settings.xml.

I had to do that in order to release a new version of a plugin yesterday.  I logged into the Jenkins Artifactory instance and had it generate an encrypted password from my profile page on that server.  I inserted that encrypted password into my ~/.m2/settings.xml file.  I'm not sure if that is the preferred way to do it, but it worked for me.

Mark Waite
 
Best.
Karsten.

On Tuesday, June 9, 2020 at 5:00:25 PM UTC+2, Oleg Nenashev wrote:

Dear all,


As you may have noticed, the release artifact uploads are currently blocked in the Jenkins Artifactory instances (https://repo.jenkins-ci.org/). We are doing a security investigation due to a partial user database loss on June 02. Today we blocked releases to the Jenkins artifactory, and there also was a temporary outage of the Artifactory downloads which was a collateral damage of the temporary permissions. You can find more details about it in this Jenkins Infra Thread and in this Dev List thread.


Current status:

  • Downloads are restored for all artifacts on https://repo.jenkins-ci.org/, Jenkins core historical releases, Remoting library and Windows Service Wrapper which were among ones reported by Jenkins users.

  • Uploads: Jenkins artifact uploads are blocked for the most of Jenkins plugin maintainers and contributors. It affects releases of Jenkins plugins, Jenkins core and modules, developer tools and all libraries hosted on https://repo.jenkins-ci.org/. Incremental and Snapshot deployments are not affected.


Quick summary: 

  • Jun 02 - There was a Kubernetes Cluster outage on June 02. During this outage we had to rebuild the cluster from scratch to get some services working again.

  • Jun 02 - After the recovery we lost three months of LDAP changes. It has happened due to the broken backup of the LDAP database.

  • Jun 02 - We identified a number of potential security risks which may be caused by the LDAP outage. Account overtake and malicious upload was one of the identified risks. FTR this issue is tracked as SECURITY-1895 as a follow-up to these discussions. Only the Security team members have access to it, so I am not sharing a link here.

  • Jun 09 - After the security risk was independently reported in public by a plugin maintainer in the dev list thread, we decided to block uploads of release artifacts to the Jenkins Artifactory instance.

  • Jun 09, 8:50AM UTC - All uploads of release artifacts were blocked (plugins, Jenkins core and modules, developer tools, etc.). Downloads of some binaries were also blocked as an unexpected collateral damage. Jenkins core historical releases, Remoting library and Windows Service Wrapper are among the affected binaries

  • Jun 09, 10AM UTC - We finished reviews of all artifact releases to https://repo.jenkins-ci.org/, which happened between the infra outage on June 02 and the blockage of the releases. There are no maliciously uploaded artifacts. Note that the common plugin release flow requires access to GitHub in order to push the release commits, so a malicious attacker would need to overtake both Jenkins and GitHub accounts of a single user to submit a legitimately-looking release.

  • Jun 09, ~1PM UTC - Artifact downloads are restored, alternate patch in the Repository Permission Updater was applied to prevent uploads. Artifact uploads are still blocking

  • Jun 09, 2PM UTC, based on repo.jenkins-ci.org and issues.jenkins-ci.org data, we restored maintainers accounts.


Our next steps would be to communicate the issue to all maintainers and contributors who might have been affected by the LDAP history loss. We will likely need to perform additional user verification steps for plugin maintainers to ensure that there are no contributors affected by the issues. Today at 3:30PM UTC we will also have a Jenkins Infrastructure team meeting where this issue will be discussed in more detail. This is a public meeting, and everyone is welcome to join. Calendar link


Thanks to Olivier Vernin, Daniel Beck and other Jenkins Infra and Security team members who contributed to this investigation.


Best regards,

Oleg Nenashev


--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/ea5483fb-6873-41dd-a82c-d5518c7de106o%40googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CAO49JtGqXd-FwrxzgVtVhJ0nki1BOwgGawSuE%3Dc4%2B940sh07XQ%40mail.gmail.com.

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CAH-3Bid1z9%2BH2n1g1aNSj56_zkP9nSek5N0WS5ZqgPCF-%2Bi7ug%40mail.gmail.com.
Reply | Threaded
Open this post in threaded view
|

Re: ANN: Jenkins release artifacts uploads blockage on June 09, and a temporary downloads issue

kjeschkies
In reply to this post by Mark Waite-2
Hi,

thanks for the advice. Hm, my ~/.m2/settings.xml had my encrypted password. The docs (https://wiki.jenkins.io/display/JENKINS/Hosting+Plugins#HostingPlugins-Releasingtojenkins-ci.org) don’t mention the API key. How can I configure Maven to use the API key instead?

Many thanks.
Karsten.


On June 17, 2020 at 14:53:22, Mark Waite ([hidden email]) wrote:



On Wed, Jun 17, 2020 at 6:44 AM Karsten Jeschkies <[hidden email]> wrote:
Hi,

thanks for you hard work. I reset my password successfully but cannot upload a release for the Mesos plugin. Are releases still blocked?


Releases are not blocked but a password reset will also reset your password to the artifact repository.  If you're receiving an HTTP 401 when you try to `mvn release perform` you may need to update your password in the ~/.m2/settings.xml.

I had to do that in order to release a new version of a plugin yesterday.  I logged into the Jenkins Artifactory instance and had it generate an encrypted password from my profile page on that server.  I inserted that encrypted password into my ~/.m2/settings.xml file.  I'm not sure if that is the preferred way to do it, but it worked for me.

Mark Waite
 
Best.
Karsten.

On Tuesday, June 9, 2020 at 5:00:25 PM UTC+2, Oleg Nenashev wrote:

Dear all,


As you may have noticed, the release artifact uploads are currently blocked in the Jenkins Artifactory instances (https://repo.jenkins-ci.org/). We are doing a security investigation due to a partial user database loss on June 02. Today we blocked releases to the Jenkins artifactory, and there also was a temporary outage of the Artifactory downloads which was a collateral damage of the temporary permissions. You can find more details about it in this Jenkins Infra Thread and in this Dev List thread.


Current status:

  • Downloads are restored for all artifacts on https://repo.jenkins-ci.org/, Jenkins core historical releases, Remoting library and Windows Service Wrapper which were among ones reported by Jenkins users.

  • Uploads: Jenkins artifact uploads are blocked for the most of Jenkins plugin maintainers and contributors. It affects releases of Jenkins plugins, Jenkins core and modules, developer tools and all libraries hosted on https://repo.jenkins-ci.org/. Incremental and Snapshot deployments are not affected.


Quick summary: 

  • Jun 02 - There was a Kubernetes Cluster outage on June 02. During this outage we had to rebuild the cluster from scratch to get some services working again.

  • Jun 02 - After the recovery we lost three months of LDAP changes. It has happened due to the broken backup of the LDAP database.

  • Jun 02 - We identified a number of potential security risks which may be caused by the LDAP outage. Account overtake and malicious upload was one of the identified risks. FTR this issue is tracked as SECURITY-1895 as a follow-up to these discussions. Only the Security team members have access to it, so I am not sharing a link here.

  • Jun 09 - After the security risk was independently reported in public by a plugin maintainer in the dev list thread, we decided to block uploads of release artifacts to the Jenkins Artifactory instance.

  • Jun 09, 8:50AM UTC - All uploads of release artifacts were blocked (plugins, Jenkins core and modules, developer tools, etc.). Downloads of some binaries were also blocked as an unexpected collateral damage. Jenkins core historical releases, Remoting library and Windows Service Wrapper are among the affected binaries

  • Jun 09, 10AM UTC - We finished reviews of all artifact releases to https://repo.jenkins-ci.org/, which happened between the infra outage on June 02 and the blockage of the releases. There are no maliciously uploaded artifacts. Note that the common plugin release flow requires access to GitHub in order to push the release commits, so a malicious attacker would need to overtake both Jenkins and GitHub accounts of a single user to submit a legitimately-looking release.

  • Jun 09, ~1PM UTC - Artifact downloads are restored, alternate patch in the Repository Permission Updater was applied to prevent uploads. Artifact uploads are still blocking

  • Jun 09, 2PM UTC, based on repo.jenkins-ci.org and issues.jenkins-ci.org data, we restored maintainers accounts.


Our next steps would be to communicate the issue to all maintainers and contributors who might have been affected by the LDAP history loss. We will likely need to perform additional user verification steps for plugin maintainers to ensure that there are no contributors affected by the issues. Today at 3:30PM UTC we will also have a Jenkins Infrastructure team meeting where this issue will be discussed in more detail. This is a public meeting, and everyone is welcome to join. Calendar link


Thanks to Olivier Vernin, Daniel Beck and other Jenkins Infra and Security team members who contributed to this investigation.


Best regards,

Oleg Nenashev


--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/ea5483fb-6873-41dd-a82c-d5518c7de106o%40googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CAO49JtGqXd-FwrxzgVtVhJ0nki1BOwgGawSuE%3Dc4%2B940sh07XQ%40mail.gmail.com.

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CAKs8YXJzCV7jFGLz18BnjePsTVjbf5ch%2B9nVBcjC-QUO5z2T8Q%40mail.gmail.com.
Reply | Threaded
Open this post in threaded view
|

Re: ANN: Jenkins release artifacts uploads blockage on June 09, and a temporary downloads issue

Tim Jacomb
it's just the same as a password to maven, so use the api key instead of a password.

On Wed, 17 Jun 2020 at 14:39, <[hidden email]> wrote:
Hi,

thanks for the advice. Hm, my ~/.m2/settings.xml had my encrypted password. The docs (https://wiki.jenkins.io/display/JENKINS/Hosting+Plugins#HostingPlugins-Releasingtojenkins-ci.org) don’t mention the API key. How can I configure Maven to use the API key instead?

Many thanks.
Karsten.


On June 17, 2020 at 14:53:22, Mark Waite ([hidden email]) wrote:



On Wed, Jun 17, 2020 at 6:44 AM Karsten Jeschkies <[hidden email]> wrote:
Hi,

thanks for you hard work. I reset my password successfully but cannot upload a release for the Mesos plugin. Are releases still blocked?


Releases are not blocked but a password reset will also reset your password to the artifact repository.  If you're receiving an HTTP 401 when you try to `mvn release perform` you may need to update your password in the ~/.m2/settings.xml.

I had to do that in order to release a new version of a plugin yesterday.  I logged into the Jenkins Artifactory instance and had it generate an encrypted password from my profile page on that server.  I inserted that encrypted password into my ~/.m2/settings.xml file.  I'm not sure if that is the preferred way to do it, but it worked for me.

Mark Waite
 
Best.
Karsten.

On Tuesday, June 9, 2020 at 5:00:25 PM UTC+2, Oleg Nenashev wrote:

Dear all,


As you may have noticed, the release artifact uploads are currently blocked in the Jenkins Artifactory instances (https://repo.jenkins-ci.org/). We are doing a security investigation due to a partial user database loss on June 02. Today we blocked releases to the Jenkins artifactory, and there also was a temporary outage of the Artifactory downloads which was a collateral damage of the temporary permissions. You can find more details about it in this Jenkins Infra Thread and in this Dev List thread.


Current status:

  • Downloads are restored for all artifacts on https://repo.jenkins-ci.org/, Jenkins core historical releases, Remoting library and Windows Service Wrapper which were among ones reported by Jenkins users.

  • Uploads: Jenkins artifact uploads are blocked for the most of Jenkins plugin maintainers and contributors. It affects releases of Jenkins plugins, Jenkins core and modules, developer tools and all libraries hosted on https://repo.jenkins-ci.org/. Incremental and Snapshot deployments are not affected.


Quick summary: 

  • Jun 02 - There was a Kubernetes Cluster outage on June 02. During this outage we had to rebuild the cluster from scratch to get some services working again.

  • Jun 02 - After the recovery we lost three months of LDAP changes. It has happened due to the broken backup of the LDAP database.

  • Jun 02 - We identified a number of potential security risks which may be caused by the LDAP outage. Account overtake and malicious upload was one of the identified risks. FTR this issue is tracked as SECURITY-1895 as a follow-up to these discussions. Only the Security team members have access to it, so I am not sharing a link here.

  • Jun 09 - After the security risk was independently reported in public by a plugin maintainer in the dev list thread, we decided to block uploads of release artifacts to the Jenkins Artifactory instance.

  • Jun 09, 8:50AM UTC - All uploads of release artifacts were blocked (plugins, Jenkins core and modules, developer tools, etc.). Downloads of some binaries were also blocked as an unexpected collateral damage. Jenkins core historical releases, Remoting library and Windows Service Wrapper are among the affected binaries

  • Jun 09, 10AM UTC - We finished reviews of all artifact releases to https://repo.jenkins-ci.org/, which happened between the infra outage on June 02 and the blockage of the releases. There are no maliciously uploaded artifacts. Note that the common plugin release flow requires access to GitHub in order to push the release commits, so a malicious attacker would need to overtake both Jenkins and GitHub accounts of a single user to submit a legitimately-looking release.

  • Jun 09, ~1PM UTC - Artifact downloads are restored, alternate patch in the Repository Permission Updater was applied to prevent uploads. Artifact uploads are still blocking

  • Jun 09, 2PM UTC, based on repo.jenkins-ci.org and issues.jenkins-ci.org data, we restored maintainers accounts.


Our next steps would be to communicate the issue to all maintainers and contributors who might have been affected by the LDAP history loss. We will likely need to perform additional user verification steps for plugin maintainers to ensure that there are no contributors affected by the issues. Today at 3:30PM UTC we will also have a Jenkins Infrastructure team meeting where this issue will be discussed in more detail. This is a public meeting, and everyone is welcome to join. Calendar link


Thanks to Olivier Vernin, Daniel Beck and other Jenkins Infra and Security team members who contributed to this investigation.


Best regards,

Oleg Nenashev


--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/ea5483fb-6873-41dd-a82c-d5518c7de106o%40googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CAO49JtGqXd-FwrxzgVtVhJ0nki1BOwgGawSuE%3Dc4%2B940sh07XQ%40mail.gmail.com.

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CAKs8YXJzCV7jFGLz18BnjePsTVjbf5ch%2B9nVBcjC-QUO5z2T8Q%40mail.gmail.com.

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CAH-3BieEJ4aA5zax1SqVf%2B_EuNL9C3-qEhCjxkiw6PRPXXakEg%40mail.gmail.com.
12