SAML with Okta is very slow

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

SAML with Okta is very slow

Mark Schroering
We have noticed it taking a very long time (up to 60s) to complete the SAML auth flow.  Here are some logs showing the bigger time gaps.  We are on version 1.1.7 of the SAML plugin and running Jenkins version 2.257. 


Sep 24, 2020 7:52:17 AM FINE org.pac4j.saml.client.SAML2Client retrieveUserProfileAdding attribute value mark.schroering@*****.com for attribute null Sep 24, 2020 7:52:17 AM FINE org.pac4j.core.profile.UserProfile addAttributeno conversion => key: email / value: [mark.schroering@*****.com] / class java.util.ArrayList Sep 24, 2020 7:52:17 AM FINE org.pac4j.core.profile.UserProfile addAttributeno conversion => key: notBefore / value: 2020-09-24T11:46:38.907Z / class org.joda.time.DateTime Sep 24, 2020 7:52:17 AM FINE org.pac4j.core.profile.UserProfile addAttributeno conversion => key: notOnOrAfter / value: 2020-09-24T11:56:38.907Z / class org.joda.time.DateTime Sep 24, 2020 7:52:17 AM FINEST org.jenkinsci.plugins.saml.OpenSAMLWrapperreset TCCL Sep 24, 2020 7:53:35 AM FINE org.jenkinsci.plugins.saml.SamlSecurityRealmSamlSecurityRealm.doCommenceLogin called. Using consumerServiceUrl https://ci.infra.lifeomic.com/securityRealm/finishLogin Sep 24, 2020 7:53:35 AM FINE org.jenkinsci.plugins.saml.SamlSecurityRealmSafe URL redirection: / Sep 24, 2020 7:53:35 AM FINEST org.jenkinsci.plugins.saml.OpenSAMLWrapperadapt TCCL Sep 24, 2020 7:53:45 AM FINE org.jenkinsci.plugins.saml.SamlSecurityRealmSamlSecurityRealm.doCommenceLogin called. Using consumerServiceUrl https://ci.infra.lifeomic.com/securityRealm/finishLogin Sep 24, 2020 7:53:45 AM FINE org.jenkinsci.plugins.saml.SamlSecurityRealmSafe URL redirection: / Sep 24, 2020 7:53:45 AM FINEST org.jenkinsci.plugins.saml.OpenSAMLWrapperadapt TCCL Sep 24, 2020 7:54:13 AM INFO org.pac4j.saml.metadata.SAML2ServiceProviderMetadataResolver Using SP entity ID https://ci.infra.lifeomic.com/securityRealm/finishLogin Sep 24, 2020 7:54:13 AM INFO org.pac4j.saml.metadata.SAML2ServiceProviderMetadataResolver resolveWriting sp metadata to /mnt/jenkins_home/saml-sp-metadata.xml Sep 24, 2020 7:54:13 AM INFO org.pac4j.saml.metadata.SAML2ServiceProviderMetadataResolver resolveAttempting to create directory structure for /mnt/jenkins_home

Looking at the browser tools on page load:

GET /securityRealm/commenceLogin  <-- 57s
GET /securityRealm/finishLogin <--- 38s

the Okta SSO parts in between seem to be quick as expected. 

Any tips on how to further debug or troubleshoot would be appreciated. 

Thanks for the help. 

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/9956335e-8175-4fd4-90d3-bca70fac4f53n%40googlegroups.com.
Reply | Threaded
Open this post in threaded view
|

Re: SAML with Okta is very slow

Mark Schroering
Here are the logs in a better format.  

Sep 24, 2020 7:52:17 AM 
FINE org.pac4j.saml.client.SAML2Client retrieveUserProfileAdding attribute value mark.schroering@*****.com for attribute null 
Sep 24, 2020 7:52:17 AM 
FINE org.pac4j.core.profile.UserProfile addAttributeno conversion => key: email / value: [mark.schroering@*****.com] / class java.util.ArrayList 
Sep 24, 2020 7:52:17 AM 
FINE org.pac4j.core.profile.UserProfile addAttributeno conversion => key: notBefore / value: 2020-09-24T11:46:38.907Z / class org.joda.time.DateTime 
Sep 24, 2020 7:52:17 AM 
FINE org.pac4j.core.profile.UserProfile addAttributeno conversion => key: notOnOrAfter / value: 2020-09-24T11:56:38.907Z / class org.joda.time.DateTime 
Sep 24, 2020 7:52:17 AM 
FINEST org.jenkinsci.plugins.saml.OpenSAMLWrapperreset TCCL 
Sep 24, 2020 7:53:35 AM 
FINE org.jenkinsci.plugins.saml.SamlSecurityRealmSamlSecurityRealm.doCommenceLogin called. Using consumerServiceUrl https://ci.infra.lifeomic.com/securityRealm/finishLogin 
Sep 24, 2020 7:53:35 AM 
FINE org.jenkinsci.plugins.saml.SamlSecurityRealmSafe URL redirection: / 
Sep 24, 2020 7:53:35 AM 
FINEST org.jenkinsci.plugins.saml.OpenSAMLWrapperadapt TCCL 
Sep 24, 2020 7:53:45 AM 
FINE org.jenkinsci.plugins.saml.SamlSecurityRealmSamlSecurityRealm.doCommenceLogin called. Using consumerServiceUrl https://ci.infra.lifeomic.com/securityRealm/finishLogin 
Sep 24, 2020 7:53:45 AM 
FINE org.jenkinsci.plugins.saml.SamlSecurityRealmSafe URL redirection: / 
Sep 24, 2020 7:53:45 AM 
FINEST org.jenkinsci.plugins.saml.OpenSAMLWrapperadapt TCCL 
Sep 24, 2020 7:54:13 AM 
INFO org.pac4j.saml.metadata.SAML2ServiceProviderMetadataResolver Using SP entity ID https://ci.infra.lifeomic.com/securityRealm/finishLogin 
Sep 24, 2020 7:54:13 AM 
INFO org.pac4j.saml.metadata.SAML2ServiceProviderMetadataResolver resolveWriting sp metadata to /mnt/jenkins_home/saml-sp-metadata.xml 
Sep 24, 2020 7:54:13 AM 
INFO org.pac4j.saml.metadata.SAML2ServiceProviderMetadataResolver resolveAttempting to create directory structure for /mnt/jenkins_home

On Thursday, September 24, 2020 at 9:37:54 AM UTC-4 Mark Schroering wrote:
We have noticed it taking a very long time (up to 60s) to complete the SAML auth flow.  Here are some logs showing the bigger time gaps.  We are on version 1.1.7 of the SAML plugin and running Jenkins version 2.257. 


Sep 24, 2020 7:52:17 AM FINE org.pac4j.saml.client.SAML2Client retrieveUserProfileAdding attribute value mark.schroering@*****.com for attribute null Sep 24, 2020 7:52:17 AM FINE org.pac4j.core.profile.UserProfile addAttributeno conversion => key: email / value: [mark.schroering@*****.com] / class java.util.ArrayList Sep 24, 2020 7:52:17 AM FINE org.pac4j.core.profile.UserProfile addAttributeno conversion => key: notBefore / value: 2020-09-24T11:46:38.907Z / class org.joda.time.DateTime Sep 24, 2020 7:52:17 AM FINE org.pac4j.core.profile.UserProfile addAttributeno conversion => key: notOnOrAfter / value: 2020-09-24T11:56:38.907Z / class org.joda.time.DateTime Sep 24, 2020 7:52:17 AM FINEST org.jenkinsci.plugins.saml.OpenSAMLWrapperreset TCCL Sep 24, 2020 7:53:35 AM FINE org.jenkinsci.plugins.saml.SamlSecurityRealmSamlSecurityRealm.doCommenceLogin called. Using consumerServiceUrl https://ci.infra.lifeomic.com/securityRealm/finishLogin Sep 24, 2020 7:53:35 AM FINE org.jenkinsci.plugins.saml.SamlSecurityRealmSafe URL redirection: / Sep 24, 2020 7:53:35 AM FINEST org.jenkinsci.plugins.saml.OpenSAMLWrapperadapt TCCL Sep 24, 2020 7:53:45 AM FINE org.jenkinsci.plugins.saml.SamlSecurityRealmSamlSecurityRealm.doCommenceLogin called. Using consumerServiceUrl https://ci.infra.lifeomic.com/securityRealm/finishLogin Sep 24, 2020 7:53:45 AM FINE org.jenkinsci.plugins.saml.SamlSecurityRealmSafe URL redirection: / Sep 24, 2020 7:53:45 AM FINEST org.jenkinsci.plugins.saml.OpenSAMLWrapperadapt TCCL Sep 24, 2020 7:54:13 AM INFO org.pac4j.saml.metadata.SAML2ServiceProviderMetadataResolver Using SP entity ID https://ci.infra.lifeomic.com/securityRealm/finishLogin Sep 24, 2020 7:54:13 AM INFO org.pac4j.saml.metadata.SAML2ServiceProviderMetadataResolver resolveWriting sp metadata to /mnt/jenkins_home/saml-sp-metadata.xml Sep 24, 2020 7:54:13 AM INFO org.pac4j.saml.metadata.SAML2ServiceProviderMetadataResolver resolveAttempting to create directory structure for /mnt/jenkins_home

Looking at the browser tools on page load:

GET /securityRealm/commenceLogin  <-- 57s
GET /securityRealm/finishLogin <--- 38s

the Okta SSO parts in between seem to be quick as expected. 

Any tips on how to further debug or troubleshoot would be appreciated. 

Thanks for the help. 

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/9b1d58a9-eaa4-46f7-9149-81fb98d9e5c6n%40googlegroups.com.
Reply | Threaded
Open this post in threaded view
|

Re: SAML with Okta is very slow

Ivan Fernandez Calvo
Is your Jenkins home in a NFS or other network storage? I think so for the mount point, when a user enter a few files are written, because your IO is slow the IO operations are blocked waiting to finish that make the login slower than expected. You probably has more performance issues, I usually recommend to not use NFS file systems for the Jenkins home, take a look to this KB https://support.cloudbees.com/hc/en-us/articles/217479948-NFS-Guide

El jueves, 24 de septiembre de 2020 a las 15:52:05 UTC+2, [hidden email] escribió:
Here are the logs in a better format.  

Sep 24, 2020 7:52:17 AM 
FINE org.pac4j.saml.client.SAML2Client retrieveUserProfileAdding attribute value mark.schroering@*****.com for attribute null 
Sep 24, 2020 7:52:17 AM 
FINE org.pac4j.core.profile.UserProfile addAttributeno conversion => key: email / value: [mark.schroering@*****.com] / class java.util.ArrayList 
Sep 24, 2020 7:52:17 AM 
FINE org.pac4j.core.profile.UserProfile addAttributeno conversion => key: notBefore / value: 2020-09-24T11:46:38.907Z / class org.joda.time.DateTime 
Sep 24, 2020 7:52:17 AM 
FINE org.pac4j.core.profile.UserProfile addAttributeno conversion => key: notOnOrAfter / value: 2020-09-24T11:56:38.907Z / class org.joda.time.DateTime 
Sep 24, 2020 7:52:17 AM 
FINEST org.jenkinsci.plugins.saml.OpenSAMLWrapperreset TCCL 
Sep 24, 2020 7:53:35 AM 
FINE org.jenkinsci.plugins.saml.SamlSecurityRealmSamlSecurityRealm.doCommenceLogin called. Using consumerServiceUrl https://ci.infra.lifeomic.com/securityRealm/finishLogin 
Sep 24, 2020 7:53:35 AM 
FINE org.jenkinsci.plugins.saml.SamlSecurityRealmSafe URL redirection: / 
Sep 24, 2020 7:53:35 AM 
FINEST org.jenkinsci.plugins.saml.OpenSAMLWrapperadapt TCCL 
Sep 24, 2020 7:53:45 AM 
FINE org.jenkinsci.plugins.saml.SamlSecurityRealmSamlSecurityRealm.doCommenceLogin called. Using consumerServiceUrl https://ci.infra.lifeomic.com/securityRealm/finishLogin 
Sep 24, 2020 7:53:45 AM 
FINE org.jenkinsci.plugins.saml.SamlSecurityRealmSafe URL redirection: / 
Sep 24, 2020 7:53:45 AM 
FINEST org.jenkinsci.plugins.saml.OpenSAMLWrapperadapt TCCL 
Sep 24, 2020 7:54:13 AM 
INFO org.pac4j.saml.metadata.SAML2ServiceProviderMetadataResolver Using SP entity ID https://ci.infra.lifeomic.com/securityRealm/finishLogin 
Sep 24, 2020 7:54:13 AM 
INFO org.pac4j.saml.metadata.SAML2ServiceProviderMetadataResolver resolveWriting sp metadata to /mnt/jenkins_home/saml-sp-metadata.xml 
Sep 24, 2020 7:54:13 AM 
INFO org.pac4j.saml.metadata.SAML2ServiceProviderMetadataResolver resolveAttempting to create directory structure for /mnt/jenkins_home

On Thursday, September 24, 2020 at 9:37:54 AM UTC-4 Mark Schroering wrote:
We have noticed it taking a very long time (up to 60s) to complete the SAML auth flow.  Here are some logs showing the bigger time gaps.  We are on version 1.1.7 of the SAML plugin and running Jenkins version 2.257. 


Sep 24, 2020 7:52:17 AM FINE org.pac4j.saml.client.SAML2Client retrieveUserProfileAdding attribute value mark.schroering@*****.com for attribute null Sep 24, 2020 7:52:17 AM FINE org.pac4j.core.profile.UserProfile addAttributeno conversion => key: email / value: [mark.schroering@*****.com] / class java.util.ArrayList Sep 24, 2020 7:52:17 AM FINE org.pac4j.core.profile.UserProfile addAttributeno conversion => key: notBefore / value: 2020-09-24T11:46:38.907Z / class org.joda.time.DateTime Sep 24, 2020 7:52:17 AM FINE org.pac4j.core.profile.UserProfile addAttributeno conversion => key: notOnOrAfter / value: 2020-09-24T11:56:38.907Z / class org.joda.time.DateTime Sep 24, 2020 7:52:17 AM FINEST org.jenkinsci.plugins.saml.OpenSAMLWrapperreset TCCL Sep 24, 2020 7:53:35 AM FINE org.jenkinsci.plugins.saml.SamlSecurityRealmSamlSecurityRealm.doCommenceLogin called. Using consumerServiceUrl https://ci.infra.lifeomic.com/securityRealm/finishLogin Sep 24, 2020 7:53:35 AM FINE org.jenkinsci.plugins.saml.SamlSecurityRealmSafe URL redirection: / Sep 24, 2020 7:53:35 AM FINEST org.jenkinsci.plugins.saml.OpenSAMLWrapperadapt TCCL Sep 24, 2020 7:53:45 AM FINE org.jenkinsci.plugins.saml.SamlSecurityRealmSamlSecurityRealm.doCommenceLogin called. Using consumerServiceUrl https://ci.infra.lifeomic.com/securityRealm/finishLogin Sep 24, 2020 7:53:45 AM FINE org.jenkinsci.plugins.saml.SamlSecurityRealmSafe URL redirection: / Sep 24, 2020 7:53:45 AM FINEST org.jenkinsci.plugins.saml.OpenSAMLWrapperadapt TCCL Sep 24, 2020 7:54:13 AM INFO org.pac4j.saml.metadata.SAML2ServiceProviderMetadataResolver Using SP entity ID https://ci.infra.lifeomic.com/securityRealm/finishLogin Sep 24, 2020 7:54:13 AM INFO org.pac4j.saml.metadata.SAML2ServiceProviderMetadataResolver resolveWriting sp metadata to /mnt/jenkins_home/saml-sp-metadata.xml Sep 24, 2020 7:54:13 AM INFO org.pac4j.saml.metadata.SAML2ServiceProviderMetadataResolver resolveAttempting to create directory structure for /mnt/jenkins_home

Looking at the browser tools on page load:

GET /securityRealm/commenceLogin  <-- 57s
GET /securityRealm/finishLogin <--- 38s

the Okta SSO parts in between seem to be quick as expected. 

Any tips on how to further debug or troubleshoot would be appreciated. 

Thanks for the help. 

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/810359b0-dd76-4756-94c9-380ab99ae118n%40googlegroups.com.
Reply | Threaded
Open this post in threaded view
|

Re: SAML with Okta is very slow

Mark Schroering
We are using AWS EFS for the Jenkins Home mount.   It was configured for burst throughput, and after reading https://aws.amazon.com/blogs/storage/best-practices-for-using-amazon-efs-for-container-storage/ we just changed it to provisioned throughput of 150 MiB/s.   The change did not help with the slow login times.  We are still digging through the logs, but are not sure what is causing the big time gaps. 



On Thursday, September 24, 2020 at 7:05:59 PM UTC-4 [hidden email] wrote:
Is your Jenkins home in a NFS or other network storage? I think so for the mount point, when a user enter a few files are written, because your IO is slow the IO operations are blocked waiting to finish that make the login slower than expected. You probably has more performance issues, I usually recommend to not use NFS file systems for the Jenkins home, take a look to this KB https://support.cloudbees.com/hc/en-us/articles/217479948-NFS-Guide

El jueves, 24 de septiembre de 2020 a las 15:52:05 UTC+2, [hidden email] escribió:
Here are the logs in a better format.  

Sep 24, 2020 7:52:17 AM 
FINE org.pac4j.saml.client.SAML2Client retrieveUserProfileAdding attribute value mark.schroering@*****.com for attribute null 
Sep 24, 2020 7:52:17 AM 
FINE org.pac4j.core.profile.UserProfile addAttributeno conversion => key: email / value: [mark.schroering@*****.com] / class java.util.ArrayList 
Sep 24, 2020 7:52:17 AM 
FINE org.pac4j.core.profile.UserProfile addAttributeno conversion => key: notBefore / value: 2020-09-24T11:46:38.907Z / class org.joda.time.DateTime 
Sep 24, 2020 7:52:17 AM 
FINE org.pac4j.core.profile.UserProfile addAttributeno conversion => key: notOnOrAfter / value: 2020-09-24T11:56:38.907Z / class org.joda.time.DateTime 
Sep 24, 2020 7:52:17 AM 
FINEST org.jenkinsci.plugins.saml.OpenSAMLWrapperreset TCCL 
Sep 24, 2020 7:53:35 AM 
FINE org.jenkinsci.plugins.saml.SamlSecurityRealmSamlSecurityRealm.doCommenceLogin called. Using consumerServiceUrl https://ci.infra.lifeomic.com/securityRealm/finishLogin 
Sep 24, 2020 7:53:35 AM 
FINE org.jenkinsci.plugins.saml.SamlSecurityRealmSafe URL redirection: / 
Sep 24, 2020 7:53:35 AM 
FINEST org.jenkinsci.plugins.saml.OpenSAMLWrapperadapt TCCL 
Sep 24, 2020 7:53:45 AM 
FINE org.jenkinsci.plugins.saml.SamlSecurityRealmSamlSecurityRealm.doCommenceLogin called. Using consumerServiceUrl https://ci.infra.lifeomic.com/securityRealm/finishLogin 
Sep 24, 2020 7:53:45 AM 
FINE org.jenkinsci.plugins.saml.SamlSecurityRealmSafe URL redirection: / 
Sep 24, 2020 7:53:45 AM 
FINEST org.jenkinsci.plugins.saml.OpenSAMLWrapperadapt TCCL 
Sep 24, 2020 7:54:13 AM 
INFO org.pac4j.saml.metadata.SAML2ServiceProviderMetadataResolver Using SP entity ID https://ci.infra.lifeomic.com/securityRealm/finishLogin 
Sep 24, 2020 7:54:13 AM 
INFO org.pac4j.saml.metadata.SAML2ServiceProviderMetadataResolver resolveWriting sp metadata to /mnt/jenkins_home/saml-sp-metadata.xml 
Sep 24, 2020 7:54:13 AM 
INFO org.pac4j.saml.metadata.SAML2ServiceProviderMetadataResolver resolveAttempting to create directory structure for /mnt/jenkins_home

On Thursday, September 24, 2020 at 9:37:54 AM UTC-4 Mark Schroering wrote:
We have noticed it taking a very long time (up to 60s) to complete the SAML auth flow.  Here are some logs showing the bigger time gaps.  We are on version 1.1.7 of the SAML plugin and running Jenkins version 2.257. 


Sep 24, 2020 7:52:17 AM FINE org.pac4j.saml.client.SAML2Client retrieveUserProfileAdding attribute value mark.schroering@*****.com for attribute null Sep 24, 2020 7:52:17 AM FINE org.pac4j.core.profile.UserProfile addAttributeno conversion => key: email / value: [mark.schroering@*****.com] / class java.util.ArrayList Sep 24, 2020 7:52:17 AM FINE org.pac4j.core.profile.UserProfile addAttributeno conversion => key: notBefore / value: 2020-09-24T11:46:38.907Z / class org.joda.time.DateTime Sep 24, 2020 7:52:17 AM FINE org.pac4j.core.profile.UserProfile addAttributeno conversion => key: notOnOrAfter / value: 2020-09-24T11:56:38.907Z / class org.joda.time.DateTime Sep 24, 2020 7:52:17 AM FINEST org.jenkinsci.plugins.saml.OpenSAMLWrapperreset TCCL Sep 24, 2020 7:53:35 AM FINE org.jenkinsci.plugins.saml.SamlSecurityRealmSamlSecurityRealm.doCommenceLogin called. Using consumerServiceUrl https://ci.infra.lifeomic.com/securityRealm/finishLogin Sep 24, 2020 7:53:35 AM FINE org.jenkinsci.plugins.saml.SamlSecurityRealmSafe URL redirection: / Sep 24, 2020 7:53:35 AM FINEST org.jenkinsci.plugins.saml.OpenSAMLWrapperadapt TCCL Sep 24, 2020 7:53:45 AM FINE org.jenkinsci.plugins.saml.SamlSecurityRealmSamlSecurityRealm.doCommenceLogin called. Using consumerServiceUrl https://ci.infra.lifeomic.com/securityRealm/finishLogin Sep 24, 2020 7:53:45 AM FINE org.jenkinsci.plugins.saml.SamlSecurityRealmSafe URL redirection: / Sep 24, 2020 7:53:45 AM FINEST org.jenkinsci.plugins.saml.OpenSAMLWrapperadapt TCCL Sep 24, 2020 7:54:13 AM INFO org.pac4j.saml.metadata.SAML2ServiceProviderMetadataResolver Using SP entity ID https://ci.infra.lifeomic.com/securityRealm/finishLogin Sep 24, 2020 7:54:13 AM INFO org.pac4j.saml.metadata.SAML2ServiceProviderMetadataResolver resolveWriting sp metadata to /mnt/jenkins_home/saml-sp-metadata.xml Sep 24, 2020 7:54:13 AM INFO org.pac4j.saml.metadata.SAML2ServiceProviderMetadataResolver resolveAttempting to create directory structure for /mnt/jenkins_home

Looking at the browser tools on page load:

GET /securityRealm/commenceLogin  <-- 57s
GET /securityRealm/finishLogin <--- 38s

the Okta SSO parts in between seem to be quick as expected. 

Any tips on how to further debug or troubleshoot would be appreciated. 

Thanks for the help. 

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/1caf58e5-020d-44ab-a682-00a974054fbcn%40googlegroups.com.
Reply | Threaded
Open this post in threaded view
|

Re: SAML with Okta is very slow

Ivan Fernandez Calvo
as I said your problem is the IO, if you enter en the Instance by ssh and check the iostats you will see more than 5-10% of your operations waiting for IO. NFS, EFS, and in general network filesystems works well with big files but with small files and write concurrence is where the problems start

El vie., 25 sept. 2020 a las 15:02, Mark Schroering (<[hidden email]>) escribió:
We are using AWS EFS for the Jenkins Home mount.   It was configured for burst throughput, and after reading https://aws.amazon.com/blogs/storage/best-practices-for-using-amazon-efs-for-container-storage/ we just changed it to provisioned throughput of 150 MiB/s.   The change did not help with the slow login times.  We are still digging through the logs, but are not sure what is causing the big time gaps. 



On Thursday, September 24, 2020 at 7:05:59 PM UTC-4 [hidden email] wrote:
Is your Jenkins home in a NFS or other network storage? I think so for the mount point, when a user enter a few files are written, because your IO is slow the IO operations are blocked waiting to finish that make the login slower than expected. You probably has more performance issues, I usually recommend to not use NFS file systems for the Jenkins home, take a look to this KB https://support.cloudbees.com/hc/en-us/articles/217479948-NFS-Guide

El jueves, 24 de septiembre de 2020 a las 15:52:05 UTC+2, [hidden email] escribió:
Here are the logs in a better format.  

Sep 24, 2020 7:52:17 AM 
FINE org.pac4j.saml.client.SAML2Client retrieveUserProfileAdding attribute value mark.schroering@*****.com for attribute null 
Sep 24, 2020 7:52:17 AM 
FINE org.pac4j.core.profile.UserProfile addAttributeno conversion => key: email / value: [mark.schroering@*****.com] / class java.util.ArrayList 
Sep 24, 2020 7:52:17 AM 
FINE org.pac4j.core.profile.UserProfile addAttributeno conversion => key: notBefore / value: 2020-09-24T11:46:38.907Z / class org.joda.time.DateTime 
Sep 24, 2020 7:52:17 AM 
FINE org.pac4j.core.profile.UserProfile addAttributeno conversion => key: notOnOrAfter / value: 2020-09-24T11:56:38.907Z / class org.joda.time.DateTime 
Sep 24, 2020 7:52:17 AM 
FINEST org.jenkinsci.plugins.saml.OpenSAMLWrapperreset TCCL 
Sep 24, 2020 7:53:35 AM 
FINE org.jenkinsci.plugins.saml.SamlSecurityRealmSamlSecurityRealm.doCommenceLogin called. Using consumerServiceUrl https://ci.infra.lifeomic.com/securityRealm/finishLogin 
Sep 24, 2020 7:53:35 AM 
FINE org.jenkinsci.plugins.saml.SamlSecurityRealmSafe URL redirection: / 
Sep 24, 2020 7:53:35 AM 
FINEST org.jenkinsci.plugins.saml.OpenSAMLWrapperadapt TCCL 
Sep 24, 2020 7:53:45 AM 
FINE org.jenkinsci.plugins.saml.SamlSecurityRealmSamlSecurityRealm.doCommenceLogin called. Using consumerServiceUrl https://ci.infra.lifeomic.com/securityRealm/finishLogin 
Sep 24, 2020 7:53:45 AM 
FINE org.jenkinsci.plugins.saml.SamlSecurityRealmSafe URL redirection: / 
Sep 24, 2020 7:53:45 AM 
FINEST org.jenkinsci.plugins.saml.OpenSAMLWrapperadapt TCCL 
Sep 24, 2020 7:54:13 AM 
INFO org.pac4j.saml.metadata.SAML2ServiceProviderMetadataResolver Using SP entity ID https://ci.infra.lifeomic.com/securityRealm/finishLogin 
Sep 24, 2020 7:54:13 AM 
INFO org.pac4j.saml.metadata.SAML2ServiceProviderMetadataResolver resolveWriting sp metadata to /mnt/jenkins_home/saml-sp-metadata.xml 
Sep 24, 2020 7:54:13 AM 
INFO org.pac4j.saml.metadata.SAML2ServiceProviderMetadataResolver resolveAttempting to create directory structure for /mnt/jenkins_home

On Thursday, September 24, 2020 at 9:37:54 AM UTC-4 Mark Schroering wrote:
We have noticed it taking a very long time (up to 60s) to complete the SAML auth flow.  Here are some logs showing the bigger time gaps.  We are on version 1.1.7 of the SAML plugin and running Jenkins version 2.257. 


Sep 24, 2020 7:52:17 AM FINE org.pac4j.saml.client.SAML2Client retrieveUserProfileAdding attribute value mark.schroering@*****.com for attribute null Sep 24, 2020 7:52:17 AM FINE org.pac4j.core.profile.UserProfile addAttributeno conversion => key: email / value: [mark.schroering@*****.com] / class java.util.ArrayList Sep 24, 2020 7:52:17 AM FINE org.pac4j.core.profile.UserProfile addAttributeno conversion => key: notBefore / value: 2020-09-24T11:46:38.907Z / class org.joda.time.DateTime Sep 24, 2020 7:52:17 AM FINE org.pac4j.core.profile.UserProfile addAttributeno conversion => key: notOnOrAfter / value: 2020-09-24T11:56:38.907Z / class org.joda.time.DateTime Sep 24, 2020 7:52:17 AM FINEST org.jenkinsci.plugins.saml.OpenSAMLWrapperreset TCCL Sep 24, 2020 7:53:35 AM FINE org.jenkinsci.plugins.saml.SamlSecurityRealmSamlSecurityRealm.doCommenceLogin called. Using consumerServiceUrl https://ci.infra.lifeomic.com/securityRealm/finishLogin Sep 24, 2020 7:53:35 AM FINE org.jenkinsci.plugins.saml.SamlSecurityRealmSafe URL redirection: / Sep 24, 2020 7:53:35 AM FINEST org.jenkinsci.plugins.saml.OpenSAMLWrapperadapt TCCL Sep 24, 2020 7:53:45 AM FINE org.jenkinsci.plugins.saml.SamlSecurityRealmSamlSecurityRealm.doCommenceLogin called. Using consumerServiceUrl https://ci.infra.lifeomic.com/securityRealm/finishLogin Sep 24, 2020 7:53:45 AM FINE org.jenkinsci.plugins.saml.SamlSecurityRealmSafe URL redirection: / Sep 24, 2020 7:53:45 AM FINEST org.jenkinsci.plugins.saml.OpenSAMLWrapperadapt TCCL Sep 24, 2020 7:54:13 AM INFO org.pac4j.saml.metadata.SAML2ServiceProviderMetadataResolver Using SP entity ID https://ci.infra.lifeomic.com/securityRealm/finishLogin Sep 24, 2020 7:54:13 AM INFO org.pac4j.saml.metadata.SAML2ServiceProviderMetadataResolver resolveWriting sp metadata to /mnt/jenkins_home/saml-sp-metadata.xml Sep 24, 2020 7:54:13 AM INFO org.pac4j.saml.metadata.SAML2ServiceProviderMetadataResolver resolveAttempting to create directory structure for /mnt/jenkins_home

Looking at the browser tools on page load:

GET /securityRealm/commenceLogin  <-- 57s
GET /securityRealm/finishLogin <--- 38s

the Okta SSO parts in between seem to be quick as expected. 

Any tips on how to further debug or troubleshoot would be appreciated. 

Thanks for the help. 

--
You received this message because you are subscribed to a topic in the Google Groups "Jenkins Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jenkinsci-users/WgAwcT0OGvk/unsubscribe.
To unsubscribe from this group and all its topics, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/1caf58e5-020d-44ab-a682-00a974054fbcn%40googlegroups.com.


--

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/CAKo5QrrMdt%2BX22e%3D7YZ5xPL1%3DODEFOg6aFMbQhRBAnY2Gt-y6A%40mail.gmail.com.
Reply | Threaded
Open this post in threaded view
|

Re: SAML with Okta is very slow

Mark Schroering
It ended up not being an IO issue.  We use https://www.jenkins.io/projects/jcasc/ and the official Jenkins docker image to deploy it to AWS ECS.  We have a startup script that does some cleanup in the mounted jenkins home directory to make sure that updated plugins are installed properly.  We noticed some saml*.xml files in the home directory.  We updated the startup script to remove these and now the auth flow with Okta works as expected.  



On Friday, September 25, 2020 at 9:07:23 AM UTC-4 [hidden email] wrote:
as I said your problem is the IO, if you enter en the Instance by ssh and check the iostats you will see more than 5-10% of your operations waiting for IO. NFS, EFS, and in general network filesystems works well with big files but with small files and write concurrence is where the problems start

El vie., 25 sept. 2020 a las 15:02, Mark Schroering (<[hidden email]>) escribió:
We are using AWS EFS for the Jenkins Home mount.   It was configured for burst throughput, and after reading https://aws.amazon.com/blogs/storage/best-practices-for-using-amazon-efs-for-container-storage/ we just changed it to provisioned throughput of 150 MiB/s.   The change did not help with the slow login times.  We are still digging through the logs, but are not sure what is causing the big time gaps. 



On Thursday, September 24, 2020 at 7:05:59 PM UTC-4 [hidden email] wrote:
Is your Jenkins home in a NFS or other network storage? I think so for the mount point, when a user enter a few files are written, because your IO is slow the IO operations are blocked waiting to finish that make the login slower than expected. You probably has more performance issues, I usually recommend to not use NFS file systems for the Jenkins home, take a look to this KB https://support.cloudbees.com/hc/en-us/articles/217479948-NFS-Guide

El jueves, 24 de septiembre de 2020 a las 15:52:05 UTC+2, [hidden email] escribió:
Here are the logs in a better format.  

Sep 24, 2020 7:52:17 AM 
FINE org.pac4j.saml.client.SAML2Client retrieveUserProfileAdding attribute value mark.schroering@*****.com for attribute null 
Sep 24, 2020 7:52:17 AM 
FINE org.pac4j.core.profile.UserProfile addAttributeno conversion => key: email / value: [mark.schroering@*****.com] / class java.util.ArrayList 
Sep 24, 2020 7:52:17 AM 
FINE org.pac4j.core.profile.UserProfile addAttributeno conversion => key: notBefore / value: 2020-09-24T11:46:38.907Z / class org.joda.time.DateTime 
Sep 24, 2020 7:52:17 AM 
FINE org.pac4j.core.profile.UserProfile addAttributeno conversion => key: notOnOrAfter / value: 2020-09-24T11:56:38.907Z / class org.joda.time.DateTime 
Sep 24, 2020 7:52:17 AM 
FINEST org.jenkinsci.plugins.saml.OpenSAMLWrapperreset TCCL 
Sep 24, 2020 7:53:35 AM 
FINE org.jenkinsci.plugins.saml.SamlSecurityRealmSamlSecurityRealm.doCommenceLogin called. Using consumerServiceUrl https://ci.infra.lifeomic.com/securityRealm/finishLogin 
Sep 24, 2020 7:53:35 AM 
FINE org.jenkinsci.plugins.saml.SamlSecurityRealmSafe URL redirection: / 
Sep 24, 2020 7:53:35 AM 
FINEST org.jenkinsci.plugins.saml.OpenSAMLWrapperadapt TCCL 
Sep 24, 2020 7:53:45 AM 
FINE org.jenkinsci.plugins.saml.SamlSecurityRealmSamlSecurityRealm.doCommenceLogin called. Using consumerServiceUrl https://ci.infra.lifeomic.com/securityRealm/finishLogin 
Sep 24, 2020 7:53:45 AM 
FINE org.jenkinsci.plugins.saml.SamlSecurityRealmSafe URL redirection: / 
Sep 24, 2020 7:53:45 AM 
FINEST org.jenkinsci.plugins.saml.OpenSAMLWrapperadapt TCCL 
Sep 24, 2020 7:54:13 AM 
INFO org.pac4j.saml.metadata.SAML2ServiceProviderMetadataResolver Using SP entity ID https://ci.infra.lifeomic.com/securityRealm/finishLogin 
Sep 24, 2020 7:54:13 AM 
INFO org.pac4j.saml.metadata.SAML2ServiceProviderMetadataResolver resolveWriting sp metadata to /mnt/jenkins_home/saml-sp-metadata.xml 
Sep 24, 2020 7:54:13 AM 
INFO org.pac4j.saml.metadata.SAML2ServiceProviderMetadataResolver resolveAttempting to create directory structure for /mnt/jenkins_home

On Thursday, September 24, 2020 at 9:37:54 AM UTC-4 Mark Schroering wrote:
We have noticed it taking a very long time (up to 60s) to complete the SAML auth flow.  Here are some logs showing the bigger time gaps.  We are on version 1.1.7 of the SAML plugin and running Jenkins version 2.257. 


Sep 24, 2020 7:52:17 AM FINE org.pac4j.saml.client.SAML2Client retrieveUserProfileAdding attribute value mark.schroering@*****.com for attribute null Sep 24, 2020 7:52:17 AM FINE org.pac4j.core.profile.UserProfile addAttributeno conversion => key: email / value: [mark.schroering@*****.com] / class java.util.ArrayList Sep 24, 2020 7:52:17 AM FINE org.pac4j.core.profile.UserProfile addAttributeno conversion => key: notBefore / value: 2020-09-24T11:46:38.907Z / class org.joda.time.DateTime Sep 24, 2020 7:52:17 AM FINE org.pac4j.core.profile.UserProfile addAttributeno conversion => key: notOnOrAfter / value: 2020-09-24T11:56:38.907Z / class org.joda.time.DateTime Sep 24, 2020 7:52:17 AM FINEST org.jenkinsci.plugins.saml.OpenSAMLWrapperreset TCCL Sep 24, 2020 7:53:35 AM FINE org.jenkinsci.plugins.saml.SamlSecurityRealmSamlSecurityRealm.doCommenceLogin called. Using consumerServiceUrl https://ci.infra.lifeomic.com/securityRealm/finishLogin Sep 24, 2020 7:53:35 AM FINE org.jenkinsci.plugins.saml.SamlSecurityRealmSafe URL redirection: / Sep 24, 2020 7:53:35 AM FINEST org.jenkinsci.plugins.saml.OpenSAMLWrapperadapt TCCL Sep 24, 2020 7:53:45 AM FINE org.jenkinsci.plugins.saml.SamlSecurityRealmSamlSecurityRealm.doCommenceLogin called. Using consumerServiceUrl https://ci.infra.lifeomic.com/securityRealm/finishLogin Sep 24, 2020 7:53:45 AM FINE org.jenkinsci.plugins.saml.SamlSecurityRealmSafe URL redirection: / Sep 24, 2020 7:53:45 AM FINEST org.jenkinsci.plugins.saml.OpenSAMLWrapperadapt TCCL Sep 24, 2020 7:54:13 AM INFO org.pac4j.saml.metadata.SAML2ServiceProviderMetadataResolver Using SP entity ID https://ci.infra.lifeomic.com/securityRealm/finishLogin Sep 24, 2020 7:54:13 AM INFO org.pac4j.saml.metadata.SAML2ServiceProviderMetadataResolver resolveWriting sp metadata to /mnt/jenkins_home/saml-sp-metadata.xml Sep 24, 2020 7:54:13 AM INFO org.pac4j.saml.metadata.SAML2ServiceProviderMetadataResolver resolveAttempting to create directory structure for /mnt/jenkins_home

Looking at the browser tools on page load:

GET /securityRealm/commenceLogin  <-- 57s
GET /securityRealm/finishLogin <--- 38s

the Okta SSO parts in between seem to be quick as expected. 

Any tips on how to further debug or troubleshoot would be appreciated. 

Thanks for the help. 

--
You received this message because you are subscribed to a topic in the Google Groups "Jenkins Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jenkinsci-users/WgAwcT0OGvk/unsubscribe.
To unsubscribe from this group and all its topics, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/1caf58e5-020d-44ab-a682-00a974054fbcn%40googlegroups.com.


--

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/39933fc2-07b4-46bc-96e7-df335091ebbfn%40googlegroups.com.