Backporting for LTS 2.60.3 started

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Backporting for LTS 2.60.3 started

ogondza
Backporting has started and the RC is scheduled for 2017-08-02.

All 4 outstanding candidates ware trivial enough and backported
immediately. There are 2 rejects that are rejected for the whole 2.60
line but I doubt we want to revisit that for .3.

Candidates: https://issues.jenkins-ci.org/issues/?filter=12146
Fixed: https://issues.jenkins-ci.org/issues/?jql=labels%20%3D%202.60.3-fixed
Rejected:
https://issues.jenkins-ci.org/issues/?jql=labels%20%3D%202.60.3-rejected
--
oliver

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/4a42c467-0a9d-4ac4-f461-2453af2c0c69%40gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Backporting for LTS 2.60.3 started

Andreas Mandel
I'm not sure if this is the right place. But please take note of JENKINS-45057 before releasing the next LTS Version.

Kind Regards,
Andreas.

Am Donnerstag, 20. Juli 2017 12:26:27 UTC+2 schrieb ogondza:
Backporting has started and the RC is scheduled for 2017-08-02.

All 4 outstanding candidates ware trivial enough and backported
immediately. There are 2 rejects that are rejected for the whole 2.60
line but I doubt we want to revisit that for .3.

Candidates: <a href="https://issues.jenkins-ci.org/issues/?filter=12146" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fissues.jenkins-ci.org%2Fissues%2F%3Ffilter%3D12146\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFV1gVcrfnbL9XL8H1i20jBPdDFJg&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fissues.jenkins-ci.org%2Fissues%2F%3Ffilter%3D12146\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFV1gVcrfnbL9XL8H1i20jBPdDFJg&#39;;return true;">https://issues.jenkins-ci.org/issues/?filter=12146
Fixed: <a href="https://issues.jenkins-ci.org/issues/?jql=labels%20%3D%202.60.3-fixed" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fissues.jenkins-ci.org%2Fissues%2F%3Fjql%3Dlabels%2520%253D%25202.60.3-fixed\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHTpgmZlfZmPM0egEbFzYxSkC5zqg&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fissues.jenkins-ci.org%2Fissues%2F%3Fjql%3Dlabels%2520%253D%25202.60.3-fixed\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHTpgmZlfZmPM0egEbFzYxSkC5zqg&#39;;return true;">https://issues.jenkins-ci.org/issues/?jql=labels%20%3D%202.60.3-fixed
Rejected:
<a href="https://issues.jenkins-ci.org/issues/?jql=labels%20%3D%202.60.3-rejected" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fissues.jenkins-ci.org%2Fissues%2F%3Fjql%3Dlabels%2520%253D%25202.60.3-rejected\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFllqqRfvsjsxO_iyw0RL3IqqUQ7g&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fissues.jenkins-ci.org%2Fissues%2F%3Fjql%3Dlabels%2520%253D%25202.60.3-rejected\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFllqqRfvsjsxO_iyw0RL3IqqUQ7g&#39;;return true;">https://issues.jenkins-ci.org/issues/?jql=labels%20%3D%202.60.3-rejected
--
oliver

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/baaa57be-049f-4d40-86a9-9ec11f33a213%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Backporting for LTS 2.60.3 started

Daniel Beck

> On 25. Jul 2017, at 23:05, Andreas Mandel <[hidden email]> wrote:
>
> I'm not sure if this is the right place. But please take note of JENKINS-45057 before releasing the next LTS Version.

LTS releases only get security fixes or backports of fixes previously in the weekly releases. See https://jenkins.io/download/lts/ for how this works.

As the comments to the issue indicate, it's still unclear when it started and what caused it, and there's no fix to be backported yet.

Daniel

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/860EACC5-C090-4215-9F1B-8C65F5F9BBC6%40beckweb.net.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Backporting for LTS 2.60.3 started

Andreas Mandel
Thanks for the feedback. Does this mean, according to plan, you release LTS versions even with blocker issues? Or do you judge the severity of the issue different?

We are trying to narrow this further down to the versions you have mentioned in the ticket but time is unfortunately quite limited.

Kind Regards, Andreas.

Am Mittwoch, 26. Juli 2017 00:10:03 UTC+2 schrieb Daniel Beck:

> On 25. Jul 2017, at 23:05, Andreas Mandel <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="yUrwuB9fBQAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">andreas...@...> wrote:
>
> I'm not sure if this is the right place. But please take note of JENKINS-45057 before releasing the next LTS Version.

LTS releases only get security fixes or backports of fixes previously in the weekly releases. See <a href="https://jenkins.io/download/lts/" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fjenkins.io%2Fdownload%2Flts%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHDo6XYf5IQg8Y0uiVGALR8TwzUSg&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fjenkins.io%2Fdownload%2Flts%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHDo6XYf5IQg8Y0uiVGALR8TwzUSg&#39;;return true;">https://jenkins.io/download/lts/ for how this works.

As the comments to the issue indicate, it's still unclear when it started and what caused it, and there's no fix to be backported yet.

Daniel

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/8c9d9f0a-9c93-4147-9e2a-e988c82ce82c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Backporting for LTS 2.60.3 started

ogondza
On 2017-07-27 07:19, Andreas Mandel wrote:
> Thanks for the feedback. Does this mean, according to plan, you release
> LTS versions even with blocker issues? Or do you judge the severity of
> the issue different?
>
> We are trying to narrow this further down to the versions you have
> mentioned in the ticket but time is unfortunately quite limited.

The severity of the issue affects the likelihood someone will work on
that in the near future but that is pretty much it. So if we do not have
the fix, the LTS will not be postponed to incorporate it. And even if we
do, I tend to prefer to have the fix soaked in weakly first unless the
bug is particularly bag to outweigh the risks.

Try to look at it this way, the regression has sneaked in, quite likely,
despite the review process and testing when community had all the time
to incorporated it. Trying to device a fix in a hurry and expedite it
into the stable branch has a potential to cause lot more damage.
Especially if we do not have a QE army to verity the result, which we do
not have.

--
oliver

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/a9d87944-da8b-e050-0bec-865f45fc4d55%40gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Backporting for LTS 2.60.3 started

Daniel Beck
In reply to this post by Andreas Mandel

> On 27. Jul 2017, at 07:19, Andreas Mandel <[hidden email]> wrote:
>
> Thanks for the feedback. Does this mean, according to plan, you release LTS versions even with blocker issues? Or do you judge the severity of the issue different?

Yes. The train model means we leave the station no matter what. Otherwise we just get bogged down in discussions what constitutes a blocker issue while holding off releases to include well-soaked bug fixes.

> but time is unfortunately quite limited.

If I'm right with my guess of 2.53 as the first affected release, the problem was around for almost 3 months between introduction of the issue and the original report, which was created only after 2.60.1 was released. As we rely on users reporting and voting on issues they encounter for things like deciding on an LTS baseline, it's no surprise this got in.

I asked a month ago in a comment for people who encounter this issue to narrow down in which release the problem was introduced. Despite having 20 watchers now, nobody seems to have taken the time to do that yet.

Does that look consistent with an urgent blocker issue to you?

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/6C6FDB86-4015-4DDA-8376-B7A1F5C28B52%40beckweb.net.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Backporting for LTS 2.60.3 started

Andreas Mandel
All valid points, for sure. This aspect of the Train Model was not clear to me... I need to find a way for me how to reduce the risk to run into such issues as much as possible for our productive system. We do tests in advance and feedback - but actually this is not very formal (on our side) and for sure it lacks some heavy load tests and tests that leave the system at work for a longer time period.

Now that I'm affected, I think it is important to somehow flag the LTS with such issues - actually one can do a Jira Query to find all known open issues or use the https://jenkins.io/changelog-stable/ page to check for the weather condition and then decide based on the likely long list of tickets if it is OK do go forward or not.

I can not say why the issue did not get more attention earlier. IMHO it is a kind of time bomb for every Jenkins installation, it might be undetected for smaller installations especially if they go with the weekly releases and so get restarted at least once a week.

At the end we try to support to get the issue solved - hope this happens soon.

Thanks and kind regards, Andreas.

Am Donnerstag, 27. Juli 2017 12:56:55 UTC+2 schrieb Daniel Beck:

> On 27. Jul 2017, at 07:19, Andreas Mandel <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="VwcjYynXBwAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">andreas...@...> wrote:
>
> Thanks for the feedback. Does this mean, according to plan, you release LTS versions even with blocker issues? Or do you judge the severity of the issue different?

Yes. The train model means we leave the station no matter what. Otherwise we just get bogged down in discussions what constitutes a blocker issue while holding off releases to include well-soaked bug fixes.

> but time is unfortunately quite limited.

If I'm right with my guess of 2.53 as the first affected release, the problem was around for almost 3 months between introduction of the issue and the original report, which was created only after 2.60.1 was released. As we rely on users reporting and voting on issues they encounter for things like deciding on an LTS baseline, it's no surprise this got in.

I asked a month ago in a comment for people who encounter this issue to narrow down in which release the problem was introduced. Despite having 20 watchers now, nobody seems to have taken the time to do that yet.

Does that look consistent with an urgent blocker issue to you?

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/7bf0a889-d95d-4fac-9c39-518e59161068%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Backporting for LTS 2.60.3 started

Oleg Nenashev
We have two pending pull requests for JENKINS-45057.
I think it worth to consider backporting them. Will try to get one of the fixes landed in the next weekly

четверг, 27 июля 2017 г., 17:07:01 UTC+2 пользователь Andreas Mandel написал:
All valid points, for sure. This aspect of the Train Model was not clear to me... I need to find a way for me how to reduce the risk to run into such issues as much as possible for our productive system. We do tests in advance and feedback - but actually this is not very formal (on our side) and for sure it lacks some heavy load tests and tests that leave the system at work for a longer time period.

Now that I'm affected, I think it is important to somehow flag the LTS with such issues - actually one can do a Jira Query to find all known open issues or use the <a href="https://jenkins.io/changelog-stable/" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fjenkins.io%2Fchangelog-stable%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNG7QqcCD5b6Q8eBjpPfmSeorkuTgw&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fjenkins.io%2Fchangelog-stable%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNG7QqcCD5b6Q8eBjpPfmSeorkuTgw&#39;;return true;">https://jenkins.io/changelog-stable/ page to check for the weather condition and then decide based on the likely long list of tickets if it is OK do go forward or not.

I can not say why the issue did not get more attention earlier. IMHO it is a kind of time bomb for every Jenkins installation, it might be undetected for smaller installations especially if they go with the weekly releases and so get restarted at least once a week.

At the end we try to support to get the issue solved - hope this happens soon.

Thanks and kind regards, Andreas.

Am Donnerstag, 27. Juli 2017 12:56:55 UTC+2 schrieb Daniel Beck:

> On 27. Jul 2017, at 07:19, Andreas Mandel <[hidden email]> wrote:
>
> Thanks for the feedback. Does this mean, according to plan, you release LTS versions even with blocker issues? Or do you judge the severity of the issue different?

Yes. The train model means we leave the station no matter what. Otherwise we just get bogged down in discussions what constitutes a blocker issue while holding off releases to include well-soaked bug fixes.

> but time is unfortunately quite limited.

If I'm right with my guess of 2.53 as the first affected release, the problem was around for almost 3 months between introduction of the issue and the original report, which was created only after 2.60.1 was released. As we rely on users reporting and voting on issues they encounter for things like deciding on an LTS baseline, it's no surprise this got in.

I asked a month ago in a comment for people who encounter this issue to narrow down in which release the problem was introduced. Despite having 20 watchers now, nobody seems to have taken the time to do that yet.

Does that look consistent with an urgent blocker issue to you?

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/aef7efb7-7708-491c-ad13-f6007716a939%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Backporting for LTS 2.60.3 started

Andres Rodriguez-2
So, what's the status of the RC? Is it going to be pushed today / tomorrow or is it waiting on some additional backport?

On Wed, Aug 2, 2017 at 8:06 PM, Oleg Nenashev <[hidden email]> wrote:
We have two pending pull requests for JENKINS-45057.
I think it worth to consider backporting them. Will try to get one of the fixes landed in the next weekly

четверг, 27 июля 2017 г., 17:07:01 UTC+2 пользователь Andreas Mandel написал:
All valid points, for sure. This aspect of the Train Model was not clear to me... I need to find a way for me how to reduce the risk to run into such issues as much as possible for our productive system. We do tests in advance and feedback - but actually this is not very formal (on our side) and for sure it lacks some heavy load tests and tests that leave the system at work for a longer time period.

Now that I'm affected, I think it is important to somehow flag the LTS with such issues - actually one can do a Jira Query to find all known open issues or use the https://jenkins.io/changelog-stable/ page to check for the weather condition and then decide based on the likely long list of tickets if it is OK do go forward or not.

I can not say why the issue did not get more attention earlier. IMHO it is a kind of time bomb for every Jenkins installation, it might be undetected for smaller installations especially if they go with the weekly releases and so get restarted at least once a week.

At the end we try to support to get the issue solved - hope this happens soon.

Thanks and kind regards, Andreas.

Am Donnerstag, 27. Juli 2017 12:56:55 UTC+2 schrieb Daniel Beck:

> On 27. Jul 2017, at 07:19, Andreas Mandel <[hidden email]> wrote:
>
> Thanks for the feedback. Does this mean, according to plan, you release LTS versions even with blocker issues? Or do you judge the severity of the issue different?

Yes. The train model means we leave the station no matter what. Otherwise we just get bogged down in discussions what constitutes a blocker issue while holding off releases to include well-soaked bug fixes.

> but time is unfortunately quite limited.

If I'm right with my guess of 2.53 as the first affected release, the problem was around for almost 3 months between introduction of the issue and the original report, which was created only after 2.60.1 was released. As we rely on users reporting and voting on issues they encounter for things like deciding on an LTS baseline, it's no surprise this got in.

I asked a month ago in a comment for people who encounter this issue to narrow down in which release the problem was introduced. Despite having 20 watchers now, nobody seems to have taken the time to do that yet.

Does that look consistent with an urgent blocker issue to you?

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/aef7efb7-7708-491c-ad13-f6007716a939%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.



--
Regards,

Andres Rodriguez

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CAM9%3DZ%2B7CpCvLYs_Vg_TEUBUrVHDOVtM1D9yaeS5L_h%3DXRWSLqQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Backporting for LTS 2.60.3 started

ogondza
I am sorry but I was not able to make it to the meeting on the last
moment. I have backported the 4 candidates and I am pushing the RC
today. JENKINS-45057 seems pretty nasty yet not backported, I am open to
revisit that for 2.60.3 once merged to master.

--
oliver

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/00f51d06-4899-73ba-0f9b-5996eae5e8b3%40gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Backporting for LTS 2.60.3 started

Andres Rodriguez-2
Thanks!

On Thu, Aug 3, 2017 at 12:34 PM, Oliver Gondža <[hidden email]> wrote:
I am sorry but I was not able to make it to the meeting on the last moment. I have backported the 4 candidates and I am pushing the RC today. JENKINS-45057 seems pretty nasty yet not backported, I am open to revisit that for 2.60.3 once merged to master.

--
oliver

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/00f51d06-4899-73ba-0f9b-5996eae5e8b3%40gmail.com.

For more options, visit https://groups.google.com/d/optout.



--
Regards,

Andres Rodriguez

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CAM9%3DZ%2B5Tn0NpxHAP6VQCEO1ptODLORhKzevb726n0uFtjPX7Zg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Backporting for LTS 2.60.3 started

Oleg Nenashev
https://github.com/jenkinsci/jenkins/pull/2954 has been integrated into the master branch

2017-08-03 13:01 GMT+02:00 Andres Rodriguez <[hidden email]>:
Thanks!

On Thu, Aug 3, 2017 at 12:34 PM, Oliver Gondža <[hidden email]> wrote:
I am sorry but I was not able to make it to the meeting on the last moment. I have backported the 4 candidates and I am pushing the RC today. JENKINS-45057 seems pretty nasty yet not backported, I am open to revisit that for 2.60.3 once merged to master.

--
oliver


--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/00f51d06-4899-73ba-0f9b-5996eae5e8b3%40gmail.com.

For more options, visit https://groups.google.com/d/optout.



--
Regards,

Andres Rodriguez

--
You received this message because you are subscribed to a topic in the Google Groups "Jenkins Developers" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jenkinsci-dev/3Vz_Db-rNzk/unsubscribe.
To unsubscribe from this group and all its topics, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CAM9%3DZ%2B5Tn0NpxHAP6VQCEO1ptODLORhKzevb726n0uFtjPX7Zg%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CAPfivLBoyZg_eWjBVTUycxWCAvEPcn18%3DLooaM9yBgXn4ieEAg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Loading...