Change 374958 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Increase db1056 weight

https://gerrit.wikimedia.org/r/374958

Change 374958 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Increase db1056 weight

https://gerrit.wikimedia.org/r/374958

Change 374969 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Restore db1056 original weight

https://gerrit.wikimedia.org/r/374969

Change 374969 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Restore db1056 original weight

https://gerrit.wikimedia.org/r/374969

Mentioned in SAL (#wikimedia-operations) [2017-08-31T11:18:22Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Restore db1056 original weight - T168661 (duration: 00m 47s)

Change 374982 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool db1084

https://gerrit.wikimedia.org/r/374982

Change 374982 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Depool db1084

https://gerrit.wikimedia.org/r/374982

Mentioned in SAL (#wikimedia-operations) [2017-08-31T12:11:46Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Depool db1084 - T168661 (duration: 00m 47s)

Mentioned in SAL (#wikimedia-operations) [2017-08-31T12:14:26Z] <marostegui> Upgrade MariaDB on db1084 - T168661

Change 374985 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Repool db1084 with low weight

https://gerrit.wikimedia.org/r/374985

Change 374985 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Repool db1084 with low weight

https://gerrit.wikimedia.org/r/374985

Mentioned in SAL (#wikimedia-operations) [2017-08-31T12:34:18Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Repool db1084 with low weight - T168661 (duration: 00m 47s)

Change 374988 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Increase weight on db1084

https://gerrit.wikimedia.org/r/374988

Change 374988 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Increase weight on db1084

https://gerrit.wikimedia.org/r/374988

Mentioned in SAL (#wikimedia-operations) [2017-08-31T13:10:34Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Increase db1084 weight - T168661 (duration: 00m 47s)

Change 374990 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Restore db1084 original weight

https://gerrit.wikimedia.org/r/374990

Change 374990 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Restore db1084 original weight

https://gerrit.wikimedia.org/r/374990

Mentioned in SAL (#wikimedia-operations) [2017-08-31T13:36:01Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Restore db1084 original weight - T168661 (duration: 00m 47s)

Change 375358 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool db1081

https://gerrit.wikimedia.org/r/375358

Change 375358 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Depool db1081

https://gerrit.wikimedia.org/r/375358

Mentioned in SAL (#wikimedia-operations) [2017-09-01T10:29:45Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Depool db1081 - T168661 (duration: 00m 43s)

Mentioned in SAL (#wikimedia-operations) [2017-09-01T10:31:18Z] <marostegui> Upgrade MariaDB to 10.0.32 on db1081 - T168661

Change 375362 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Repool db1081 with low weight

https://gerrit.wikimedia.org/r/375362

Change 375362 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Repool db1081 with low weight

https://gerrit.wikimedia.org/r/375362

Mentioned in SAL (#wikimedia-operations) [2017-09-01T11:23:25Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Repool db1081 with low weight - T168661 (duration: 00m 44s)

All s4 hosts have been upgraded to 10.0.32.
Obviously not the master (see below)

So the remaining steps are:

• DBAs: to alter s1 master (db1052) early in the morning (I will attempt to do this on Monday at 5:00 UTC)
• @MarkTraceur to help arranging a read-only time for s4 so we can upgrade s4 master to 10.0.32 and run the instant alter there.

Change 375363 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Increase traffic for db1081

https://gerrit.wikimedia.org/r/375363

Change 375363 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Increase traffic for db1081

https://gerrit.wikimedia.org/r/375363

Mentioned in SAL (#wikimedia-operations) [2017-09-01T11:50:03Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Increase db1081 weight - T168661 (duration: 00m 43s)

Change 375367 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Restore db1081 original weight

https://gerrit.wikimedia.org/r/375367

Change 375367 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Restore db1081 original weight

https://gerrit.wikimedia.org/r/375367

Mentioned in SAL (#wikimedia-operations) [2017-09-01T12:32:39Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Restore db1081 original weight - T168661 (duration: 00m 43s)

Mentioned in SAL (#wikimedia-operations) [2017-09-04T04:52:00Z] <marostegui> Deploy alter table on db1052 - T168661

I have done the alter table on enwiki master (db1052) and it has gone fine. I pre-warmed the big tables (image and filearchive) before and it went thru relatively quickly.
Before running the alter I checked the tables last modifications and indeed they were having low traffic at this time, so while running the alter I didn't see any connection waiting or errors happening so looks like it didn't impact anyone.
Image table took around 1 minute to be altered, and filearchive around 3 minutes.

root@neodymium:~# for i in filearchive image oldimage uploadstash; do echo $i; mysql --skip-ssl -hdb1052 enwiki -e "show create table $i\G" | grep "3D";one
> ^C
root@neodymium:~# for i in filearchive image oldimage uploadstash; do echo $i; mysql --skip-ssl -hdb1052 enwiki -e "show create table $i\G" | grep "3D";done
filearchive
  `fa_media_type` enum('UNKNOWN','BITMAP','DRAWING','AUDIO','VIDEO','MULTIMEDIA','OFFICE','TEXT','EXECUTABLE','ARCHIVE','3D') DEFAULT NULL,
image
  `img_media_type` enum('UNKNOWN','BITMAP','DRAWING','AUDIO','VIDEO','MULTIMEDIA','OFFICE','TEXT','EXECUTABLE','ARCHIVE','3D') DEFAULT NULL,
oldimage
  `oi_media_type` enum('UNKNOWN','BITMAP','DRAWING','AUDIO','VIDEO','MULTIMEDIA','OFFICE','TEXT','EXECUTABLE','ARCHIVE','3D') DEFAULT NULL,
uploadstash
  `us_media_type` enum('UNKNOWN','BITMAP','DRAWING','AUDIO','VIDEO','MULTIMEDIA','OFFICE','TEXT','EXECUTABLE','ARCHIVE','3D') DEFAULT NULL,

Just for the record, I am checking again that the alter has been done across all the host and all the wikis and the only pending one would be commonswiki on s4 master - db1068

Mentioned in SAL (#wikimedia-operations) [2017-09-04T07:03:18Z] <marostegui> Deploy alter table on s3 codfw master with replication on kpbwiki - T168661

The new 3D enum value was missing on the new created wiki kbpwiki on s3, as probably the patch was merged after it was created.
The rest of wikis look good across all the shards.

@MarkTraceur any update on this? Were you able to try to arrange a read-only time for s4?

Oh, I'm sorry, I missed some updates here (lots of noise on this ticket), and misunderstood that I needed to go through some process to complete it. I've never requested read-only time before, is it appreciably more complex than e-mailing @greg and telling him what's up? We already have a 1-hour deploy window for testwiki this Wednesday at noon UTC, maybe we can pull together a 30-minute window before that or on Tuesday? What works best?

In T168661#3632148, @MarkTraceur wrote:

Oh, I'm sorry, I missed some updates here (lots of noise on this ticket), and misunderstood that I needed to go through some process to complete it. I've never requested read-only time before, is it appreciably more complex than e-mailing @greg and telling him what's up? We already have a 1-hour deploy window for testwiki this Wednesday at noon UTC, maybe we can pull together a 30-minute window before that or on Tuesday? What works best?

I assume it is a bit more complicate than that, because it implies getting commonswiki on read only, meaning that no one would be able to upload files :-)
I guess that needs to be announced somewhere so people are aware of that maintenance coming on.
@klove maybe can point us on the right direction here?

What @Marostegui said, the only really more complicated part is the announce/delay so that people are aware.

I'm not sure @klove is the right person to ask here? Who were you thinking of Manuel?

In T168661#3632757, @greg wrote:

What @Marostegui said, the only really more complicated part is the announce/delay so that people are aware.

I'm not sure @klove is the right person to ask here? Who were you thinking of Manuel?

Hey @greg!
I might be wrong indeed, but I thought @klove was the one coordinating the announcements of the DC switchover for instance?
Basically I was looking for some that can coordinate the announcement of commons going read only for a bit with us :-)

Who do you think can help us here?

I would ask the Multimedia team who their standard community liaison is (they must have one due to the Structured Data on Commons work). cc @MarkTraceur

That would be @CKoerner_WMF ...

Change 382380 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] db1068: Update socket path

https://gerrit.wikimedia.org/r/382380

This ticket has a lot of comments so I might have missed it, but has someone verified that putting Commons into read only mode won't affect other wikis when you try and add an image to an article - IIRC the globalimagelinks table is also on commonswiki database and we write to that every time a new Commons image is used in an article.

Also there's features like cross-wiki image upload - how will those handle read only mode?

The procedure I had in mind to upgrade the master:

• Disable all replication alerts on s4.
• Set commonswiki on read-only by merging: https://gerrit.wikimedia.org/r/#/c/382379/1
• Double check indeed writes have stopped on the master.
• Once checked: stop mysql on the master
• apt-get update && apt-get upgrade
• Update socket path by merging: https://gerrit.wikimedia.org/r/#/c/382380/
• Run puppet
• Start MySQL on the master
• mysql_upgrade on the master
• Stop MySQL
• Start MySQL
• Make sure the master has read_only=OFF
• Make sure all the slave reconnect
• Revert read-only by reverting: https://gerrit.wikimedia.org/r/#/c/382379/
• Check that writes are happening

@jcrespo can you double check this procedure and let me know if it looks good to you?

In T168661#3659656, @Legoktm wrote:

This ticket has a lot of comments so I might have missed it, but has someone verified that putting Commons into read only mode won't affect other wikis when you try and add an image to an article - IIRC the globalimagelinks table is also on commonswiki database and we write to that every time a new Commons image is used in an article.

Also there's features like cross-wiki image upload - how will those handle read only mode?

Thanks for the input!
I don't think anyone has checked that, who could be a good starting point to double check it on mediawiki code?

In T168661#3659662, @Marostegui wrote:

In T168661#3659656, @Legoktm wrote:

This ticket has a lot of comments so I might have missed it, but has someone verified that putting Commons into read only mode won't affect other wikis when you try and add an image to an article - IIRC the globalimagelinks table is also on commonswiki database and we write to that every time a new Commons image is used in an article.

Also there's features like cross-wiki image upload - how will those handle read only mode?

Thanks for the input!
I don't think anyone has checked that, who could be a good starting point to double check it on mediawiki code?

Probably @aaron or @Bawolff ?

OK, I tested it by putting Commons into read-only mode using the same method that is planned. My hunch was correct, and that adding a file to a page while Commons is read only throws an exception after the page is saved, and the globalimagelinks table is *not* updated:

Second, for whatever reason, the setting being used to mark Commons as read only doesn't actually display the provided message to the user:

Thanks @Legoktm for testing this.
@aaron @Bawolff any advise?

Once a Commons administrator implements https://commons.wikimedia.org/wiki/MediaWiki_talk:Readonlywarning that will fix the issue of the read only message not being displayed to users.

I would compress the steps:

Set commonswiki on read-only by merging: https://gerrit.wikimedia.org/r/#/c/382379/1
Double check indeed writes have stopped on the master.
Once checked: stop mysql on the master
apt-get update && apt-get upgrade
Update socket path by merging: https://gerrit.wikimedia.org/r/#/c/382380/
Run puppet
Start MySQL on the master
mysql_upgrade on the master
Stop MySQL
Start MySQL
Make sure the master has read_only=OFF
Make sure all the slave reconnect

Into

apt update && apt full-upgrade
Set commonswiki on read-only by merging: https://gerrit.wikimedia.org/r/#/c/382379/1
Double check indeed writes have stopped on the master.
Once checked: stop mysql on the master
Update socket path by merging: https://gerrit.wikimedia.org/r/#/c/382380/
Run puppet
Start MySQL on the master
Make sure the master has read_only=OFF
Make sure all the slave reconnect
Revert read-only by reverting: https://gerrit.wikimedia.org/r/#/c/382379/
Run mysql_upgrade

mysql_upgrade, in this case, is unlikely to change any inner structure 10.0.23 -> 10.0.32, and by double-rebooting it, you are losing the buffer pool dump. We can also speed up the warmup by running a SELECT/ALTER TABLE on image just after reboot (metadata locking may be problematic once it is fully pooled).

In T168661#3663735, @jcrespo wrote:

Into

apt update && apt full-upgrade
Set commonswiki on read-only by merging: https://gerrit.wikimedia.org/r/#/c/382379/1
Double check indeed writes have stopped on the master.
Once checked: stop mysql on the master
Update socket path by merging: https://gerrit.wikimedia.org/r/#/c/382380/
Run puppet
Start MySQL on the master
Make sure the master has read_only=OFF
Make sure all the slave reconnect
Revert read-only by reverting: https://gerrit.wikimedia.org/r/#/c/382379/
Run mysql_upgrade

mysql_upgrade, in this case, is unlikely to change any inner structure 10.0.23 -> 10.0.32, and by double-rebooting it, you are losing the buffer pool dump. We can also speed up the warmup by running a SELECT/ALTER TABLE on image just after reboot (metadata locking may be problematic once it is fully pooled).

One note: you would still want to run mysql_upgrade after read-only is gone?

you would still want to run mysql_upgrade after read-only is gone?

Yes, precisely because it will be noop, so no need to block on that. Not necessarily after, reversion and run should happen at the same time, most likely it will end before all deployment is done.

Cool! Fine by me, so let's do that procedure then - thanks!:

Disable all replication alerts on s4.
apt update && apt full-upgrade
Set commonswiki on read-only by merging: https://gerrit.wikimedia.org/r/#/c/382379/1
Double check indeed writes have stopped on the master.
Once checked: stop mysql on the master
Update socket path by merging: https://gerrit.wikimedia.org/r/#/c/382380/
Run puppet
Start MySQL on the master
ALTER the tables
Make sure the master has read_only=OFF
Make sure all the slave reconnect
Revert read-only by reverting: https://gerrit.wikimedia.org/r/#/c/382379/
Run mysql_upgrade

I have tested the procedure described at T168661#3663752 with another core host and it worked perfectly.

Mentioned in SAL (#wikimedia-operations) [2017-10-11T05:06:42Z] <marostegui> Dump buffer pool on s4 primary master db1068 - T168661

Mentioned in SAL (#wikimedia-operations) [2017-10-11T05:13:39Z] <marostegui> Disable alerts on s4 for 2 hours - T168661

I'm worried that T168661#3663516 is still an issue, but I suppose we can just clean it up afterwards by running refreshLinks.php everywhere.

In T168661#3674551, @Legoktm wrote:

I'm worried that T168661#3663516 is still an issue, but I suppose we can just clean it up afterwards by running refreshLinks.php everywhere.

Yeah, let's do that.
We are hoping not to be in RO a long time though.

Mentioned in SAL (#wikimedia-operations) [2017-10-11T05:35:14Z] <marostegui> Disable puppet on db1068 (s4 primary master) - T168661

Change 382380 merged by Marostegui:
[operations/puppet@production] db1068: Update socket path

https://gerrit.wikimedia.org/r/382380

Mentioned in SAL (#wikimedia-operations) [2017-10-11T05:48:54Z] <marostegui> Package update on db1068 s4 primary master - T168661

Mentioned in SAL (#wikimedia-operations) [2017-10-11T06:00:56Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Set s4 in read-only for maintenance - T168661 (duration: 00m 47s)

Mentioned in SAL (#wikimedia-operations) [2017-10-11T06:01:03Z] <marostegui> Set read-only on db1068 s4 primary master - T168661

Mentioned in SAL (#wikimedia-operations) [2017-10-11T06:07:14Z] <marostegui> Deploy alter table on s4 primary master - T168661

Mentioned in SAL (#wikimedia-operations) [2017-10-11T06:14:11Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Set s4 to writable mode after maintenance - T168661 (duration: 00m 47s)

Mentioned in SAL (#wikimedia-operations) [2017-10-11T06:14:18Z] <marostegui> Set db1068 s4 primary master to read-only OFF - T168661

This maintenance is finished

Read only time started at: 6:01
Read only time finished at: 6:14

Total 13 minutes

The master has now the new types:

filearchive
  `fa_media_type` enum('UNKNOWN','BITMAP','DRAWING','AUDIO','VIDEO','MULTIMEDIA','OFFICE','TEXT','EXECUTABLE','ARCHIVE','3D') DEFAULT NULL,
image
  `img_media_type` enum('UNKNOWN','BITMAP','DRAWING','AUDIO','VIDEO','MULTIMEDIA','OFFICE','TEXT','EXECUTABLE','ARCHIVE','3D') DEFAULT NULL,
oldimage
  `oi_media_type` enum('UNKNOWN','BITMAP','DRAWING','AUDIO','VIDEO','MULTIMEDIA','OFFICE','TEXT','EXECUTABLE','ARCHIVE','3D') DEFAULT NULL,
uploadstash
  `us_media_type` enum('UNKNOWN','BITMAP','DRAWING','AUDIO','VIDEO','MULTIMEDIA','OFFICE','TEXT','EXECUTABLE','ARCHIVE','3D') DEFAULT NULL,

This is all done - I am going to check again (T168661#3576396) across all the fleet to make sure this is done everywhere before closing this task.

@Legoktm will you want to file a ticket for something like "smoother read only" for other projects when commons or wikidata are down/ro or are you ok with how things went?