For gadgets using mw.Api() (which is hopefully most of them), the nice solution would be to add a new constructor option to pass a scheduler. Then within mw.Api, we'd just have to change mw.Api.ajax() so that it gets a promise from the scheduler and chains the API call to that, and in gadgets you only need to make some trivial change like new mw.Api() -> new mw.Api( { scheduler: mw.Api.defaultScheduler } ). The scheduler could then decide whether to sequentialize requests or limit them to a fixed number of connections or inject delay or whatever.

In T316303#9916132, @Tgr wrote:

[autocreation from the primary authentication provider after a successful login] would probably take a lot of fiddling to fix because there are a ton of permission checks in AuthManager and SpecialUserlogin and we'd need to ensure every single one of those uses an anonymous account with a name set. But this scenario will probably go away anyway after T348388. That leaves central autologin, which seems easy to fix.

Per the discussion with Security that happened as part of T367995: Security Preview for shared login domain, we should probably fix this before making the copyright message from all wikis show up on the same shared login domain.

First stab:

{
  "query": {
    "bool": {
      "minimum_should_match": 1,
      "should": [
        {
          "match": {
            "channel": "resourceloader"
          }
        },
        {
          "bool": {
            "must": [
              {
                "terms": {
                  "channel.keyword": [ "OAuth", "authentication", "session", "SQLBagOStuff" ]
                }
              },
              {
                "terms": {
                  "level.keyword": [ "ERROR" ]
                }
              }
            ]
          }
        },
        {
          "match_phrase": {
            "exception.trace": "ResourceLoader"
          }
        },
        {
          "bool": {
            "must": [
              {
                "term": {
                  "channel.keyword": "exception"
                }
              },
              {
                "match_phrase": {
                  "exception.trace": "includes/auth/"
                }
              }
            ]
          }
        },
        {
          "bool": {
            "must": [
              {
                "term": {
                  "channel.keyword": "exception"
                }
              },
              {
                "match_phrase": {
                  "exception.trace": "includes/session/"
                }
              }
            ]
          }
        },
        {
          "bool": {
            "must": [
              {
                "term": {
                  "channel.keyword": "exception"
                }
              },
              {
                "match_phrase": {
                  "exception.trace": "includes/password/"
                }
              }
            ]
          }
        },
        {
          "bool": {
            "must": [
              {
                "term": {
                  "channel.keyword": "exception"
                }
              },
              {
                "match_phrase": {
                  "exception.trace": "maintenance/benchmarks/"
                }
              }
            ]
          }
        },
        {
          "bool": {
            "must": [
              {
                "term": {
                  "channel.keyword": "exception"
                }
              },
              {
                "match_phrase": {
                  "exception.trace": "includes/profiler"
                }
              }
            ]
          }
        },
        {
          "bool": {
            "must": [
              {
                "term": {
                  "channel.keyword": "exception"
                }
              },
              {
                "match_phrase": {
                  "exception.trace": "includes/libs/Stats/"
                }
              }
            ]
          }
        },
        {
          "bool": {
            "must": [
              {
                "term": {
                  "channel.keyword": "exception"
                }
              },
              {
                "match_phrase": {
                  "exception.trace": "vendor/cssjanus/cssjanus"
                }
              }
            ]
          }
        },
        {
          "bool": {
            "must": [
              {
                "term": {
                  "channel.keyword": "exception"
                }
              },
              {
                "match_phrase": {
                  "exception.trace": "vendor/wikimedia/less.php"
                }
              }
            ]
          }
        },
        {
          "bool": {
            "must": [
              {
                "term": {
                  "channel.keyword": "exception"
                }
              },
              {
                "match_phrase": {
                  "exception.trace": "vendor/wikimedia/minify"
                }
              }
            ]
          }
        },
        {
          "bool": {
            "must": [
              {
                "term": {
                  "channel.keyword": "exception"
                }
              },
              {
                "match_phrase": {
                  "exception.trace": "vendor/wikimedia/php-session-serializer"
                }
              }
            ]
          }
        },
        {
          "bool": {
            "must": [
              {
                "term": {
                  "channel.keyword": "exception"
                }
              },
              {
                "match_phrase": {
                  "exception.trace": "vendor/wikimedia/relpath"
                }
              }
            ]
          }
        },
        {
          "bool": {
            "must": [
              {
                "term": {
                  "channel.keyword": "exception"
                }
              },
              {
                "match_phrase": {
                  "exception.trace": "vendor/wikimedia/wrappedstring"
                }
              }
            ]
          }
        },
        {
          "bool": {
            "must": [
              {
                "term": {
                  "channel.keyword": "exception"
                }
              },
              {
                "match_phrase": {
                  "exception.file": "includes/poolcounter/"
                }
              }
            ]
          }
        },
        {
          "bool": {
            "must": [
              {
                "term": {
                  "channel.keyword": "exception"
                }
              },
              {
                "match_phrase": {
                  "exception.file": "includes/objectcache/"
                }
              }
            ]
          }
        },
        {
          "bool": {
            "must": [
              {
                "term": {
                  "channel.keyword": "exception"
                }
              },
              {
                "match_phrase": {
                  "exception.file": "includes/libs/objectcache/"
                }
              }
            ]
          }
        }
      ]
    }
  }
}

(test)

Conceptually, file revisions are completely unrelated to page revisions. MediaWiki happens to generate a null page revision when a new file revision is uploaded, but AFAIK that's just for product reasons (makes the page history more helpful). I'm not sure it happens for all file changes (revert? revision delete? undelete?). Also once you move to Wikipedias, there are two potential page event streams that are associated with a file (if it exists locally, the local file page; if it doesn't exist locally, the Commons file page), which is complexity that you could hide with a file event stream but not so much with a page event stream.

A somewhat oversimplified diagram of how the cookie exchange between domains works: https://prezi.com/p/euajtznrs59r/?present=1

So until now any API involving page images broke when used with a CA token? Huh.

In T373504#10099116, @Tgr wrote:

If it doesn't occur any more

This seems like a pre-existing bug that became more severe due to stricter type hinting added in that patch. Not sure what exactly is going on - it seems the API sees the centralauthtoken URL parameter at the beginning of the request so it loads the CentralAuth session provider, but then doesn't see the parameter anymore by the point it actually starts processing the request. The first check is via WebRequest::getVal() and the second one via ApiMain::getVal() so it's presumably some sort of discrepancy between those.

If it doesn't occur any more, it's simply a side effect of deploying a change that adds a new field to a data structure stored in the session, while some live sessions were holding the old version of the data structure. (Sorry, I should have foreseen that happening.) Authentication sessions expire in five minutes so not a big deal, a few logins (up to a hundred or so, I'd guess based on the size difference between group1 and group2) will fail with a strange error message and users will have to retry. Not great but not worth an emergency patch, IMO.

Caused by rECAUe59ff145fbbb: Use TokenManager for existing token operations.

Caused by rMWcde00b558542: auth: Add AuthManagerFilterProviders hook. Not sure if it's a bug in that patch or it just exposed a bug from earlier, will look into it.

Updated some more. I think it's fixed now but will leave it open so @DAlangi_WMF
can review.

We might need to tweak what providers to filter in the future, but that will only require changing a list.

Google says

As Chrome pursues a new path for Privacy Sandbox on the web, preserving user-critical functionality that relies on third-party cookies remains a priority. For the 1% of users who have had third-party cookies restricted for testing, Chrome is extending the grace period that allows sites and services demonstrating user-facing, non-ads breakage to continue to access third-party cookies.

The origin associated with this application is currently taking advantage of the grace period and does not need to deploy deprecation trial tokens. If you have already deployed tokens, you do not need to keep them in place. We plan to maintain the grace period at least until we introduce a new experience in Chrome that lets people make an informed choice that applies across their web browsing.

Oh, right, forgot that I filed it already.

Also, locks of temporary accounts canbot be undone while blocks can. Not that it matters much, but an accidental incorrect lock will disconnect the user from their messages and notifications.

In T369467#10092251, @Krinkle wrote:

I added as-is, it would redirect straight back when already centrally logged-in. Is that correct?

@Mondo it seems after getting logged out, you logged in twice in quick succession (two minutes apart), do you remember why that was necessary? Did you get logged out on two different devices at the same time?

In T372943#10087406, @Ladsgroup wrote:

I think we do that already? Check DB errors dashboard in logstash.

In T372943#10081783, @CDanis wrote:

It would be so nice to have an x-request-id (and ideally also a traceparent) accessible in the query text inside MariaDB. As I understand we need to be careful or otherwise we'd break a bunch of aggregation that happens in Logstash and also for Information_Schema (or things that use information_schema like our slow query logger...?)

In T372702#10077189, @Tgr wrote:

maybe the crawler just switched from normal autologin URLs to top-level autologin URLs for whatever reason?

Thanks @Vestrian24Bio! Do you remember / can check the answers to the other questions by any chance?

What's the state of using JSON for metadata? If we refresh all files anyway, that would be nice to update. It would make refreshing small subsets of images much more efficient in the future since MySQL could introspect the metadata, rather than having to do string matching on PHP-serialized arrays.

With T298485: MW scripts should reload the database config fixed, runtime should be less of an issue now, but probably still shouldn't run for months uninterrupted, due to DC switchover, code changes, memory leaks etc. Also the script waits for replication but doesn't commit - it uses deferred updates, not sure how those interact with transactions in a maintenance script, so maybe not actually a problem. But replacing waitForReplication() with commitTransaction() should be harmless and at the very least make the code look safer.

Also:

• Is it happening on the same wiki where you logged in (ie. where you filled out a login form by entering your username and password) or only on other wikis?
• Are you sure you have set the "keep logged in" checkbox? If you don't do that, a login only lasting for 24 hours is normal. (More accurately, the login expires once you are inactive for half a day or so, but no sooner than 24 hours from login.)

Some details that would be helpful if you are experiencing this:

• How frequently is it happening? (One hypothesis would be that login expires after 24 hours, which is the length data is retained in the central session store. Does that sound about right?)
• Is it correlated to some action you are taking, especially visiting another wiki which you have not visited for a while?
• Which of these cookies does your browser have? centralauth_Session, centralauth_Token, <wiki name>Session, <wiki name>Token (do NOT share the values of the cookies!) When will they expire? Extra useful if you can check right when you get logged out. (You can check in your browser's developer toolbar, e.g. on Chrome it's F12 and then select the Application tab, and the Cookies menu item on the left.)
• Are you logged out on all wikis at the same time? If you use other devices (e.g. a mobile phone), are you logged out on those at the same time as well?
• If you experience getting logged out again, the time range when it happened (the time when you opened a page and were logged out, and the last time before that when you opened a page and were still logged in, as accurately as you can remember).

In T367995#10033086, @sbassett wrote:

Would all of this be ready for discussion in a couple of weeks or so?

In T371040#10016268, @Umherirrender wrote:

Between two new temp usernames a significant change in the name is at the end, so it could end up in the same gap on the unique index.
There are some browser tests failing also on fast user creation (T199393).

In T370304#10073845, @Ladsgroup wrote:

It's actually the same rows being updated and causing contention: it's updating user_editcount for the same user and it's updating category membership numbers for the same categories

As CheckUser is not installed on the beta clusters

Although that discussion is about a single user so not necessarily related.

Reported on enwiki as well: https://en.wikipedia.org/wiki/Wikipedia:Village_pump_(technical)#%22Remember_me%22_not_working_as_intended_for_me

Globally, the number of logins and similar statistics haven't changed (except for the big spike in top-level autologin but that's some sort of bot). I don't think we have merged anything interesting recently - maybe rECAUdfea2e7879dc: Split CentralAuthTokenManager from CentralAuthSessionManager, but AIUI that's too recent.

Is it happening to specific people? (Ie. the same person getting logged out several times in a row, as opposed to many people getting logged out once?) If it's happening to specific people, do they have a centralauth_Token cookie?

Note there is also https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/tools/cookiecutter-library/. Probably the same tag can be used for both.

Thanks, I see now there is https://gitlab.wikimedia.org/explore/groups to list namespaces.

Is the issue reproducible in Beta? If not, I think the path of least resistance is to book a deploy session, add some logging on one of the production debug hosts, and check via X-Wikimedia-Debug whether 1) the hook handler actually executes, 2) if not, which hook handlers execute and which don't, 3) if yes, just dump a lot of information inside the hook and hope to spot what's going wrong.

IMO a wikispore namespace would be nice if it's not too much work. (It seems Cloud VPS projects have no supergroup like toolforge-repos for Toolforge projects.)

In T363699#9983910, @Bugreporter wrote:

Next time user may directly browse the login page in shared domain without visiting Special:Userlogin in local wiki. So either we need to guarantee either the login procedure work will succeed if starting in step 4 (not having a local session), or try to create a local session (by redirect user first to local wiki then back to login domain)

In T191156#10056343, @stjn wrote:

I get the point, but I think that the responsibility of creating the tasks for something that was not broken before should lie with the person who broke it. I reported around 10 issues here already, creating a task for every single one of them is extremely tedious.

In T371977#10054698, @DavidBrooks wrote:

To the comment on breaking-or-not siteinfo, please note that the AWB break (now fixed) was due to the flag removed from userinfo. That may come to the same thing; I don't know the internals.

Packaging the vagrant box (via vagrant package) for export fails with

tgr@wikispore-test:/srv/mediawiki-vagrant$ vagrant package
==> default: Compressing container's rootfs...
Traceback (most recent call last):
...
/srv/vagrant-data/gems/2.5.5/gems/vagrant-lxc-1.4.3/lib/vagrant-lxc/driver.rb:53:in `rootfs_path': undefined method `[]' for nil:NilClass (NoMethodError)

This is apparently a known bug: https://github.com/fgrehm/vagrant-lxc/issues/475. As suggested there, replacing

config_entry = config_string.match(/^lxc\.rootfs\s+=\s+(.+)$/)[1]

with

config_entry = config_string.match(/^lxc\.rootfs.path\s+=\s+(.+)$/)[1]

in line 53 of driver.rb helps.

In T372006#10055542, @Seddon wrote:

it is possible that the discordance between job queue versions created the issue.

Fixed the older stuck renames using a script generated with

tgr@mwmaint1002:~$ sql centralauth -- --silent --raw > fix.sh <<EOF
select CONCAT( "mwscript extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=", ru_wiki, " --ignorestatus --logwiki=metawiki '",  ru_oldname, "' '", ru_newname,  "'\nsleep 120") from renameuser_status join renameuser_queue on ru_oldname = rq_name and ru_newname = rq_newname where ru_status = 'inprogress' and rq_completed_ts < '20240810000000';
EOF

(the sleep is probably not needed but we haven't tried in a long time to mass-fix many stuck renames simultaneously, so just in case)

(This is related to Cookiecutter which doesn't seem to have a dedicated Phabricator tag.)

In T372017#10052902, @bd808 wrote:

@Reedy, @Tgr, @Jdforrester-WMF: is the mw-core code removal worth this collateral damage?

The other major fallout was T372017: AWB reads old writeapi user right, so stops working on wikis when it is removed. AWB is still using SVN so that sounds like a challenge.

In T372017#10054015, @Kitayama wrote:

Site:sv.wikipedia.
Updated to 6.3.1. Created page "Wikipedia:AutoWikiBrowser/CheckPageJSON" as it didn't exist before. Now I get "Check page failed to load".

In T371977#10049882, @Krinkle wrote:

I specifically amended https://gerrit.wikimedia.org/r/c/mediawiki/core/+/392542, which removed the "writeapi" feature to not break the stable API for "siteinfo", by leaving this behind as constant and non-deprecated boolean field.

In T294397#10049715, @bjh21 wrote:

A quick note that this change caused some disruption on Commons because four gadget definitions were conditional on the writeapi right. This has been fixed on Commons now, but similar problems might affect others.