If you want to use change dispatching from wikidata, which is a proven mechanism, then you'd probably be better off keeping the lexeme data within the wiki structure, and pass it to the orchestrator as a parameter of the function. That would allow you to re-parse the wiki page using the normal mechanism that's already established for wikis, and solve a lot of problems for you (including I think how to fetch the items, but I'd let @LucasWerkmeister comment on that).

In T368654#9942854, @DMartin-WMF wrote:

In Wikidata:Data_access, it says:

The following URL formats are used by the user interface and by the query service updater, respectively, so if you use one of the same URL formats there’s a good chance you’ll get faster (cached) responses:

https://www.wikidata.org/wiki/Special:EntityData/Q42.json?revision=1600533266 (JSON)

Does that mean we would have to specify a specific revision to get the benefit of caching?

Special pages are not cached at the edge, so there is no caching for that url, independently of indicating a revision or not:

$ curl -Is https://www.wikidata.org/wiki/Special:EntityData/Q42.json  | grep cache-control
cache-control: private, s-maxage=0, max-age=0, must-revalidate
$ curl -Is https://www.wikidata.org/wiki/Special:EntityData/Q42.json?revision=1600533266  | grep cache-control
cache-control: private, s-maxage=0, max-age=0, must-revalidate

In T374394#10133179, @Ladsgroup wrote:

In T374394#10132257, @jeremyb wrote:

That's not possible, the whole point of the API query is to decide whether or not to show the banner. (on enwiki the categories I'm filtering on for article (ns0) views are on the corresponding Talk. (ns1) so they're also not available in wgCategories.) I was worried more about performance for the API requests. I tested pretty thoroughly on a few devices so I was less worried about JS errors but of course WP has an enormous variety of clients.

CN banners work on a different level, one API request might be fast but if you add it to let's say enwiki CN, you will trigger a flood of several thousand requests a second every second. Reaching billions more requests to production in a day. That can easily bring down everything specially since API requests don't get CDN cached.

If something is not possible natively in CN, it's better not to circumvent it by API calls.

For the record, the cause was a relatively aggressive crawler filling up all resources. While we've rate-limited this bot, I think we should use robots.txt to ban crawling from most pages.

I'm finally seeing bounces get processed in logstash https://logstash.wikimedia.org/goto/3d34190bb82088f19669b0c66331d7c9

I still see the errors in the logs, and it's baffling. In fact, I've tried the command now listed in exim's configuration:

The check is done using Webrequest::getIP() which uses REMOTE_ADDR as a source for the address, and then overrides that using X-Forwarded-For only for trusted proxies.

Let me add some prospect, as I've heard people are complaining about this.

In T371633#10102614, @ItamarWMDE wrote:

Is this somehow related to T362084: [SW] [WBQC] shellbox-constraints returning 500 on preg_match error?

For the record, the reason we wanted to support large file uploads was not to worsen the performance of upload-by-url, which has since been fixed by making the process asynchronous. Better performance handling large files would be welcome, though.

I don't think we really need this, because I can't remember one episode in which this check has acutally fired and it wasn't expected/a false positive.

Now prometheus only reports scraping the correct ports https://prometheus-eqiad.wikimedia.org/k8s/targets?scrapePool=k8s-pods-metrics&search=statsd-exporter

In T371885#10049340, @Scott_French wrote:

Edit: Whoops, I completely missed T371885#10048618 onward before posting this. In any case, question still stands re: the annotation :)

For the endpoints marked down: it looks as if prometheus is scraping both container ports - i.e., 9102 (correct) and 9125 (statsd listen port, incorrect).

Not sure if that could somehow cause problems like those described in the task description, but it would at least explain T371885#10048571.

I wonder if we need to add an explicit prometheus.io/port annotation to ensure only 9102 is scraped?

I think we need to be able to pass to the release script a set of base images for mediawiki and web, and build the final images for each of those pairs.

I should add, wgLocalHTTPProxy and all those mechanisms are hacks, and if we want to do such things the right way, we should have a proper configuration table for read-only and read-write paths for different domains, and then instruct the application writers to use one or the other. I'd avoid replicating the hack we have at the traffic layer, because say one request gets erroneously sent to the wrong datacenter - now instead of paying 30 ms in total of penalty, we'll pay 30 ms per query to the mysql master.

There's quite a bit of incorrect information in the wall of text above, but to actually keep it short:
We went with just pointing to the read-write api because there's no system, within MediaWiki, to split requests between write and read and we didn't want to add ad-hoc logic to the service mesh just for that.

Thanks for the thorough explanation! I know the traffic folks were a bit worried about controlling stick tables from requestctl but I think this format is ok.

While I do get why this solution seems attractive, I don't think it's really viable at any level.

To remove even more doubt, can you make an example of what you'd expect in haproxy dsl with the following rule:

Before I think of implementations, I'd like to understand better what you want to offer:

In T215217#10010991, @bd808 wrote:

I have been thinking about this general problem as well this quarter. I am interested in putting a Kubernetes cluster into deployment-prep, possibly by using Magnum and OpenTofu to provision the service. In theory we could then get folks to add config for this new environment to their Helm charts for MediaWiki and other things that are hosted via Kubernetes in the production realm.

In T370808#10009552, @Volans wrote:

@joanna_borun I think this should probably be discussed in the next SRE I/F meeting for approval.

I can confirm that with our account we can access the form linked above. If approved I'll paste here all the fields required to submit a request to register the bot.

@bd808 to add to your point: We're near another inflection point for deployment-prep: soon the puppet code for configuring mediawiki in deployment-prep is going to be dismissed and not used in the production environment anymore. By the end of the calendar year, we count on having moved all of production (hopefully, all on kubernetes) to using php 8.x.

In T369606#9985617, @CDanis wrote:

As @Fabfur points out, in haproxy 3.0+ (but not haproxy 2.8.x) we have the option of evaluating many ACLs together with negation, as part of a fetching samples.

In T352650#10002138, @Milimetric wrote:

Sorry to ask this very basic question, but I found a bunch of others didn't know: how exactly is Dumps blocking the php 8 upgrade? Like, if we leave everything exactly as-is and just upgrade PHP, would it not run the way it is currently set up? On surface I see no big difference between the current setup and a containerized MW running on the same servers, so I'm curious about the nuance I'm missing here.

The SIGILL thing happened on bare metal as well, albeit quite rarely. We never properly tracked down what happened, but it seemed to have some relation to accessing the shared anonymous memory and the related semaphores, so I guess one of apcu and opcache are responsible. I'm starting to think we might need a liveness probe of some kind for the pod to depend on that?

Some of the standard filters on the logstash dashboard didn't make me see all entries, this is the right link: https://logstash.wikimedia.org/goto/77120810b3dbe1cfe36bcf3478ebeaf9 where we can also find the request IDs reported in the bug originally. So the blast radius was quite ample, although I still think this might have been caused by the job running, I'm not 100% sure anymore.

This looks like a genuine issue of overloading and seem to be caused by MediaModerationScanFileJob. So I'd say the protection worked as intended; I am wary of having circuit breaking on shared-by-everything databases - but here it looks like the only web view that failed also did so due to MetaModeration failing.

There's an interesting problem to manage with haproxy, which is making me think we should support a much simplified syntax.

As @Fabfur made me notice, conditions can also be expressed inline:

Haproxy has a logic that's very different from varnish, but it should be possible to translate most of our current patterns or ipblocks to something haproxy can read.

To clarify a bit, I didn't take the route described in the task. In fact, we want:

A couple of notes:

In T317794#8289928, @jbond wrote:

While implementing the the dry-run version of the current hotlinking patch i found out that its not possible to update the resp object in the vcl_hit hook. As such its not possible to update the X-Analytics header with the current suggested approach. This impacts two aspects of the design

Changing the priority to high as we had yet other instances where being able to limit bandwidth on uploads would've helped us.

Yesterday we merged https://gitlab.wikimedia.org/repos/sre/conftool/-/merge_requests/3 which triggered the job building the debian packages for bullseye: https://gitlab.wikimedia.org/repos/sre/conftool/-/jobs/308138

I don't see how the two as conflicting options, although in the case of conftool, I'd prefer to keep distributing it in production as deb packages, which makes a lot of the 'required' stages in python-release kind of not useful.

I'll boldly consider this task resolved, please reopen it if the problem is still present.

I think this is a waste of time and resources to make such an endpoint part of the rest api, including the fact it's going to get published and other clients might start using it. But more importantly, it's yet another specialized endpoint that I can't just see clearly in the same place where others are. I've learned the hard way when moving to k8s that php files under docroots often are sour surprises. Even as part of the rest api, this will at a minimum need to be restricted to just mediawiki.org (how?) and monitored somehow.
It also introduces yet another set of rewrite rules for stuff under rest.php, which we managed to avoid up to now.

The priority of this task has become high, as doing this is currently a blocker to finishing the k8s migration of mediawiki and thus moving to php 8.x, which is desirable for various reasons and is a request from a lot of teams.

Personal opinion, we should never ever turn off DirectorySlash. It's a perfect landmine waiting to be stepped onto, apart from the other risks.

From an SRE prespective, I'd expect us to build some automation around what addWiki.php does and probably move some of its functions outside of the realm of MediaWiki for specific things we might not want MediaWiki to be able to do (for instance: create databases and/or elasticsearch indexes).

https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1051428 and followups should fix the issue

This should be solved with this morning's release.

This should be solved with the deployment in production.

Whenever you find a new occurence of a bug after months the original one has been resolved, please open a new task. The origin of your problem is most probably very different than the one we solved here.

In T368544#9930822, @Vgutierrez wrote:

theoretically speaking we could keep low-traffic on liberica/IPVS (instead of liberica/Katran) to be able to get rid of pybal entirely. Besides k8s based services we have some other services on low-traffic that need load balancing AFAIK.

But even if we stay on Liberica/IPVS for low-traffic, having the ability of running healthchecks properly, using the same network path as incoming requests would be a nice benefit for low-traffic services.

Uh this task was solved on that day, not sure why I forgot to close it. Sorry Andre if this messes with your UBN stats :)

In T368544#9930740, @Vgutierrez wrote:

In T368544#9930584, @Joe wrote:

I'd go ahead and take a step back: why do we need to switch to IPIP encapsulation for backend services?

Is there a compelling reason why it's better than our current solution?

I believe we wanna move away from all the VXLAN setup and stop requiring L2 connectivity between load balancers and realservers.

It is pretty clear to me that the only way to have fair load balancing with maglev is if we do the consistent hashing using the remote port as well.

I'd go ahead and take a step back: why do we need to switch to IPIP encapsulation for backend services?

In T364400#9780622, @BBlack wrote:

In T364400#9779996, @hnowlan wrote:

Could we implement this remapping at the ATS layer rather than the Apache one, in a manner that would mean that when we need to cache we only need to store each effective URL once?

It would be preferable to not do so. The caching gains would be minimal, but more importantly: we hope to minimize the details of the application layer that are spread into the cache configuration (there will always be necessary cases, but the more we avoid it, the easier things are in the future).

The degradation seems to have started around midnight between June 18th and June 19th.