User Details
- User Since
- Oct 3 2014, 5:57 AM (518 w, 6 d)
- Availability
- Available
- LDAP User
- Giuseppe Lavagetto
- MediaWiki User
- GLavagetto (WMF) [ Global Accounts ]
Yesterday
If you want to use change dispatching from wikidata, which is a proven mechanism, then you'd probably be better off keeping the lexeme data within the wiki structure, and pass it to the orchestrator as a parameter of the function. That would allow you to re-parse the wiki page using the normal mechanism that's already established for wikis, and solve a lot of problems for you (including I think how to fetch the items, but I'd let @LucasWerkmeister comment on that).
Special pages are not cached at the edge, so there is no caching for that url, independently of indicating a revision or not:
$ curl -Is https://www.wikidata.org/wiki/Special:EntityData/Q42.json | grep cache-control cache-control: private, s-maxage=0, max-age=0, must-revalidate $ curl -Is https://www.wikidata.org/wiki/Special:EntityData/Q42.json?revision=1600533266 | grep cache-control cache-control: private, s-maxage=0, max-age=0, must-revalidate
Tue, Sep 10
Mon, Sep 9
For the record, the cause was a relatively aggressive crawler filling up all resources. While we've rate-limited this bot, I think we should use robots.txt to ban crawling from most pages.
Thu, Sep 5
I'm finally seeing bounces get processed in logstash https://logstash.wikimedia.org/goto/3d34190bb82088f19669b0c66331d7c9
I still see the errors in the logs, and it's baffling. In fact, I've tried the command now listed in exim's configuration:
The check is done using Webrequest::getIP() which uses REMOTE_ADDR as a source for the address, and then overrides that using X-Forwarded-For only for trusted proxies.
Fri, Aug 30
Let me add some prospect, as I've heard people are complaining about this.
Wed, Aug 28
Tue, Aug 27
Mon, Aug 26
For the record, the reason we wanted to support large file uploads was not to worsen the performance of upload-by-url, which has since been fixed by making the process asynchronous. Better performance handling large files would be welcome, though.
Aug 8 2024
I don't think we really need this, because I can't remember one episode in which this check has acutally fired and it wasn't expected/a false positive.
Now prometheus only reports scraping the correct ports https://prometheus-eqiad.wikimedia.org/k8s/targets?scrapePool=k8s-pods-metrics&search=statsd-exporter
Aug 6 2024
I think we need to be able to pass to the release script a set of base images for mediawiki and web, and build the final images for each of those pairs.
Aug 5 2024
I should add, wgLocalHTTPProxy and all those mechanisms are hacks, and if we want to do such things the right way, we should have a proper configuration table for read-only and read-write paths for different domains, and then instruct the application writers to use one or the other. I'd avoid replicating the hack we have at the traffic layer, because say one request gets erroneously sent to the wrong datacenter - now instead of paying 30 ms in total of penalty, we'll pay 30 ms per query to the mysql master.
There's quite a bit of incorrect information in the wall of text above, but to actually keep it short:
We went with just pointing to the read-write api because there's no system, within MediaWiki, to split requests between write and read and we didn't want to add ad-hoc logic to the service mesh just for that.
Jul 30 2024
Thanks for the thorough explanation! I know the traffic folks were a bit worried about controlling stick tables from requestctl but I think this format is ok.
Jul 29 2024
While I do get why this solution seems attractive, I don't think it's really viable at any level.
To remove even more doubt, can you make an example of what you'd expect in haproxy dsl with the following rule:
Before I think of implementations, I'd like to understand better what you want to offer:
Jul 25 2024
Jul 24 2024
@bd808 to add to your point: We're near another inflection point for deployment-prep: soon the puppet code for configuring mediawiki in deployment-prep is going to be dismissed and not used in the production environment anymore. By the end of the calendar year, we count on having moved all of production (hopefully, all on kubernetes) to using php 8.x.
Jul 23 2024
Jul 22 2024
The SIGILL thing happened on bare metal as well, albeit quite rarely. We never properly tracked down what happened, but it seemed to have some relation to accessing the shared anonymous memory and the related semaphores, so I guess one of apcu and opcache are responsible. I'm starting to think we might need a liveness probe of some kind for the pod to depend on that?
Jul 17 2024
Some of the standard filters on the logstash dashboard didn't make me see all entries, this is the right link: https://logstash.wikimedia.org/goto/77120810b3dbe1cfe36bcf3478ebeaf9 where we can also find the request IDs reported in the bug originally. So the blast radius was quite ample, although I still think this might have been caused by the job running, I'm not 100% sure anymore.
This looks like a genuine issue of overloading and seem to be caused by MediaModerationScanFileJob. So I'd say the protection worked as intended; I am wary of having circuit breaking on shared-by-everything databases - but here it looks like the only web view that failed also did so due to MetaModeration failing.
Jul 16 2024
There's an interesting problem to manage with haproxy, which is making me think we should support a much simplified syntax.
Jul 15 2024
As @Fabfur made me notice, conditions can also be expressed inline:
Haproxy has a logic that's very different from varnish, but it should be possible to translate most of our current patterns or ipblocks to something haproxy can read.
To clarify a bit, I didn't take the route described in the task. In fact, we want:
Jul 12 2024
A couple of notes:
Jul 11 2024
Changing the priority to high as we had yet other instances where being able to limit bandwidth on uploads would've helped us.
Yesterday we merged https://gitlab.wikimedia.org/repos/sre/conftool/-/merge_requests/3 which triggered the job building the debian packages for bullseye: https://gitlab.wikimedia.org/repos/sre/conftool/-/jobs/308138
Jul 10 2024
Jul 9 2024
I don't see how the two as conflicting options, although in the case of conftool, I'd prefer to keep distributing it in production as deb packages, which makes a lot of the 'required' stages in python-release kind of not useful.
I'll boldly consider this task resolved, please reopen it if the problem is still present.
I think this is a waste of time and resources to make such an endpoint part of the rest api, including the fact it's going to get published and other clients might start using it. But more importantly, it's yet another specialized endpoint that I can't just see clearly in the same place where others are. I've learned the hard way when moving to k8s that php files under docroots often are sour surprises. Even as part of the rest api, this will at a minimum need to be restricted to just mediawiki.org (how?) and monitored somehow.
It also introduces yet another set of rewrite rules for stuff under rest.php, which we managed to avoid up to now.
Jul 8 2024
The priority of this task has become high, as doing this is currently a blocker to finishing the k8s migration of mediawiki and thus moving to php 8.x, which is desirable for various reasons and is a request from a lot of teams.
Personal opinion, we should never ever turn off DirectorySlash. It's a perfect landmine waiting to be stepped onto, apart from the other risks.
Jul 7 2024
From an SRE prespective, I'd expect us to build some automation around what addWiki.php does and probably move some of its functions outside of the realm of MediaWiki for specific things we might not want MediaWiki to be able to do (for instance: create databases and/or elasticsearch indexes).
Jul 2 2024
https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1051428 and followups should fix the issue
This should be solved with this morning's release.
This should be solved with the deployment in production.
Whenever you find a new occurence of a bug after months the original one has been resolved, please open a new task. The origin of your problem is most probably very different than the one we solved here.
Jun 28 2024
Jun 27 2024
Uh this task was solved on that day, not sure why I forgot to close it. Sorry Andre if this messes with your UBN stats :)
It is pretty clear to me that the only way to have fair load balancing with maglev is if we do the consistent hashing using the remote port as well.
I'd go ahead and take a step back: why do we need to switch to IPIP encapsulation for backend services?
Jun 26 2024
The degradation seems to have started around midnight between June 18th and June 19th.