Svoboda | Graniru | BBC Russia | Golosameriki | Facebook
BBC RussianHomePhabricator
Log In
Maniphest T364400

map the /api/ prefix to /w/rest.php
Open, Stalled, HighPublic

Description

Currently, we are exposing REST APIs for each wiki from two pathes:

  • /api/rest_v1/ for APIs handled by standalone services (restbase and otherwise).
  • /w/rest.php/ for APIs handled by MediaWiki (core and extensions)

We should consolidate the two, so clients do not have to be aware how an API is implemented, and we are free to change how it is implemented without breaking clients. Also, shared prefix for all REST APIsmakes configuration of the edge caches more straight forward.

Proposal

This can be achived by introducing a rewrite rule into the Apache config that maps /api/ to /w/rest.php exactly in the same way we map /wiki/ to /w/index.php.

In the ATS layer, we need a rule that will send traffic for /api/ to MW app servers in the API cluster. However, traffic for the
/api/rest_v1/ prefix still needs to be sent to RESTbase / REST Gateway.

Once the routing rules are in place, the $wgRestPath setting can be changed to /api/, analogous to $wgArticlePath. But that's outside of the scoep iof this ticket, and needs to be announced widely.

Impact

APIs that are currently exposed under /w/rest.php will also become availabe under /api/, for example:

  • /w/rest.php/v1/page/{title}/html will become available under /api/v1/page/{title}/html.
  • /w/rest.php/oauth2/authorize will become available under /api/oauth2/authorize.
  • /w/rest.php/ipinfo/v0/revision/{id} will become available under /api/ipinfo/v0/revision/{id}.

Note that we will have /api/rest_v1/ prefix side by side with core's /api/v1/ prefix, which is slightly confusing. The idea is to move APIs out of both generic prefixes over time:

  • With the sunsetting of RESTbase, we can move APIs out of the /api/rest_v1/ prefix to dedicated prefixes, e.g. /api/citation.v1/ instead of /api/rest_v1/data/citation/.
  • With the introducion of modules into the MediaWiki REST framework, we can move APIs out of the /api/v1/ prefix to dedicated prefixes, e.g. /api/content.v1/page/ instead of /api/v1/page/.

Future

If we have a need to route different paths under /api/ to different services (beyond rest_v1) , we can route all requests for the /api/ prefix through an API gateway. This would also provide unified intrumentation for all REST endpoints.

Aproval

This idea has been floated and discussed for a while. This task should be used to discuss any remaining issuesd, and document consensus.

References

Event Timeline

Restricted Application added a subscriber: Aklapper. ยท View Herald TranscriptMay 7 2024, 3:59 PM

This seems like a pretty reasonable idea to me. Could we implement this remapping at the ATS layer rather than the Apache one, in a manner that would mean that when we need to cache we only need to store each effective URL once?

are /api/v1/page/{title}/html, /api/oauth2/authorize and /api/ipinfo/v0/revision/{id} all new API paths or existing ones?

Could we implement this remapping at the ATS layer rather than the Apache one, in a manner that would mean that when we need to cache we only need to store each effective URL once?

It would be preferable to not do so. The caching gains would be minimal, but more importantly: we hope to minimize the details of the application layer that are spread into the cache configuration (there will always be necessary cases, but the more we avoid it, the easier things are in the future).

This seems like a pretty reasonable idea to me. Could we implement this remapping at the ATS layer rather than the Apache one, in a manner that would mean that when we need to cache we only need to store each effective URL once?

I hope we can stop serving from the old prefix in the not-too-far-away future. We could turn it into a 308.

In any case, from the MW perspective, I don't care where the mapping is. I just recvalled Giuseppe saying that he'd prefer to have it in Apache. And as far as I know, that's where the mapping for index.php is. It's probably a good idea to keep them side by side.

are /api/v1/page/{title}/html, /api/oauth2/authorize and /api/ipinfo/v0/revision/{id} all new API paths or existing ones?

The endpoints all exist, but currently under /w/rest.php, not under /api/.

@daniel thank you for al the prep work! This is good to go :-)

Could we implement this remapping at the ATS layer rather than the Apache one, in a manner that would mean that when we need to cache we only need to store each effective URL once?

It would be preferable to not do so. The caching gains would be minimal, but more importantly: we hope to minimize the details of the application layer that are spread into the cache configuration (there will always be necessary cases, but the more we avoid it, the easier things are in the future).

Even more to @BBlack's comment, I would just have apache funnel anything under /api it receives to an endpoint in mediawiki, and do routing there.

Even more to @BBlack's comment, I would just have apache funnel anything under /api it receives to an endpoint in mediawiki, and do routing there.

For everything, including /api/rest_v1? I was thinking that we should keep routing /api/rest_v1 directly to ApiGateway/RestGateway. Doing that routing in MediaWiki will require some prep work, and would likely come with a latency panelty for the PCS endpoints.

But everything under /api outside of /api/rest_v1 should indeed just go to /w/rest.php.

This is unblocked from our side - which team is going to take care of creating the routing rule? Traffic or serviceops?

Please hold for a final round of alignment.

Change #1052745 had a related patch set uploaded (by Alexandros Kosiaris; author: Alexandros Kosiaris):

[operations/puppet@production] ats: Route /api/ to /w/rest.php on mw-api-ext

https://gerrit.wikimedia.org/r/1052745

Could we implement this remapping at the ATS layer rather than the Apache one, in a manner that would mean that when we need to cache we only need to store each effective URL once?

It would be preferable to not do so. The caching gains would be minimal, but more importantly: we hope to minimize the details of the application layer that are spread into the cache configuration (there will always be necessary cases, but the more we avoid it, the easier things are in the future).

Even more to @BBlack's comment, I would just have apache funnel anything under /api it receives to an endpoint in mediawiki, and do routing there.

At least if we want to keep our separation of concerns (visible end user traffic vs API traffic), we 'll need 1 rule in ATS to at least route to mw-api-ext requests for /api/, there is not way we can do that in MediaWiki.

1 patch doing the above uploaded at https://gerrit.wikimedia.org/r/c/operations/puppet/+/1052745, let me know what you think.

Change #1052745 merged by Alexandros Kosiaris:

[operations/puppet@production] ats: Route /api/ to /w/rest.php on mw-api-ext

https://gerrit.wikimedia.org/r/1052745

Change #1070032 had a related patch set uploaded (by Clรฉment Goubert; author: Clรฉment Goubert):

[operations/puppet@production] trafficserver: Fix /w/rest.php and /api/ regex_map

https://gerrit.wikimedia.org/r/1070032

Mentioned in SAL (#wikimedia-operations) [2024-09-02T14:58:55Z] <claime> Disabling puppet on O:cache::text to merge 1070032 - T364400

Change #1070032 merged by Clรฉment Goubert:

[operations/puppet@production] trafficserver: Fix /w/rest.php and /api/ regex_map

https://gerrit.wikimedia.org/r/1070032

Mentioned in SAL (#wikimedia-operations) [2024-09-02T15:02:54Z] <claime> Enabling puppet on cp2027.codfw.wmnet to test 1070032 - T364400

Mentioned in SAL (#wikimedia-operations) [2024-09-02T15:39:18Z] <claime> Enabling puppet on O:cache::text for 1070032 - T364400

Our ATS configuration was wrong and as such, traffic was being sent to the mw-web cluster instead of mw-api-ext. This had functionally no impact as the clusters are identical except for the amount of resources they have, but it is now fixed.

Change #1070274 had a related patch set uploaded (by Alexandros Kosiaris; author: Alexandros Kosiaris):

[operations/puppet@production] ats: Fix issue with /api/ pointing to /w/rest.php

https://gerrit.wikimedia.org/r/1070274

Change #1070274 merged by Alexandros Kosiaris:

[operations/puppet@production] ats: Fix issue with /api/ pointing to /w/rest.php

https://gerrit.wikimedia.org/r/1070274

@daniel, We 've already setup the /api/ part to point to mw-ext-api, which has introduced a minor regression per T373998. We can always rollback this, but I was wondering if we have enough consensus to proceed in the path of having /api/ be rewritten in MediaWiki to /w/rest.php

I was wondering if we have enough consensus to proceed in the path of having /api/ be rewritten in MediaWiki to /w/rest.php

We do not have a consensus at this time. Conversations remain underway, but we have no definite time frame in which we can guarantee an answer. Might be soon, might not.

Sorry, I wish we had a different answer (one way or another).

I was wondering if we have enough consensus to proceed in the path of having /api/ be rewritten in MediaWiki to /w/rest.php

We do not have a consensus at this time. Conversations remain underway, but we have no definite time frame in which we can guarantee an answer. Might be soon, might not.

Sorry, I wish we had a different answer (one way or another).

Thanks for the update @BPirkle. In hindsight, I was probably too eager to see /api becoming a thing.

On our part, we 'll revert the CDN part of the changes then until we get consensus.

Change #1071229 had a related patch set uploaded (by Alexandros Kosiaris; author: Alexandros Kosiaris):

[operations/puppet@production] ats: Revert the /api/ changes on the CDN side

https://gerrit.wikimedia.org/r/1071229

Change #1071229 merged by Alexandros Kosiaris:

[operations/puppet@production] ats: Revert the /api/ changes on the CDN side

https://gerrit.wikimedia.org/r/1071229

akosiaris changed the task status from Open to Stalled.Fri, Sep 6, 3:13 PM

Change reverted, I 'll switch this to stalled until we have enough consensus. Let us know of any updates or if you need help with anything.