I cleared the RESTBase cache that mobile-html endpoint relies on as described here.

The issue should be gone by now, could you please confirm and resolve the ticket?

Assuming this task is about the Mobile-Content-Service code project, hence adding that project tag so other people who don't know or don't care about team tags can also find this task when searching via projects. (Please set appropriate project tags when possible. Thanks!)

Thanks @Aklapper these project tags are confusing, I'm changing to PCS which is more appropriate.

@MadriCR a total cache refresh is possible but it will take some time and coordination to not create an incident due to increased load on DBs. I'll coordinate that change ASAP, thanks for reaching out.

By looking at the objects in cache for one affected page, page/html has the correct content but page/summary does not, even if their objects have been stored in cache *after* the last edit to the template. Specifically these objject in cache were saved 0.3 seconds apart:

cassandra@cqlsh:wikipedia_T_page__summary> select dateof(tid) from "wikipedia_T_parsoidphp".data where "_domain" = 'es.wikipedia.org' AND key = 'James_Buchanan';

 system.dateof(tid)
---------------------------------
 2023-01-28 06:13:55.394000+0000

(1 rows)
cassandra@cqlsh:wikipedia_T_page__summary> select dateof(tid) from data where "_domain" = 'es.wikipedia.org' AND key = 'James_Buchanan';

 system.dateof(tid)
---------------------------------
 2023-01-28 06:13:55.671000+0000

(1 rows)

Now, re-rendering that page in restbase takes more than 0.3 seconds, so it's possible we have a race condition here so that:

• Revert of the vandalism happens
• change-propagation sends a request to re-render page/html
• change-propagation sends a request to re-render page/summary
  • restbase calls PCS, which in turns calls restbase to get page/html
  • if the new version of page/html wasn't ready, PCS gets served the old, vandalized version

This should probably verified in the code of restbase, if we do remove the old cached version or we just replace it once the new one is ready, this is likely what happened.

If I am right, a null edit to the template should be enough to fix the issue. @MadriCR would you be able to do a null edit to the template?

Done; please let me know.

In T271184#8585977, @Joe wrote:

By looking at the objects in cache for one affected page, page/html has the correct content but page/summary does not, even if their objects have been stored in cache *after* the last edit to the template. Specifically these objject in cache were saved 0.3 seconds apart:

cassandra@cqlsh:wikipedia_T_page__summary> select dateof(tid) from "wikipedia_T_parsoidphp".data where "_domain" = 'es.wikipedia.org' AND key = 'James_Buchanan';

 system.dateof(tid)
---------------------------------
 2023-01-28 06:13:55.394000+0000

(1 rows)
cassandra@cqlsh:wikipedia_T_page__summary> select dateof(tid) from data where "_domain" = 'es.wikipedia.org' AND key = 'James_Buchanan';

 system.dateof(tid)
---------------------------------
 2023-01-28 06:13:55.671000+0000

(1 rows)

Now, re-rendering that page in restbase takes more than 0.3 seconds, so it's possible we have a race condition here so that:

• Revert of the vandalism happens
• change-propagation sends a request to re-render page/html
• change-propagation sends a request to re-render page/summary
  • restbase calls PCS, which in turns calls restbase to get page/html
  • if the new version of page/html wasn't ready, PCS gets served the old, vandalized version

This should probably verified in the code of restbase, if we do remove the old cached version or we just replace it once the new one is ready, this is likely what happened.

If I am right, a null edit to the template should be enough to fix the issue. @MadriCR would you be able to do a null edit to the template?

In T271184#8585983, @MadriCR wrote:

Done; please let me know.

I would expect this to take anywhere between a few minutes and a few hours to propagate fully, depending on how much the jobqueue is busy, but it was our best solution to try to fix the issue consistently, at least for this time.

I think this is an issue with Page-Previews / PageImages - I've filed T328880: Clear abusive image from PagePreviews/Images on eswiki accordingly per https://www.mediawiki.org/wiki/Extension:PageImages#How_can_I_purge_a_bad_image%3F since Purging will not work. For larger emergencies please file a Phabricator ticket.

Closing after T329532