Svoboda | Graniru | BBC Russia | Golosameriki | Facebook
BBC RussianHomePhabricator
Log In
Create Task
Maniphest T184759

Sqoop cu_changes table for geowiki
Closed, ResolvedPublic21 Estimated Story Points
Actions

Description

Sqoop https://www.mediawiki.org/wiki/Extension:CheckUser/cu_changes_table

This sqoop is a bit different from the "grab everything and overwrite" approach of our current sqoop script. We need to make sure we have a full month of raw cu_changes data, process it, and then delete it. According to Asaf, the filter we need is:

(scope creep: the patch that resolves this ended up fixing the sqoop script to handle logging and error reporting better, and do more robust success flag setting)

Details

SubjectRepoBranchLines +/-
Update the cron command with the new sqoop scriptoperations/puppetproduction+30 -12
Refactor sqoop jobs and add from_timestampanalytics/refinerymaster+675 -444
Customize query in gerrit

Related Objects
Search...

Event Timeline

• Nuria triaged this task as High priority.Jan 11 2018, 6:47 PM
• Nuria created this task.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 11 2018, 6:47 PM
Milimetric renamed this task from Scoop https://www.mediawiki.org/wiki/Extension:CheckUser/cu_changes_table to Sqoop cu_changes table for geowiki.Jan 12 2018, 2:39 PM
Milimetric updated the task description. (Show Details)
Milimetric moved this task from Next Up to In Progress on the Analytics-Kanban board.Jan 12 2018, 4:18 PM
• Nuria set the point value for this task to 5.Jan 12 2018, 5:07 PM
Milimetric added a comment.Jan 12 2018, 5:40 PM

Some thoughts from post-standup:

  • snapshot partition name doesn't apply to this use case, change it to like temporary or something like that
  • sqoop only one month of data
  • after processing, delete the second to last month
Milimetric moved this task from In Progress to Paused on the Analytics-Kanban board.Feb 5 2018, 3:23 PM
Milimetric moved this task from Paused to In Progress on the Analytics-Kanban board.Feb 6 2018, 1:44 PM
Milimetric changed the point value for this task from 5 to 13.Feb 11 2018, 2:00 PM

(points increased a bit because I have to split this code up, but on track for being done early this week)

gerritbot subscribed.Feb 16 2018, 6:56 PM

Change 408848 had a related patch set uploaded (by Milimetric; owner: Milimetric):
[analytics/refinery@master] [WIP] Saving in case laptop catches on fire

https://gerrit.wikimedia.org/r/408848

gerritbot added a project: Patch-For-Review.Feb 16 2018, 6:56 PM
Milimetric moved this task from In Progress to In Code Review on the Analytics-Kanban board.Feb 22 2018, 5:04 PM
gerritbot added a comment.Feb 28 2018, 3:55 AM

Change 415217 had a related patch set uploaded (by Milimetric; owner: Milimetric):
[operations/puppet@production] [WIP] Merge requires coordination

https://gerrit.wikimedia.org/r/415217

gerritbot added a comment.Feb 28 2018, 3:43 PM

Change 408848 merged by Milimetric:
[analytics/refinery@master] Refactor sqoop jobs and add from_timestamp

https://gerrit.wikimedia.org/r/408848

gerritbot added a comment.Feb 28 2018, 10:10 PM

Change 415217 merged by Ottomata:
[operations/puppet@production] Update the cron command with the new sqoop script

https://gerrit.wikimedia.org/r/415217

Milimetric moved this task from In Code Review to Done on the Analytics-Kanban board.Feb 28 2018, 10:18 PM
Milimetric updated the task description. (Show Details)
Milimetric changed the point value for this task from 13 to 21.
• Nuria closed this task as Resolved.Mar 2 2018, 9:03 PM
• Nuria closed subtask T188113: Oozie job to compute geowiki on top of sqooped data as Resolved.Apr 12 2018, 10:08 PM
• 238482n375 removed Milimetric as the assignee of this task.Jun 15 2018, 8:04 AM
• 238482n375 lowered the priority of this task from High to Lowest.
• 238482n375 moved this task from Done to In Code Review on the Analytics-Kanban board.
• 238482n375 added projects: acl*security, Wikimedia-VE-Campaigns (S2-2018), Scap (Scap3-Adoption-Phase2), AbuseFilter, Data-release, Hashtags, LabsDB-Auditor, Ladies-That-FOSS-MediaWiki, Language-2018-Apr-June, Language-2018-Jan-Mar, HHVM, HAWelcome.
• 238482n375 edited subscribers, added: Milimetric, • 238482n375; removed: Aklapper.

SG9tZVBoYWJyaWNhdG9yCk5vIG1lc3NhZ2VzLiBObyBub3RpZmljYXRpb25zLgoKICAgIFNlYXJjaAoKQ3JlYXRlIFRhc2sKTWFuaXBoZXN0ClQxOTcyODEKRml4IGZhaWxpbmcgd2VicmVxdWVzdCBob3VycyAodXBsb2FkIGFuZCB0ZXh0IDIwMTgtMDYtMTQtMTEpCk9wZW4sIE5lZWRzIFRyaWFnZVB1YmxpYwoKICAgIEVkaXQgVGFzawogICAgRWRpdCBSZWxhdGVkIFRhc2tzLi4uCiAgICBFZGl0IFJlbGF0ZWQgT2JqZWN0cy4uLgogICAgUHJvdGVjdCBhcyBzZWN1cml0eSBpc3N1ZQoKICAgIE11dGUgTm90aWZpY2F0aW9ucwogICAgQXdhcmQgVG9rZW4KICAgIEZsYWcgRm9yIExhdGVyCgpUYWdzCgogICAgQW5hbHl0aWNzLUthbmJhbiAoSW4gUHJvZ3Jlc3MpCgpTdWJzY3JpYmVycwpBa2xhcHBlciwgSkFsbGVtYW5kb3UKQXNzaWduZWQgVG8KSkFsbGVtYW5kb3UKQXV0aG9yZWQgQnkKSkFsbGVtYW5kb3UsIEZyaSwgSnVuIDE1CkRlc2NyaXB0aW9uCgpPb3ppZSBqb2JzIGhhdmUgYmVlbiBmYWlsaW5nIGF0IGxlYXN0IGEgZmV3IHRpbWVzIGVhY2guIE1vcmUgaW52ZXN0aWdhdGlvbiBuZWVkZWQuCkpBbGxlbWFuZG91IGNyZWF0ZWQgdGhpcyB0YXNrLkZyaSwgSnVuIDE1LCA3OjIxIEFNCkhlcmFsZCBhZGRlZCBhIHN1YnNjcmliZXI6IEFrbGFwcGVyLiC3IFZpZXcgSGVyYWxkIFRyYW5zY3JpcHRGcmksIEp1biAxNSwgNzoyMSBBTQpKQWxsZW1hbmRvdSBjbGFpbWVkIHRoaXMgdGFzay5GcmksIEp1biAxNSwgNzoyMiBBTQpKQWxsZW1hbmRvdSB1cGRhdGVkIHRoZSB0YXNrIGRlc2NyaXB0aW9uLiAoU2hvdyBEZXRhaWxzKQpKQWxsZW1hbmRvdSBhZGRlZCBhIHByb2plY3Q6IEFuYWx5dGljcy1LYW5iYW4uCkpBbGxlbWFuZG91IG1vdmVkIHRoaXMgdGFzayBmcm9tIE5leHQgVXAgdG8gSW4gUHJvZ3Jlc3Mgb24gdGhlIEFuYWx5dGljcy1LYW5iYW4gYm9hcmQuCkNoYW5nZSBTdWJzY3JpYmVycwpDaGFuZ2UgUHJpb3JpdHkKQXNzaWduIC8gQ2xhaW0KTW92ZSBvbiBXb3JrYm9hcmQKQ2hhbmdlIFByb2plY3QgVGFncwpBbmFseXRpY3MtS2FuYmFuCtcKU2VjdXJpdHkK1wpXaWtpbWVkaWEtVkUtQ2FtcGFpZ25zIChTMi0yMDE4KQrXClNjYXAK1wpTY2FwIChTY2FwMy1BZG9wdGlvbi1QaGFzZTIpCtcKQWJ1c2VGaWx0ZXIK1wpEYXRhLXJlbGVhc2UK1wpIYXNodGFncwrXCkxhYnNEQi1BdWRpdG9yCtcKTGFkaWVzLVRoYXQtRk9TUy1NZWRpYVdpa2kK1wpMYW5ndWFnZS0yMDE4LUFwci1KdW5lCtcKTGFuZ3VhZ2UtMjAxOC1KYW4tTWFyCtcKSEhWTQrXCkhBV2VsY29tZQrXCkJvbGQKSXRhbGljcwpNb25vc3BhY2VkCkxpbmsKQnVsbGV0ZWQgTGlzdApOdW1iZXJlZCBMaXN0CkNvZGUgQmxvY2sKUXVvdGUKVGFibGUKVXBsb2FkIEZpbGUKTWVtZQpQcmV2aWV3CkhlbHAKRnVsbHNjcmVlbiBNb2RlClBpbiBGb3JtIE9uIFNjcmVlbgoyMzg0ODJuMzc1IGFkZGVkIHByb2plY3RzOiBTZWN1cml0eSwgV2lraW1lZGlhLVZFLUNhbXBhaWducyAoUzItMjAxOCksIFNjYXAgKFNjYXAzLUFkb3B0aW9uLVBoYXNlMiksIEFidXNlRmlsdGVyLCBEYXRhLXJlbGVhc2UsIEhhc2h0YWdzLCBMYWJzREItQXVkaXRvciwgTGFkaWVzLVRoYXQtRk9TUy1NZWRpYVdpa2ksIExhbmd1YWdlLTIwMTgtQXByLUp1bmUsIExhbmd1YWdlLTIwMTgtSmFuLU1hciwgSEhWTSwgSEFXZWxjb21lLlBSRVZJRVcKMjM4NDgybjM3NSBtb3ZlZCB0aGlzIHRhc2sgZnJvbSBJbiBQcm9ncmVzcyB0byBJbiBDb2RlIFJldmlldyBvbiB0aGUgQW5hbHl0aWNzLUthbmJhbiBib2FyZC4KMjM4NDgybjM3NSByZW1vdmVkIEpBbGxlbWFuZG91IGFzIHRoZSBhc3NpZ25lZSBvZiB0aGlzIHRhc2suCjIzODQ4Mm4zNzUgdHJpYWdlZCB0aGlzIHRhc2sgYXMgTG93ZXN0IHByaW9yaXR5LgoyMzg0ODJuMzc1IHJlbW92ZWQgc3Vic2NyaWJlcnM6IEFrbGFwcGVyLCBKQWxsZW1hbmRvdS4KQ29udGVudCBsaWNlbnNlZCB1bmRlciBDcmVhdGl2ZSBDb21tb25zIEF0dHJpYnV0aW9uLVNoYXJlQWxpa2UgMy4wIChDQy1CWS1TQSkgdW5sZXNzIG90aGVyd2lzZSBub3RlZDsgY29kZSBsaWNlbnNlZCB1bmRlciBHTlUgR2VuZXJhbCBQdWJsaWMgTGljZW5zZSAoR1BMKSBvciBvdGhlciBvcGVuIHNvdXJjZSBsaWNlbnNlcy4gQnkgdXNpbmcgdGhpcyBzaXRlLCB5b3UgYWdyZWUgdG8gdGhlIFRlcm1zIG9mIFVzZSwgUHJpdmFjeSBQb2xpY3ksIGFuZCBDb2RlIG9mIENvbmR1Y3QuILcgV2lraW1lZGlhIEZvdW5kYXRpb24gtyBQcml2YWN5IFBvbGljeSC3IENvZGUgb2YgQ29uZHVjdCC3IFRlcm1zIG9mIFVzZSC3IERpc2NsYWltZXIgtyBDQy1CWS1TQSC3IEdQTApZb3VyIGJyb3dzZXIgdGltZXpvbmUgc2V0dGluZyBkaWZmZXJzIGZyb20gdGhlIHRpbWV6b25lIHNldHRpbmcgaW4geW91ciBwcm9maWxlLCBjbGljayB0byByZWNvbmNpbGUu

Aklapper raised the priority of this task from Lowest to High.Jun 15 2018, 1:07 PM
Aklapper removed projects: HAWelcome, HHVM, Language-2018-Jan-Mar, Language-2018-Apr-June, Ladies-That-FOSS-MediaWiki, LabsDB-Auditor, Hashtags, Data-release, AbuseFilter, Scap (Scap3-Adoption-Phase2), Wikimedia-VE-Campaigns (S2-2018), acl*security, Analytics-Kanban.
Aklapper edited subscribers, added: Aklapper; removed: • 238482n375.
Aklapper assigned this task to Milimetric.Jun 15 2018, 1:09 PM
Aklapper added a project: Analytics-Kanban.
Milimetric added a comment.Jun 18 2018, 2:09 PM

Thanks for the fixes, @Aklapper. In case you didn't decipher the text yet, just do atob() on it in the browser console. It looks like the HTML of the page encoded in base64.

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits