I can create and delete containers in 'testlabs' and 'service' but not in 'taavi-test-project' or 'cloudvirt-canary'.
Let's hope this is configurable someplace! So far all I see is rules about bucket naming, at the top of https://documentation.suse.com/en-us/ses/7.1/html/ses-all/cha-ceph-gw.html
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | aborrero | T296411 cloud: decide on general idea for having cloud-dedicated hardware provide service in the cloud realm & the internet | |||
| Resolved | aborrero | T297596 have cloud hardware servers in the cloud realm using a dedicated LB layer | |||
| Resolved | aborrero | T324992 cloudlb: create PoC on codfw | |||
| Resolved | aborrero | T338937 cloudlb: review swift/radosgw status | |||
| Resolved | Andrew | T341509 radosgw+keystone chokes on projects with '-' in their id | |||
| Resolved | Andrew | T343158 Support OpenStack projects where project name != project id | |||
| Open | None | T366679 openstack-browser support for projects where id != name | |||
| Resolved | Andrew | T347927 Better swift error messages for projects with invalid chars | |||
| Resolved | Andrew | T366301 Document and enforce restrictions on openstack project names |
Here's a log from a recent bucket creation attempt:
Nothing helpful in there as far as I can see.
I think this is because radosgw is expecting the uuid of a project, which is alphanumeric only. Our hack to force project name == project ID is finally costing us!
I see return isalnum(ch) || ch == '_'; is the validation step. Is there anywhere that we can insert remapping of names? At least in Toolforge we use naming rules that roughly match DNS label restrictions which means alphanumeric plus -. If we could remap - to _ things should comply with the radosgw constraints.
Most of the UUID standards I know use - as a separator in their canonical textual representations, so I'm not sure this is actually the cause of our woe.
I think this is because radosgw is expecting the uuid of a project, which is alphanumeric only. Our hack to force project name == project ID is finally costing us!
Most of the UUID standards I know use - as a separator in their canonical textual representations, so I'm not sure this is actually the cause of our woe.
Yeah, David and I puzzled over this for a while too but it seems that internally keystone project UUIDs are represented without '-'s
This is an interesting idea! The order of operation is the wrong way around to make this easy...
Right now the project id flows directly from the client (openstack CLI, curl, horizon, etc) to radosgw. Then radosgw uses the project ID to check auth against keystone before proceeding. So to map things we would need to insert middleware somewhere between the internet and radosgw, and then also provide some kind of de-mapping middleware in front of keystone (or hack keystone to regard - and _ as equivalent when doing a project lookup.)
The good news is that currently NO projects use _ in their names, despite it being an allowed character. So we could hack in a validation rule in Keystone that ensures no future projects have _ either and thus avoid namespace collisions.
This seems silly, but possible!
Oh, also, because surely someone is going to ask, here are all the eqiad1 projects with 'invalid' characters:
account-creation-assistance admin-monitoring automation-framework capacity-exchange centralnotice-staging checkuser-beta-wiki civicrm-prototype clouddb-services cloudinfra-nfs cloudvirt-canary collection-alt-renderer commons-corruption-checker deployment-prep dump-references-processor entity-detection fa-wp fr-tech-dev gitlab-runners gitlab-test google-api-proxy huwiki-dev idm-dev image-suggestion-api ldap-dev library-upgrader lta-tracker machine-learning maps-experiments media-streaming mediawiki-vagrant mix-n-match mwv-apt netops-clab paws-0b0584a2-6aa9-463d-8466-e86d62201c3b paws-ee853c6a-df8a-4f8d-8154-3eab320229c6 policy-test-project procbot-4a50d61f-b315-4b94-8790-efa182205e30 project-proxy puppet-dev puppet-diffs quarry-61d3d72c-3909-467f-af76-2b1dedf67dd5 reading-web-staging recommendation-api research-collaborations-api security-tools shiny-r spi-tools sre-sandbox superset-7e016ecd-2e3e-4643-ab40-7a04d8133ad5 text-to-speech tf-infra-test wikibase-registry wikidata-dev wikidata-history-query-service wikidata-query wikidata-reconciliation wikimania-mautic wlm-it-visual wm-bot wmcs-uptime wmcz-stats wmde-dashboards wmf-dumps-playground wmf-research-tools
The laziest possible option is to prevent any future project from having invalid chars, and then just declare those projects to be unlucky and unable to use object storage. We'd be leaving a lot of major projects out in the cold.
And of course another way forward is to switch to using UUIDs for project IDs. I'm not sure how much work that would be, other than 'probably quite a bit'.
The upcoming OpenStack rebuild is I think the best opportunity for this that we've had in ages.
I've reviewed the ceph code that vets tenant name and it doesn't seem to be restricted to keystone integration; the rule is applied to ALL Ceph tenant names. So it's unlikely that a patch removing that naming restriction will be accepted.
Tonight it seems like the simplest path forward is to add the same naming restrictions in our cloud, and then rename projects (possible wholesale, or possibly as needed.) My main concern about /that/ strategy is that I see projects with names like 'paws-0b0584a2-6aa9-463d-8466-e86d62201c3b' which I suspect are auto-generated by magnum. So if magnum requires dashes and ceph abhors them... something will have to give.
DNS labels are technically supposed to use hyphens and not underscores. We actually had to make some validation changes for Toolforge tool names because of this (T176027: Tools with "_" in their name or names longer than 63 characters do not get Kubernetes namespaces created). We currently use the Cloud VPS project name as a DNS label in both the $instance.$project.$region.wikimedia.cloud instance names and also in the $project.wmcloud.org zone that each project is given control of.
DNS labels are technically supposed to use hyphens and not underscores.
So, so that means that use of underscores in ceph tenant names /is/ broken, since presumably the tenant names make it into the urls used for object access.
"Broken" in this case would only apply for use as a DNS label, meaning as a hostname component. Just about anything can be used in the path component of a URL when URL encoded. The challenge here is really that different use cases for a Cloud VPS project name are placing different constraints on the allowed character set. I think lowest common denominator would be to only allow alphanumeric characters from the ASCII set, potentially with an additional restriction that the label must start with a letter.
If we adopt this as the most workable solution for interoperability with radosgw for Cloud VPS projects I wonder if we would also need to make the same change for Toolforge tool names? Have we thought much yet about how radosgw and Toolforge things should interoperate?
Change 941464 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):
[operations/puppet@production] keystone.conf: turn on character restrictions for new projects and domains.
Change 941465 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):
[operations/puppet@production] keystone: hack to reject all non-alphanumerical project or domain names
Change 941464 merged by Andrew Bogott:
[operations/puppet@production] keystone.conf: turn on character restrictions for new projects and domains.
Change 941465 merged by Andrew Bogott:
[operations/puppet@production] keystone: hack to reject all new non-alphanumerical project or domain names
Change 942778 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):
[operations/puppet@production] Revert "keystone: hack to reject all new non-alphanumerical project or domain names"
Change 942778 merged by Andrew Bogott:
[operations/puppet@production] Revert "keystone: hack to reject all new non-alphanumerical project or domain names"
Change 963323 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):
[openstack/horizon/horizon@main] Container dashboard: inject polite error page for projects w/out object support
Change 963323 merged by Andrew Bogott:
[openstack/horizon/horizon@main] Container dashboard: inject polite error page for projects w/out object support
Change 963728 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):
[openstack/horizon/[email protected]] Container dashboard: inject polite error page for projects w/out object support
Change 963728 merged by Andrew Bogott:
[openstack/horizon/[email protected]] Container dashboard: inject polite error page for projects w/out object support
Change 963730 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):
[operations/puppet@production] Horizon: update codfw1dev horizon version
Change 963730 merged by Andrew Bogott:
[operations/puppet@production] Horizon: update codfw1dev horizon version
Change 963733 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):
[operations/puppet@production] Horizon: update eqiad1 horizon version
Change 963733 merged by Andrew Bogott:
[operations/puppet@production] Horizon: update eqiad1 horizon version