Cloud Run PM here. Sorry for the inconvenience. This is an issue for which we have a fix, but the fix still needs to roll out (internal bug 154368323 for googlers reading, we're still evaluating the best rollout strategy). Until then, I'll add it to https://cloud.google.com/run/docs/issues.

For those not familiar with Cloud Run, it's a Google Cloud service that allows you to run Docker containers without managing the server it runs on.

https://cloud.google.com/run

The solution here is to use `$XDG_CACHE_HOME` instead of `$HOME/.cache`, which may not be where a user wants their cache to live: https://specifications.freedesktop.org/basedir-spec/basedir-...

As an alternative, don’t use root for Dockerfiles. Ever. It only takes a few lines to create a user, group, and use said user, and closes a whole class of security issues.

That sucks.

My only advice is, when using this or cloud functions always start by creating a function/image that prints all env variables.

For cloud functions these changes tremendously between Python version. Also logging changes completely between some Python versions, to the point where upping the runtime causes logs to stop being saved on cloud logging.

Does Huggingface not have a way to override the cache directory? I always specify directories like this manually when using docker specifically to avoid issues like the one in TFA.

Closes security loopholes, increases operational burden (if only slightly). A tale as old as time.

Can confirm, using a non-root user also solves the issue (which I should have been doing anyway). It guess Cloud Run assumes $HOME is set to /home/something and since root uses /root it gets confused?

Are you sure Cloud Run only overrides $HOME when the container runs as root? Because if it always overrides $HOME then what user the container runs as is meaningless (for this particular issue).

True! That would have also been a smart move

Why some people do it: there are problems with bind mounts when using a non privileged user inside of a container. Not only that, but many containers either don't have instructions for running with custom users, make bad assumptions about what users will be used (chmod/chown) or even will have cryptic error messages when you try to run them with custom users.

In short, it's a pain for the average person just trying to get something running.

For example, you want to mount /docker/my-app/mysql-container/var/lib/mysql into /var/lib/mysql of the container. Maybe you're doing this so you can view the files on the host system and not use the Docker Volumes abstraction, maybe you want to do backups on a per directory basis and Docker doesn't let you move separate volumes into separate directories, or any other reason. Some orchestrators like Docker Swarm won't create the local directory for you, so you just end up doing that yourself, with UID:GID of 1004:1004 (or whatever your local server user has).

Now, if you run the container with the default settings, which indeed use the root user, there is a good chance of it working (illustrated here with Docker):

  > docker run --rm -e MYSQL_ROOT_PASSWORD="something-hopefully-long" -v "/docker/my-app/mysql-container/var/lib/mysql:/var/lib/mysql" mysql:5.7
  ... lots of output here
  [Note] mysqld: ready for connections.

Because by default, even MySQL uses root inside of the container:

  > docker run --rm mysql:5.7 id
  uid=0(root) gid=0(root) groups=0(root)

When you change it to another user without knowing which one you need, which is pretty common, it breaks:

  > docker run --rm -u 1010:1010 -e MYSQL_ROOT_PASSWORD="something-hopefully-long" -v "/docker/my-app/mysql-container/var/lib/mysql:/var/lib/mysql" mysql:5.7
  [ERROR] InnoDB: The innodb_system data file 'ibdata1' must be writable
  [ERROR] InnoDB: The innodb_system data file 'ibdata1' must be writable
  [ERROR] InnoDB: Plugin initialization aborted with error Generic error
  [ERROR] Plugin 'InnoDB' init function returned error.
  [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
  [ERROR] Failed to initialize builtin plugins.
  [ERROR] Aborting

As you can see, the error messages don't come from Docker or any other system that knows what's actually happening, but from the piece of software itself, in this case, MySQL. Not only that, but it doesn't give you actionable steps on its own and "Generic error" will only serve to confuse many of the newer users, who'll fail to understand what "The innodb_system data file 'ibdata1' must be writable" means.

How many users do you think will understand what the error message means and will know how to solve it? How many container images out there won't give user friendly error messages and instead will just crash?

How many users do you think will notice the "Running as an arbitrary user" under the instructions for that particular container image [0]? How many container images will even support running as arbitrary users?

How many users do you think will be able to find the documentation for the parameters that they need for either Docker Compose [1] or specifying a user in Docker [2]?

To avoid that problem, you need to:

  - know about users/groups and permissions management in GNU/Linux
  - know what your user/group is if you're creating anything like bind mounts
  - know how to set the user/group that's going to run inside of the container
  - know that the external container you use will handle them properly (but also know how it's initialized, e.g. no additional config necessary, since some badly containerized pieces of software will start additional processes with UID/GIDs that have been set in some configuration file somewhere in the container)

And with all of that, how many users do you think will decide that none of the above is worth the hassle and just run their containers as root, regardless of any other potential risks in the future, as opposed to having problems now? If something is hard to do, it simply won't be done in practice, especially if things work without doing it.

Though i'm not sure what can be done to improve the DX here, without making Docker aware of the attempts to access files inside of the container. Plus, managing users and groups has far too many approaches to begin with [3].

Links:

  [0] https://hub.docker.com/_/mysql
  [1] https://docs.docker.com/compose/compose-file/compose-file-v3/
  [2] https://docs.docker.com/engine/reference/run/#user
  [3] https://blog.giovannidemizio.eu/2021/05/24/how-to-set-user-and-group-in-docker-compose/