"/usr/share/nginx/html/index.html" is forbidden (13: Permission denied)

Does the user have permission to that file?

As far as I understand, it should as it’s embedded inside the actual image file and not mounted.

I’m having other strange behaviour on non-1000 uid accounts. For example, running docker.io/library/httpd using the podman examples page on an account that isn’t userid 1000 will create an issue where it complains that libgcc_0 isn’t installed. This works flawlessly on the uid 1000 user.

It does make me think that something needs to be configured for non-uid 1000 users on podman. This is not replicable on arch linux on my raspberry pi, fwiw.

Shell into the container and find out.

Complete shot in the dark, but I would start by comparing podman system info between the two user accounts, with the most importance on the idMappings. Podman is usually configured to use rootless by default, and this can result in wildly different behavior based on whether /etc/subuid and /etc/subgid are configured to give your user IDs or not. See tutorial if your user 1001 is missing.

Modern distro installers tend to give the user created during setup 65536 IDs, but useradd usually doesn't do that by default. This is really distro-specific stuff though and I don't know how modern Fedora handles it, so I might be grasping at nothing here.

I think I have uids set up, but I’m not positive they’re set up correctly.

Here’s an example of /etc/subuid for example, with it a bit anonymised for my specific user setup:

core:524288:65536
syncthing:589824:65536
userA:655360:65536
userB:720896:65536
userC:786432:65536

Syncthing in this case isn’t running any podman containers, all sharing is done bare-metal. userA is working, but userB and userC do look like they’re configured “correctly” at first glance. podman system info seems to be outputting the results above

  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1101
      size: 1
    - container_id: 1
      host_id: 786432
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1101
      size: 1
    - container_id: 1
      host_id: 786432
      size: 65536

Note: The user in question here is uid 1101, another new user I created for testing purposes.

That looks correct to me if uid 1101 == userC. Sorry, don't really know what it might be then.

Sorry for not answering to your question directly, but why not just use subuids for separation instead?

podman run -d --restart unless-stopped \
  --uidmap "+0:@100001:1" \
  --name nginx-1 \
  --port 8081:80 \
  docker.io/library/nginx

podman run -d --restart unless-stopped \
  --uidmap "+0:@100002:1" \
  --name nginx-2 \
  --port 8082:80 \
  docker.io/library/nginx

I mean, I might be wrong, but this only changes the mapping right? In the case that user broke containment, I would assume that they would have the user id of the user that’s running the process?

Granted this is all in theory so I’m not sure exactly – that might be a perfectly reasonable way to do it.

This is due to a security design deicison of Podman. Each user’s network(s) is only available to that user. This is great for most services, but can cause issues for some services - especially reverse proxies. Unfortunately, I’m not aware of an ideal solution. The only solution I’ve seen is moving the reverse proxy to another host and exposing the services’ ports on the localhost. I hope someone can provide a better solution!

OK now this would make some sense but would definitely be a bit of a show stopper for me. I’ll give it a try though.

Edit: Unfortunately this didn’t solve the issue. I thought it did for a minute, but it was the browser cache I think auto-filling the correct result.

The solution to this can be multiple reverse proxies. Each user runs it’s own reverse proxy (if you actually need one, otherwise you let the container bind directly to a port on the host system). Then you run one main reverse proxy on port 80/443 that proxies to those other ports based on hostname.

The upside of this construction is that containers running as different users can’t directly access each others internal networks, which is much better for security, as those networks often contain barely protected services, which is why you’re using a reverse proxy in the first place.

Edit: there are different options with custom networks and such, but they’re even more complicated so I wouldn’t advise them.

I think this is what OP was originally trying, but this approach breaks when each service’s Podman service runs on its own localhost user.

You can use the per-user reverse proxies as a bridge between the host network and the user-specific internal network. So for a user coming from outside the path looks like this:

User -> main reverse proxy on main IP and port 80/443 -> user-specific proxy listening on main IP with port 8080 and proxying to user-specific internal network -> destination container listening on user-specific internal network.

And for a container running as a different user the path will be the same, but the user-specific reverse proxy will be listening on port 8081 and higher for example.

This would require the main proxy running as root or with some other sort of elevated privileges to allow cross-user network access though, right? If so, wouldn’t that essentially make the user-specific reverse proxy unnecessary in most cases?

It can be run as root, but it doesn’t have to, as it is only accessing the host network, not any user-specific network. Crossing the boundaries from host network to user-specific networks is left to the reverse proxies running as those specific users.

Port 80 can normally only be bound as root, but you can work around that with either firewall rules or by using something like this in your sysctl config:

net.ipv4.ip_unprivileged_port_start = 80

I’m not trying to convince you of this solution (I personally don’t like it), but I am curious what didn’t work for you. Were you unable to get the reverse proxy to serve each service? Were you unable to have the services behind the reverse proxy to talk to each other?

No expert but don’t the logs look like the requests are coming through?

I didn’t see the logs when I originally posted. However, I’m not sure what the logs indicate. It could be that nginx successfully received the request and received an authorization error from Podman’s networking stack then returned a 503 error to the client (or logged the 503 response that was returned to nginx).

You use podman-compose with the Podman backend?

And there are no volumes mounted, just to be sure?

No volumes mounted and yes, it’s using the podman backend. Just double checked it wasn’t using the docker one.

The service starts as it should, right? The errors appear only on request, right? Any other sections in the compose logs that seem relevant?

The best way to run Podman is root with UserNS to dole out UID/GID protection. Running Podman as root allows you to share networks between containers while having the containers run under different users. If you go rootless, you’d need to run under one user to share the user’s network space with all the containers you want.

As for your issue, I can’t really divine what the problem is from the errors. I avoid nginx because it’s coded to not play well with user abstraction and changing the user with the files it wants to write to etc. Gotta write into a ton of random folders! So not sure exactly what is up. But with Podman root it is easy to run as root 0 internally and make nginx think it has all the control it could ever want.

Try this setup (it is in Podman Quadlet format, apologies I don’t know the compose versions). It runs the container as root 0 internally, externally it runs as some random UID/GID - secure! It uses Volume idmap to map the internal root 0 user to 1001 for write access to the Volume.

Note that in Debian 13 symlinks are broken and won’t work with idmap, just point to the original source. If you need symlinks, I have an alternate UserNS that maps internal user root 0 to external user 1001 directly. You’d drop the idmap in Volume then and use that. You lose some extra security - now the container is running as external user 1001 instead of some random UID/GID - but that’s a pretty minor hit as long as your external user doesn’t have access to tons of things.

# Volumes to mount -> the @ is essential for saying "1001 is absolute and external" basically. 0 is internal. size of 1. You can map 1001 to 0 and 1002 to 1 with @1001-0-2, etc., etc., etc.  
Volume=/mnt/something:/etc/nginx/wants/to/write/here:rw,noexec,nosuid,nodev,Z,idmap=uids=@1001-0-1;gids=@1001-0-1  

# Run as user running the container  
UserNS=auto  
# [use this if req symlink b/c idmap does NOT work with symlinks] -> I tested and it is fixed in at least Podman v5.8.3, so Debian 14 will work with idmap and symlinks directly ! drop the idmap if using !  
# UserNS=auto:uidmapping=0:@1001:1,gidmapping=0:@1001:1  

# Security time  
NoNewPrivileges=true  
# https://man7.org/linux/man-pages/man7/capabilities.7.html  
DropCapability=all  
ReadOnly=true  
ReadOnlyTmpfs=True  
# These capabilities are needed for linuxserver's s6 "launcher" thing  
#AddCapability=CAP_CHOWN  
#AddCapability=CAP_DAC_OVERRIDE  
#AddCapability=CAP_FOWNER  
#AddCapability=CAP_SETGID  
#AddCapability=CAP_SETUID  
# Needs this if the container tries to bind below port 1024. I'm not sure if it is needed if it only tries to bind internally.  
#AddCapability=CAP_NET_BIND_SERVICE  

# TempFS for ReadOnly fixes I've used for nginx - may not be relevant for you. These are from getting Frigate running.  
PodmanArgs=--tmpfs /usr/local/nginx/conf:size=1M,rw,noexec,nosuid,nodev  
PodmanArgs=--tmpfs /usr/local/nginx/logs:size=40M,rw,noexec,nosuid,nodev  
PodmanArgs=--tmpfs /usr/local/nginx/client_body_temp:size=1M,rw,noexec,nosuid,nodev  
PodmanArgs=--tmpfs /usr/local/nginx/proxy_temp:size=1M,rw,noexec,nosuid,nodev  
PodmanArgs=--tmpfs /usr/local/nginx/fastcgi_temp:size=1M,rw,noexec,nosuid,nodev  
PodmanArgs=--tmpfs /usr/local/nginx/uwsgi_temp:size=1M,rw,noexec,nosuid,nodev  
PodmanArgs=--tmpfs /usr/local/nginx/scgi_temp:size=1M,rw,noexec,nosuid,nodev  
PodmanArgs=--tmpfs /etc/letsencrypt:size=1M,rw,noexec,nosuid,nodev  

Root Podman and UserNS=auto needs a containers user to pull uid/gid from.

# Root Podman needs a `containers` "user" (not really a user, just a reserved uid/gid space)  
sudo echo "containers:2147483647:2147483648" >> /etc/subuid  
sudo echo "containers:2147483647:2147483648" >> /etc/subgid  

The documentation for Podman is critically lacking in the “hobbyist” space. Hope this helps.

Edit: This approach works well because most Docker containers are built assuming they’ll run as root 0. That’s why Linuxserver uses the S6 overlay thing to jump from root 0 to something else. The container can be built to run as any user though, if you look at the Dockerfile for the container you’re using, you’ll see what user they’re declaring it will run as (and likely what user owns all the files). Root 0 usually gets around that problem - unless they “cleverly” code it to try to prevent you from running the container as root 0 (I’ve run into this before! It was Heimdall from the Linuxserver people).

Edit2: I’ve noticed you said no Volumes, so drop that. But you can still use the UserNS mapping to run it as root internally which should fix the internal permissions issues. The tmpfs stuff is if you declare ReadOnly for extra security - it’s a great idea - but nginx is extra difficult in that regard. Disregard it while you get going.

# Run as user running the container  
UserNS=auto  

# Security time  
NoNewPrivileges=true  
# https://man7.org/linux/man-pages/man7/capabilities.7.html  
DropCapability=all  
# These capabilities are needed for linuxserver's s6 "launcher" thing  
#AddCapability=CAP_CHOWN  
#AddCapability=CAP_DAC_OVERRIDE  
#AddCapability=CAP_FOWNER  
#AddCapability=CAP_SETGID  
#AddCapability=CAP_SETUID  
# Needs this if the container tries to bind below port 1024. I'm not sure if it is needed if it only tries to bind internally.  
#AddCapability=CAP_NET_BIND_SERVICE  

Edit3: Use sudo podman top nginx user huser group hgroup groups hgroups to see the internal user/host user (huser) mappings easily for debug.

USER    HUSER       GROUP   HGROUP      GROUPS  HGROUPS
root        2147485695  root        2147485695  105     105

Here’s an output from my frigate container. Internally (USER) it is root, externally (HUSER) it’s some random UID. I’ve also mapped the internal group (GROUPS) 105 to the external group (HGROUPS) 105 so that it has render access.

Note that sudo echo does not work as non root because the shell redirection is attempted before the command runs. You want echo foo | sudo tee -a /bar instead

So the sudo echo does echo as sudo but doesn’t carry over? Makes sense damn I hate bash! Any way to keep the >>? Or do you need to tee? Cause the >> is pretty cool

That’s the thing about computers, they do exactly what you tell them. sudo echo works, but the pipe and the redirect are interpreted by the bash shell you’re in, which isn’t being run with sudo.

You should also know that echo is a bash shell builtin, so the echo you’re running with echo and sudo echo are two very different things.

What are the assigned subuids and subgids for userA and userB?

You might want to switch to rootless quadlets with subuids

https://www.golinuxcloud.com/rootless-podman-quadlet/

You’ll get unified logging through systemctl and journalctl, you’ll also get segmented user space and if you need persistent storage that you can gate at subuid.