From: Sujithra P <hidden> Date: 2021-07-21 18:46:37
Hi SELinux Experts,
The following issue is described in the below post as well.
https://github.com/containers/container-selinux/issues/141
Occasionally running into the following selinux denials for docker
type=AVC msg=audit(1626732057.636:4583): avc: denied { associate }
for pid=57450 comm="dockerd" name="/" dev="tmpfs" ino=150014
scontext=system_u:object_r:container_file_t:s0:c263,c914
tcontext=system_u:object_r:lib_t:s0 tclass=filesystem permissive=0
type=AVC msg=audit(1626812823.170:9434): avc: denied { associate }
for pid=20027 comm="dockerd" name="/" dev="tmpfs" ino=198147
scontext=system_u:object_r:container_file_t:s0:c578,c672
tcontext=system_u:object_r:locale_t:s0 tclass=filesystem permissive=0
level=error msg="Handler for POST
/v1.40/containers/a3a875e7896384e3bff53b8317e91ed4301a13957f42187eb227f28e09bd877c/start
returned error: error setting label on mount source
'/var/lib/kubelet/pods/f7cee5b2-bcd9-4aa1-9d67-c75b677ba2a1/volumes/kubernetes.io~secret/secret':
failed to set file label on
/var/lib/kubelet/pods/f7cee5b2-bcd9-4aa1-9d67-c75b677ba2a1/volumes/kubernetes.io~secret/secret:
permission denied"
Docker is not able to set labels for these tmpfs mounts because they
end up having wrong labels when they are created (sometimes
"locale_t", sometimes "lib_t" which of course is not the
default/correct context for tmpfs fs).
Apparently semodule -R and deleting these tmps files or reboot of the
node fixes the problem.
Not sure what is causing the tmpfs mounts to get wrong labels in the
first place.
Everything seems to be fine to begin with, but as the system keeps
scheduling pods on the node, this behavior is observed sometimes (not
consistent always).
OS Details:
NAME="CentOS Linux"
VERSION="8 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="8"
PLATFORM_ID="platform:el8"
PRETTY_NAME="CentOS Linux 8 (Core)"
Docker Version:
Client: Docker Engine - Community
Version: 19.03.13
API version: 1.40
Go version: go1.13.15
Git commit: 4484c46d9d
Built: Wed Sep 16 17:02:36 2020
OS/Arch: linux/amd64
Experimental: false
Kubernetes Version*
v1.20.8-gke.1500
Any help on how to debug this issue would be greatly appreciated.
Thanks
Sujithra.
From: Paul Moore <paul@paul-moore.com> Date: 2021-07-21 21:02:04
On Wed, Jul 21, 2021 at 2:46 PM Sujithra P [off-list ref] wrote:
Hi SELinux Experts,
The following issue is described in the below post as well.
https://github.com/containers/container-selinux/issues/141
Occasionally running into the following selinux denials for docker
type=AVC msg=audit(1626732057.636:4583): avc: denied { associate }
for pid=57450 comm="dockerd" name="/" dev="tmpfs" ino=150014
scontext=system_u:object_r:container_file_t:s0:c263,c914
tcontext=system_u:object_r:lib_t:s0 tclass=filesystem permissive=0
type=AVC msg=audit(1626812823.170:9434): avc: denied { associate }
for pid=20027 comm="dockerd" name="/" dev="tmpfs" ino=198147
scontext=system_u:object_r:container_file_t:s0:c578,c672
tcontext=system_u:object_r:locale_t:s0 tclass=filesystem permissive=0
level=error msg="Handler for POST
/v1.40/containers/a3a875e7896384e3bff53b8317e91ed4301a13957f42187eb227f28e09bd877c/start
returned error: error setting label on mount source
'/var/lib/kubelet/pods/f7cee5b2-bcd9-4aa1-9d67-c75b677ba2a1/volumes/kubernetes.io~secret/secret':
failed to set file label on
/var/lib/kubelet/pods/f7cee5b2-bcd9-4aa1-9d67-c75b677ba2a1/volumes/kubernetes.io~secret/secret:
permission denied"
Docker is not able to set labels for these tmpfs mounts because they
end up having wrong labels when they are created (sometimes
"locale_t", sometimes "lib_t" which of course is not the
default/correct context for tmpfs fs).
Apparently semodule -R and deleting these tmps files or reboot of the
node fixes the problem.
Not sure what is causing the tmpfs mounts to get wrong labels in the
first place.
Everything seems to be fine to begin with, but as the system keeps
scheduling pods on the node, this behavior is observed sometimes (not
consistent always).
OS Details:
NAME="CentOS Linux"
VERSION="8 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="8"
PLATFORM_ID="platform:el8"
PRETTY_NAME="CentOS Linux 8 (Core)"
Docker Version:
Client: Docker Engine - Community
Version: 19.03.13
API version: 1.40
Go version: go1.13.15
Git commit: 4484c46d9d
Built: Wed Sep 16 17:02:36 2020
OS/Arch: linux/amd64
Experimental: false
Kubernetes Version*
v1.20.8-gke.1500
Any help on how to debug this issue would be greatly appreciated.
This sounds like it might be a problem with CentOS and/or your Docker
install, have you tried talking with the RH/CentOS folks about this
problem? We focus mostly on upstream issues here and it isn't clear
to me at this moment that this is an upstream issue.
--
paul moore
www.paul-moore.com
From: Sujithra P <hidden> Date: 2021-07-21 22:17:53
Thanks Paul!
Is there any specific centos/RH mailing list that I can ask? Not sure
whether it is a problem with kernel/docker/kubelet.
semodule -R seems to fix the problem, but not sure what is causing the
loaded policy to get corrupt.
Any insight on how to figure this out would be very much appreciated.
Thanks
Sujithra.
On Wed, Jul 21, 2021 at 2:01 PM Paul Moore [off-list ref] wrote:
On Wed, Jul 21, 2021 at 2:46 PM Sujithra P [off-list ref] wrote:
quoted
Hi SELinux Experts,
The following issue is described in the below post as well.
https://github.com/containers/container-selinux/issues/141
Occasionally running into the following selinux denials for docker
type=AVC msg=audit(1626732057.636:4583): avc: denied { associate }
for pid=57450 comm="dockerd" name="/" dev="tmpfs" ino=150014
scontext=system_u:object_r:container_file_t:s0:c263,c914
tcontext=system_u:object_r:lib_t:s0 tclass=filesystem permissive=0
type=AVC msg=audit(1626812823.170:9434): avc: denied { associate }
for pid=20027 comm="dockerd" name="/" dev="tmpfs" ino=198147
scontext=system_u:object_r:container_file_t:s0:c578,c672
tcontext=system_u:object_r:locale_t:s0 tclass=filesystem permissive=0
level=error msg="Handler for POST
/v1.40/containers/a3a875e7896384e3bff53b8317e91ed4301a13957f42187eb227f28e09bd877c/start
returned error: error setting label on mount source
'/var/lib/kubelet/pods/f7cee5b2-bcd9-4aa1-9d67-c75b677ba2a1/volumes/kubernetes.io~secret/secret':
failed to set file label on
/var/lib/kubelet/pods/f7cee5b2-bcd9-4aa1-9d67-c75b677ba2a1/volumes/kubernetes.io~secret/secret:
permission denied"
Docker is not able to set labels for these tmpfs mounts because they
end up having wrong labels when they are created (sometimes
"locale_t", sometimes "lib_t" which of course is not the
default/correct context for tmpfs fs).
Apparently semodule -R and deleting these tmps files or reboot of the
node fixes the problem.
Not sure what is causing the tmpfs mounts to get wrong labels in the
first place.
Everything seems to be fine to begin with, but as the system keeps
scheduling pods on the node, this behavior is observed sometimes (not
consistent always).
OS Details:
NAME="CentOS Linux"
VERSION="8 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="8"
PLATFORM_ID="platform:el8"
PRETTY_NAME="CentOS Linux 8 (Core)"
Docker Version:
Client: Docker Engine - Community
Version: 19.03.13
API version: 1.40
Go version: go1.13.15
Git commit: 4484c46d9d
Built: Wed Sep 16 17:02:36 2020
OS/Arch: linux/amd64
Experimental: false
Kubernetes Version*
v1.20.8-gke.1500
Any help on how to debug this issue would be greatly appreciated.
This sounds like it might be a problem with CentOS and/or your Docker
install, have you tried talking with the RH/CentOS folks about this
problem? We focus mostly on upstream issues here and it isn't clear
to me at this moment that this is an upstream issue.
--
paul moore
www.paul-moore.com
From: Daniel Walsh <hidden> Date: 2021-07-22 09:38:24
On 7/21/21 18:17, Sujithra P wrote:
Thanks Paul!
Is there any specific centos/RH mailing list that I can ask? Not sure
whether it is a problem with kernel/docker/kubelet.
semodule -R seems to fix the problem, but not sure what is causing the
loaded policy to get corrupt.
Any insight on how to figure this out would be very much appreciated.
Thanks
Sujithra.
I am guessing that one of the containers is loading policy. You should
be able to see something in the auditlog, about a policy load.
On Wed, Jul 21, 2021 at 2:01 PM Paul Moore [off-list ref] wrote:
quoted
On Wed, Jul 21, 2021 at 2:46 PM Sujithra P [off-list ref] wrote:
quoted
Hi SELinux Experts,
The following issue is described in the below post as well.
https://github.com/containers/container-selinux/issues/141
Occasionally running into the following selinux denials for docker
type=AVC msg=audit(1626732057.636:4583): avc: denied { associate }
for pid=57450 comm="dockerd" name="/" dev="tmpfs" ino=150014
scontext=system_u:object_r:container_file_t:s0:c263,c914
tcontext=system_u:object_r:lib_t:s0 tclass=filesystem permissive=0
type=AVC msg=audit(1626812823.170:9434): avc: denied { associate }
for pid=20027 comm="dockerd" name="/" dev="tmpfs" ino=198147
scontext=system_u:object_r:container_file_t:s0:c578,c672
tcontext=system_u:object_r:locale_t:s0 tclass=filesystem permissive=0
level=error msg="Handler for POST
/v1.40/containers/a3a875e7896384e3bff53b8317e91ed4301a13957f42187eb227f28e09bd877c/start
returned error: error setting label on mount source
'/var/lib/kubelet/pods/f7cee5b2-bcd9-4aa1-9d67-c75b677ba2a1/volumes/kubernetes.io~secret/secret':
failed to set file label on
/var/lib/kubelet/pods/f7cee5b2-bcd9-4aa1-9d67-c75b677ba2a1/volumes/kubernetes.io~secret/secret:
permission denied"
Docker is not able to set labels for these tmpfs mounts because they
end up having wrong labels when they are created (sometimes
"locale_t", sometimes "lib_t" which of course is not the
default/correct context for tmpfs fs).
Apparently semodule -R and deleting these tmps files or reboot of the
node fixes the problem.
Not sure what is causing the tmpfs mounts to get wrong labels in the
first place.
Everything seems to be fine to begin with, but as the system keeps
scheduling pods on the node, this behavior is observed sometimes (not
consistent always).
OS Details:
NAME="CentOS Linux"
VERSION="8 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="8"
PLATFORM_ID="platform:el8"
PRETTY_NAME="CentOS Linux 8 (Core)"
Docker Version:
Client: Docker Engine - Community
Version: 19.03.13
API version: 1.40
Go version: go1.13.15
Git commit: 4484c46d9d
Built: Wed Sep 16 17:02:36 2020
OS/Arch: linux/amd64
Experimental: false
Kubernetes Version*
v1.20.8-gke.1500
Any help on how to debug this issue would be greatly appreciated.
This sounds like it might be a problem with CentOS and/or your Docker
install, have you tried talking with the RH/CentOS folks about this
problem? We focus mostly on upstream issues here and it isn't clear
to me at this moment that this is an upstream issue.
--
paul moore
www.paul-moore.com
From: Sujithra P <hidden> Date: 2021-07-23 06:06:28
Thanks Daniel.
I'm not seeing anything suspicious in the audit logs, the. following
are the MAC_POLICY_LOAD events
type=SYSCALL msg=audit(1622991228.547:183): arch=c000003e syscall=1
success=yes exit=8577048 a0=4 a1=7fd682c61000 a2=82e018 a3=0 items=0
ppid=62178 pid=62186 auid=1004 uid=0 gid=0 euid=0 suid=0 fsuid=0
egid=0 sgid=0 fsgid=0 tty=pts0 ses=1 comm="load_policy"
exe="/usr/sbin/load_policy"
subj=unconfined_u:unconfined_r:load_policy_t:s0-s0:c0.c1023 key=(null)
type=MAC_POLICY_LOAD msg=audit(1627002776.825:6075): auid=4294967295
ses=4294967295 lsm=selinux res=1
----
type=MAC_POLICY_LOAD msg=audit(1627008064.852:7615): auid=4294967295
ses=4294967295 lsm=selinux res=1
----
type=MAC_POLICY_LOAD msg=audit(1627008273.029:7617): auid=4294967295
ses=4294967295 lsm=selinux res=1
----
type=MAC_POLICY_LOAD msg=audit(1627009159.383:8392): auid=4294967295
ses=4294967295 lsm=selinux res=1
----
On Thu, Jul 22, 2021 at 2:38 AM Daniel Walsh [off-list ref] wrote:
On 7/21/21 18:17, Sujithra P wrote:
quoted
Thanks Paul!
Is there any specific centos/RH mailing list that I can ask? Not sure
whether it is a problem with kernel/docker/kubelet.
semodule -R seems to fix the problem, but not sure what is causing the
loaded policy to get corrupt.
Any insight on how to figure this out would be very much appreciated.
Thanks
Sujithra.
I am guessing that one of the containers is loading policy. You should
be able to see something in the auditlog, about a policy load.
quoted
On Wed, Jul 21, 2021 at 2:01 PM Paul Moore [off-list ref] wrote:
quoted
On Wed, Jul 21, 2021 at 2:46 PM Sujithra P [off-list ref] wrote:
quoted
Hi SELinux Experts,
The following issue is described in the below post as well.
https://github.com/containers/container-selinux/issues/141
Occasionally running into the following selinux denials for docker
type=AVC msg=audit(1626732057.636:4583): avc: denied { associate }
for pid=57450 comm="dockerd" name="/" dev="tmpfs" ino=150014
scontext=system_u:object_r:container_file_t:s0:c263,c914
tcontext=system_u:object_r:lib_t:s0 tclass=filesystem permissive=0
type=AVC msg=audit(1626812823.170:9434): avc: denied { associate }
for pid=20027 comm="dockerd" name="/" dev="tmpfs" ino=198147
scontext=system_u:object_r:container_file_t:s0:c578,c672
tcontext=system_u:object_r:locale_t:s0 tclass=filesystem permissive=0
level=error msg="Handler for POST
/v1.40/containers/a3a875e7896384e3bff53b8317e91ed4301a13957f42187eb227f28e09bd877c/start
returned error: error setting label on mount source
'/var/lib/kubelet/pods/f7cee5b2-bcd9-4aa1-9d67-c75b677ba2a1/volumes/kubernetes.io~secret/secret':
failed to set file label on
/var/lib/kubelet/pods/f7cee5b2-bcd9-4aa1-9d67-c75b677ba2a1/volumes/kubernetes.io~secret/secret:
permission denied"
Docker is not able to set labels for these tmpfs mounts because they
end up having wrong labels when they are created (sometimes
"locale_t", sometimes "lib_t" which of course is not the
default/correct context for tmpfs fs).
Apparently semodule -R and deleting these tmps files or reboot of the
node fixes the problem.
Not sure what is causing the tmpfs mounts to get wrong labels in the
first place.
Everything seems to be fine to begin with, but as the system keeps
scheduling pods on the node, this behavior is observed sometimes (not
consistent always).
OS Details:
NAME="CentOS Linux"
VERSION="8 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="8"
PLATFORM_ID="platform:el8"
PRETTY_NAME="CentOS Linux 8 (Core)"
Docker Version:
Client: Docker Engine - Community
Version: 19.03.13
API version: 1.40
Go version: go1.13.15
Git commit: 4484c46d9d
Built: Wed Sep 16 17:02:36 2020
OS/Arch: linux/amd64
Experimental: false
Kubernetes Version*
v1.20.8-gke.1500
Any help on how to debug this issue would be greatly appreciated.
This sounds like it might be a problem with CentOS and/or your Docker
install, have you tried talking with the RH/CentOS folks about this
problem? We focus mostly on upstream issues here and it isn't clear
to me at this moment that this is an upstream issue.
--
paul moore
www.paul-moore.com