403 Forbidden

[joaobernardo-87]

I'm trying to use Headlamp with Microsoft Entra ID as a OIDC provider. I'm able to authenticate, but not able to see any resources. I would like to allow users to only list pods from the dxjmse-dev namespace and Headlamp pod is deployed on kube-system namesapace. The users are attached to an Entra ID group and the former one is associated to the Azure Enterprise Application. When opening the web development tools, I checked that I'm receiving a 403 Forbidden status once logged in the headlamp UI. (see image below)

This is the jwt token decoded that is being passed:

{ "typ": "JWT", "alg": "RS256", "kid": "CNv0OI3RwqlHFEVnaoMAshCH2XE" }.{ "aud": "ad6d237d-1680-4f42-8e9f-cd8680b3c093", "iss": "https://login.microsoftonline.com/c6e0547c-0aa6-491a-9fdc-cf5ef5c021df/v2.0", "iat": 1747147357, "nbf": 1747147357, "exp": 1747151257, "email": "joaobernardo87@yahoo.com", "groups": [ "3685d112-4420-4336-9072-eb9ea1f8c4a0" ], "idp": "mail", "name": "João Mesquita", "oid": "1c7f8bea-678e-450d-b51e-78194b50b0aa", "preferred_username": "joaobernardo87@yahoo.com", "rh": "1.Aa8AfFTgxqYKGkmf3M9e9cAh330jba2AFkJPjp_NhoCzwJOvACGvAA.", "sid": "004e4ef9-dca0-176c-fff9-83cdee7c8e3b", "sub": "urFygHaohkwT-PChe6UqV9g4JGyPngdWqhCdhVnhX7M", "tid": "c6e0547c-0aa6-491a-9fdc-cf5ef5c021df", "uti": "iPQhuw7_QkeHwfKFXAc8AA", "ver": "2.0", "wids": [ "13bd1c72-6f4a-4dcf-985f-18d3b80f208a" ] }.[Signature]

I've also created the the following role and rolebinding:

`apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"rbac.authorization.k8s.io/v1","kind":"Role","metadata":{"annotations":{},"name":"headlamp-access-role","namespace":"dxjmse-dev"},"rules":[{"apiGroups":[""],"resources":["pods","services"],"verbs":["get","list","watch","delete"]}]}
creationTimestamp: "2025-04-17T17:29:15Z"
name: headlamp-access-role
namespace: dxjmse-dev
resourceVersion: "156352280"
uid: e8bf67f2-e593-45b9-b242-7d4b4ebc9c96
rules:

• apiGroups:
  • ""
    resources:
  • pods
    verbs:
  • get
  • list
  • watch
    apiVersion: rbac.authorization.k8s.io/v1
    kind: RoleBinding
    metadata:
    annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
    {"apiVersion":"rbac.authorization.k8s.io/v1","kind":"RoleBinding","metadata":{"annotations":{},"name":"headlamp-access-rolebinding","namespace":"dxjmse-dev"},"roleRef":{"apiGroup":"rbac.authorization.k8s.io","kind":"Role","name":"headlamp-access-role"},"subjects":[{"apiGroup":"rbac.authorization.k8s.io","kind":"Group","name":"3685d112-4420-4336-9072-eb9ea1f8c4a0"}]}
    creationTimestamp: "2025-04-17T17:29:46Z"
    name: headlamp-access-rolebinding
    namespace: dxjmse-dev
    resourceVersion: "150558527"
    uid: 594337c8-e915-4b9c-b51a-e49935e1370a
    roleRef:
    apiGroup: rbac.authorization.k8s.io
    kind: Role
    name: headlamp-access-role
    subjects:
• apiGroup: rbac.authorization.k8s.io
  kind: Group
  name: 3685d112-4420-4336-9072-eb9ea1f8c4a0
  `

I also have the following clusterrole and clusterolebinding:

`apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
annotations:
rbac.authorization.kubernetes.io/autoupdate: "true"
creationTimestamp: "2024-04-15T16:47:22Z"
labels:
kubernetes.io/bootstrapping: rbac-defaults
name: cluster-admin
resourceVersion: "69"
uid: cc3e592f-2575-4174-955f-833628f96948
rules:

• apiGroups:
  • '*'
    resources:
  • '*'
    verbs:
  • '*'
• nonResourceURLs:
  • '*'
    verbs:
  • '*'
    apiVersion: rbac.authorization.k8s.io/v1
    kind: ClusterRoleBinding
    metadata:
    annotations:
    rbac.authorization.kubernetes.io/autoupdate: "true"
    creationTimestamp: "2024-04-15T16:47:23Z"
    labels:
    kubernetes.io/bootstrapping: rbac-defaults
    name: cluster-admin
    resourceVersion: "154160506"
    uid: e23a847f-01e1-462d-9ccf-db62ce205283
    roleRef:
    apiGroup: rbac.authorization.k8s.io
    kind: ClusterRole
    name: cluster-admin
    subjects:
• apiGroup: rbac.authorization.k8s.io
  kind: Group
  name: system:masters`

This is the headlamp deployment:

`apiVersion: apps/v1
kind: Deployment
metadata:
annotations:
deployment.kubernetes.io/revision: "20"
meta.helm.sh/release-name: my-headlamp
meta.helm.sh/release-namespace: kube-system
creationTimestamp: "2025-04-14T17:23:48Z"
generation: 21
labels:
app.kubernetes.io/instance: my-headlamp
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: headlamp
app.kubernetes.io/version: 0.30.0
helm.sh/chart: headlamp-0.30.1
name: my-headlamp
namespace: kube-system
resourceVersion: "156346810"
uid: 759aefeb-f9f9-412a-a004-43543f1d16cb
spec:
progressDeadlineSeconds: 600
replicas: 1
revisionHistoryLimit: 10
selector:
matchLabels:
app.kubernetes.io/instance: my-headlamp
app.kubernetes.io/name: headlamp
strategy:
rollingUpdate:
maxSurge: 25%
maxUnavailable: 25%
type: RollingUpdate
template:
metadata:
annotations:
kubectl.kubernetes.io/restartedAt: "2025-02-19T15:19:29Z"
creationTimestamp: null
labels:
app.kubernetes.io/instance: my-headlamp
app.kubernetes.io/name: headlamp
spec:
containers:
- args:
- -in-cluster
- -plugins-dir=/headlamp/plugins
env:
- name: HEADLAMP_CONFIG_OIDC_CLIENT_ID
value: ad6d237d-1680-4f42-8e9f-cd8680b3c093
- name: HEADLAMP_CONFIG_OIDC_CLIENT_SECRET
value: (Removed due to sensitive information)
- name: HEADLAMP_CONFIG_OIDC_IDP_ISSUER_URL
value: https://login.microsoftonline.com/c6e0547c-0aa6-491a-9fdc-cf5ef5c021df/v2.0
- name: HEADLAMP_CONFIG_OIDC_SCOPES
value: openid profile email
image: ghcr.io/headlamp-k8s/headlamp:v0.30.0
imagePullPolicy: IfNotPresent
livenessProbe:
failureThreshold: 3
httpGet:
path: /
port: http
scheme: HTTP
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
name: headlamp
ports:
- containerPort: 4466
name: http
protocol: TCP
readinessProbe:
failureThreshold: 3
httpGet:
path: /
port: http
scheme: HTTP
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
resources: {}
securityContext:
privileged: false
runAsGroup: 101
runAsNonRoot: true
runAsUser: 100
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
dnsPolicy: ClusterFirst
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
serviceAccount: my-headlamp
serviceAccountName: my-headlamp
terminationGracePeriodSeconds: 30
status:
availableReplicas: 1
conditions:

• lastTransitionTime: "2025-04-30T15:24:16Z"
  lastUpdateTime: "2025-04-30T15:24:16Z"
  message: Deployment has minimum availability.
  reason: MinimumReplicasAvailable
  status: "True"
  type: Available
• lastTransitionTime: "2025-04-14T17:23:48Z"
  lastUpdateTime: "2025-04-30T15:42:45Z"
  message: ReplicaSet "my-headlamp-56d569ffdb" has successfully progressed.
  reason: NewReplicaSetAvailable
  status: "True"
  type: Progressing
  observedGeneration: 21
  readyReplicas: 1
  replicas: 1
  updatedReplicas: 1
  `

The image below shows how the oidc provider is associated to the eks cluster:

Those are the claims configured in microsoft entra ID app registration:

These are the API permissions set in the microsoft entra id enterprise application:

Did anyone experience this issue before?

Let me know if I need to provide more evidences

Thanks

[illume]

There was support for this merged in #3177 and it will be in the next release. In there there's also some documentation.

A community member wrote a tutorial on how to do it using a reverse proxy here: https://soapfault.com/2025/03/01/azure-aks-entra-oidc-rbac-headlamp/

[joaobernardo-87]

Hi @illume ,

thanks for your response. Altough I'm using an EKS cluster, and not an AKS, like described in the tutorial, I followed the steps from there, but it didn't work. I'm still receiving the 403 forbidden and I'm not able to list the resources.
I also observed that the token is still getting the HEADLAMP_CONFIG_OIDC_CLIENT_ID and not the HEADLAMP_CONFIG_OIDC_VALIDATOR_CLIENT_ID in the "aud" claim. Same thing for the "iss" claim where it's still getting the value from HEADLAMP_CONFIG_OIDC_IDP_ISSUER_URL and not from HEADLAMP_CONFIG_OIDC_VALIDATOR_IDP_ISSUER_URL.

Here follows the headlamp's deployment yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
annotations:
deployment.kubernetes.io/revision: "26"
meta.helm.sh/release-name: my-headlamp
meta.helm.sh/release-namespace: kube-system
creationTimestamp: "2025-04-14T17:23:48Z"
generation: 27
labels:
app.kubernetes.io/instance: my-headlamp
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: headlamp
app.kubernetes.io/version: 0.30.0
helm.sh/chart: headlamp-0.30.1
name: my-headlamp
namespace: kube-system
resourceVersion: "168865206"
uid: 759aefeb-f9f9-412a-a004-43543f1d16cb
spec:
progressDeadlineSeconds: 600
replicas: 1
revisionHistoryLimit: 10
selector:
matchLabels:
app.kubernetes.io/instance: my-headlamp
app.kubernetes.io/name: headlamp
strategy:
rollingUpdate:
maxSurge: 25%
maxUnavailable: 25%
type: RollingUpdate
template:
metadata:
annotations:
kubectl.kubernetes.io/restartedAt: "2025-02-19T15:19:29Z"
creationTimestamp: null
labels:
app.kubernetes.io/instance: my-headlamp
app.kubernetes.io/name: headlamp
spec:
containers:
- args:
- -in-cluster
- -plugins-dir=/headlamp/plugins
env:
- name: HEADLAMP_CONFIG_OIDC_CLIENT_ID
value: ad6d237d-1680-4f42-8e9f-cd8680b3c093
- name: HEADLAMP_CONFIG_OIDC_CLIENT_SECRET
value: xxxxxxxxxxxxxxxxxxxxxx
- name: HEADLAMP_CONFIG_OIDC_IDP_ISSUER_URL
value: https://login.microsoftonline.com/c6e0547c-0aa6-491a-9fdc-cf5ef5c021df/v2.0
- name: HEADLAMP_CONFIG_OIDC_SCOPES
value: openid profile email groups user.read GroupMember.Read.All
- name: HEADLAMP_CONFIG_OIDC_VALIDATOR_CLIENT_ID
value: 6dae42f8-4368-4678-94ff-3960e28e3630
- name: HEADLAMP_CONFIG_OIDC_VALIDATOR_IDP_ISSUER_URL
value: https://sts.windows.net/c6e0547c-0aa6-491a-9fdc-cf5ef5c021df/
- name: HEADLAMP_CONFIG_OIDC_USE_ACCESS_TOKEN
value: "true"
image: ghcr.io/headlamp-k8s/headlamp:v0.30.0
imagePullPolicy: IfNotPresent
livenessProbe:
failureThreshold: 3
httpGet:
path: /
port: http
scheme: HTTP
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
name: headlamp
ports:
- containerPort: 4466
name: http
protocol: TCP
readinessProbe:
failureThreshold: 3
httpGet:
path: /
port: http
scheme: HTTP
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
resources: {}
securityContext:
privileged: false
runAsGroup: 101
runAsNonRoot: true
runAsUser: 100
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
dnsPolicy: ClusterFirst
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
serviceAccount: my-headlamp
serviceAccountName: my-headlamp
terminationGracePeriodSeconds: 30
status:
availableReplicas: 1
conditions:

• lastTransitionTime: "2025-04-30T15:24:16Z"
  lastUpdateTime: "2025-04-30T15:24:16Z"
  message: Deployment has minimum availability.
  reason: MinimumReplicasAvailable
  status: "True"
  type: Available
• lastTransitionTime: "2025-04-14T17:23:48Z"
  lastUpdateTime: "2025-05-28T14:59:48Z"
  message: ReplicaSet "my-headlamp-67d4fc85c5" has successfully progressed.
  reason: NewReplicaSetAvailable
  status: "True"
  type: Progressing
  observedGeneration: 27
  readyReplicas: 1
  replicas: 1
  updatedReplicas: 1

And it's a screenshot of how token configuration is set in Azure application:

Thanks