Kubernetes Cluster IP no such host : kube-state-metrics 설치

버전 설명

쿠버네티스 버전 1.26.1

구성 CLI 버전 1.21.1

해당 노드는 taint제거한 단일노드임

필요성

kubernetes의 정보를 활용하는 metricbeat-kubernetes.yml파일을 실행해보면 로그 메세지에 service/kubernetes의 Cluster IP 주소에 대해서 no such host 라는 로그가 찍힌적이 있다

그래서 무슨 이유인가 확인해봤다.

도커이미지에서는 확인이 안되었고, Helm으로 띄운 metricbeat에서 manifest를 확인해보니까 해당이미지를 사용하는걸 확인했다.

• 관련링크 : https://medium.com/finda-tech/kube-state-metrics에-대해서-1303b10fb8f8

주의사항 : 버전확인

kube-state-metrics는 kubernetes의 버전에 엄청 민감한것으로 보인다.

꼭 readme.md의 Compatibility matrix 항목을 확인해야한다.

 

 

Kubernetes client-go Version이라고 적혀있지만, 그냥 Kubernetes의 버전과 동일한것같았다.

kubectl get nodes -o wide

를 통해서 쿠버네티스의 버전을 확인해보고, 해당 버전에 맞는 kube-state-metrics 버전을 설치해야한다.

설치

현재 노드에서 K8s의 버전이 1.26.1 으로 확인되었고, 이는 main의 버전과 동일(2.8.0) 하므로, 바로 클론해준다

git clone <https://github.com/kubernetes/kube-state-metrics> 

실행

cd kube-state-metrics/examples

kubectl apply -f standard

이러면 standard 안에 있는 파일들이 실행되는데, 하나의 파일로 합치면 다음과 같다

실행결과 : 로그

I0220 09:21:11.077299       1 wrapper.go:98] "Starting kube-state-metrics"                                                                                 │
│ I0220 09:21:11.077631       1 server.go:186] "Used default resources"                                                                                      │
│ I0220 09:21:11.077725       1 types.go:184] "Using all namespaces"                                                                                         │
│ I0220 09:21:11.077792       1 server.go:219] "Metric allow-denylisting" allowDenyStatus="Excluding the following lists that were on denylist: "            │
│ W0220 09:21:11.077827       1 client_config.go:618] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.     │
│ I0220 09:21:11.078856       1 server.go:364] "Tested communication with server"                                                                            │
│ I0220 09:21:11.086812       1 server.go:369] "Run with Kubernetes cluster version" major="1" minor="26" gitVersion="v1.26.1" gitTreeState="clean" gitCommi │
│ I0220 09:21:11.086842       1 server.go:370] "Communication with server successful"                                                                        │
│ I0220 09:21:11.087467       1 server.go:316] "Started metrics server" metricsServerAddress=":8080"                                                         │
│ I0220 09:21:11.087917       1 metrics_handler.go:99] "Autosharding disabled"                                                                               │
│ I0220 09:21:11.088062       1 server.go:74] levelinfomsgListening onaddress:8080                                                                           │
│ I0220 09:21:11.088116       1 server.go:74] levelinfomsgTLS is disabled.http2falseaddress:8080                                                             │
│ I0220 09:21:11.088753       1 builder.go:254] "Active resources" activeStoreNames="certificatesigningrequests,configmaps,cronjobs,daemonsets,deployments,e │
│ I0220 09:21:11.089973       1 server.go:305] "Started kube-state-metrics self metrics server" telemetryAddress=":8081"                                     │
│ I0220 09:21:11.090161       1 server.go:74] levelinfomsgListening onaddress:8081                                                                           │
│ I0220 09:21:11.090229       1 server.go:74] levelinfomsgTLS is disabled.http2falseaddress:8081                                                             │
│ Stream closed EOF for kube-system/kube-state-metrics-7fd4bb7bb5-pwm78 (kube-state-metrics)

임의의 포트를 물고, 리버스 프록시를 해주는것같다.

단일파일

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  labels:
    app.kubernetes.io/component: exporter
    app.kubernetes.io/name: kube-state-metrics
    app.kubernetes.io/version: 2.8.0
  name: kube-state-metrics
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: kube-state-metrics
subjects:
- kind: ServiceAccount
  name: kube-state-metrics
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  labels:
    app.kubernetes.io/component: exporter
    app.kubernetes.io/name: kube-state-metrics
    app.kubernetes.io/version: 2.8.0
  name: kube-state-metrics
rules:
- apiGroups:
  - ""
  resources:
  - configmaps
  - secrets
  - nodes
  - pods
  - services
  - serviceaccounts
  - resourcequotas
  - replicationcontrollers
  - limitranges
  - persistentvolumeclaims
  - persistentvolumes
  - namespaces
  - endpoints
  verbs:
  - list
  - watch
- apiGroups:
  - apps
  resources:
  - statefulsets
  - daemonsets
  - deployments
  - replicasets
  verbs:
  - list
  - watch
- apiGroups:
  - batch
  resources:
  - cronjobs
  - jobs
  verbs:
  - list
  - watch
- apiGroups:
  - autoscaling
  resources:
  - horizontalpodautoscalers
  verbs:
  - list
  - watch
- apiGroups:
  - authentication.k8s.io
  resources:
  - tokenreviews
  verbs:
  - create
- apiGroups:
  - authorization.k8s.io
  resources:
  - subjectaccessreviews
  verbs:
  - create
- apiGroups:
  - policy
  resources:
  - poddisruptionbudgets
  verbs:
  - list
  - watch
- apiGroups:
  - certificates.k8s.io
  resources:
  - certificatesigningrequests
  verbs:
  - list
  - watch
- apiGroups:
  - discovery.k8s.io
  resources:
  - endpointslices
  verbs:
  - list
  - watch
- apiGroups:
  - storage.k8s.io
  resources:
  - storageclasses
  - volumeattachments
  verbs:
  - list
  - watch
- apiGroups:
  - admissionregistration.k8s.io
  resources:
  - mutatingwebhookconfigurations
  - validatingwebhookconfigurations
  verbs:
  - list
  - watch
- apiGroups:
  - networking.k8s.io
  resources:
  - networkpolicies
  - ingressclasses
  - ingresses
  verbs:
  - list
  - watch
- apiGroups:
  - coordination.k8s.io
  resources:
  - leases
  verbs:
  - list
  - watch
- apiGroups:
  - rbac.authorization.k8s.io
  resources:
  - clusterrolebindings
  - clusterroles
  - rolebindings
  - roles
  verbs:
  - list
  - watch
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app.kubernetes.io/component: exporter
    app.kubernetes.io/name: kube-state-metrics
    app.kubernetes.io/version: 2.8.0
  name: kube-state-metrics
  namespace: kube-system
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: kube-state-metrics
  template:
    metadata:
      labels:
        app.kubernetes.io/component: exporter
        app.kubernetes.io/name: kube-state-metrics
        app.kubernetes.io/version: 2.8.0
    spec:
      automountServiceAccountToken: true
      containers:
      - image: registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.8.0
        livenessProbe:
          httpGet:
            path: /healthz
            port: 8080
          initialDelaySeconds: 5
          timeoutSeconds: 5
        name: kube-state-metrics
        ports:
        - containerPort: 8080
          name: http-metrics
        - containerPort: 8081
          name: telemetry
        readinessProbe:
          httpGet:
            path: /
            port: 8081
          initialDelaySeconds: 5
          timeoutSeconds: 5
        securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            drop:
            - ALL
          readOnlyRootFilesystem: true
          runAsUser: 65534
      nodeSelector:
        kubernetes.io/os: linux
      serviceAccountName: kube-state-metrics
---
apiVersion: v1
automountServiceAccountToken: false
kind: ServiceAccount
metadata:
  labels:
    app.kubernetes.io/component: exporter
    app.kubernetes.io/name: kube-state-metrics
    app.kubernetes.io/version: 2.8.0
  name: kube-state-metrics
  namespace: kube-system
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app.kubernetes.io/component: exporter
    app.kubernetes.io/name: kube-state-metrics
    app.kubernetes.io/version: 2.8.0
  name: kube-state-metrics
  namespace: kube-system
spec:
  clusterIP: None
  ports:
  - name: http-metrics
    port: 8080
    targetPort: http-metrics
  - name: telemetry
    port: 8081
    targetPort: telemetry
  selector:
    app.kubernetes.io/name: kube-state-metrics