最近の Prism Central には、Microservice Infrastructure（MSP）という Kubernetes を利用した仕組みが導入されています。
以前に構築した Prism Central pc.2023.3 で、MSP の様子を見てみます。
今回の内容です。
kubectl 実行のポイント
MSP では、Prism Central の CVM（PCVM）が Kubernetes のノードになります。そこで、PCVM に SSH でログインして様子を見てみます。
しかしながら、nutanix / admin ユーザでは、パーミッションが不足していて、kubectl が利用できません。
nutanix@NTNX-192-168-20-15-A-PCVM:~$ id uid=1000(nutanix) gid=1000(nutanix) groups=1000(nutanix),160(snmp),600(java),990(docker) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 nutanix@NTNX-192-168-20-15-A-PCVM:~$ kubectl get nodes Error in configuration: * unable to read client-cert /var/nutanix/etc/kubernetes/ssl/master.pem for kubelet due to open /var/nutanix/etc/kubernetes/ssl/master.pem: permission denied * unable to read client-key /var/nutanix/etc/kubernetes/ssl/master-key.pem for kubelet due to open /var/nutanix/etc/kubernetes/ssl/master-key.pem: permission denied * unable to read certificate-authority /var/nutanix/etc/kubernetes/ssl/ca.pem for prism-central-699db5 due to open /var/nutanix/etc/kubernetes/ssl/ca.pem: permission denied
そこで、root ユーザにスイッチして作業します。
nutanix@NTNX-192-168-20-15-A-PCVM:~$ sudo su - Last login: Mon Dec 4 05:36:32 PST 2023 on pts/0 root@NTNX-192-168-20-15-A-PCVM:~#
この Prism Central は、シングル ノード構成です。そのため Kubernetes クラスタとしても、シングルノード構成です。
root@NTNX-192-168-20-15-A-PCVM:~# kubectl get nodes NAME STATUS ROLES AGE VERSION ntnx-192-168-20-15-a-pcvm Ready control-plane,etcd,master,node 28d v1.17.17-18+1e6914054e41d3-ntnx
MSP の Kubernetes リソースの様子
Pod とコンテナ イメージ
起動されている Pod の様子です。ちなみに、親リソースにあたる Deployment や StatefulSet については権限不足で参照できませんでした。
root@NTNX-192-168-20-15-A-PCVM:~# kubectl get pods -A -o wide NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES default security-dashboard-786dfb58d7-ppm9p 1/1 Running 1 28d 10.100.0.151 ntnx-192-168-20-15-a-pcvm <none> <none> kube-system cloud-controller-manager-5cwdp 1/1 Running 1 28d 10.100.0.147 ntnx-192-168-20-15-a-pcvm <none> <none> kube-system coredns-5477fcbb7d-l8wkv 1/1 Running 1 28d 10.100.0.136 ntnx-192-168-20-15-a-pcvm <none> <none> kube-system kube-apiserver-ntnx-192-168-20-15-a-pcvm 3/3 Running 3 28d 192.168.5.2 ntnx-192-168-20-15-a-pcvm <none> <none> kube-system kube-flannel-ds-cmkkb 1/1 Running 2 28d 192.168.5.2 ntnx-192-168-20-15-a-pcvm <none> <none> kube-system kube-proxy-ds-tss97 1/1 Running 1 28d 192.168.5.2 ntnx-192-168-20-15-a-pcvm <none> <none> kube-system lb-controller-deployment-0 1/1 Running 1 28d 10.100.0.145 ntnx-192-168-20-15-a-pcvm <none> <none> kube-system mspdns-tr9nh 1/1 Running 1 28d 10.100.0.140 ntnx-192-168-20-15-a-pcvm <none> <none> kube-system mspserviceregistry-65b585b987-dpm2k 1/1 Running 1 28d 10.100.0.131 ntnx-192-168-20-15-a-pcvm <none> <none> ntnx-base alerts-broker-6d4fcc656-cnjjd 1/1 Running 1 28d 10.100.0.138 ntnx-192-168-20-15-a-pcvm <none> <none> ntnx-base backrest-backup-cape-n5569 0/1 Completed 0 67m 10.100.0.155 ntnx-192-168-20-15-a-pcvm <none> <none> ntnx-base cape-5dbf8994c6-kszqn 2/2 Running 2 28d 10.100.0.142 ntnx-192-168-20-15-a-pcvm <none> <none> ntnx-base cape-backrest-shared-repo-c6d9b6c58-6kkhl 1/1 Running 1 28d 10.100.0.134 ntnx-192-168-20-15-a-pcvm <none> <none> ntnx-base cape-full-sch-backup-lhpxg 0/1 Completed 0 46m 10.100.0.156 ntnx-192-168-20-15-a-pcvm <none> <none> ntnx-base cape-gfjc-769cb54d7f-rxkfx 2/2 Running 2 28d 10.100.0.139 ntnx-192-168-20-15-a-pcvm <none> <none> ntnx-base cape-stanza-create-j4d2t 0/1 Completed 0 28d 10.100.0.24 ntnx-192-168-20-15-a-pcvm <none> <none> ntnx-base iam-bootstrap-mpzgd 0/1 Completed 0 28d 10.100.0.20 ntnx-192-168-20-15-a-pcvm <none> <none> ntnx-base iam-proxy-6b9689f9bf-928q6 2/2 Running 2 28d 10.100.0.135 ntnx-192-168-20-15-a-pcvm <none> <none> ntnx-base iam-proxy-control-plane-5895966c95-kb5pw 1/1 Running 1 28d 10.100.0.154 ntnx-192-168-20-15-a-pcvm <none> <none> ntnx-base iam-themis-58bb476-q5m4w 1/1 Running 1 28d 10.100.0.132 ntnx-192-168-20-15-a-pcvm <none> <none> ntnx-base iam-ui-7d6755888-6txhj 1/1 Running 1 28d 10.100.0.141 ntnx-192-168-20-15-a-pcvm <none> <none> ntnx-base iam-user-authn-7c99bf7789-ndpxl 1/1 Running 2 28d 10.100.0.152 ntnx-192-168-20-15-a-pcvm <none> <none> ntnx-base pgo-deploy-7gz6p 0/1 Completed 0 28d 10.100.0.16 ntnx-192-168-20-15-a-pcvm <none> <none> ntnx-base pgo-pull-image-4jdkt 0/28 Completed 0 28d 10.100.0.17 ntnx-192-168-20-15-a-pcvm <none> <none> ntnx-base postgres-operator-6bd777cb4f-jvt6s 4/4 Running 6 28d 10.100.0.148 ntnx-192-168-20-15-a-pcvm <none> <none> ntnx-base redis-standalone-667f77748f-fd2xw 3/3 Running 3 28d 10.100.0.149 ntnx-192-168-20-15-a-pcvm <none> <none> ntnx-base svcmgr-7bb69656d-2qngt 1/1 Running 1 28d 10.100.0.146 ntnx-192-168-20-15-a-pcvm <none> <none> ntnx-system alertmanager-main-0 2/2 Running 2 28d 10.100.0.133 ntnx-192-168-20-15-a-pcvm <none> <none> ntnx-system csi-node-ntnx-plugin-pwvmt 2/2 Running 2 28d 192.168.5.2 ntnx-192-168-20-15-a-pcvm <none> <none> ntnx-system csi-provisioner-ntnx-plugin-0 2/2 Running 2 28d 10.100.0.153 ntnx-192-168-20-15-a-pcvm <none> <none> ntnx-system kube-state-metrics-74b9c847b5-bztzd 3/3 Running 3 28d 10.100.0.130 ntnx-192-168-20-15-a-pcvm <none> <none> ntnx-system mutator-webhook-dep-754df4bfd7-lb8q4 1/1 Running 1 28d 192.168.5.2 ntnx-192-168-20-15-a-pcvm <none> <none> ntnx-system node-exporter-9d5dz 2/2 Running 2 28d 192.168.5.2 ntnx-192-168-20-15-a-pcvm <none> <none> ntnx-system ntnx-cluster-maintainer-d8974cf44-lxbn6 1/1 Running 1 28d 10.100.0.143 ntnx-192-168-20-15-a-pcvm <none> <none> ntnx-system ntnx-k8s-cluster-maintainer-operator-9df9746b-24dxx 1/1 Running 1 28d 10.100.0.150 ntnx-192-168-20-15-a-pcvm <none> <none> ntnx-system prometheus-k8s-0 3/3 Running 4 28d 10.100.0.129 ntnx-192-168-20-15-a-pcvm <none> <none> ntnx-system prometheus-operator-85846db5bc-nq29d 1/1 Running 1 28d 10.100.0.144 ntnx-192-168-20-15-a-pcvm <none> <none> nusights msp-prometheus-collector-fd998fcf8-f8hcb 1/1 Running 1 28d 192.168.5.2 ntnx-192-168-20-15-a-pcvm <none> <none>
コンテナ イメージは、MSP 内蔵のレジストリからダウンロードしているようです。
root@NTNX-192-168-20-15-A-PCVM:~# kubectl get pods -o yaml -A | grep image: | sort -u
image: msp-registry.prism-central.cluster.local:5001/alertmanager:v0.18.0
image: msp-registry.prism-central.cluster.local:5001/cluster-maintainer-controller:0.1.0
image: msp-registry.prism-central.cluster.local:5001/cluster-maintainer-service:0.1.0
image: msp-registry.prism-central.cluster.local:5001/configmap-reload:v0.3.0
image: msp-registry.prism-central.cluster.local:5001/coredns:1.6.6
image: msp-registry.prism-central.cluster.local:5001/csi-node-driver-registrar:v2.2.0
image: msp-registry.prism-central.cluster.local:5001/csi-provisioner:v2.2.0
image: msp-registry.prism-central.cluster.local:5001/flannel:v0.15.1-amd64
image: msp-registry.prism-central.cluster.local:5001/hyperkube:v1.17.17-8-msp-ntnx
image: msp-registry.prism-central.cluster.local:5001/iam-bootstrap:2503
image: msp-registry.prism-central.cluster.local:5001/iam-proxy:1094
image: msp-registry.prism-central.cluster.local:5001/iam-proxy-control-plane:1094
image: msp-registry.prism-central.cluster.local:5001/iam-ratelimit:78
image: msp-registry.prism-central.cluster.local:5001/iam-themis:7385
image: msp-registry.prism-central.cluster.local:5001/iam-ui:2695
image: msp-registry.prism-central.cluster.local:5001/iam-user-authn:275
image: msp-registry.prism-central.cluster.local:5001/iam-user-authn:8456
image: msp-registry.prism-central.cluster.local:5001/k8s-mutator:1.1.0
image: msp-registry.prism-central.cluster.local:5001/kube-rbac-proxy:v0.4.1
image: msp-registry.prism-central.cluster.local:5001/kube-state-metrics:v1.9.2
image: msp-registry.prism-central.cluster.local:5001/lb-controller:2.3.5
image: msp-registry.prism-central.cluster.local:5001/msp-service-registry:2.2.3
image: msp-registry.prism-central.cluster.local:5001/node-exporter:v0.18.1
image: msp-registry.prism-central.cluster.local:5001/ntnx-api-pc-service-manager:1426
image: msp-registry.prism-central.cluster.local:5001/ntnx-api-security-dashboard-pc:1697
- image: msp-registry.prism-central.cluster.local:5001/ntnx-api-security-dashboard-pc:1697
image: msp-registry.prism-central.cluster.local:5001/ntnx-cloudprovider:2.2.2
image: msp-registry.prism-central.cluster.local:5001/ntnx-csi:v2.4.7
image: msp-registry.prism-central.cluster.local:5001/nusights-msp-prometheus-collector:68
image: msp-registry.prism-central.cluster.local:5001/pg-service-operator:crunchy-postgres-exporter-v2
image: msp-registry.prism-central.cluster.local:5001/pg-service-operator:crunchy-postgres-exporter-v3
image: msp-registry.prism-central.cluster.local:5001/pg-service-operator:crunchy-postgres-ha-v2
image: msp-registry.prism-central.cluster.local:5001/pg-service-operator:crunchy-postgres-ha-v3
image: msp-registry.prism-central.cluster.local:5001/pg-service-operator:pgo-apiserver-op-v3
image: msp-registry.prism-central.cluster.local:5001/pg-service-operator:pgo-apiserver-v2
image: msp-registry.prism-central.cluster.local:5001/pg-service-operator:pgo-backrest-repo-sync-v2
image: msp-registry.prism-central.cluster.local:5001/pg-service-operator:pgo-backrest-repo-sync-v3
image: msp-registry.prism-central.cluster.local:5001/pg-service-operator:pgo-backrest-repo-v2
image: msp-registry.prism-central.cluster.local:5001/pg-service-operator:pgo-backrest-repo-v3
image: msp-registry.prism-central.cluster.local:5001/pg-service-operator:pgo-backrest-restore-v2
image: msp-registry.prism-central.cluster.local:5001/pg-service-operator:pgo-backrest-restore-v3
image: msp-registry.prism-central.cluster.local:5001/pg-service-operator:pgo-backrest-v2
image: msp-registry.prism-central.cluster.local:5001/pg-service-operator:pgo-backrest-v3
image: msp-registry.prism-central.cluster.local:5001/pg-service-operator:pgo-base-v2
image: msp-registry.prism-central.cluster.local:5001/pg-service-operator:pgo-base-v3
image: msp-registry.prism-central.cluster.local:5001/pg-service-operator:pgo-client-v2
image: msp-registry.prism-central.cluster.local:5001/pg-service-operator:pgo-client-v3
image: msp-registry.prism-central.cluster.local:5001/pg-service-operator:pgo-deployer-op-v3
image: msp-registry.prism-central.cluster.local:5001/pg-service-operator:pgo-deployer-v2
image: msp-registry.prism-central.cluster.local:5001/pg-service-operator:pgo-event-op-v3
image: msp-registry.prism-central.cluster.local:5001/pg-service-operator:pgo-event-v2
image: msp-registry.prism-central.cluster.local:5001/pg-service-operator:pgo-rmdata-v2
image: msp-registry.prism-central.cluster.local:5001/pg-service-operator:pgo-rmdata-v3
image: msp-registry.prism-central.cluster.local:5001/pg-service-operator:pgo-scheduler-op-v3
image: msp-registry.prism-central.cluster.local:5001/pg-service-operator:pgo-scheduler-v2
image: msp-registry.prism-central.cluster.local:5001/pg-service-operator:postgres-operator-op-v3
image: msp-registry.prism-central.cluster.local:5001/pg-service-operator:postgres-operator-v2
image: msp-registry.prism-central.cluster.local:5001/prometheus-config-reloader:v0.35.1
image: msp-registry.prism-central.cluster.local:5001/prometheus-operator:v0.35.1
image: msp-registry.prism-central.cluster.local:5001/prometheus:v2.11.0
image: msp-registry.prism-central.cluster.local:5001/redis_exporter:v1.0.0-alpine
image: msp-registry.prism-central.cluster.local:5001/redis:v1
ストレージ（CSI）
CSI ドライバーが、デフォルトで構成されています。
root@NTNX-192-168-20-15-A-PCVM:~# kubectl get csidrivers NAME CREATED AT csi.nutanix.com 2023-11-06T10:49:47Z
kubectl では StorageClass や PV、PVC などは権限不足で確認できませんでしたが、Prism Central からであれば、「pvc-～」という名前の VG が作成されていることがわかります。
Nutanix CSI Driver で管理しているようで、VG の Description 確認することで、利用している Kubernetes リソースがわかります。
Service リソース
Service リソースの様子です。EXTERNAL-IP で利用されている 192.168.20.15 は、Prism Central に Web ブラウザなどからアクセスする IP アドレスです。
PCVM のデプロイ中にデフォルト値として指定した 192.168.5.2 も、一部のサービスで利用されていることがわかります。この IP アドレスは、Nutanix HCI としてはハイパーバイザー → CVM の接続でかならず必要にあることで有名なものです。しかし MSP の内部のみで利用されるコンポーネントからは直接 CVM に接続する必要がないため、PCVM に 192.168.5.2 が設定されていても問題ないようです。
root@NTNX-192-168-20-15-A-PCVM:~# kubectl get svc -A NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE default kubernetes ClusterIP 10.200.32.1 <none> 443/TCP 28d default security-dashboard LoadBalancer 10.200.32.78 192.168.20.15 8801:31700/TCP,8802:30645/TCP,8800:32568/TCP,8803:30970/TCP,8805:31538/TCP,8804:32362/TCP 28d kube-system etcd ClusterIP None <none> 2379/TCP 28d kube-system kube-controller-manager-prometheus-discovery ClusterIP None <none> 10252/TCP 28d kube-system kube-dns ClusterIP 10.200.32.10 <none> 53/UDP,53/TCP,9153/TCP 28d kube-system kube-scheduler-prometheus-discovery ClusterIP None <none> 10251/TCP 28d kube-system kubelet ClusterIP None <none> 10250/TCP 28d kube-system lb-controller-service ClusterIP 10.200.32.205 <none> 9812/TCP 28d kube-system mspdns NodePort 10.200.32.21 <none> 53:31000/UDP,53:31000/TCP 28d kube-system mspserviceregistry NodePort 10.200.32.22 <none> 2085:31002/TCP 28d ntnx-base alerts-broker ClusterIP 10.200.32.29 <none> 6000/TCP 28d ntnx-base cape ClusterIP 10.200.32.83 <none> 9187/TCP,2022/TCP,5432/TCP 28d ntnx-base cape-backrest-shared-repo ClusterIP 10.200.32.174 <none> 2022/TCP 28d ntnx-base cape-replica ClusterIP 10.200.32.141 <none> 9187/TCP,2022/TCP,5432/TCP 28d ntnx-base iam-proxy LoadBalancer 10.200.32.33 192.168.20.15 8440:32705/TCP,8443:32736/TCP,8445:32092/TCP 28d ntnx-base iam-proxy-admin ClusterIP 10.200.32.80 <none> 8001/TCP 28d ntnx-base iam-proxy-cac LoadBalancer 10.200.32.166 192.168.20.15 9441:32396/TCP,9442:31264/TCP 28d ntnx-base iam-proxy-control-plane ClusterIP 10.200.32.165 <none> 8446/TCP,8447/TCP 28d ntnx-base iam-themis ClusterIP 10.200.32.110 <none> 5558/TCP,5560/TCP 28d ntnx-base iam-ui ClusterIP 10.200.32.145 <none> 5557/TCP 28d ntnx-base iam-user-authn ClusterIP 10.200.32.81 <none> 5556/TCP,5558/TCP 28d ntnx-base postgres-operator ClusterIP 10.200.32.72 <none> 8443/TCP,4171/TCP,4150/TCP 28d ntnx-base redis-exporter ClusterIP 10.200.32.140 <none> 9121/TCP 28d ntnx-base redis-sentinel ClusterIP 10.200.32.225 <none> 26379/TCP,6379/TCP 28d ntnx-base stolon-keeper ClusterIP 10.200.32.209 <none> 9187/TCP,5432/TCP 28d ntnx-base stolon-proxy-service ClusterIP 10.200.32.13 <none> 9187/TCP,5432/TCP 28d ntnx-base svcmgr LoadBalancer 10.200.32.252 192.168.5.2 2083:32191/TCP 28d ntnx-system alertmanager-main ClusterIP 10.200.32.143 <none> 9093/TCP 28d ntnx-system alertmanager-operated ClusterIP None <none> 9093/TCP,9094/TCP,9094/UDP 28d ntnx-system cmsp-envoy-1 ExternalName <none> 192.168.5.2 9902/TCP 28d ntnx-system csi-metrics-service ClusterIP 10.200.32.162 <none> 9809/TCP 28d ntnx-system csi-provisioner-ntnx-plugin ClusterIP 10.200.32.25 <none> 12345/TCP 28d ntnx-system kube-state-metrics ClusterIP None <none> 8443/TCP,9443/TCP 28d ntnx-system mutator-webhook-service ClusterIP 10.200.32.12 <none> 9815/TCP 28d ntnx-system node-exporter ClusterIP None <none> 9100/TCP 28d ntnx-system prometheus-k8s ClusterIP 10.200.32.2 <none> 9090/TCP 28d ntnx-system prometheus-operated ClusterIP None <none> 9090/TCP 28d ntnx-system prometheus-operator ClusterIP None <none> 8080/TCP 28d nusights msp-prometheus-collector ClusterIP 10.200.32.37 <none> 2411/TCP,2410/TCP
CNI には、Flannel（VXLAN）が使用されています。
root@NTNX-192-168-20-15-A-PCVM:~# kubectl get pod -A | grep flannel
kube-system kube-flannel-ds-cmkkb 1/1 Running 2 28d
root@NTNX-192-168-20-15-A-PCVM:~# kubectl get nodes -o yaml | head
apiVersion: v1
items:
- apiVersion: v1
kind: Node
metadata:
annotations:
csi.volume.kubernetes.io/nodeid: '{"csi.nutanix.com":"ntnx-192-168-20-15-a-pcvm"}'
flannel.alpha.coreos.com/backend-data: '{"VNI":1,"VtepMAC":"56:63:ea:90:12:94"}'
flannel.alpha.coreos.com/backend-type: vxlan
flannel.alpha.coreos.com/kube-subnet-manager: "true"
PCVM のネットワーク構成
シングル ノード構成なのですが、Flannel のインターフェース（flannel.1）で VXLAN が構成されています。
root@NTNX-192-168-20-15-A-PCVM:~# ip -d link show dev flannel.1
4: flannel.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue state UNKNOWN mode DEFAULT group default
link/ether 56:63:ea:90:12:94 brd ff:ff:ff:ff:ff:ff promiscuity 0
vxlan id 1 local 192.168.5.2 dev eth1 srcport 0 0 dstport 8472 nolearning ageing 300 udpcsum noudp6zerocsumtx noudp6zerocsumrx addrgenmode eui64 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535
PCVM はコンテナ ホストも兼ねているので、ネットワーク インターフェースも多く作成されています。
root@NTNX-192-168-20-15-A-PCVM:~# ip address show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 50:6b:8d:a2:ce:a9 brd ff:ff:ff:ff:ff:ff
inet 192.168.20.15/24 brd 192.168.20.255 scope global eth0
valid_lft forever preferred_lft forever
inet6 fe80::526b:8dff:fea2:cea9/64 scope link
valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default qlen 1000
link/ether b2:e6:4c:6b:b0:04 brd ff:ff:ff:ff:ff:ff
inet 192.168.5.2/24 scope global eth1
valid_lft forever preferred_lft forever
inet6 fe80::b0e6:4cff:fe6b:b004/64 scope link
valid_lft forever preferred_lft forever
4: flannel.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue state UNKNOWN group default
link/ether 56:63:ea:90:12:94 brd ff:ff:ff:ff:ff:ff
inet 10.100.0.0/32 brd 10.100.0.0 scope global flannel.1
valid_lft forever preferred_lft forever
inet6 fe80::5463:eaff:fe90:1294/64 scope link
valid_lft forever preferred_lft forever
5: cni0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue state UP group default qlen 1000
link/ether be:47:c3:45:4c:0b brd ff:ff:ff:ff:ff:ff
inet 10.100.0.1/24 brd 10.100.0.255 scope global cni0
valid_lft forever preferred_lft forever
inet6 fe80::bc47:c3ff:fe45:4c0b/64 scope link
valid_lft forever preferred_lft forever
6: veth8a7f70ef@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue master cni0 state UP group default
link/ether ea:8f:c5:a8:a7:88 brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet6 fe80::e88f:c5ff:fea8:a788/64 scope link
valid_lft forever preferred_lft forever
7: veth98df1cd1@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue master cni0 state UP group default
link/ether b2:5d:15:e0:54:2d brd ff:ff:ff:ff:ff:ff link-netnsid 1
inet6 fe80::b05d:15ff:fee0:542d/64 scope link
valid_lft forever preferred_lft forever
8: veth2e00cbad@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue master cni0 state UP group default
link/ether 4e:31:f1:d3:21:ba brd ff:ff:ff:ff:ff:ff link-netnsid 2
inet6 fe80::4c31:f1ff:fed3:21ba/64 scope link
valid_lft forever preferred_lft forever
9: veth02cceaa5@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue master cni0 state UP group default
link/ether be:9f:0d:6c:37:55 brd ff:ff:ff:ff:ff:ff link-netnsid 3
inet6 fe80::bc9f:dff:fe6c:3755/64 scope link
valid_lft forever preferred_lft forever
10: veth504e2c8c@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue master cni0 state UP group default
link/ether 22:16:31:77:b2:44 brd ff:ff:ff:ff:ff:ff link-netnsid 4
inet6 fe80::2016:31ff:fe77:b244/64 scope link
valid_lft forever preferred_lft forever
11: veth2a464585@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue master cni0 state UP group default
link/ether ba:b7:11:55:ea:43 brd ff:ff:ff:ff:ff:ff link-netnsid 5
inet6 fe80::b8b7:11ff:fe55:ea43/64 scope link
valid_lft forever preferred_lft forever
12: vethec5211d8@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue master cni0 state UP group default
link/ether a2:84:fb:4a:05:a7 brd ff:ff:ff:ff:ff:ff link-netnsid 6
inet6 fe80::a084:fbff:fe4a:5a7/64 scope link
valid_lft forever preferred_lft forever
13: veth70642072@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue master cni0 state UP group default
link/ether be:85:19:ed:10:0e brd ff:ff:ff:ff:ff:ff link-netnsid 7
inet6 fe80::bc85:19ff:feed:100e/64 scope link
valid_lft forever preferred_lft forever
14: vethedf225db@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue master cni0 state UP group default
link/ether 0a:0d:c5:f1:99:10 brd ff:ff:ff:ff:ff:ff link-netnsid 8
inet6 fe80::80d:c5ff:fef1:9910/64 scope link
valid_lft forever preferred_lft forever
16: veth2ee814a5@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue master cni0 state UP group default
link/ether 46:21:1a:61:66:21 brd ff:ff:ff:ff:ff:ff link-netnsid 10
inet6 fe80::4421:1aff:fe61:6621/64 scope link
valid_lft forever preferred_lft forever
17: vethc7b28c46@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue master cni0 state UP group default
link/ether 26:63:bd:4d:f3:05 brd ff:ff:ff:ff:ff:ff link-netnsid 11
inet6 fe80::2463:bdff:fe4d:f305/64 scope link
valid_lft forever preferred_lft forever
18: vethc500a097@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue master cni0 state UP group default
link/ether b2:f2:f1:83:8b:ab brd ff:ff:ff:ff:ff:ff link-netnsid 12
inet6 fe80::b0f2:f1ff:fe83:8bab/64 scope link
valid_lft forever preferred_lft forever
19: veth2876fcb9@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue master cni0 state UP group default
link/ether 8e:cd:a0:07:83:1e brd ff:ff:ff:ff:ff:ff link-netnsid 13
inet6 fe80::8ccd:a0ff:fe07:831e/64 scope link
valid_lft forever preferred_lft forever
20: vethe5f95d7a@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue master cni0 state UP group default
link/ether 86:fc:d9:bd:18:17 brd ff:ff:ff:ff:ff:ff link-netnsid 14
inet6 fe80::84fc:d9ff:febd:1817/64 scope link
valid_lft forever preferred_lft forever
21: vethdbb8ee00@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue master cni0 state UP group default
link/ether a6:45:fc:0a:8c:86 brd ff:ff:ff:ff:ff:ff link-netnsid 15
inet6 fe80::a445:fcff:fe0a:8c86/64 scope link
valid_lft forever preferred_lft forever
22: vetha11605fd@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue master cni0 state UP group default
link/ether 12:3b:62:1c:d3:03 brd ff:ff:ff:ff:ff:ff link-netnsid 16
inet6 fe80::103b:62ff:fe1c:d303/64 scope link
valid_lft forever preferred_lft forever
23: veth6ea45431@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue master cni0 state UP group default
link/ether b2:2b:48:bd:f9:13 brd ff:ff:ff:ff:ff:ff link-netnsid 17
inet6 fe80::b02b:48ff:febd:f913/64 scope link
valid_lft forever preferred_lft forever
24: veth554b098b@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue master cni0 state UP group default
link/ether d6:51:b7:5c:3b:8f brd ff:ff:ff:ff:ff:ff link-netnsid 18
inet6 fe80::d451:b7ff:fe5c:3b8f/64 scope link
valid_lft forever preferred_lft forever
25: vethc8e65cc0@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue master cni0 state UP group default
link/ether 7a:d6:6e:5d:d4:a4 brd ff:ff:ff:ff:ff:ff link-netnsid 19
inet6 fe80::78d6:6eff:fe5d:d4a4/64 scope link
valid_lft forever preferred_lft forever
26: veth3297435e@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue master cni0 state UP group default
link/ether 5e:70:5e:d5:e9:ca brd ff:ff:ff:ff:ff:ff link-netnsid 20
inet6 fe80::5c70:5eff:fed5:e9ca/64 scope link
valid_lft forever preferred_lft forever
27: vethde10b1d9@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue master cni0 state UP group default
link/ether 6e:d6:69:22:37:b5 brd ff:ff:ff:ff:ff:ff link-netnsid 21
inet6 fe80::6cd6:69ff:fe22:37b5/64 scope link
valid_lft forever preferred_lft forever
28: veth0567b693@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue master cni0 state UP group default
link/ether 9a:7c:0f:7c:d4:08 brd ff:ff:ff:ff:ff:ff link-netnsid 22
inet6 fe80::987c:fff:fe7c:d408/64 scope link
valid_lft forever preferred_lft forever
29: vethf2152c3c@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue master cni0 state UP group default
link/ether 2e:3a:55:94:fa:11 brd ff:ff:ff:ff:ff:ff link-netnsid 23
inet6 fe80::2c3a:55ff:fe94:fa11/64 scope link
valid_lft forever preferred_lft forever
30: veth0b0f0045@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue master cni0 state UP group default
link/ether f2:36:9a:48:ee:a3 brd ff:ff:ff:ff:ff:ff link-netnsid 9
inet6 fe80::f036:9aff:fe48:eea3/64 scope link
valid_lft forever preferred_lft forever
31: veth8cc7df8d@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue master cni0 state UP group default
link/ether 2e:9c:22:fd:fa:f9 brd ff:ff:ff:ff:ff:ff link-netnsid 24
inet6 fe80::2c9c:22ff:fefd:faf9/64 scope link
valid_lft forever preferred_lft forever
34: vetha304935f@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue master cni0 state UP group default
link/ether f2:27:1b:a5:f5:eb brd ff:ff:ff:ff:ff:ff link-netnsid 25
inet6 fe80::f027:1bff:fea5:f5eb/64 scope link
valid_lft forever preferred_lft forever
さきほど話題にした 192.168.5.2 は、eth1 に割り当てられています。
root@NTNX-192-168-20-15-A-PCVM:~# ip address show dev eth1
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default qlen 1000
link/ether b2:e6:4c:6b:b0:04 brd ff:ff:ff:ff:ff:ff
inet 192.168.5.2/24 scope global eth1
valid_lft forever preferred_lft forever
inet6 fe80::b0e6:4cff:fe6b:b004/64 scope link
valid_lft forever preferred_lft forever
以上。
