NTNX>日記

個人的な趣味による Nutanix Community Edition 日記。Japanese のみですみません。

Nested Nutanix CE で Multi-Node Cluster の Open vSwitch に Bridge / Bond port を追加してみる。

複数の NIC がある Nutanix CE で、Open vSwitch に Bridge を追加してみます。Prism の Web UI からだと設定できないので、CVM にログインして CLI を使用します。

  • Bridge の作成は、manage_ovs コマンドを使用します。
  • 基本的に CVM から操作します。(AHV へのコマンド実行も CVM から)
  • CE のバージョンは ce-2018.01.31-stable です。

今回は、ESXi で起動している Nested Nutanix CE です。これは、下記のようにセットアップしています。

Nutanix CE は、4 ノードのクラスタ構成にしています。Nutanix Cmdlets でクラスタに参加しているノードを見ておきます。

NTNX> Get-NTNXHost | select @{N="Cluster";E={(Get-NTNXCluster -Id $_.ClusterUuid).name}},name,hypervisorAddress,serviceVMExternalIP,hypervisorFullName | sort hypervisorAddress | ft -AutoSize

Cluster     name            hypervisorAddress serviceVMExternalIP hypervisorFullName
-------     ----            ----------------- ------------------- ------------------
ce-ntnx-n10 NTNX-54e60e4a-A 192.168.1.181     192.168.1.185       Nutanix 20180123.170
ce-ntnx-n10 NTNX-5eab51d4-A 192.168.1.182     192.168.1.186       Nutanix 20180123.170
ce-ntnx-n10 NTNX-a17f7da5-A 192.168.1.183     192.168.1.187       Nutanix 20180123.170
ce-ntnx-n10 NTNX-7cb824ec-A 192.168.1.184     192.168.1.188       Nutanix 20180123.170

ESXi 上の VM にしているので、PowerCLI で Nutanix CE をインストールしている VM の構成を見ておきます。

PowerCLI> Get-Folder nutanix-4node | Get-VM | Get-VMGuest | select VM,{$_.IPAddress | where {$_ -like "192*"}} | sort VM

VM       $_.IPAddress | where {$_ -like "192*"}
--       --------------------------------------
ntnx-n11 {192.168.1.181, 192.168.5.1}
ntnx-n12 {192.168.1.182, 192.168.5.1}
ntnx-n13 {192.168.1.183, 192.168.5.1}
ntnx-n14 {192.168.1.184, 192.168.5.1}

それぞれの ESXi の VM に、vNIC(Nutainx AHV から見ると物理 NIC)が 4つあります。

PowerCLI> Get-Folder nutanix-4node | Get-VM | Get-NetworkAdapter | select Parent,Name,MacAddress | sort Parent,Name,PortGroupName

Parent   Name              MacAddress
------   ----              ----------
ntnx-n11 Network adapter 1 00:50:56:8a:f9:d2
ntnx-n11 Network adapter 2 00:50:56:8a:26:8d
ntnx-n11 Network adapter 3 00:50:56:8a:8b:b0
ntnx-n11 Network adapter 4 00:50:56:8a:40:b2
ntnx-n12 Network adapter 1 00:50:56:8a:32:d4
ntnx-n12 Network adapter 2 00:50:56:8a:f6:86
ntnx-n12 Network adapter 3 00:50:56:8a:d6:4b
ntnx-n12 Network adapter 4 00:50:56:8a:c1:15
ntnx-n13 Network adapter 1 00:50:56:8a:76:f8
ntnx-n13 Network adapter 2 00:50:56:8a:6a:3c
ntnx-n13 Network adapter 3 00:50:56:8a:10:1b
ntnx-n13 Network adapter 4 00:50:56:8a:d1:13
ntnx-n14 Network adapter 1 00:50:56:8a:5c:94
ntnx-n14 Network adapter 2 00:50:56:8a:5c:b1
ntnx-n14 Network adapter 3 00:50:56:8a:f7:50
ntnx-n14 Network adapter 4 00:50:56:8a:24:7e

CVM から見た AHV のネットワーク構成

インターフェースは 4つです。

nutanix@NTNX-7cb824ec-A-CVM:192.168.1.188:~$ PS1='CVM$ '
CVM$ manage_ovs show_interfaces
  name  mode  link speed
ens161 10000 False  None
ens192 10000  True 10000
ens224 10000 False  None
ens256 10000 False  None

すべての AHV でまとめて確認できます。

CVM$ allssh manage_ovs show_interfaces
================== 192.168.1.185 =================
  name  mode  link speed
ens161 10000 False  None
ens192 10000  True 10000
ens224 10000 False  None
ens256 10000 False  None
================== 192.168.1.186 =================
  name  mode  link speed
ens161 10000 False  None
ens192 10000  True 10000
ens224 10000 False  None
ens256 10000 False  None
================== 192.168.1.187 =================
  name  mode  link speed
ens161 10000 False  None
ens192 10000  True 10000
ens224 10000 False  None
ens256 10000 False  None
================== 192.168.1.188 =================
  name  mode  link speed
ens161 10000 False  None
ens192 10000  True 10000
ens224 10000 False  None
ens256 10000 False  None
CVM$

MAC アドレスをもとに確認すると、NIC の名前は下記になっています。

  • vNIC#1: ens192
  • vNIC#2: ens224
  • vNIC#3: ens256
  • vNIC#4: ens161
CVM$ hostssh "ip link | grep ens -A1"
============= 192.168.1.184 ============
2: ens192: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master ovs-system state UP mode DEFAULT qlen 1000
    link/ether 00:50:56:8a:5c:94 brd ff:ff:ff:ff:ff:ff
--
15: ens224: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT qlen 1000
    link/ether 00:50:56:8a:5c:b1 brd ff:ff:ff:ff:ff:ff
16: ens256: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT qlen 1000
    link/ether 00:50:56:8a:f7:50 brd ff:ff:ff:ff:ff:ff
17: ens161: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT qlen 1000
    link/ether 00:50:56:8a:24:7e brd ff:ff:ff:ff:ff:ff
============= 192.168.1.183 ============
2: ens192: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master ovs-system state UP mode DEFAULT qlen 1000
    link/ether 00:50:56:8a:76:f8 brd ff:ff:ff:ff:ff:ff
--
15: ens224: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT qlen 1000
    link/ether 00:50:56:8a:6a:3c brd ff:ff:ff:ff:ff:ff
16: ens256: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT qlen 1000
    link/ether 00:50:56:8a:10:1b brd ff:ff:ff:ff:ff:ff
17: ens161: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT qlen 1000
    link/ether 00:50:56:8a:d1:13 brd ff:ff:ff:ff:ff:ff
============= 192.168.1.182 ============
2: ens192: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master ovs-system state UP mode DEFAULT qlen 1000
    link/ether 00:50:56:8a:32:d4 brd ff:ff:ff:ff:ff:ff
--
15: ens224: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT qlen 1000
    link/ether 00:50:56:8a:f6:86 brd ff:ff:ff:ff:ff:ff
16: ens256: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT qlen 1000
    link/ether 00:50:56:8a:d6:4b brd ff:ff:ff:ff:ff:ff
17: ens161: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT qlen 1000
    link/ether 00:50:56:8a:c1:15 brd ff:ff:ff:ff:ff:ff
============= 192.168.1.181 ============
2: ens192: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master ovs-system state UP mode DEFAULT qlen 1000
    link/ether 00:50:56:8a:f9:d2 brd ff:ff:ff:ff:ff:ff
--
15: ens224: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT qlen 1000
    link/ether 00:50:56:8a:26:8d brd ff:ff:ff:ff:ff:ff
16: ens256: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT qlen 1000
    link/ether 00:50:56:8a:8b:b0 brd ff:ff:ff:ff:ff:ff
17: ens161: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT qlen 1000
    link/ether 00:50:56:8a:40:b2 brd ff:ff:ff:ff:ff:ff
CVM$

Bridge は br0 だけあります。

CVM$ allssh manage_ovs show_bridges
================== 192.168.1.185 =================
Bridges:
br0
================== 192.168.1.186 =================
Bridges:
br0
================== 192.168.1.187 =================
Bridges:
br0
================== 192.168.1.188 =================
Bridges:
br0
CVM$

Bridge のアップリンクには、ens192 だけ接続されています。

CVM$ allssh manage_ovs show_uplinks
================== 192.168.1.185 =================
Bridge br0:
  Uplink ports: ens192
  Uplink ifaces: ens192

================== 192.168.1.186 =================
Bridge br0:
  Uplink ports: ens192
  Uplink ifaces: ens192

================== 192.168.1.187 =================
Bridge br0:
  Uplink ports: ens192
  Uplink ifaces: ens192

================== 192.168.1.188 =================
Bridge br0:
  Uplink ports: ens192
  Uplink ifaces: ens192

CVM$

現状では、Bondig されていません。

CVM$ hostssh ovs-appctl bond/show
============= 192.168.1.184 ============
============= 192.168.1.183 ============
============= 192.168.1.182 ============
============= 192.168.1.181 ============
CVM$

1台だけ、Open vSwitch の全容を見ておきます。Nutanix CE は ce-2018.01.31-stable から突然 Open vSwitch の様子が激変しました。多分マイクロセグメンテーション(microseg とかもあり)関連の影響かなと思います。

CVM$ ssh root@192.168.5.1 ovs-vsctl show
FIPS mode initialized
d83db5ae-0c09-4795-b82e-cc54fad99400
    Bridge br.dmx
        Port br.dmx.u
            Interface br.dmx.u
                type: patch
                options: {peer=br.nf.d}
        Port br.dmx
            Interface br.dmx
                type: internal
        Port "br.dmx.d.br0"
            Interface "br.dmx.d.br0"
                type: patch
                options: {peer="br0.u"}
    Bridge br.nf
        Port br.nf.u
            Interface br.nf.u
                type: patch
                options: {peer=br.microseg.d}
        Port br.nf
            Interface br.nf
                type: internal
        Port br.nf.d
            Interface br.nf.d
                type: patch
                options: {peer=br.dmx.u}
    Bridge "br0.local"
        Port "br0.local"
            Interface "br0.local"
                type: internal
        Port "br0.local.d"
            Interface "br0.local.d"
                type: patch
                options: {peer="br.mx.u.br0"}
    Bridge "br0"
        Port "br0.u"
            Interface "br0.u"
                type: patch
                options: {peer="br.dmx.d.br0"}
        Port "br0-dhcp"
            Interface "br0-dhcp"
                type: vxlan
                options: {key="1", remote_ip="192.168.1.187"}
        Port "vnet0"
            Interface "vnet0"
        Port "ens192"
            Interface "ens192"
        Port "br0-arp"
            Interface "br0-arp"
                type: vxlan
                options: {key="1", remote_ip="192.168.5.2"}
        Port "br0"
            Interface "br0"
                type: internal
    Bridge br.mx
        Port br.mx.d
            Interface br.mx.d
                type: patch
                options: {peer=br.microseg.u}
        Port br.mx
            Interface br.mx
                type: internal
        Port "br.mx.u.br0"
            Interface "br.mx.u.br0"
                type: patch
                options: {peer="br0.local.d"}
    Bridge br.microseg
        Port br.microseg.d
            Interface br.microseg.d
                type: patch
                options: {peer=br.nf.u}
        Port br.microseg.u
            Interface br.microseg.u
                type: patch
                options: {peer=br.mx.d}
        Port br.microseg
            Interface br.microseg
                type: internal
    ovs_version: "2.5.2"
CVM$

Bond Port の構成(br0-up)

まず、br0 に接続している NIC を Bond します。Bonding port は、br0-up という名前にしておきます。vNIC#1 と vNIC#2 にあたる、ens192 と ens224 を Bonding します。

CVM$ allssh "manage_ovs --bridge_name br0 --bond_name br0-up --interfaces ens192,ens224 --require_link=false update_uplinks"
================== 192.168.1.185 =================
2018-03-19 22:58:09 WARNING manage_ovs:344 Interface ens224 does not have link state
2018-03-19 22:58:09 INFO manage_ovs:394 Deleting OVS ports: ens192
2018-03-19 22:58:09 INFO manage_ovs:408 Adding bond br0-up for OVS ports: ens192 ens224
2018-03-19 22:58:11 WARNING manage_ovs:460 Failed to get IP for br0, not sending gratuitous ARPs
================== 192.168.1.186 =================
2018-03-19 22:58:20 WARNING manage_ovs:344 Interface ens224 does not have link state
2018-03-19 22:58:20 INFO manage_ovs:394 Deleting OVS ports: ens192
2018-03-19 22:58:21 INFO manage_ovs:408 Adding bond br0-up for OVS ports: ens192 ens224
2018-03-19 22:58:22 WARNING manage_ovs:460 Failed to get IP for br0, not sending gratuitous ARPs
================== 192.168.1.187 =================
2018-03-19 22:58:31 WARNING manage_ovs:344 Interface ens224 does not have link state
2018-03-19 22:58:31 INFO manage_ovs:394 Deleting OVS ports: ens192
2018-03-19 22:58:31 INFO manage_ovs:408 Adding bond br0-up for OVS ports: ens192 ens224
2018-03-19 22:58:33 WARNING manage_ovs:460 Failed to get IP for br0, not sending gratuitous ARPs
================== 192.168.1.188 =================
2018-03-19 22:58:42 WARNING manage_ovs:344 Interface ens224 does not have link state
2018-03-19 22:58:42 INFO manage_ovs:394 Deleting OVS ports: ens192
2018-03-19 22:58:42 INFO manage_ovs:408 Adding bond br0-up for OVS ports: ens192 ens224
2018-03-19 22:58:43 WARNING manage_ovs:460 Failed to get IP for br0, not sending gratuitous ARPs
CVM$

br0-up が構成できました。

CVM$ allssh manage_ovs show_uplinks
================== 192.168.1.185 =================
Bridge br0:
  Uplink ports: br0-up
  Uplink ifaces: ens192 ens224

================== 192.168.1.186 =================
Bridge br0:
  Uplink ports: br0-up
  Uplink ifaces: ens192 ens224

================== 192.168.1.187 =================
Bridge br0:
  Uplink ports: br0-up
  Uplink ifaces: ens192 ens224

================== 192.168.1.188 =================
Bridge br0:
  Uplink ports: br0-up
  Uplink ifaces: ens192 ens224

CVM$

Bonding は、デフォルトでは active-backup です。

CVM$ hostssh ovs-appctl bond/show br0-up
============= 192.168.1.184 ============
---- br0-up ----
bond_mode: active-backup
bond may use recirculation: no, Recirc-ID : -1
bond-hash-basis: 0
updelay: 0 ms
downdelay: 0 ms
lacp_status: off
active slave mac: 00:50:56:8a:5c:94(ens192)

slave ens192: enabled
        active slave
        may_enable: true

slave ens224: disabled
        may_enable: false

============= 192.168.1.183 ============
---- br0-up ----
bond_mode: active-backup
bond may use recirculation: no, Recirc-ID : -1
bond-hash-basis: 0
updelay: 0 ms
downdelay: 0 ms
lacp_status: off
active slave mac: 00:50:56:8a:76:f8(ens192)

slave ens192: enabled
        active slave
        may_enable: true

slave ens224: disabled
        may_enable: false

============= 192.168.1.182 ============
---- br0-up ----
bond_mode: active-backup
bond may use recirculation: no, Recirc-ID : -1
bond-hash-basis: 0
updelay: 0 ms
downdelay: 0 ms
lacp_status: off
active slave mac: 00:50:56:8a:32:d4(ens192)

slave ens192: enabled
        active slave
        may_enable: true

slave ens224: disabled
        may_enable: false

============= 192.168.1.181 ============
---- br0-up ----
bond_mode: active-backup
bond may use recirculation: no, Recirc-ID : -1
bond-hash-basis: 0
updelay: 0 ms
downdelay: 0 ms
lacp_status: off
active slave mac: 00:50:56:8a:f9:d2(ens192)

slave ens192: enabled
        active slave
        may_enable: true

slave ens224: disabled
        may_enable: false

CVM$

Bridge の作成(br1)

br1 を作成します。

CVM$ allssh "manage_ovs --bridge_name br1 create_single_bridge"
================== 192.168.1.185 =================
================== 192.168.1.186 =================
================== 192.168.1.187 =================
================== 192.168.1.188 =================
CVM$

br1 が作成されました。

CVM$ allssh manage_ovs show_bridges
================== 192.168.1.185 =================
Bridges:
br1
br0
================== 192.168.1.186 =================
Bridges:
br1
br0
================== 192.168.1.187 =================
Bridges:
br1
br0
================== 192.168.1.188 =================
Bridges:
br1
br0
CVM$

Bond Port の構成(br1-up)

br1 にBond port として br1-up を作成します。

CVM$ allssh "manage_ovs --bridge_name br1 --bond_name br1-up --interfaces ens256,ens161 --require_link=false update_uplinks"
================== 192.168.1.185 =================
2018-03-19 23:06:02 WARNING manage_ovs:344 Interface ens256 does not have link state
2018-03-19 23:06:02 WARNING manage_ovs:344 Interface ens161 does not have link state
2018-03-19 23:06:02 INFO manage_ovs:394 Deleting OVS ports:
2018-03-19 23:06:02 INFO manage_ovs:408 Adding bond br1-up for OVS ports: ens256 ens161
2018-03-19 23:06:05 WARNING manage_ovs:460 Failed to get IP for br1, not sending gratuitous ARPs
================== 192.168.1.186 =================
2018-03-19 23:06:16 WARNING manage_ovs:344 Interface ens256 does not have link state
2018-03-19 23:06:16 WARNING manage_ovs:344 Interface ens161 does not have link state
2018-03-19 23:06:16 INFO manage_ovs:394 Deleting OVS ports:
2018-03-19 23:06:16 INFO manage_ovs:408 Adding bond br1-up for OVS ports: ens256 ens161
2018-03-19 23:06:17 WARNING manage_ovs:460 Failed to get IP for br1, not sending gratuitous ARPs
================== 192.168.1.187 =================
2018-03-19 23:06:28 WARNING manage_ovs:344 Interface ens256 does not have link state
2018-03-19 23:06:28 WARNING manage_ovs:344 Interface ens161 does not have link state
2018-03-19 23:06:28 INFO manage_ovs:394 Deleting OVS ports:
2018-03-19 23:06:28 INFO manage_ovs:408 Adding bond br1-up for OVS ports: ens256 ens161
2018-03-19 23:06:29 WARNING manage_ovs:460 Failed to get IP for br1, not sending gratuitous ARPs
================== 192.168.1.188 =================
2018-03-19 23:06:39 WARNING manage_ovs:344 Interface ens256 does not have link state
2018-03-19 23:06:39 WARNING manage_ovs:344 Interface ens161 does not have link state
2018-03-19 23:06:39 INFO manage_ovs:394 Deleting OVS ports:
2018-03-19 23:06:39 INFO manage_ovs:408 Adding bond br1-up for OVS ports: ens256 ens161
2018-03-19 23:06:41 WARNING manage_ovs:460 Failed to get IP for br1, not sending gratuitous ARPs
CVM$

br1-up が作成されました。

CVM$ allssh manage_ovs show_uplinks
================== 192.168.1.185 =================
Bridge br1:
  Uplink ports: br1-up
  Uplink ifaces: ens161 ens256

Bridge br0:
  Uplink ports: br0-up
  Uplink ifaces: ens192 ens224

================== 192.168.1.186 =================
Bridge br1:
  Uplink ports: br1-up
  Uplink ifaces: ens161 ens256

Bridge br0:
  Uplink ports: br0-up
  Uplink ifaces: ens192 ens224

================== 192.168.1.187 =================
Bridge br1:
  Uplink ports: br1-up
  Uplink ifaces: ens161 ens256

Bridge br0:
  Uplink ports: br0-up
  Uplink ifaces: ens192 ens224

================== 192.168.1.188 =================
Bridge br1:
  Uplink ports: br1-up
  Uplink ifaces: ens161 ens256

Bridge br0:
  Uplink ports: br0-up
  Uplink ifaces: ens192 ens224

CVM$

ちなみに、今回は Nutanix CE インストール後に NIC を追加していて、追加分の 3 NIC は Link Down のままになっています。これは、下記のように Link Up にできます。

Bridge / Bond port 作成後の Open vSwitch の全容は下記のようになっています。(これは AHV ホスト 1台だけの情報です)

CVM$ ssh root@192.168.5.1 ovs-vsctl show
FIPS mode initialized
e97f53fc-c1c9-499f-bc14-f4768401d8b0
    Bridge "br0.local"
        Port "br0.local.d"
            Interface "br0.local.d"
                type: patch
                options: {peer="br.mx.u.br0"}
        Port "br0.local"
            Interface "br0.local"
                type: internal
    Bridge "br1.local"
        Port "br1.local"
            Interface "br1.local"
                type: internal
        Port "br1.local.d"
            Interface "br1.local.d"
                type: patch
                options: {peer="br.mx.u.br1"}
    Bridge "br0"
        Port "br0-up"
            Interface "ens224"
            Interface "ens192"
        Port "br0"
            Interface "br0"
                type: internal
        Port "br0-arp"
            Interface "br0-arp"
                type: vxlan
                options: {key="1", remote_ip="192.168.5.2"}
        Port "vnet0"
            Interface "vnet0"
        Port "br0-dhcp"
            Interface "br0-dhcp"
                type: vxlan
                options: {key="1", remote_ip="192.168.1.187"}
        Port "br0.u"
            Interface "br0.u"
                type: patch
                options: {peer="br.dmx.d.br0"}
    Bridge br.nf
        Port br.nf
            Interface br.nf
                type: internal
        Port br.nf.u
            Interface br.nf.u
                type: patch
                options: {peer=br.microseg.d}
        Port br.nf.d
            Interface br.nf.d
                type: patch
                options: {peer=br.dmx.u}
    Bridge br.dmx
        Port "br.dmx.d.br1"
            Interface "br.dmx.d.br1"
                type: patch
                options: {peer="br1.u"}
        Port br.dmx.u
            Interface br.dmx.u
                type: patch
                options: {peer=br.nf.d}
        Port "br.dmx.d.br0"
            Interface "br.dmx.d.br0"
                type: patch
                options: {peer="br0.u"}
        Port br.dmx
            Interface br.dmx
                type: internal
    Bridge "br1"
        Port "br1"
            Interface "br1"
                type: internal
        Port "br1-dhcp"
            Interface "br1-dhcp"
                type: vxlan
                options: {key="2", remote_ip="192.168.1.187"}
        Port "br1-arp"
            Interface "br1-arp"
                type: vxlan
                options: {key="2", remote_ip="192.168.5.2"}
        Port "br1-up"
            Interface "ens256"
            Interface "ens161"
        Port "br1.u"
            Interface "br1.u"
                type: patch
                options: {peer="br.dmx.d.br1"}
    Bridge br.mx
        Port "br.mx.u.br0"
            Interface "br.mx.u.br0"
                type: patch
                options: {peer="br0.local.d"}
        Port "br.mx.u.br1"
            Interface "br.mx.u.br1"
                type: patch
                options: {peer="br1.local.d"}
        Port br.mx
            Interface br.mx
                type: internal
        Port br.mx.d
            Interface br.mx.d
                type: patch
                options: {peer=br.microseg.u}
    Bridge br.microseg
        Port br.microseg.u
            Interface br.microseg.u
                type: patch
                options: {peer=br.mx.d}
        Port br.microseg.d
            Interface br.microseg.d
                type: patch
                options: {peer=br.nf.u}
        Port br.microseg
            Interface br.microseg
                type: internal
    ovs_version: "2.5.2"
CVM$

つづく。

©2024 gowatana
クリエイティブ・コモンズ・ライセンスこの 作品 は クリエイティブ・コモンズ 表示 4.0 国際 ライセンスの下に提供されています。