複数の NIC がある Nutanix CE で、Open vSwitch に Bridge を追加してみます。Prism の Web UI からだと設定できないので、CVM にログインして CLI を使用します。
- Bridge の作成は、manage_ovs コマンドを使用します。
- 基本的に CVM から操作します。（AHV へのコマンド実行も CVM から）
- CE のバージョンは ce-2018.01.31-stable です。
今回は、ESXi で起動している Nested Nutanix CE です。これは、下記のようにセットアップしています。
Nutanix CE は、4 ノードのクラスタ構成にしています。Nutanix Cmdlets でクラスタに参加しているノードを見ておきます。
NTNX> Get-NTNXHost | select @{N="Cluster";E={(Get-NTNXCluster -Id $_.ClusterUuid).name}},name,hypervisorAddress,serviceVMExternalIP,hypervisorFullName | sort hypervisorAddress | ft -AutoSize
Cluster name hypervisorAddress serviceVMExternalIP hypervisorFullName
------- ---- ----------------- ------------------- ------------------
ce-ntnx-n10 NTNX-54e60e4a-A 192.168.1.181 192.168.1.185 Nutanix 20180123.170
ce-ntnx-n10 NTNX-5eab51d4-A 192.168.1.182 192.168.1.186 Nutanix 20180123.170
ce-ntnx-n10 NTNX-a17f7da5-A 192.168.1.183 192.168.1.187 Nutanix 20180123.170
ce-ntnx-n10 NTNX-7cb824ec-A 192.168.1.184 192.168.1.188 Nutanix 20180123.170
ESXi 上の VM にしているので、PowerCLI で Nutanix CE をインストールしている VM の構成を見ておきます。
PowerCLI> Get-Folder nutanix-4node | Get-VM | Get-VMGuest | select VM,{$_.IPAddress | where {$_ -like "192*"}} | sort VM
VM $_.IPAddress | where {$_ -like "192*"}
-- --------------------------------------
ntnx-n11 {192.168.1.181, 192.168.5.1}
ntnx-n12 {192.168.1.182, 192.168.5.1}
ntnx-n13 {192.168.1.183, 192.168.5.1}
ntnx-n14 {192.168.1.184, 192.168.5.1}
それぞれの ESXi の VM に、vNIC（Nutainx AHV から見ると物理 NIC）が 4つあります。
PowerCLI> Get-Folder nutanix-4node | Get-VM | Get-NetworkAdapter | select Parent,Name,MacAddress | sort Parent,Name,PortGroupName Parent Name MacAddress ------ ---- ---------- ntnx-n11 Network adapter 1 00:50:56:8a:f9:d2 ntnx-n11 Network adapter 2 00:50:56:8a:26:8d ntnx-n11 Network adapter 3 00:50:56:8a:8b:b0 ntnx-n11 Network adapter 4 00:50:56:8a:40:b2 ntnx-n12 Network adapter 1 00:50:56:8a:32:d4 ntnx-n12 Network adapter 2 00:50:56:8a:f6:86 ntnx-n12 Network adapter 3 00:50:56:8a:d6:4b ntnx-n12 Network adapter 4 00:50:56:8a:c1:15 ntnx-n13 Network adapter 1 00:50:56:8a:76:f8 ntnx-n13 Network adapter 2 00:50:56:8a:6a:3c ntnx-n13 Network adapter 3 00:50:56:8a:10:1b ntnx-n13 Network adapter 4 00:50:56:8a:d1:13 ntnx-n14 Network adapter 1 00:50:56:8a:5c:94 ntnx-n14 Network adapter 2 00:50:56:8a:5c:b1 ntnx-n14 Network adapter 3 00:50:56:8a:f7:50 ntnx-n14 Network adapter 4 00:50:56:8a:24:7e
CVM から見た AHV のネットワーク構成
インターフェースは 4つです。
nutanix@NTNX-7cb824ec-A-CVM:192.168.1.188:~$ PS1='CVM$ ' CVM$ manage_ovs show_interfaces name mode link speed ens161 10000 False None ens192 10000 True 10000 ens224 10000 False None ens256 10000 False None
すべての AHV でまとめて確認できます。
CVM$ allssh manage_ovs show_interfaces ================== 192.168.1.185 ================= name mode link speed ens161 10000 False None ens192 10000 True 10000 ens224 10000 False None ens256 10000 False None ================== 192.168.1.186 ================= name mode link speed ens161 10000 False None ens192 10000 True 10000 ens224 10000 False None ens256 10000 False None ================== 192.168.1.187 ================= name mode link speed ens161 10000 False None ens192 10000 True 10000 ens224 10000 False None ens256 10000 False None ================== 192.168.1.188 ================= name mode link speed ens161 10000 False None ens192 10000 True 10000 ens224 10000 False None ens256 10000 False None CVM$
MAC アドレスをもとに確認すると、NIC の名前は下記になっています。
- vNIC#1: ens192
- vNIC#2: ens224
- vNIC#3: ens256
- vNIC#4: ens161
CVM$ hostssh "ip link | grep ens -A1"
============= 192.168.1.184 ============
2: ens192: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master ovs-system state UP mode DEFAULT qlen 1000
link/ether 00:50:56:8a:5c:94 brd ff:ff:ff:ff:ff:ff
--
15: ens224: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT qlen 1000
link/ether 00:50:56:8a:5c:b1 brd ff:ff:ff:ff:ff:ff
16: ens256: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT qlen 1000
link/ether 00:50:56:8a:f7:50 brd ff:ff:ff:ff:ff:ff
17: ens161: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT qlen 1000
link/ether 00:50:56:8a:24:7e brd ff:ff:ff:ff:ff:ff
============= 192.168.1.183 ============
2: ens192: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master ovs-system state UP mode DEFAULT qlen 1000
link/ether 00:50:56:8a:76:f8 brd ff:ff:ff:ff:ff:ff
--
15: ens224: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT qlen 1000
link/ether 00:50:56:8a:6a:3c brd ff:ff:ff:ff:ff:ff
16: ens256: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT qlen 1000
link/ether 00:50:56:8a:10:1b brd ff:ff:ff:ff:ff:ff
17: ens161: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT qlen 1000
link/ether 00:50:56:8a:d1:13 brd ff:ff:ff:ff:ff:ff
============= 192.168.1.182 ============
2: ens192: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master ovs-system state UP mode DEFAULT qlen 1000
link/ether 00:50:56:8a:32:d4 brd ff:ff:ff:ff:ff:ff
--
15: ens224: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT qlen 1000
link/ether 00:50:56:8a:f6:86 brd ff:ff:ff:ff:ff:ff
16: ens256: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT qlen 1000
link/ether 00:50:56:8a:d6:4b brd ff:ff:ff:ff:ff:ff
17: ens161: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT qlen 1000
link/ether 00:50:56:8a:c1:15 brd ff:ff:ff:ff:ff:ff
============= 192.168.1.181 ============
2: ens192: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master ovs-system state UP mode DEFAULT qlen 1000
link/ether 00:50:56:8a:f9:d2 brd ff:ff:ff:ff:ff:ff
--
15: ens224: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT qlen 1000
link/ether 00:50:56:8a:26:8d brd ff:ff:ff:ff:ff:ff
16: ens256: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT qlen 1000
link/ether 00:50:56:8a:8b:b0 brd ff:ff:ff:ff:ff:ff
17: ens161: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT qlen 1000
link/ether 00:50:56:8a:40:b2 brd ff:ff:ff:ff:ff:ff
CVM$
Bridge は br0 だけあります。
CVM$ allssh manage_ovs show_bridges ================== 192.168.1.185 ================= Bridges: br0 ================== 192.168.1.186 ================= Bridges: br0 ================== 192.168.1.187 ================= Bridges: br0 ================== 192.168.1.188 ================= Bridges: br0 CVM$
Bridge のアップリンクには、ens192 だけ接続されています。
CVM$ allssh manage_ovs show_uplinks ================== 192.168.1.185 ================= Bridge br0: Uplink ports: ens192 Uplink ifaces: ens192 ================== 192.168.1.186 ================= Bridge br0: Uplink ports: ens192 Uplink ifaces: ens192 ================== 192.168.1.187 ================= Bridge br0: Uplink ports: ens192 Uplink ifaces: ens192 ================== 192.168.1.188 ================= Bridge br0: Uplink ports: ens192 Uplink ifaces: ens192 CVM$
現状では、Bondig されていません。
CVM$ hostssh ovs-appctl bond/show ============= 192.168.1.184 ============ ============= 192.168.1.183 ============ ============= 192.168.1.182 ============ ============= 192.168.1.181 ============ CVM$
1台だけ、Open vSwitch の全容を見ておきます。Nutanix CE は ce-2018.01.31-stable から突然 Open vSwitch の様子が激変しました。多分マイクロセグメンテーション（microseg とかもあり）関連の影響かなと思います。
CVM$ ssh root@192.168.5.1 ovs-vsctl show
FIPS mode initialized
d83db5ae-0c09-4795-b82e-cc54fad99400
Bridge br.dmx
Port br.dmx.u
Interface br.dmx.u
type: patch
options: {peer=br.nf.d}
Port br.dmx
Interface br.dmx
type: internal
Port "br.dmx.d.br0"
Interface "br.dmx.d.br0"
type: patch
options: {peer="br0.u"}
Bridge br.nf
Port br.nf.u
Interface br.nf.u
type: patch
options: {peer=br.microseg.d}
Port br.nf
Interface br.nf
type: internal
Port br.nf.d
Interface br.nf.d
type: patch
options: {peer=br.dmx.u}
Bridge "br0.local"
Port "br0.local"
Interface "br0.local"
type: internal
Port "br0.local.d"
Interface "br0.local.d"
type: patch
options: {peer="br.mx.u.br0"}
Bridge "br0"
Port "br0.u"
Interface "br0.u"
type: patch
options: {peer="br.dmx.d.br0"}
Port "br0-dhcp"
Interface "br0-dhcp"
type: vxlan
options: {key="1", remote_ip="192.168.1.187"}
Port "vnet0"
Interface "vnet0"
Port "ens192"
Interface "ens192"
Port "br0-arp"
Interface "br0-arp"
type: vxlan
options: {key="1", remote_ip="192.168.5.2"}
Port "br0"
Interface "br0"
type: internal
Bridge br.mx
Port br.mx.d
Interface br.mx.d
type: patch
options: {peer=br.microseg.u}
Port br.mx
Interface br.mx
type: internal
Port "br.mx.u.br0"
Interface "br.mx.u.br0"
type: patch
options: {peer="br0.local.d"}
Bridge br.microseg
Port br.microseg.d
Interface br.microseg.d
type: patch
options: {peer=br.nf.u}
Port br.microseg.u
Interface br.microseg.u
type: patch
options: {peer=br.mx.d}
Port br.microseg
Interface br.microseg
type: internal
ovs_version: "2.5.2"
CVM$
Bond Port の構成（br0-up）
まず、br0 に接続している NIC を Bond します。Bonding port は、br0-up という名前にしておきます。vNIC#1 と vNIC#2 にあたる、ens192 と ens224 を Bonding します。
CVM$ allssh "manage_ovs --bridge_name br0 --bond_name br0-up --interfaces ens192,ens224 --require_link=false update_uplinks" ================== 192.168.1.185 ================= 2018-03-19 22:58:09 WARNING manage_ovs:344 Interface ens224 does not have link state 2018-03-19 22:58:09 INFO manage_ovs:394 Deleting OVS ports: ens192 2018-03-19 22:58:09 INFO manage_ovs:408 Adding bond br0-up for OVS ports: ens192 ens224 2018-03-19 22:58:11 WARNING manage_ovs:460 Failed to get IP for br0, not sending gratuitous ARPs ================== 192.168.1.186 ================= 2018-03-19 22:58:20 WARNING manage_ovs:344 Interface ens224 does not have link state 2018-03-19 22:58:20 INFO manage_ovs:394 Deleting OVS ports: ens192 2018-03-19 22:58:21 INFO manage_ovs:408 Adding bond br0-up for OVS ports: ens192 ens224 2018-03-19 22:58:22 WARNING manage_ovs:460 Failed to get IP for br0, not sending gratuitous ARPs ================== 192.168.1.187 ================= 2018-03-19 22:58:31 WARNING manage_ovs:344 Interface ens224 does not have link state 2018-03-19 22:58:31 INFO manage_ovs:394 Deleting OVS ports: ens192 2018-03-19 22:58:31 INFO manage_ovs:408 Adding bond br0-up for OVS ports: ens192 ens224 2018-03-19 22:58:33 WARNING manage_ovs:460 Failed to get IP for br0, not sending gratuitous ARPs ================== 192.168.1.188 ================= 2018-03-19 22:58:42 WARNING manage_ovs:344 Interface ens224 does not have link state 2018-03-19 22:58:42 INFO manage_ovs:394 Deleting OVS ports: ens192 2018-03-19 22:58:42 INFO manage_ovs:408 Adding bond br0-up for OVS ports: ens192 ens224 2018-03-19 22:58:43 WARNING manage_ovs:460 Failed to get IP for br0, not sending gratuitous ARPs CVM$
br0-up が構成できました。
CVM$ allssh manage_ovs show_uplinks ================== 192.168.1.185 ================= Bridge br0: Uplink ports: br0-up Uplink ifaces: ens192 ens224 ================== 192.168.1.186 ================= Bridge br0: Uplink ports: br0-up Uplink ifaces: ens192 ens224 ================== 192.168.1.187 ================= Bridge br0: Uplink ports: br0-up Uplink ifaces: ens192 ens224 ================== 192.168.1.188 ================= Bridge br0: Uplink ports: br0-up Uplink ifaces: ens192 ens224 CVM$
Bonding は、デフォルトでは active-backup です。
CVM$ hostssh ovs-appctl bond/show br0-up
============= 192.168.1.184 ============
---- br0-up ----
bond_mode: active-backup
bond may use recirculation: no, Recirc-ID : -1
bond-hash-basis: 0
updelay: 0 ms
downdelay: 0 ms
lacp_status: off
active slave mac: 00:50:56:8a:5c:94(ens192)
slave ens192: enabled
active slave
may_enable: true
slave ens224: disabled
may_enable: false
============= 192.168.1.183 ============
---- br0-up ----
bond_mode: active-backup
bond may use recirculation: no, Recirc-ID : -1
bond-hash-basis: 0
updelay: 0 ms
downdelay: 0 ms
lacp_status: off
active slave mac: 00:50:56:8a:76:f8(ens192)
slave ens192: enabled
active slave
may_enable: true
slave ens224: disabled
may_enable: false
============= 192.168.1.182 ============
---- br0-up ----
bond_mode: active-backup
bond may use recirculation: no, Recirc-ID : -1
bond-hash-basis: 0
updelay: 0 ms
downdelay: 0 ms
lacp_status: off
active slave mac: 00:50:56:8a:32:d4(ens192)
slave ens192: enabled
active slave
may_enable: true
slave ens224: disabled
may_enable: false
============= 192.168.1.181 ============
---- br0-up ----
bond_mode: active-backup
bond may use recirculation: no, Recirc-ID : -1
bond-hash-basis: 0
updelay: 0 ms
downdelay: 0 ms
lacp_status: off
active slave mac: 00:50:56:8a:f9:d2(ens192)
slave ens192: enabled
active slave
may_enable: true
slave ens224: disabled
may_enable: false
CVM$
Bridge の作成（br1）
br1 を作成します。
CVM$ allssh "manage_ovs --bridge_name br1 create_single_bridge" ================== 192.168.1.185 ================= ================== 192.168.1.186 ================= ================== 192.168.1.187 ================= ================== 192.168.1.188 ================= CVM$
br1 が作成されました。
CVM$ allssh manage_ovs show_bridges ================== 192.168.1.185 ================= Bridges: br1 br0 ================== 192.168.1.186 ================= Bridges: br1 br0 ================== 192.168.1.187 ================= Bridges: br1 br0 ================== 192.168.1.188 ================= Bridges: br1 br0 CVM$
Bond Port の構成（br1-up）
br1 にBond port として br1-up を作成します。
CVM$ allssh "manage_ovs --bridge_name br1 --bond_name br1-up --interfaces ens256,ens161 --require_link=false update_uplinks" ================== 192.168.1.185 ================= 2018-03-19 23:06:02 WARNING manage_ovs:344 Interface ens256 does not have link state 2018-03-19 23:06:02 WARNING manage_ovs:344 Interface ens161 does not have link state 2018-03-19 23:06:02 INFO manage_ovs:394 Deleting OVS ports: 2018-03-19 23:06:02 INFO manage_ovs:408 Adding bond br1-up for OVS ports: ens256 ens161 2018-03-19 23:06:05 WARNING manage_ovs:460 Failed to get IP for br1, not sending gratuitous ARPs ================== 192.168.1.186 ================= 2018-03-19 23:06:16 WARNING manage_ovs:344 Interface ens256 does not have link state 2018-03-19 23:06:16 WARNING manage_ovs:344 Interface ens161 does not have link state 2018-03-19 23:06:16 INFO manage_ovs:394 Deleting OVS ports: 2018-03-19 23:06:16 INFO manage_ovs:408 Adding bond br1-up for OVS ports: ens256 ens161 2018-03-19 23:06:17 WARNING manage_ovs:460 Failed to get IP for br1, not sending gratuitous ARPs ================== 192.168.1.187 ================= 2018-03-19 23:06:28 WARNING manage_ovs:344 Interface ens256 does not have link state 2018-03-19 23:06:28 WARNING manage_ovs:344 Interface ens161 does not have link state 2018-03-19 23:06:28 INFO manage_ovs:394 Deleting OVS ports: 2018-03-19 23:06:28 INFO manage_ovs:408 Adding bond br1-up for OVS ports: ens256 ens161 2018-03-19 23:06:29 WARNING manage_ovs:460 Failed to get IP for br1, not sending gratuitous ARPs ================== 192.168.1.188 ================= 2018-03-19 23:06:39 WARNING manage_ovs:344 Interface ens256 does not have link state 2018-03-19 23:06:39 WARNING manage_ovs:344 Interface ens161 does not have link state 2018-03-19 23:06:39 INFO manage_ovs:394 Deleting OVS ports: 2018-03-19 23:06:39 INFO manage_ovs:408 Adding bond br1-up for OVS ports: ens256 ens161 2018-03-19 23:06:41 WARNING manage_ovs:460 Failed to get IP for br1, not sending gratuitous ARPs CVM$
br1-up が作成されました。
CVM$ allssh manage_ovs show_uplinks ================== 192.168.1.185 ================= Bridge br1: Uplink ports: br1-up Uplink ifaces: ens161 ens256 Bridge br0: Uplink ports: br0-up Uplink ifaces: ens192 ens224 ================== 192.168.1.186 ================= Bridge br1: Uplink ports: br1-up Uplink ifaces: ens161 ens256 Bridge br0: Uplink ports: br0-up Uplink ifaces: ens192 ens224 ================== 192.168.1.187 ================= Bridge br1: Uplink ports: br1-up Uplink ifaces: ens161 ens256 Bridge br0: Uplink ports: br0-up Uplink ifaces: ens192 ens224 ================== 192.168.1.188 ================= Bridge br1: Uplink ports: br1-up Uplink ifaces: ens161 ens256 Bridge br0: Uplink ports: br0-up Uplink ifaces: ens192 ens224 CVM$
ちなみに、今回は Nutanix CE インストール後に NIC を追加していて、追加分の 3 NIC は Link Down のままになっています。これは、下記のように Link Up にできます。
Bridge / Bond port 作成後の Open vSwitch の全容は下記のようになっています。（これは AHV ホスト 1台だけの情報です）
CVM$ ssh root@192.168.5.1 ovs-vsctl show
FIPS mode initialized
e97f53fc-c1c9-499f-bc14-f4768401d8b0
Bridge "br0.local"
Port "br0.local.d"
Interface "br0.local.d"
type: patch
options: {peer="br.mx.u.br0"}
Port "br0.local"
Interface "br0.local"
type: internal
Bridge "br1.local"
Port "br1.local"
Interface "br1.local"
type: internal
Port "br1.local.d"
Interface "br1.local.d"
type: patch
options: {peer="br.mx.u.br1"}
Bridge "br0"
Port "br0-up"
Interface "ens224"
Interface "ens192"
Port "br0"
Interface "br0"
type: internal
Port "br0-arp"
Interface "br0-arp"
type: vxlan
options: {key="1", remote_ip="192.168.5.2"}
Port "vnet0"
Interface "vnet0"
Port "br0-dhcp"
Interface "br0-dhcp"
type: vxlan
options: {key="1", remote_ip="192.168.1.187"}
Port "br0.u"
Interface "br0.u"
type: patch
options: {peer="br.dmx.d.br0"}
Bridge br.nf
Port br.nf
Interface br.nf
type: internal
Port br.nf.u
Interface br.nf.u
type: patch
options: {peer=br.microseg.d}
Port br.nf.d
Interface br.nf.d
type: patch
options: {peer=br.dmx.u}
Bridge br.dmx
Port "br.dmx.d.br1"
Interface "br.dmx.d.br1"
type: patch
options: {peer="br1.u"}
Port br.dmx.u
Interface br.dmx.u
type: patch
options: {peer=br.nf.d}
Port "br.dmx.d.br0"
Interface "br.dmx.d.br0"
type: patch
options: {peer="br0.u"}
Port br.dmx
Interface br.dmx
type: internal
Bridge "br1"
Port "br1"
Interface "br1"
type: internal
Port "br1-dhcp"
Interface "br1-dhcp"
type: vxlan
options: {key="2", remote_ip="192.168.1.187"}
Port "br1-arp"
Interface "br1-arp"
type: vxlan
options: {key="2", remote_ip="192.168.5.2"}
Port "br1-up"
Interface "ens256"
Interface "ens161"
Port "br1.u"
Interface "br1.u"
type: patch
options: {peer="br.dmx.d.br1"}
Bridge br.mx
Port "br.mx.u.br0"
Interface "br.mx.u.br0"
type: patch
options: {peer="br0.local.d"}
Port "br.mx.u.br1"
Interface "br.mx.u.br1"
type: patch
options: {peer="br1.local.d"}
Port br.mx
Interface br.mx
type: internal
Port br.mx.d
Interface br.mx.d
type: patch
options: {peer=br.microseg.u}
Bridge br.microseg
Port br.microseg.u
Interface br.microseg.u
type: patch
options: {peer=br.mx.d}
Port br.microseg.d
Interface br.microseg.d
type: patch
options: {peer=br.nf.u}
Port br.microseg
Interface br.microseg
type: internal
ovs_version: "2.5.2"
CVM$
つづく。
