TenForward

技術ブログ。はてなダイアリーから移転しました

SR-IOV を有効にする

Linux Virtualization

Ubuntu Linux 11.10 で SR-IOV を有効にするメモ．

ホストは 日本HP Redirect 日本HP 旧モデル情報 8100 Elite SF−スペック （Directplusモデル）の Core i5 でメモリ 8GB 積んだマシンです．

# lspci | grep 82576
01:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
01:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)

こんなチップを載せたカード．

# lspci -v -s 01:00.0
01:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
	Subsystem: Intel Corporation Gigabit ET Dual Port Server Adapter
	Flags: bus master, fast devsel, latency 0, IRQ 16
	Memory at f0800000 (32-bit, non-prefetchable) [size=128K]
	Memory at f0000000 (32-bit, non-prefetchable) [size=4M]
	I/O ports at 1100 [size=32]
	Memory at f0840000 (32-bit, non-prefetchable) [size=16K]
	[virtual] Expansion ROM at f8000000 [disabled] [size=4M]
	Capabilities: [40] Power Management version 3
	Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
	Capabilities: [70] MSI-X: Enable+ Count=10 Masked-
	Capabilities: [a0] Express Endpoint, MSI 00
	Capabilities: [100] Advanced Error Reporting
	Capabilities: [140] Device Serial Number 00-1b-21-ff-ff-a5-72-48
	Capabilities: [150] Alternative Routing-ID Interpretation (ARI)
	Capabilities: [160] Single Root I/O Virtualization (SR-IOV)
	Kernel driver in use: igb
	Kernel modules: igb

SR-IOV 機能搭載．この機能を有効にするために /etc/default/grub で以下のように起動オプションを与えます．

GRUB_CMDLINE_LINUX="intel_iommu=on pci=assign-busses"

update-grub コマンドを実行してから，リブート．

# modprobe -r igb && modprobe igb max_vfs=7
# lspci | grep 82576
01:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
01:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
02:10.0 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
02:10.1 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
02:10.2 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
02:10.3 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
02:10.4 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
02:10.5 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
02:10.6 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
02:10.7 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
02:11.0 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
02:11.1 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
02:11.2 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
02:11.3 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
02:11.4 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
02:11.5 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)

使ってないけどたくさんインターフェースが見えますね．ホストからもこういう風に見えるんだ...

$ ip addr show | egrep "eth[0-9]*:"
3: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master br100 state UP qlen 1000
8: eth1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
9: eth2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
10: eth3: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
11: eth4: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
12: eth5: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
13: eth6: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
14: eth7: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
15: eth8: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
16: eth9: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
17: eth10: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
18: eth11: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
19: eth12: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
20: eth13: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
21: eth14: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
22: eth15: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
23: eth16: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000

Redhat の文書だと intel_iommu=on のカーネルオプションのみ記載されていたりしますが，これだけだと

SR-IOV: bus number out of range

こんなエラーが出たりしていました．