4. Installation with DPDK

This section describes how to install DPDK-enabled Lagopus on Linux installed bear-metal server with DPDK-enabled NICs and its basic configuration. Lagopus software switch leverages DPDK for high-performance packet processing and forwarding. DPDK [1] provide a set of high-performance network-related library and user-space polling-based drivers in order to accelerate network I/O and forwarding performance on top of PC servers with standard NICs. DPDK allows both packet processing and NIC device handling in user-space not in kernel-space, therefore, special configuration are required compared with the Lagopus with raw-socket configuration (none-DPDK configuration).

To enable DPDK configuration on Lagopus software switch, you have to configure DPDK environment before its execution. Usually Ethernet NIC are under the control of Linux kernel. Therefore you can see these NIC name by ip command, for example, ethX or emX. DPDK-enabled NICs are supposed to be managed under the userspace DPDK process and userspace drivers. As a result, DPDK-enabled NICs are supposed to be out of the control of Linux kernel. Then you will not see the name of DPDK-controlled NICs by ip command.

DPDK-enabled Lagopus can run on a hypervisor-based virtual machine, such as KVM, VMware, and VirtualBox. Therefore, you can test DPDK-enabled Lagopus on your environment. This installation steps are confirmed on VirtualBox VM, which should be identical to the steps on bear-metal server. If you want to run Lagopus with raw-socket configuration instead of DPDK-enabled Lagopus, refer to Installation with raw-socket configuration.

[1]For detail DPDK information, check http://dpdk.org/

4.1. Minimum hardware requirement

Lagopus software switch with DPDK configuration uses multiple CPU cores to achieve high-performance network I/O and processing performance. Therefore, you have to prepare a host machine that meets the following minimum hardware requirements.

  • # of CPU cores: 2 or more
  • Memory size: 2 GB or more
  • NIC: DPDK-enabled NICs

Warning

You need 2 or more processors (CPU cores) to run Lagopus software switch using DPDK.

4.2. Software Versions

4.3. Installation steps

  • Install Ubuntu to a bear-metal server or a VM.

  • Install necessary packages.

    $ sudo apt-get update
$ sudo apt-get install build-essential linux-headers-$(uname -r) \
  libexpat-dev libgmp-dev \
  libssl-dev libpcap-dev libnuma-dev byacc flex git \
  python-dev python-pastedeploy python-paste python-twisted

  • Download Lagopus source code.

    You have two options to get Lagopus source code by downloading tar file or by git.

    $ wget https://github.com/lagopus/lagopus/archive/v0.2.10.tar.gz
$ tar xvf v0.2.10.tar.gz
$ ls
lagopus-0.2.10  v0.2.10.tar.gz
$

    Or

    $ git clone -b v0.2.10 --recursive https://github.com/lagopus/lagopus.git
$ ls
lagopus
$

  • Compile DPDK-enabled Lagopus software switch

    The source code of DPDK library is automatically download by git submodule mechanism. You don’t have to download by manual.

    $ ./configure
$ make

    Note

    If you find an error in configure process or make process, check git submodule setting. Sometimes git submodule operation may be failed due to firewall or proxy setting on your environment.

    /home/<usr>/lagopus-0.2.10/mk/make_dpdk.sh /home/<usr>/lagopus-0.2.10 src/dpdk \
            "x86_64" "linuxapp" gcc
fatal: Not a git repository (or any of the parent directories): .git
make[1]: *** [dpdk] Error 1
make[1]: Leaving directory `/home/<usr>/lagopus-0.2.3'
make: *** [prerequisite] Error 2
configure: error: Prerequisite failure.

  • Install lagopus package

    $ sudo make install

4.4. Setup DPDK

To enable DPDK on your environment, you have three steps, kernel module setup, hugepage setup, DPDK-enabled NIC assignment. The above compilation process of Lagopus software switch, DPDK library and drivers are compiled automatically. Thus you just perform the following DPDK configurations.

4.4.1. Setup kernel module

DPDK provides two kernel modules, igb_uio and rte_kni, to realize userspace NIC drivers and network packet processsing. The igb_uio is a wrapper module for DPDK-enabled NICs on top of UIO module of Linux kernel. The igb_uio allows userspace DPDK driver access to memory-mapped registers on NICs directly. The rte_kni module enables packet frame exchanges between an user space DPDK application and network stack in kernel space.

  • Load UIO and kernel modules.

    The kernel modules built are available in the src/dpdk/build/kmod directory.

    $ sudo modprobe uio
$ cd lagopus
$ sudo insmod ./src/dpdk/build/kmod/igb_uio.ko
$ sudo insmod ./src/dpdk/build/kmod/rte_kni.ko
$ lsmod | egrep 'uio|kni'
rte_kni               282624  0
igb_uio                16384  0
uio                    20480  1 igb_uio
$

    Note

    You have to perform the above steps above after the OS reboot.

4.4.2. Setup Hugepages

Make hugepages available to DPDK. You can setup hugepage in two ways:

  1. Manual configuration: Repeate steps after reboot if you select this.
  2. Script configuration: Select this to keep it permanent after reboot.

Note

When configured manually with , you need the same steps after rebooting OS.

  1. Manual confguration (undone after reboot)

    Perform the following commands

    $ sudo sh -c "echo 256 >  /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages"
$ sudo mkdir -p /mnt/huge
$ sudo mount -t hugetlbfs nodev /mnt/huge

  2. Script configuration (permanent after reboot)

    • Reserve 256 pages of 2 MB hugepages in linux by adding the following line in /etc/sysctl.conf.

      $ sudo vi /etc/sysctl.conf
vm.nr_hugepages = 256
$

    • Enable permanent across reboots. Add a directory for hugepages and the following line to /etc/fstab so that mount point can be made permanent across reboots.

      $ sudo mkdir -p /mnt/huge
$ sudo vi /etc/fstab
nodev /mnt/huge hugetlbfs defaults 0 0

    • Confirm HugePages are configured correctly by the below commands.

      $ grep -i "HugePages" /proc/meminfo
AnonHugePages:         0 kB
HugePages_Total:     256
HugePages_Free:      256
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
$ mount | grep huge
nodev on /mnt/huge type hugetlbfs (rw)
$

4.4.3. NIC (Network Interface Card) assignment

The following steps detatch the control and management of NICs, which will be used for DPDK application, from Linux kernel.

In this example, the host has three NICs, we are going to use eth1, eth2 for Lagopus software swtich. Perform the following steps to enable NICs for DPDK application.

Note

  • You need to unbound NIC from kernel (ixgbe driver) before using it with DPDK.
  • You will lose connection to the OS if you unbound NIC used for management plane (ex: ssh).

  • Check PCI ID of the NICs you want to use for DPDK with dpdk-devbind.py script.

    dpdk-devbind.py script displays DPDK-enabled NIC information, such as PCI ID and interface name in Linux kernel.

    $ cd lagopus
$ sudo ./src/dpdk/tools/dpdk-devbind.py --status

Network devices using DPDK-compatible driver
============================================
<none>

Network devices using kernel driver
===================================
0000:00:03.0 '82540EM Gigabit Ethernet Controller' if=eth0 drv=e1000 unused= *Active*
0000:00:08.0 '82545EM Gigabit Ethernet Controller (Copper)' if=eth1 drv=e1000 unused=
0000:00:09.0 '82545EM Gigabit Ethernet Controller (Copper)' if=eth2 drv=e1000 unused=

Other network devices
=====================
<none>
$

    In this example, You can see PCI IDs of eth1 and eth2 from the output, 0000:00:08.0 and 0000:00:09.0.

    NIC configuation before NIC unbound operation
    PCI ID Linux IF name Bounded by Linux
    0000:00:03.0 eth0 Yes
    0000:00:08.0 eth1 Yes
    0000:00:09.0 eth2 Yes

  • Unbound NICs from ixgbe driver and registerd with igb_uio driver.

    Perform the dpdk-devbind with the PCI IDs to be unbounded from Linux kernel.

    ~/lagopus$ sudo ./src/dpdk/tools/dpdk-devbind.py --bind=igb_uio 0000:00:08.0 0000:00:09.0
~/lagopus$ sudo ./src/dpdk/tools/dpdk-devbind.py --status

Network devices using DPDK-compatible driver
============================================
0000:00:08.0 '82545EM Gigabit Ethernet Controller (Copper)' drv=igb_uio unused=
0000:00:09.0 '82545EM Gigabit Ethernet Controller (Copper)' drv=igb_uio unused=

Network devices using kernel driver
===================================
0000:00:03.0 '82540EM Gigabit Ethernet Controller' if=eth0 drv=e1000 unused=igb_uio *Active*

Other network devices
=====================
<none>

  • Memorize the DPDK NIC configuration

    After the unbound NICs from Linux kernel, the NIC which was bounded to eth1 is accesssed by DPDK port #0 or PCI ID (0000:00:08.0) directly. The NIC which was bounded by eth2 is also accessed by DPDK port #1 or PCI ID (0000:00:09.0).

    NIC configuation after NIC unbound operation
    PCI ID Linux IF name Bounded by Linux DPDK ready DPDK port #
    0000:00:03.0 eth0 Yes No  
    0000:00:08.0 eth1 No Yes 0
    0000:00:09.0 eth2 No Yes 1

4.5. Setup Lagopus configuration file

Example Lagopus configuration (DSL format) can be found at “misc/examples/lagopus.dsl”. lagopus.dsl file must be located at the same directory of the executable of lagopus, or under /usr/local/etc/lagopus/.

  • Copy sample configuration file under /usr/local/etc/lagopus/.

    $ sudo mkdir /usr/local/etc/lagopus/
$ cd ~/lagopus-0.2.10
$ sudo cp misc/examples/lagopus.dsl /usr/local/etc/lagopus/lagopus.dsl

  • Edit configuration file suited to your environment.

    • Example:

      • One OpenFlow controller: “127.0.0.1”
      • eth0: management interface. (Thus does not appear in the configuration)
      • DPDK port #0 which was eth1 and DPDK port #1 which was eth2: Lagopus dataplane ports. These two ports are accessed with DPDK.
      $ sudo vi /usr/local/etc/lagopus/lagopus.dsl
channel channel01 create -dst-addr 127.0.0.1 -protocol tcp
controller controller01 create -channel channel01 -role equal -connection-type main
interface interface01 create -type ethernet-dpdk-phy -port-number 0
interface interface02 create -type ethernet-dpdk-phy -device :0000:00:09.0 # Since v0.2.10, interface can be specified by PCI ID.
port port01 create -interface interface01
port port02 create -interface interface02
bridge bridge01 create -controller controller01 -port port01 1 -port port02 2 -dpid 0x1
bridge bridge01 enable
$

4.6. Running / Stopping Lagopus software switch

4.6.1. DPDK command option

In order to run Lagopus softwarwe switch with DPDK configuration, you need to specify DPDK-related options in lagopus command: which CPU cores are assigned to packet processing (-c), how many memory channels the host has (-n).

In this example, the host has four CPU cores and two memory channel and you try to assign CPU core #1 and CPU core #0 for network I/O and processing and DPDK port #0 and DPDK port #1. With -c option, you should specify which CPU cores are assigned to Lagopus software switch. The value of -c option uses the hexadecimal notation that its N -bit shows whehter CPU core # N is used or not for DPDK. The following table help your understanding of CPU flags.

DPDK CPU flag
cpu core # 3 cpu core # 2 cpu core # 1 cpu core # 0 flag in binary flag in hexadecimal
0 0 1 1 0x0011 0x3

4.6.2. Run Lagopus software switch

  • Peform the following command to run Lagopus software switch in foreground with debug mode.

    $ sudo lagopus -d -- -c3 -n2 --

Or

  • Peform the following command to run Lagopus software switch in background.

    $ sudo lagopus -- -c3 -n2 --

4.6.3. Operation of Lagopus software switch with lagosh

  • Enter show version command from lagosh to confirm it’s running.

  • Enter stop command from lagosh to stop Lagopus vswitch.

    $ lagosh
Lagosh> show version
{
    "product-name": "Lagopus",
    "version": "0.2.10-release"
}
Lagosh> stop
Lagosh> show version
Socket connection refused.  Lagopus is not running?
Lagosh> exit
$

  • For more configuration options, refer to Lagopus software switch command options and examples

  • For more information about lagosh, refer to Using Lagopus CLI (lagosh)

Note

Lagopus software switch with DPDK use pooling-based packet processing. Therefore you will see lagopus consume 100% of CPUs by top command.

top - 15:50:26 up 6 min,  1 user,  load average: 0.34, 0.13, 0.07
Tasks:  83 total,   2 running,  81 sleeping,   0 stopped,   0 zombie
%Cpu0  :  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu1  :100.0 us,  0.0 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem:   3081312 total,   686804 used,  2394508 free,    20676 buffers
KiB Swap:  3143676 total,        0 used,  3143676 free.    78792 cached Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
 1205 root      20   0 1037728  16264   6764 S 100.2  0.5   0:25.58 lagopus