LINE

LINE is a network emulator designed for realism, accuracy and powerful measurement capabilities. It is capable of emulating in real-time networks on layer 3, while forwarding network traffic generated by real Linux network stacks.

LINE is released under the GPLv2 license.

Get the latest LINE: 

git clone https://git.epfl.ch/repo/line.git

Or get the setup used in our SIGCOMM 2014 paper:

git clone https://git.epfl.ch/repo/line-sigcomm14.git

Screencast

Architecture

A typical LINE setup consists of three computers, each having a specific role.

Network emulator

This machine performs the emulation of the routers in the network. It should be a system optimized for low latency.

Hardware requirements

CPU: 64-bit quad-core with good single thread performance. Our current setup uses an Intel Xeon E3-1290 V2 (Ivy Bridge) CPU, but there might be other good options. We found this classification useful: PassMark - CPU Mark Single Thread Performance.

Network: two network cards, one for WAN and another one dedicated for exchanging traffic with the traffic generator. The dedicated card should run at 10 Gbps (1 Gbps will work too, but will limit the amount of traffic that can be sent through the emulated network to < 1 Gbps); we use Intel 10-Gigabit X540-AT2. The traffic generator and the network emulator should be connected directly through a network cable (no switches/routers, to avoid background traffic and other effects).

Software requirements

OS: Ubuntu 12.04.4 (LTS) 64-bit server.

The machine should not be used for anything else, and should be connected to a secure network, since we are going to disable the firewall.

Traffic generator

This machine generates the traffic that will be routed through the emulated network. It should be a system optimized for good network I/O performance.

Hardware requirements

CPU: 64-bit quad-core or better with good overall performance. Our current setup uses an Intel Xeon E3-1290 V2 (Ivy Bridge) CPU, but a CPU with more cores or a multi-CPU setup might work well. This classification may be useful: PassMark - CPU Mark High End CPUs.

Network: two network cards, one for WAN and another one dedicated for exchanging traffic with the network emulator. The dedicated card should run at the same speed as the dedicated card of the network emulator.

Software requirements

OS: Ubuntu 12.04.4 (LTS) 64-bit server.

The machine should not be used for anything else, and should be connected to a secure network, since we are going to disable the firewall.

Control center

This machine is used for configuring the experiments, and controlling remotely the router emulator and the traffic generator.

Hardware requirements

CPU: 64-bit, with modest performance.

Network: network connectivity is required in order to run experiments.

Software requirements

OS: Ubuntu 12.04.4 (LTS) 64-bit desktop.

System configuration

Network emulator

These instructions assume that the system is running a clean Ubuntu 12.04.4 (LTS) 64-bit server installation. The instructions should be executed as root (run sudo su).

Configure password-less SSH authentication with keys for root

See https://help.ubuntu.com/community/SSH/OpenSSH/Keys.

Disable hyperthreading from BIOS

Hyper-threading does not work well with our low-latency configuration, and is not particularly useful, so we disable it.

Install GCC & friends:

apt-get install -y build-essential libssl-dev flex bison gperf libicu-dev libxslt-dev ruby libcups2-dev libglib2.0-dev

Set a static IP address on the dedicated interface (in our case, eth7):

# In /etc/network/interfaces:
auto eth7
iface eth7 inet static
    address 192.168.77.1
    netmask 255.255.255.0

Install some utilities:

apt-get install -y wget mc lm-sensors ipmitool htop stress zip bc traceroute ethtool irqbalance strace tcpdump bwm-ng netcat pciutils

Compile and install iproute2 and ethtool for kernel 3.11:

apt-get install -y iptables-dev libdb4.8-dev

wget https://www.kernel.org/pub/linux/utils/net/iproute2/iproute2-3.11.0.tar.xz
tar -xJf iproute2-3.11.0.tar.xz
cd iproute2-3.11.0
./configure --prefix=/usr
make -j7

make install
cd ..

wget https://www.kernel.org/pub/software/network/ethtool/ethtool-3.11.tar.xz
tar -xJf ethtool-3.11.tar.xz
cd ethtool-3.11
./configure --prefix=/usr
make -j7

make install
rm -f /sbin/ethttool
cp /usr/sbin/ethtool /sbin/ethttool
cd ..

Fix a strange IPMI problem (optional, in our case there were errors in dmesg)

rm /etc/modprobe.d/ipmi.conf

Install GCC 4.7.3 (for C++11 atomic ops)

apt-get install -y software-properties-common python-software-properties
add-apt-repository -y ppa:ubuntu-toolchain-r/test
apt-get update
apt-cache search "g\+\+"
apt-get install -y gcc-4.7 g++-4.7

ls -lh /usr/bin/g++
update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-4.6 60 --slave /usr/bin/g++ g++ /usr/bin/g++-4.6
update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-4.7 40 --slave /usr/bin/g++ g++ /usr/bin/g++-4.7
update-alternatives --config gcc
# Select 4.7, then check:
g++ --version

Compile and install Qt 4 (without GUI)

apt-get install -y libssl-dev flex bison gperf libicu-dev libxslt-dev ruby libcups2-dev libglib2.0-dev

pushd .
cd /tmp
rm -rf qt-everywhere-opensource-src-4.8.5
wget http://download.qt.io/archive/qt/4.8/4.8.5/qt-everywhere-opensource-src-4.8.5.tar.gz
tar xzf qt-everywhere-opensource-src-4.8.5.tar.gz
cd qt-everywhere-opensource-src-4.8.5
./configure --prefix=/usr/local --opensource --confirm-license --graphicssystem=raster --release -no-gui -nomake examples -nomake demos
make -j4

make install
cd ..
popd

Install PF_RING dependencies

apt-get install -y libnuma-dev flex bison

Install LINE dependencies

apt-get install -y libunwind7-dev

wget https://gperftools.googlecode.com/files/gperftools-2.1.tar.gz
tar xzf gperftools-2.1.tar.gz
cd gperftools-2.1
./configure --prefix=/usr
make -j4

make install
cd ..

apt-get install -y libnl-dev libev-dev

Compile and install a custom kernel (configured with low-latency optimizations)

Run these commands as non-root:

sudo apt-get install -y ncurses-dev kernel-package
mkdir ~/src
cd ~/src
apt-get source linux-image-$(uname -r)
cd linux-lts-trusty-3.13.0
make menuconfig

Edit the following options:

  • enable CONFIG_NO_HZ_FULL (General setup -> Timers subsystem -> Timer tick handling -> Full dynticks system (tickless));
  • enable CONFIG_NO_HZ_FULL_ALL (General setup -> Timers subsystem -> Timer tick handling -> Full dynticks system on all CPUs by default);
  • enable CONFIG_PREEMPT (Processor type and features -> Preemption model -> Preemptible kernel (Low-Latency Desktop));
  • select your CPU model instead of generic x86_64 (Processor type and features -> Processor family -> Core 2/newer Xeon).

export CONCURRENCY_LEVEL=4
time fakeroot make-kpkg --initrd --append-to-version=-line kernel-image kernel-headers
cd ..
sudo dpkg -i linux-image-* linux-headers-*

Reboot

reboot

Disable the firewall and in-kernel connection tracking (needs to be done after every kernel update)

service ufw stop
echo "manual" > /etc/init/ufw.override
echo "blacklist iptable_filter" >> /etc/modprobe.d/blacklist.conf
echo "blacklist ip_tables" >> /etc/modprobe.d/blacklist.conf
echo "blacklist x_tables" >> /etc/modprobe.d/blacklist.conf
for d in /lib/modules/* ; do mv $d/kernel/net/ipv4/netfilter $d/kernel/net/ipv4/netfilter-old || /bin/true ; done

Set a custom kernel command line

# In /etc/default/grub, add to GRUB_CMDLINE_LINUX_DEFAULT and GRUB_CMDLINE_LINUX:
isolcpus=1,2,3 nohz_full=1,2,3 rcu_nocbs=1,2,3 acpi=noirq nosoftlockup intel_idle.max_cstate=0 processor.max_cstate=1 idle=halt mce=ignore_ce memtest

Update the bootloader:

update-grub
Parameters explained:
  • isolcpus=1,2,3: isolates cores 1, 2 and 3 from the rest of the system (they will be dedicated for LINE, i.e. no other applications will be scheduled to run on these cores); core 0 will be kept for housekeeping;
  • nohz_full=1,2,3: disables the local timer on cores 1, 2 and 3 (to reduce latency/jitter);
  • rcu_nocbs=1,2,3: disables RCU callback execution on cores 1, 2 and 3 (to reduce latency/jitter);
  • acpi=noirq: whether this is useful is highly dependent on your motherboard and/or BIOS. We used it because a large number of "Rescheduled interrupts" was listed in /proc/interrupts. YMMV. See ReschedulingInterrupts;
  • nosoftlockup: disables the kernel soft-lockup detector. Setting this is recommended in the low-latency configuration manual of our HP server (included in servers/HP Proliant Low Latency Guide.pdf);
  • intel_idle.max_cstate=0 processor.max_cstate=1 idle=halt mce=ignore_ce: do not allow any core to go into a sleep state (reduces latency/jitter);
  • memtest: runs a memory test at boot. Optional.

Disable irqbalance (LINE maps manually all IRQs except for the dedicated network card to CPU 0):

# Set ENABLED=0 in /etc/default/irqbalance

Reboot

reboot

Traffic generator

These instructions assume that the system is running a clean Ubuntu 12.04.4 (LTS) 64-bit server installation. The instructions should be executed as root (run sudo su).

Configure password-less SSH authentication with keys for root

See https://help.ubuntu.com/community/SSH/OpenSSH/Keys.

Install GCC & friends:

apt-get install -y build-essential libssl-dev flex bison gperf libicu-dev libxslt-dev ruby libcups2-dev libglib2.0-dev

Set a static IP address on the dedicated interface (in our case, eth7):

# In /etc/network/interfaces:
auto eth7
iface eth7 inet static
    address 192.168.77.2
    netmask 255.255.255.0

Install some utilities:

apt-get install -y wget mc lm-sensors ipmitool htop stress zip bc traceroute ethtool irqbalance strace tcpdump bwm-ng netcat pciutils

Compile and install iproute2 and ethtool for kernel 3.11:

apt-get install -y iptables-dev libdb4.8-dev

wget https://www.kernel.org/pub/linux/utils/net/iproute2/iproute2-3.11.0.tar.xz
tar -xJf iproute2-3.11.0.tar.xz
cd iproute2-3.11.0
./configure --prefix=/usr
make -j7

make install
cd ..

wget https://www.kernel.org/pub/software/network/ethtool/ethtool-3.11.tar.xz
tar -xJf ethtool-3.11.tar.xz
cd ethtool-3.11
./configure --prefix=/usr
make -j7

make install
rm -f /sbin/ethttool
cp /usr/sbin/ethtool /sbin/ethttool
cd ..

Fix a strange IPMI problem (optional, in our case there were errors in dmesg)

rm /etc/modprobe.d/ipmi.conf

Install GCC 4.7.3 (for C++11 atomic ops)

apt-get install -y software-properties-common python-software-properties
add-apt-repository -y ppa:ubuntu-toolchain-r/test
apt-get update
apt-cache search "g\+\+"
apt-get install -y gcc-4.7 g++-4.7

ls -lh /usr/bin/g++
update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-4.6 60 --slave /usr/bin/g++ g++ /usr/bin/g++-4.6
update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-4.7 40 --slave /usr/bin/g++ g++ /usr/bin/g++-4.7
update-alternatives --config gcc
# Select 4.7, then check:
g++ --version

Compile and install Qt 4 (without GUI)

apt-get install -y libssl-dev flex bison gperf libicu-dev libxslt-dev ruby libcups2-dev libglib2.0-dev

pushd .
cd /tmp
rm -rf qt-everywhere-opensource-src-4.8.5
wget http://download.qt.io/archive/qt/4.8/4.8.5/qt-everywhere-opensource-src-4.8.5.tar.gz
tar xzf qt-everywhere-opensource-src-4.8.5.tar.gz
cd qt-everywhere-opensource-src-4.8.5
./configure --prefix=/usr/local --opensource --confirm-license --graphicssystem=raster --release -no-gui -nomake examples -nomake demos
make -j4

make install
cd ..
popd

Install LINE dependencies

apt-get install -y libunwind7-dev

wget https://gperftools.googlecode.com/files/gperftools-2.1.tar.gz
tar xzf gperftools-2.1.tar.gz
cd gperftools-2.1
./configure --prefix=/usr
make -j4

make install
cd ..

apt-get install -y libnl-dev libev-dev

Disable the firewall and in-kernel connection tracking (needs to be done after every kernel update)

service ufw stop
echo "manual" > /etc/init/ufw.override
echo "blacklist iptable_filter" >> /etc/modprobe.d/blacklist.conf
echo "blacklist ip_tables" >> /etc/modprobe.d/blacklist.conf
echo "blacklist x_tables" >> /etc/modprobe.d/blacklist.conf
for d in /lib/modules/* ; do mv $d/kernel/net/ipv4/netfilter $d/kernel/net/ipv4/netfilter-old || /bin/true ; done

Reboot

reboot

Control center

This machine requires a graphical desktop and password-less SSH access as root to the network emulator machine and the traffic generator machine.

Install GCC & friends and git:

apt-get install -y build-essential git

Install Qt4

apt-get install -y libqt4-dev qt4-dev-tools

Install other dependencies

apt-get install -y libunwind7-dev

Download LINE

git clone https://git.epfl.ch/repo/line.git
cd line

Configure LINE

cp remote-config-template.sh remote-config.sh

Edit remote-config.sh and set the following variables:

  • REMOTE_HOST_ROUTER: The IP address of the network emulator;
  • REMOTE_DEDICATED_IF_ROUTER: The name of the dedicated network interface on the network emulator (in our network configuration example from above, eth7);
  • REMOTE_HOST_HOSTS: The IP address of the traffic generator;
  • REMOTE_DEDICATED_IF_HOSTS: The name of the dedicated network interface on the traffic generator (in our network configuration example from above, eth7).

Compile line-gui (control center GUI)

mkdir build-line-gui
cd build-line-gui
qmake ../line-gui/line-gui.pro -spec linux-g++-64 CONFIG+=debug CONFIG+=declarative_debug
make -j4
cd ..

Compile line-runner (control center backend)

mkdir build-line-runner
cd build-line-runner
qmake ../line-runner/line-runner.pro -spec linux-g++-64 CONFIG+=debug CONFIG+=declarative_debug
make -j4
cd ..

Compile line-router (network emulator application)

cd line-router
./make-remote.sh
cd ..

Compile line-traffic (traffic generator application)

cd line-traffic
./make-remote.sh
cd ..

Start line-gui

cd build-line-gui
./line-gui

Using LINE

Running the first experiment

Video: testing-the-setup.webm

Editing the topology

Video: editing-the-topology.webm

Generating traffic

Video: generating-traffic.webm

Inspecting the experiment results

Video: inspecting-the-results.webm

Credits

LINE uses the following third party libraries/components/applications: