[ovs-discuss] OpenVSwitch latency issues

[Chris]

Hello,

actually I'm coming from OpenStack. But we use the L2 Open vSwitch (1.11.0),
so this is maybe the right place to ask.

The Hypervisor is KVM the Guest Windows 2012 R2 we use the latest virtio
driver for the guest.

We are facing some network performance issues on our webserver VMs. For
example each DB connection or internal API query adds an additional latency
by factor 2x.

This accumulates and make the service latency wise to slow.



We did some performance testing and the throughput from/to the VM is a
little spikey.

Especially the VM receiving from a bare metal (BM) machine seems to be
significant slower.

Loading Image...



Any tips to debug Open vSwitch regarding the latency or tuning tips?



Thanks in advance!



Greetings

Chris

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://openvswitch.org/pipermail/discuss/attachments/20141012/79ef7c56/attachment.html>

[Ben Pfaff]

Post by Chris
Hello,
actually I'm coming from OpenStack. But we use the L2 Open vSwitch (1.11.0),
so this is maybe the right place to ask.
The Hypervisor is KVM the Guest Windows 2012 R2 we use the latest virtio
driver for the guest.
We are facing some network performance issues on our webserver VMs. For
example each DB connection or internal API query adds an additional latency
by factor 2x.
This accumulates and make the service latency wise to slow.
We did some performance testing and the throughput from/to the VM is a
little spikey.
Especially the VM receiving from a bare metal (BM) machine seems to be
significant slower.
http://i.imgur.com/TByQsjZ.png
Any tips to debug Open vSwitch regarding the latency or tuning tips?

The easiest thing to do is to upgrade to version 2.3. We've been
concentrating on performance for the last several releases, so every
version you upgrade should improve performance.

[Chris]

Hello

We upgraded the ovs user space version to 2.3.
The kernel module is still the shipped one: CentOS 6.5 Linux
2.6.32-431.23.3.el6.x86_64
We didn't see a big performance improvement.

Here some more data, it's hard to see the hits & miss they seemed to be
unchanged, but the flows has reduced drastically. The ovs-vswitchd consumes
around 5% - 10 % CPU on both and almost no memory.

Version 1.11:
- around 5000 flows
- around 3500 hits per second
- around 700 miss per second

After upgrade version 2.3:
- around 2500 flows
- around 3500 hits per second
- around 700 miss per second

Any further performance suggestions?

Thank you!

Greetings
Chris

-----Original Message-----
From: Ben Pfaff [mailto:blp at nicira.com]
Sent: Monday, October 13, 2014 00:36
To: Chris
Cc: discuss at openvswitch.org
Subject: Re: [ovs-discuss] OpenVSwitch latency issues

Post by Chris
Hello,
actually I'm coming from OpenStack. But we use the L2 Open vSwitch
(1.11.0), so this is maybe the right place to ask.
The Hypervisor is KVM the Guest Windows 2012 R2 we use the latest
virtio driver for the guest.
We are facing some network performance issues on our webserver VMs.
For example each DB connection or internal API query adds an
additional latency by factor 2x.
This accumulates and make the service latency wise to slow.
We did some performance testing and the throughput from/to the VM is a
little spikey.
Especially the VM receiving from a bare metal (BM) machine seems to
be significant slower.
http://i.imgur.com/TByQsjZ.png
Any tips to debug Open vSwitch regarding the latency or tuning tips?

The easiest thing to do is to upgrade to version 2.3. We've been
concentrating on performance for the last several releases, so every version
you upgrade should improve performance.

[Ben Pfaff]

Post by Chris
We upgraded the ovs user space version to 2.3.
The kernel module is still the shipped one: CentOS 6.5 Linux
2.6.32-431.23.3.el6.x86_64

Linux 2.6.32 doesn't come with an openvswitch kernel module, so what
version of the kernel module do you have? Some performance improvements
will only come with a sufficiently new kernel module.

[Chris]

Hi

It looks like it's the one which comes with OpenStack:

# modinfo openvswitch
filename:
/lib/modules/2.6.32-431.23.3.el6.x86_64/kernel/net/openvswitch/openvswitch.k
o
license: GPL
description: Open vSwitch switching datapath
srcversion: 993363C44DF474BD67B03CC
depends: vxlan
vermagic: 2.6.32-431.23.3.el6.x86_64 SMP mod_unload modversions

-----Original Message-----
From: Ben Pfaff [mailto:blp at nicira.com]
Sent: Wednesday, October 15, 2014 22:45
To: Chris
Cc: discuss at openvswitch.org
Subject: Re: [ovs-discuss] OpenVSwitch latency issues

Post by Chris
We upgraded the ovs user space version to 2.3.
The kernel module is still the shipped one: CentOS 6.5 Linux
2.6.32-431.23.3.el6.x86_64

Linux 2.6.32 doesn't come with an openvswitch kernel module, so what version
of the kernel module do you have? Some performance improvements will only
come with a sufficiently new kernel module.

[Ben Pfaff]

How can I find out what version that is? I am not familiar with how
OpenStack is distributed.

Post by Chris
Hi
# modinfo openvswitch
/lib/modules/2.6.32-431.23.3.el6.x86_64/kernel/net/openvswitch/openvswitch.k
o
license: GPL
description: Open vSwitch switching datapath
srcversion: 993363C44DF474BD67B03CC
depends: vxlan
vermagic: 2.6.32-431.23.3.el6.x86_64 SMP mod_unload modversions
-----Original Message-----
From: Ben Pfaff [mailto:blp at nicira.com]
Sent: Wednesday, October 15, 2014 22:45
To: Chris
Cc: discuss at openvswitch.org
Subject: Re: [ovs-discuss] OpenVSwitch latency issues

Post by Chris
We upgraded the ovs user space version to 2.3.
The kernel module is still the shipped one: CentOS 6.5 Linux
2.6.32-431.23.3.el6.x86_64

Linux 2.6.32 doesn't come with an openvswitch kernel module, so what version
of the kernel module do you have? Some performance improvements will only
come with a sufficiently new kernel module.