Discussion:
[ovs-discuss] Open vSwitch performance with UDP traffic
McGarvey, Kevin
2014-01-21 22:50:30 UTC
Permalink
I am sending UDP DNS queries to a KVM guest through Open vSwitch running on the RHEL hypervisor. With traffic of only 5K request/response pairs per second the CPU consumption of the ovs-vswitchd process is over 60%. The source IP and port change with each request, simulating normal UDP DNS traffic. It appears that each request is being handled as a new flow and is being sent to the userspace process. When I dump data flows with ovs-dpctl, there are thousands of flows that are never used.

Is there a way to get reasonable performance from Open vSwitch with this type of traffic? I tried adding a flow with ovs-ofctl, specifying the destination MAC of the guest and everything else wildcarded, but I saw no change in the CPU consumption of ovs-vswitchd.

Kevin McGarvey
Verisign Inc.
"This message (including any attachments) is intended only for the use of the individual or entity to which it is addressed, and may contain information that is non-public, proprietary, privileged, confidential and exempt from disclosure under applicable law or may be constituted as attorney work product. If you are not the intended recipient, you are hereby notified that any use, dissemination, distribution, or copying of this communication is strictly prohibited. If you have received this message in error, notify sender immediately and delete this message immediately."
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://openvswitch.org/pipermail/discuss/attachments/20140121/28e259d8/attachment.htm>
Ben Pfaff
2014-01-21 22:55:45 UTC
Permalink
Post by McGarvey, Kevin
I am sending UDP DNS queries to a KVM guest through Open vSwitch
running on the RHEL hypervisor. With traffic of only 5K
request/response pairs per second the CPU consumption of the
ovs-vswitchd process is over 60%. The source IP and port change with
each request, simulating normal UDP DNS traffic. It appears that
each request is being handled as a new flow and is being sent to the
userspace process. When I dump data flows with ovs-dpctl, there are
thousands of flows that are never used.
Is there a way to get reasonable performance from Open vSwitch with
this type of traffic? I tried adding a flow with ovs-ofctl,
specifying the destination MAC of the guest and everything else
wildcarded, but I saw no change in the CPU consumption of
ovs-vswitchd.
You didn't say what version of OVS you're using. The CPU% used for
this kind of traffic should drop dramatically with an upgrade to 1.11
or later.
Ben Pfaff
2014-01-21 23:17:37 UTC
Permalink
This post might be inappropriate. Click to display it.
McGarvey, Kevin
2014-01-22 17:35:40 UTC
Permalink
Post by Ben Pfaff
Please don't drop the mailing list.
My apologies for dropping the mailing list.
Post by Ben Pfaff
I'd expect a dramatic drop in CPU consumption in that case. There are
a few special cases where the upgrade wouldn't help. One is if
in-band control is in use, another is if NetFlow is turned on, a third
is if LACP bonds with L4 port based hashing are turned on, and there
are probably a few others that don't come to mind immediately.
I plan to rerun the test to rule out some mistake on my part.

Could you provide more information about the nature of the change made in
1.11 that improves performance for this type of traffic? Is the kernel
module able to forward UDP DNS packets without sending them to userspace,
or was it an optimization of the userspace processing? What roughly is
the level of performance I should see?
Post by Ben Pfaff
Version 2.0 helped with this more, too.
Sorry, I left out details in order to get an answer about whether kernel
forwarding of single packet flows like this is even possible.
I was initially using the version that ships with RHEL 6.4, but I upgraded
to openvswitch-1.11.0_8ce28d-1.el6ost.x86_64 and upgraded the kernel to
2.6.32-358.123.4.openstack.el6.x86_64. I upgraded the kernel first, at
the recommendation of one of our engineers who has been working a lot with
OpenStack. When that didn't reduce the CPU consumption I upgraded Open
vSwitch. But I didn't notice a drop in CPU consumption.
The server is a Dell 710
The NICs are Broadcom bcm5709
We have 2 bridges configured, one for each NIC
We are using VLANs
Kevin
Post by Ben Pfaff
Post by McGarvey, Kevin
I am sending UDP DNS queries to a KVM guest through Open vSwitch
running on the RHEL hypervisor. With traffic of only 5K
request/response pairs per second the CPU consumption of the
ovs-vswitchd process is over 60%. The source IP and port change with
each request, simulating normal UDP DNS traffic. It appears that
each request is being handled as a new flow and is being sent to the
userspace process. When I dump data flows with ovs-dpctl, there are
thousands of flows that are never used.
Is there a way to get reasonable performance from Open vSwitch with
this type of traffic? I tried adding a flow with ovs-ofctl,
specifying the destination MAC of the guest and everything else
wildcarded, but I saw no change in the CPU consumption of
ovs-vswitchd.
You didn't say what version of OVS you're using. The CPU% used for
this kind of traffic should drop dramatically with an upgrade to 1.11
or later.
Ben Pfaff
2014-01-22 17:39:14 UTC
Permalink
Post by McGarvey, Kevin
Post by Ben Pfaff
I'd expect a dramatic drop in CPU consumption in that case. There are
a few special cases where the upgrade wouldn't help. One is if
in-band control is in use, another is if NetFlow is turned on, a third
is if LACP bonds with L4 port based hashing are turned on, and there
are probably a few others that don't come to mind immediately.
I plan to rerun the test to rule out some mistake on my part.
Could you provide more information about the nature of the change made in
1.11 that improves performance for this type of traffic? Is the kernel
module able to forward UDP DNS packets without sending them to userspace,
or was it an optimization of the userspace processing? What roughly is
the level of performance I should see?
In 1.11 and later, for simple OpenFlow tables (I don't think you
mentioned whether you are using a controller or which one), Open
vSwitch can set up only a single kernel flow that covers many possible
flows, for example all possible UDP destination ports, rather than
setting up an individual kernel flow for each UDP packet. When that
works, it eliminates most of the kernel/userspace traffic, improving
performance. Version 2.0 is better at analyzing OpenFlow flow tables
to see when this is possible, so it can better take advantage of the
ability.
Ben Pfaff
2014-01-22 17:44:32 UTC
Permalink
Post by Ben Pfaff
Post by McGarvey, Kevin
Post by Ben Pfaff
I'd expect a dramatic drop in CPU consumption in that case. There are
a few special cases where the upgrade wouldn't help. One is if
in-band control is in use, another is if NetFlow is turned on, a third
is if LACP bonds with L4 port based hashing are turned on, and there
are probably a few others that don't come to mind immediately.
I plan to rerun the test to rule out some mistake on my part.
Could you provide more information about the nature of the change made in
1.11 that improves performance for this type of traffic? Is the kernel
module able to forward UDP DNS packets without sending them to userspace,
or was it an optimization of the userspace processing? What roughly is
the level of performance I should see?
In 1.11 and later, for simple OpenFlow tables (I don't think you
mentioned whether you are using a controller or which one), Open
vSwitch can set up only a single kernel flow that covers many possible
flows, for example all possible UDP destination ports, rather than
setting up an individual kernel flow for each UDP packet. When that
works, it eliminates most of the kernel/userspace traffic, improving
performance. Version 2.0 is better at analyzing OpenFlow flow tables
to see when this is possible, so it can better take advantage of the
ability.
I see that I didn't answer your question about performance.

When this optimization kicks in fully, I guess that the performance
should be about the same as for traffic with long flows (like the
netperf TCP_STREAM test, for example) in terms of packets per second.
McGarvey, Kevin
2014-01-22 20:17:05 UTC
Permalink
This post might be inappropriate. Click to display it.
Ben Pfaff
2014-01-22 20:23:18 UTC
Permalink
Post by McGarvey, Kevin
Post by Ben Pfaff
Post by McGarvey, Kevin
Post by McGarvey, Kevin
Post by Ben Pfaff
I'd expect a dramatic drop in CPU consumption in that case. There
are
Post by McGarvey, Kevin
Post by Ben Pfaff
a few special cases where the upgrade wouldn't help. One is if
in-band control is in use, another is if NetFlow is turned on, a
third
Post by McGarvey, Kevin
Post by Ben Pfaff
is if LACP bonds with L4 port based hashing are turned on, and there
are probably a few others that don't come to mind immediately.
I plan to rerun the test to rule out some mistake on my part.
Could you provide more information about the nature of the change
made in
Post by McGarvey, Kevin
1.11 that improves performance for this type of traffic? Is the
kernel
Post by McGarvey, Kevin
module able to forward UDP DNS packets without sending them to
userspace,
Post by McGarvey, Kevin
or was it an optimization of the userspace processing? What roughly
is
Post by McGarvey, Kevin
the level of performance I should see?
In 1.11 and later, for simple OpenFlow tables (I don't think you
mentioned whether you are using a controller or which one), Open
vSwitch can set up only a single kernel flow that covers many possible
flows, for example all possible UDP destination ports, rather than
setting up an individual kernel flow for each UDP packet. When that
works, it eliminates most of the kernel/userspace traffic, improving
performance. Version 2.0 is better at analyzing OpenFlow flow tables
to see when this is possible, so it can better take advantage of the
ability.
I see that I didn't answer your question about performance.
When this optimization kicks in fully, I guess that the performance
should be about the same as for traffic with long flows (like the
netperf TCP_STREAM test, for example) in terms of packets per second.
Thanks. This is encouraging. The only question is why isn't the
optimization kicking in?
I repeated the test, and under a load of 10K DNS requests/responses per
second ovs-vswitchd is using 82% of a core.
I wasn't sure whether in-band control was on or off by default, so I
disabled it with the command below and restarted openvswitch, but the cpu
ovs-vsctl set bridge <bridge> other-config:disable-in-band=true
I did not set up the configuration, but as far as I can tell Netflow is
not turned on. The output of 'ovsdb-tool -show-log | grep -i netflow' is
empty.
There are no bonded interfaces. The 2 NICs used for DNS traffic are
associated with separate bridges.
We are not using a controller.
In your response you mentioned that for simple OpenFlow tables Open
vSwitch can set up a single kernel flow that covers many possible flows.
I think this is exactly what I need. Do I need to add a flow using
ovs-ofctl?
No. With the settings you describe, it should kick in automatically.

Here is an experiment that might help. Take one of the flows that
"ovs-dpctl dump-flows" prints, then feed that flow back into
"ovs-appctl ofproto/trace", and show us the results. (You might have
to spend a few minutes reading the ovs-vswitchd manpage to learn the
ofproto/trace syntax, if you don't already know it.)
McGarvey, Kevin
2014-01-22 21:04:48 UTC
Permalink
Post by Ben Pfaff
Post by McGarvey, Kevin
Post by Ben Pfaff
Post by McGarvey, Kevin
Post by McGarvey, Kevin
Post by Ben Pfaff
I'd expect a dramatic drop in CPU consumption in that case. There
are
Post by McGarvey, Kevin
Post by Ben Pfaff
a few special cases where the upgrade wouldn't help. One is if
in-band control is in use, another is if NetFlow is turned on, a
third
Post by McGarvey, Kevin
Post by Ben Pfaff
is if LACP bonds with L4 port based hashing are turned on, and
there
Post by Ben Pfaff
Post by McGarvey, Kevin
Post by McGarvey, Kevin
Post by Ben Pfaff
are probably a few others that don't come to mind immediately.
I plan to rerun the test to rule out some mistake on my part.
Could you provide more information about the nature of the change
made in
Post by McGarvey, Kevin
1.11 that improves performance for this type of traffic? Is the
kernel
Post by McGarvey, Kevin
module able to forward UDP DNS packets without sending them to
userspace,
Post by McGarvey, Kevin
or was it an optimization of the userspace processing? What
roughly
Post by Ben Pfaff
Post by McGarvey, Kevin
is
Post by McGarvey, Kevin
the level of performance I should see?
In 1.11 and later, for simple OpenFlow tables (I don't think you
mentioned whether you are using a controller or which one), Open
vSwitch can set up only a single kernel flow that covers many
possible
Post by Ben Pfaff
Post by McGarvey, Kevin
flows, for example all possible UDP destination ports, rather than
setting up an individual kernel flow for each UDP packet. When that
works, it eliminates most of the kernel/userspace traffic, improving
performance. Version 2.0 is better at analyzing OpenFlow flow tables
to see when this is possible, so it can better take advantage of the
ability.
I see that I didn't answer your question about performance.
When this optimization kicks in fully, I guess that the performance
should be about the same as for traffic with long flows (like the
netperf TCP_STREAM test, for example) in terms of packets per second.
Thanks. This is encouraging. The only question is why isn't the
optimization kicking in?
I repeated the test, and under a load of 10K DNS requests/responses per
second ovs-vswitchd is using 82% of a core.
I wasn't sure whether in-band control was on or off by default, so I
disabled it with the command below and restarted openvswitch, but the cpu
ovs-vsctl set bridge <bridge> other-config:disable-in-band=true
I did not set up the configuration, but as far as I can tell Netflow is
not turned on. The output of 'ovsdb-tool -show-log | grep -i netflow' is
empty.
There are no bonded interfaces. The 2 NICs used for DNS traffic are
associated with separate bridges.
We are not using a controller.
In your response you mentioned that for simple OpenFlow tables Open
vSwitch can set up a single kernel flow that covers many possible flows.
I think this is exactly what I need. Do I need to add a flow using
ovs-ofctl?
No. With the settings you describe, it should kick in automatically.
Here is an experiment that might help. Take one of the flows that
"ovs-dpctl dump-flows" prints, then feed that flow back into
"ovs-appctl ofproto/trace", and show us the results. (You might have
to spend a few minutes reading the ovs-vswitchd manpage to learn the
ofproto/trace syntax, if you don't already know it.)
Below is the ofproto/trace output for an inbound request to bridge brsvr2.
One more piece of information is that the packets are going through a
load balancer.

ovs-appctl ofproto/trace brsvr2
"in_port(4),eth(src=00:1f:26:3d:e8:00,dst=52:54:00:a1:27:1e),eth_type(0x810
0),vlan(vid=138,pcp=0),encap(eth_type(0x0800),ipv4(src=40.90.75.212,dst=10.
63.101.234,proto=17,tos=0,ttl=60,frag=no),udp(src=26011,dst=53))"
Bridge: brsvr2
Flow:
udp,metadata=0,in_port=1,dl_vlan=138,dl_vlan_pcp=0,dl_src=00:1f:26:3d:e8:00
,dl_dst=52:54:00:a1:27:1e,nw_src=40.90.75.212,nw_dst=10.63.101.234,nw_tos=0
,nw_ecn=0,nw_ttl=60,tp_src=26011,tp_dst=53
Rule: table=0 cookie=0 priority=0
OpenFlow actions=NORMAL
forwarding to learned port

Final flow: unchanged
Relevant fields:
skb_priority=0,ip,in_port=1,dl_vlan=138,dl_src=00:1f:26:3d:e8:00,dl_dst=52:
54:00:a1:27:1e,nw_frag=no
Datapath actions: 9


And here is the output for a response from the VM:

ovs-appctl ofproto/trace brsvr2
"in_port(9),eth(src=52:54:00:a1:27:1e,dst=90:e2:ba:26:89:45),eth_type(0x810
0),vlan(vid=138,pcp=0),encap(eth_type(0x0800),ipv4(src=10.63.101.234,dst=40
.90.85.153,proto=17,tos=0,ttl=64,frag=no),udp(src=53,dst=26292))"
Bridge: brsvr2
Flow:
udp,metadata=0,in_port=2,dl_vlan=138,dl_vlan_pcp=0,dl_src=52:54:00:a1:27:1e
,dl_dst=90:e2:ba:26:89:45,nw_src=10.63.101.234,nw_dst=40.90.85.153,nw_tos=0
,nw_ecn=0,nw_ttl=64,tp_src=53,tp_dst=26292
Rule: table=0 cookie=0 priority=0
OpenFlow actions=NORMAL
forwarding to learned port

Final flow: unchanged
Relevant fields:
skb_priority=0,ip,in_port=2,dl_vlan=138,dl_src=52:54:00:a1:27:1e,dl_dst=90:
e2:ba:26:89:45,nw_frag=no
Datapath actions: 4
Ben Pfaff
2014-01-22 21:10:27 UTC
Permalink
Post by McGarvey, Kevin
Post by Ben Pfaff
Post by McGarvey, Kevin
Post by Ben Pfaff
Post by McGarvey, Kevin
Post by McGarvey, Kevin
Post by Ben Pfaff
I'd expect a dramatic drop in CPU consumption in that case. There
are
Post by McGarvey, Kevin
Post by Ben Pfaff
a few special cases where the upgrade wouldn't help. One is if
in-band control is in use, another is if NetFlow is turned on, a
third
Post by McGarvey, Kevin
Post by Ben Pfaff
is if LACP bonds with L4 port based hashing are turned on, and
there
Post by Ben Pfaff
Post by McGarvey, Kevin
Post by McGarvey, Kevin
Post by Ben Pfaff
are probably a few others that don't come to mind immediately.
I plan to rerun the test to rule out some mistake on my part.
Could you provide more information about the nature of the change
made in
Post by McGarvey, Kevin
1.11 that improves performance for this type of traffic? Is the
kernel
Post by McGarvey, Kevin
module able to forward UDP DNS packets without sending them to
userspace,
Post by McGarvey, Kevin
or was it an optimization of the userspace processing? What
roughly
Post by Ben Pfaff
Post by McGarvey, Kevin
is
Post by McGarvey, Kevin
the level of performance I should see?
In 1.11 and later, for simple OpenFlow tables (I don't think you
mentioned whether you are using a controller or which one), Open
vSwitch can set up only a single kernel flow that covers many
possible
Post by Ben Pfaff
Post by McGarvey, Kevin
flows, for example all possible UDP destination ports, rather than
setting up an individual kernel flow for each UDP packet. When that
works, it eliminates most of the kernel/userspace traffic, improving
performance. Version 2.0 is better at analyzing OpenFlow flow tables
to see when this is possible, so it can better take advantage of the
ability.
I see that I didn't answer your question about performance.
When this optimization kicks in fully, I guess that the performance
should be about the same as for traffic with long flows (like the
netperf TCP_STREAM test, for example) in terms of packets per second.
Thanks. This is encouraging. The only question is why isn't the
optimization kicking in?
I repeated the test, and under a load of 10K DNS requests/responses per
second ovs-vswitchd is using 82% of a core.
I wasn't sure whether in-band control was on or off by default, so I
disabled it with the command below and restarted openvswitch, but the cpu
ovs-vsctl set bridge <bridge> other-config:disable-in-band=true
I did not set up the configuration, but as far as I can tell Netflow is
not turned on. The output of 'ovsdb-tool -show-log | grep -i netflow' is
empty.
There are no bonded interfaces. The 2 NICs used for DNS traffic are
associated with separate bridges.
We are not using a controller.
In your response you mentioned that for simple OpenFlow tables Open
vSwitch can set up a single kernel flow that covers many possible flows.
I think this is exactly what I need. Do I need to add a flow using
ovs-ofctl?
No. With the settings you describe, it should kick in automatically.
Here is an experiment that might help. Take one of the flows that
"ovs-dpctl dump-flows" prints, then feed that flow back into
"ovs-appctl ofproto/trace", and show us the results. (You might have
to spend a few minutes reading the ovs-vswitchd manpage to learn the
ofproto/trace syntax, if you don't already know it.)
Below is the ofproto/trace output for an inbound request to bridge brsvr2.
One more piece of information is that the packets are going through a
load balancer.
It looks very much to me like you are using an OVS kernel module that
is too old to support this feature. Are you using the kernel module
that came with OVS 1.11, or a kernel module that came with your kernel
(which kernel version), or some other module? ("dmesg|grep Open" can
help find out.)
McGarvey, Kevin
2014-01-22 21:40:11 UTC
Permalink
Post by Ben Pfaff
Post by McGarvey, Kevin
Post by Ben Pfaff
Post by McGarvey, Kevin
Post by Ben Pfaff
Post by McGarvey, Kevin
Post by McGarvey, Kevin
Post by Ben Pfaff
I'd expect a dramatic drop in CPU consumption in that case.
There
Post by Ben Pfaff
Post by McGarvey, Kevin
Post by Ben Pfaff
Post by McGarvey, Kevin
are
Post by McGarvey, Kevin
Post by Ben Pfaff
a few special cases where the upgrade wouldn't help. One is if
in-band control is in use, another is if NetFlow is turned on,
a
Post by Ben Pfaff
Post by McGarvey, Kevin
Post by Ben Pfaff
Post by McGarvey, Kevin
third
Post by McGarvey, Kevin
Post by Ben Pfaff
is if LACP bonds with L4 port based hashing are turned on, and
there
Post by Ben Pfaff
Post by McGarvey, Kevin
Post by McGarvey, Kevin
Post by Ben Pfaff
are probably a few others that don't come to mind immediately.
I plan to rerun the test to rule out some mistake on my part.
Could you provide more information about the nature of the
change
Post by Ben Pfaff
Post by McGarvey, Kevin
Post by Ben Pfaff
Post by McGarvey, Kevin
made in
Post by McGarvey, Kevin
1.11 that improves performance for this type of traffic? Is the
kernel
Post by McGarvey, Kevin
module able to forward UDP DNS packets without sending them to
userspace,
Post by McGarvey, Kevin
or was it an optimization of the userspace processing? What
roughly
Post by Ben Pfaff
Post by McGarvey, Kevin
is
Post by McGarvey, Kevin
the level of performance I should see?
In 1.11 and later, for simple OpenFlow tables (I don't think you
mentioned whether you are using a controller or which one), Open
vSwitch can set up only a single kernel flow that covers many
possible
Post by Ben Pfaff
Post by McGarvey, Kevin
flows, for example all possible UDP destination ports, rather than
setting up an individual kernel flow for each UDP packet. When
that
Post by Ben Pfaff
Post by McGarvey, Kevin
Post by Ben Pfaff
Post by McGarvey, Kevin
works, it eliminates most of the kernel/userspace traffic,
improving
Post by Ben Pfaff
Post by McGarvey, Kevin
Post by Ben Pfaff
Post by McGarvey, Kevin
performance. Version 2.0 is better at analyzing OpenFlow flow
tables
Post by Ben Pfaff
Post by McGarvey, Kevin
Post by Ben Pfaff
Post by McGarvey, Kevin
to see when this is possible, so it can better take advantage of
the
Post by Ben Pfaff
Post by McGarvey, Kevin
Post by Ben Pfaff
Post by McGarvey, Kevin
ability.
I see that I didn't answer your question about performance.
When this optimization kicks in fully, I guess that the performance
should be about the same as for traffic with long flows (like the
netperf TCP_STREAM test, for example) in terms of packets per
second.
Post by Ben Pfaff
Post by McGarvey, Kevin
Thanks. This is encouraging. The only question is why isn't the
optimization kicking in?
I repeated the test, and under a load of 10K DNS requests/responses
per
Post by Ben Pfaff
Post by McGarvey, Kevin
second ovs-vswitchd is using 82% of a core.
I wasn't sure whether in-band control was on or off by default, so I
disabled it with the command below and restarted openvswitch, but the cpu
ovs-vsctl set bridge <bridge> other-config:disable-in-band=true
I did not set up the configuration, but as far as I can tell Netflow
is
Post by Ben Pfaff
Post by McGarvey, Kevin
not turned on. The output of 'ovsdb-tool -show-log | grep -i
netflow'
Post by Ben Pfaff
Post by McGarvey, Kevin
is
empty.
There are no bonded interfaces. The 2 NICs used for DNS traffic are
associated with separate bridges.
We are not using a controller.
In your response you mentioned that for simple OpenFlow tables Open
vSwitch can set up a single kernel flow that covers many possible
flows.
Post by Ben Pfaff
Post by McGarvey, Kevin
I think this is exactly what I need. Do I need to add a flow using
ovs-ofctl?
No. With the settings you describe, it should kick in automatically.
Here is an experiment that might help. Take one of the flows that
"ovs-dpctl dump-flows" prints, then feed that flow back into
"ovs-appctl ofproto/trace", and show us the results. (You might have
to spend a few minutes reading the ovs-vswitchd manpage to learn the
ofproto/trace syntax, if you don't already know it.)
Below is the ofproto/trace output for an inbound request to bridge brsvr2.
One more piece of information is that the packets are going through a
load balancer.
It looks very much to me like you are using an OVS kernel module that
is too old to support this feature. Are you using the kernel module
that came with OVS 1.11, or a kernel module that came with your kernel
(which kernel version), or some other module? ("dmesg|grep Open" can
help find out.)
Here's the dmesg output:

dmesg|grep Open
ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
openvswitch: Open vSwitch switching datapath

The dmesg output didn't seem very informative, so I ran modinfo:

modinfo openvswitch
filename:
/lib/modules/2.6.32-358.123.4.openstack.el6.x86_64/kernel/net/openvswitch/o
penvswitch.ko
license: GPL
description: Open vSwitch switching datapath
srcversion: 19E48B3ED642482269914B5
depends: vxlan
vermagic: 2.6.32-358.123.4.openstack.el6.x86_64 SMP mod_unload
modversions


The ovs kernel module came with the kernel which is below. I upgraded to
this kernel on the recommendation of one of our engineers who works a lot
with OpenStack.

2.6.32-358.123.4.openstack.el6.x86_64 #1 SMP Wed Oct 30 13:52:57 EDT 2013
x86_64 x86_64 x86_64 GNU/Linux
Ben Pfaff
2014-01-22 21:42:24 UTC
Permalink
Post by Ben Pfaff
Post by Ben Pfaff
Post by McGarvey, Kevin
Post by Ben Pfaff
Post by McGarvey, Kevin
Post by Ben Pfaff
Post by McGarvey, Kevin
Post by McGarvey, Kevin
Post by Ben Pfaff
I'd expect a dramatic drop in CPU consumption in that case.
There
Post by Ben Pfaff
Post by McGarvey, Kevin
Post by Ben Pfaff
Post by McGarvey, Kevin
are
Post by McGarvey, Kevin
Post by Ben Pfaff
a few special cases where the upgrade wouldn't help. One is if
in-band control is in use, another is if NetFlow is turned on,
a
Post by Ben Pfaff
Post by McGarvey, Kevin
Post by Ben Pfaff
Post by McGarvey, Kevin
third
Post by McGarvey, Kevin
Post by Ben Pfaff
is if LACP bonds with L4 port based hashing are turned on, and
there
Post by Ben Pfaff
Post by McGarvey, Kevin
Post by McGarvey, Kevin
Post by Ben Pfaff
are probably a few others that don't come to mind immediately.
I plan to rerun the test to rule out some mistake on my part.
Could you provide more information about the nature of the
change
Post by Ben Pfaff
Post by McGarvey, Kevin
Post by Ben Pfaff
Post by McGarvey, Kevin
made in
Post by McGarvey, Kevin
1.11 that improves performance for this type of traffic? Is the
kernel
Post by McGarvey, Kevin
module able to forward UDP DNS packets without sending them to
userspace,
Post by McGarvey, Kevin
or was it an optimization of the userspace processing? What
roughly
Post by Ben Pfaff
Post by McGarvey, Kevin
is
Post by McGarvey, Kevin
the level of performance I should see?
In 1.11 and later, for simple OpenFlow tables (I don't think you
mentioned whether you are using a controller or which one), Open
vSwitch can set up only a single kernel flow that covers many
possible
Post by Ben Pfaff
Post by McGarvey, Kevin
flows, for example all possible UDP destination ports, rather than
setting up an individual kernel flow for each UDP packet. When
that
Post by Ben Pfaff
Post by McGarvey, Kevin
Post by Ben Pfaff
Post by McGarvey, Kevin
works, it eliminates most of the kernel/userspace traffic,
improving
Post by Ben Pfaff
Post by McGarvey, Kevin
Post by Ben Pfaff
Post by McGarvey, Kevin
performance. Version 2.0 is better at analyzing OpenFlow flow
tables
Post by Ben Pfaff
Post by McGarvey, Kevin
Post by Ben Pfaff
Post by McGarvey, Kevin
to see when this is possible, so it can better take advantage of
the
Post by Ben Pfaff
Post by McGarvey, Kevin
Post by Ben Pfaff
Post by McGarvey, Kevin
ability.
I see that I didn't answer your question about performance.
When this optimization kicks in fully, I guess that the performance
should be about the same as for traffic with long flows (like the
netperf TCP_STREAM test, for example) in terms of packets per
second.
Post by Ben Pfaff
Post by McGarvey, Kevin
Thanks. This is encouraging. The only question is why isn't the
optimization kicking in?
I repeated the test, and under a load of 10K DNS requests/responses
per
Post by Ben Pfaff
Post by McGarvey, Kevin
second ovs-vswitchd is using 82% of a core.
I wasn't sure whether in-band control was on or off by default, so I
disabled it with the command below and restarted openvswitch, but the cpu
ovs-vsctl set bridge <bridge> other-config:disable-in-band=true
I did not set up the configuration, but as far as I can tell Netflow
is
Post by Ben Pfaff
Post by McGarvey, Kevin
not turned on. The output of 'ovsdb-tool -show-log | grep -i
netflow'
Post by Ben Pfaff
Post by McGarvey, Kevin
is
empty.
There are no bonded interfaces. The 2 NICs used for DNS traffic are
associated with separate bridges.
We are not using a controller.
In your response you mentioned that for simple OpenFlow tables Open
vSwitch can set up a single kernel flow that covers many possible
flows.
Post by Ben Pfaff
Post by McGarvey, Kevin
I think this is exactly what I need. Do I need to add a flow using
ovs-ofctl?
No. With the settings you describe, it should kick in automatically.
Here is an experiment that might help. Take one of the flows that
"ovs-dpctl dump-flows" prints, then feed that flow back into
"ovs-appctl ofproto/trace", and show us the results. (You might have
to spend a few minutes reading the ovs-vswitchd manpage to learn the
ofproto/trace syntax, if you don't already know it.)
Below is the ofproto/trace output for an inbound request to bridge brsvr2.
One more piece of information is that the packets are going through a
load balancer.
It looks very much to me like you are using an OVS kernel module that
is too old to support this feature. Are you using the kernel module
that came with OVS 1.11, or a kernel module that came with your kernel
(which kernel version), or some other module? ("dmesg|grep Open" can
help find out.)
dmesg|grep Open
ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
openvswitch: Open vSwitch switching datapath
modinfo openvswitch
/lib/modules/2.6.32-358.123.4.openstack.el6.x86_64/kernel/net/openvswitch/o
penvswitch.ko
license: GPL
description: Open vSwitch switching datapath
srcversion: 19E48B3ED642482269914B5
depends: vxlan
vermagic: 2.6.32-358.123.4.openstack.el6.x86_64 SMP mod_unload
modversions
The ovs kernel module came with the kernel which is below. I upgraded to
this kernel on the recommendation of one of our engineers who works a lot
with OpenStack.
2.6.32-358.123.4.openstack.el6.x86_64 #1 SMP Wed Oct 30 13:52:57 EDT 2013
x86_64 x86_64 x86_64 GNU/Linux
My guess is that that kernel module doesn't include the "megaflow"
support required for the improved performance here. I recommend
trying the module that comes with the OVS version that you are using.
McGarvey, Kevin
2014-01-22 22:54:25 UTC
Permalink
Post by Ben Pfaff
Post by McGarvey, Kevin
Post by Ben Pfaff
Post by McGarvey, Kevin
Post by Ben Pfaff
Post by McGarvey, Kevin
Post by Ben Pfaff
On Wed, Jan 22, 2014 at 05:35:40PM +0000, McGarvey, Kevin
Post by McGarvey, Kevin
Post by Ben Pfaff
I'd expect a dramatic drop in CPU consumption in that case.
There
Post by Ben Pfaff
Post by McGarvey, Kevin
Post by Ben Pfaff
are
Post by McGarvey, Kevin
Post by Ben Pfaff
a few special cases where the upgrade wouldn't help. One
is if
Post by Ben Pfaff
Post by McGarvey, Kevin
Post by Ben Pfaff
Post by McGarvey, Kevin
Post by Ben Pfaff
Post by McGarvey, Kevin
Post by Ben Pfaff
in-band control is in use, another is if NetFlow is turned
on,
Post by Ben Pfaff
Post by McGarvey, Kevin
a
Post by Ben Pfaff
Post by McGarvey, Kevin
Post by Ben Pfaff
third
Post by McGarvey, Kevin
Post by Ben Pfaff
is if LACP bonds with L4 port based hashing are turned on,
and
Post by Ben Pfaff
Post by McGarvey, Kevin
Post by Ben Pfaff
Post by McGarvey, Kevin
there
Post by Ben Pfaff
Post by McGarvey, Kevin
Post by Ben Pfaff
are probably a few others that don't come to mind
immediately.
Post by Ben Pfaff
Post by McGarvey, Kevin
Post by Ben Pfaff
Post by McGarvey, Kevin
Post by Ben Pfaff
Post by McGarvey, Kevin
I plan to rerun the test to rule out some mistake on my part.
Could you provide more information about the nature of the
change
Post by Ben Pfaff
Post by McGarvey, Kevin
Post by Ben Pfaff
made in
Post by McGarvey, Kevin
1.11 that improves performance for this type of traffic? Is
the
Post by Ben Pfaff
Post by McGarvey, Kevin
Post by Ben Pfaff
Post by McGarvey, Kevin
Post by Ben Pfaff
kernel
Post by McGarvey, Kevin
module able to forward UDP DNS packets without sending them
to
Post by Ben Pfaff
Post by McGarvey, Kevin
Post by Ben Pfaff
Post by McGarvey, Kevin
Post by Ben Pfaff
userspace,
Post by McGarvey, Kevin
or was it an optimization of the userspace processing? What
roughly
Post by Ben Pfaff
is
Post by McGarvey, Kevin
the level of performance I should see?
In 1.11 and later, for simple OpenFlow tables (I don't think
you
Post by Ben Pfaff
Post by McGarvey, Kevin
Post by Ben Pfaff
Post by McGarvey, Kevin
Post by Ben Pfaff
mentioned whether you are using a controller or which one),
Open
Post by Ben Pfaff
Post by McGarvey, Kevin
Post by Ben Pfaff
Post by McGarvey, Kevin
Post by Ben Pfaff
vSwitch can set up only a single kernel flow that covers many
possible
Post by Ben Pfaff
flows, for example all possible UDP destination ports, rather
than
Post by Ben Pfaff
Post by McGarvey, Kevin
Post by Ben Pfaff
Post by McGarvey, Kevin
Post by Ben Pfaff
setting up an individual kernel flow for each UDP packet. When
that
Post by Ben Pfaff
Post by McGarvey, Kevin
Post by Ben Pfaff
works, it eliminates most of the kernel/userspace traffic,
improving
Post by Ben Pfaff
Post by McGarvey, Kevin
Post by Ben Pfaff
performance. Version 2.0 is better at analyzing OpenFlow flow
tables
Post by Ben Pfaff
Post by McGarvey, Kevin
Post by Ben Pfaff
to see when this is possible, so it can better take advantage
of
Post by Ben Pfaff
Post by McGarvey, Kevin
the
Post by Ben Pfaff
Post by McGarvey, Kevin
Post by Ben Pfaff
ability.
I see that I didn't answer your question about performance.
When this optimization kicks in fully, I guess that the
performance
Post by Ben Pfaff
Post by McGarvey, Kevin
Post by Ben Pfaff
Post by McGarvey, Kevin
Post by Ben Pfaff
should be about the same as for traffic with long flows (like the
netperf TCP_STREAM test, for example) in terms of packets per
second.
Post by Ben Pfaff
Post by McGarvey, Kevin
Thanks. This is encouraging. The only question is why isn't the
optimization kicking in?
I repeated the test, and under a load of 10K DNS
requests/responses
Post by Ben Pfaff
Post by McGarvey, Kevin
per
Post by Ben Pfaff
Post by McGarvey, Kevin
second ovs-vswitchd is using 82% of a core.
I wasn't sure whether in-band control was on or off by default,
so I
Post by Ben Pfaff
Post by McGarvey, Kevin
Post by Ben Pfaff
Post by McGarvey, Kevin
disabled it with the command below and restarted openvswitch, but
the
Post by Ben Pfaff
Post by McGarvey, Kevin
Post by Ben Pfaff
Post by McGarvey, Kevin
cpu
ovs-vsctl set bridge <bridge> other-config:disable-in-band=true
I did not set up the configuration, but as far as I can tell
Netflow
Post by Ben Pfaff
Post by McGarvey, Kevin
is
Post by Ben Pfaff
Post by McGarvey, Kevin
not turned on. The output of 'ovsdb-tool -show-log | grep -i
netflow'
Post by Ben Pfaff
Post by McGarvey, Kevin
is
empty.
There are no bonded interfaces. The 2 NICs used for DNS traffic
are
Post by Ben Pfaff
Post by McGarvey, Kevin
Post by Ben Pfaff
Post by McGarvey, Kevin
associated with separate bridges.
We are not using a controller.
In your response you mentioned that for simple OpenFlow tables
Open
Post by Ben Pfaff
Post by McGarvey, Kevin
Post by Ben Pfaff
Post by McGarvey, Kevin
vSwitch can set up a single kernel flow that covers many possible
flows.
Post by Ben Pfaff
Post by McGarvey, Kevin
I think this is exactly what I need. Do I need to add a flow
using
Post by Ben Pfaff
Post by McGarvey, Kevin
Post by Ben Pfaff
Post by McGarvey, Kevin
ovs-ofctl?
No. With the settings you describe, it should kick in
automatically.
Post by Ben Pfaff
Post by McGarvey, Kevin
Post by Ben Pfaff
Here is an experiment that might help. Take one of the flows that
"ovs-dpctl dump-flows" prints, then feed that flow back into
"ovs-appctl ofproto/trace", and show us the results. (You might
have
Post by Ben Pfaff
Post by McGarvey, Kevin
Post by Ben Pfaff
to spend a few minutes reading the ovs-vswitchd manpage to learn the
ofproto/trace syntax, if you don't already know it.)
Below is the ofproto/trace output for an inbound request to bridge brsvr2.
One more piece of information is that the packets are going through
a
Post by Ben Pfaff
Post by McGarvey, Kevin
load balancer.
It looks very much to me like you are using an OVS kernel module that
is too old to support this feature. Are you using the kernel module
that came with OVS 1.11, or a kernel module that came with your kernel
(which kernel version), or some other module? ("dmesg|grep Open" can
help find out.)
dmesg|grep Open
ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
openvswitch: Open vSwitch switching datapath
modinfo openvswitch
/lib/modules/2.6.32-358.123.4.openstack.el6.x86_64/kernel/net/openvswitch
/o
penvswitch.ko
license: GPL
description: Open vSwitch switching datapath
srcversion: 19E48B3ED642482269914B5
depends: vxlan
vermagic: 2.6.32-358.123.4.openstack.el6.x86_64 SMP mod_unload
modversions
The ovs kernel module came with the kernel which is below. I upgraded to
this kernel on the recommendation of one of our engineers who works a lot
with OpenStack.
2.6.32-358.123.4.openstack.el6.x86_64 #1 SMP Wed Oct 30 13:52:57 EDT 2013
x86_64 x86_64 x86_64 GNU/Linux
My guess is that that kernel module doesn't include the "megaflow"
support required for the improved performance here. I recommend
trying the module that comes with the OVS version that you are using.
Where can I find that? I installed OVS from an rpm. I ran rpm -qlp on
the package and I didn't see the kernel module. Do I need to build it?
Ben Pfaff
2014-01-22 23:15:08 UTC
Permalink
Post by McGarvey, Kevin
Post by Ben Pfaff
Post by McGarvey, Kevin
Post by Ben Pfaff
Post by McGarvey, Kevin
Post by Ben Pfaff
Post by McGarvey, Kevin
Post by Ben Pfaff
On Wed, Jan 22, 2014 at 05:35:40PM +0000, McGarvey, Kevin
Post by McGarvey, Kevin
Post by Ben Pfaff
I'd expect a dramatic drop in CPU consumption in that case.
There
Post by Ben Pfaff
Post by McGarvey, Kevin
Post by Ben Pfaff
are
Post by McGarvey, Kevin
Post by Ben Pfaff
a few special cases where the upgrade wouldn't help. One
is if
Post by Ben Pfaff
Post by McGarvey, Kevin
Post by Ben Pfaff
Post by McGarvey, Kevin
Post by Ben Pfaff
Post by McGarvey, Kevin
Post by Ben Pfaff
in-band control is in use, another is if NetFlow is turned
on,
Post by Ben Pfaff
Post by McGarvey, Kevin
a
Post by Ben Pfaff
Post by McGarvey, Kevin
Post by Ben Pfaff
third
Post by McGarvey, Kevin
Post by Ben Pfaff
is if LACP bonds with L4 port based hashing are turned on,
and
Post by Ben Pfaff
Post by McGarvey, Kevin
Post by Ben Pfaff
Post by McGarvey, Kevin
there
Post by Ben Pfaff
Post by McGarvey, Kevin
Post by Ben Pfaff
are probably a few others that don't come to mind
immediately.
Post by Ben Pfaff
Post by McGarvey, Kevin
Post by Ben Pfaff
Post by McGarvey, Kevin
Post by Ben Pfaff
Post by McGarvey, Kevin
I plan to rerun the test to rule out some mistake on my part.
Could you provide more information about the nature of the
change
Post by Ben Pfaff
Post by McGarvey, Kevin
Post by Ben Pfaff
made in
Post by McGarvey, Kevin
1.11 that improves performance for this type of traffic? Is
the
Post by Ben Pfaff
Post by McGarvey, Kevin
Post by Ben Pfaff
Post by McGarvey, Kevin
Post by Ben Pfaff
kernel
Post by McGarvey, Kevin
module able to forward UDP DNS packets without sending them
to
Post by Ben Pfaff
Post by McGarvey, Kevin
Post by Ben Pfaff
Post by McGarvey, Kevin
Post by Ben Pfaff
userspace,
Post by McGarvey, Kevin
or was it an optimization of the userspace processing? What
roughly
Post by Ben Pfaff
is
Post by McGarvey, Kevin
the level of performance I should see?
In 1.11 and later, for simple OpenFlow tables (I don't think
you
Post by Ben Pfaff
Post by McGarvey, Kevin
Post by Ben Pfaff
Post by McGarvey, Kevin
Post by Ben Pfaff
mentioned whether you are using a controller or which one),
Open
Post by Ben Pfaff
Post by McGarvey, Kevin
Post by Ben Pfaff
Post by McGarvey, Kevin
Post by Ben Pfaff
vSwitch can set up only a single kernel flow that covers many
possible
Post by Ben Pfaff
flows, for example all possible UDP destination ports, rather
than
Post by Ben Pfaff
Post by McGarvey, Kevin
Post by Ben Pfaff
Post by McGarvey, Kevin
Post by Ben Pfaff
setting up an individual kernel flow for each UDP packet. When
that
Post by Ben Pfaff
Post by McGarvey, Kevin
Post by Ben Pfaff
works, it eliminates most of the kernel/userspace traffic,
improving
Post by Ben Pfaff
Post by McGarvey, Kevin
Post by Ben Pfaff
performance. Version 2.0 is better at analyzing OpenFlow flow
tables
Post by Ben Pfaff
Post by McGarvey, Kevin
Post by Ben Pfaff
to see when this is possible, so it can better take advantage
of
Post by Ben Pfaff
Post by McGarvey, Kevin
the
Post by Ben Pfaff
Post by McGarvey, Kevin
Post by Ben Pfaff
ability.
I see that I didn't answer your question about performance.
When this optimization kicks in fully, I guess that the
performance
Post by Ben Pfaff
Post by McGarvey, Kevin
Post by Ben Pfaff
Post by McGarvey, Kevin
Post by Ben Pfaff
should be about the same as for traffic with long flows (like the
netperf TCP_STREAM test, for example) in terms of packets per
second.
Post by Ben Pfaff
Post by McGarvey, Kevin
Thanks. This is encouraging. The only question is why isn't the
optimization kicking in?
I repeated the test, and under a load of 10K DNS
requests/responses
Post by Ben Pfaff
Post by McGarvey, Kevin
per
Post by Ben Pfaff
Post by McGarvey, Kevin
second ovs-vswitchd is using 82% of a core.
I wasn't sure whether in-band control was on or off by default,
so I
Post by Ben Pfaff
Post by McGarvey, Kevin
Post by Ben Pfaff
Post by McGarvey, Kevin
disabled it with the command below and restarted openvswitch, but
the
Post by Ben Pfaff
Post by McGarvey, Kevin
Post by Ben Pfaff
Post by McGarvey, Kevin
cpu
ovs-vsctl set bridge <bridge> other-config:disable-in-band=true
I did not set up the configuration, but as far as I can tell
Netflow
Post by Ben Pfaff
Post by McGarvey, Kevin
is
Post by Ben Pfaff
Post by McGarvey, Kevin
not turned on. The output of 'ovsdb-tool -show-log | grep -i
netflow'
Post by Ben Pfaff
Post by McGarvey, Kevin
is
empty.
There are no bonded interfaces. The 2 NICs used for DNS traffic
are
Post by Ben Pfaff
Post by McGarvey, Kevin
Post by Ben Pfaff
Post by McGarvey, Kevin
associated with separate bridges.
We are not using a controller.
In your response you mentioned that for simple OpenFlow tables
Open
Post by Ben Pfaff
Post by McGarvey, Kevin
Post by Ben Pfaff
Post by McGarvey, Kevin
vSwitch can set up a single kernel flow that covers many possible
flows.
Post by Ben Pfaff
Post by McGarvey, Kevin
I think this is exactly what I need. Do I need to add a flow
using
Post by Ben Pfaff
Post by McGarvey, Kevin
Post by Ben Pfaff
Post by McGarvey, Kevin
ovs-ofctl?
No. With the settings you describe, it should kick in
automatically.
Post by Ben Pfaff
Post by McGarvey, Kevin
Post by Ben Pfaff
Here is an experiment that might help. Take one of the flows that
"ovs-dpctl dump-flows" prints, then feed that flow back into
"ovs-appctl ofproto/trace", and show us the results. (You might
have
Post by Ben Pfaff
Post by McGarvey, Kevin
Post by Ben Pfaff
to spend a few minutes reading the ovs-vswitchd manpage to learn the
ofproto/trace syntax, if you don't already know it.)
Below is the ofproto/trace output for an inbound request to bridge brsvr2.
One more piece of information is that the packets are going through
a
Post by Ben Pfaff
Post by McGarvey, Kevin
load balancer.
It looks very much to me like you are using an OVS kernel module that
is too old to support this feature. Are you using the kernel module
that came with OVS 1.11, or a kernel module that came with your kernel
(which kernel version), or some other module? ("dmesg|grep Open" can
help find out.)
dmesg|grep Open
ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
openvswitch: Open vSwitch switching datapath
modinfo openvswitch
/lib/modules/2.6.32-358.123.4.openstack.el6.x86_64/kernel/net/openvswitch
/o
penvswitch.ko
license: GPL
description: Open vSwitch switching datapath
srcversion: 19E48B3ED642482269914B5
depends: vxlan
vermagic: 2.6.32-358.123.4.openstack.el6.x86_64 SMP mod_unload
modversions
The ovs kernel module came with the kernel which is below. I upgraded to
this kernel on the recommendation of one of our engineers who works a lot
with OpenStack.
2.6.32-358.123.4.openstack.el6.x86_64 #1 SMP Wed Oct 30 13:52:57 EDT 2013
x86_64 x86_64 x86_64 GNU/Linux
My guess is that that kernel module doesn't include the "megaflow"
support required for the improved performance here. I recommend
trying the module that comes with the OVS version that you are using.
Where can I find that? I installed OVS from an rpm. I ran rpm -qlp on
the package and I didn't see the kernel module. Do I need to build it?
I'm not an expert on the RPM packaging, so I don't know. I guess that
someone else will have to jump in to help at this point.
Gurucharan Shetty
2014-01-22 23:30:10 UTC
Permalink
Post by McGarvey, Kevin
Where can I find that? I installed OVS from an rpm. I ran rpm -qlp on
the package and I didn't see the kernel module. Do I need to build it?
Where did you get your RPMS from? Did you build it?

RHEL comes with a default OVS kernel module. I think you are using it.

Each OVS version comes with 2 rpm packages. One includes user-space
utilities and another one includes the kernel module. You can read
INSTALL.RHEL for more details.
Post by McGarvey, Kevin
_______________________________________________
discuss mailing list
discuss at openvswitch.org
http://openvswitch.org/mailman/listinfo/discuss
McGarvey, Kevin
2014-01-22 23:36:18 UTC
Permalink
Post by Gurucharan Shetty
Post by McGarvey, Kevin
Where can I find that? I installed OVS from an rpm. I ran rpm -qlp on
the package and I didn't see the kernel module. Do I need to build it?
Where did you get your RPMS from? Did you build it?
RHEL comes with a default OVS kernel module. I think you are using it.
Each OVS version comes with 2 rpm packages. One includes user-space
utilities and another one includes the kernel module. You can read
INSTALL.RHEL for more details.
I got them from an in-house clone of the RDO repo. I looked for a
kmod-openvswitch rpm but didn't see one. I assume that redhat packages
the kernel module with the kernel.
Post by Gurucharan Shetty
Post by McGarvey, Kevin
_______________________________________________
discuss mailing list
discuss at openvswitch.org
http://openvswitch.org/mailman/listinfo/discuss
Gurucharan Shetty
2014-01-22 23:39:02 UTC
Permalink
Post by McGarvey, Kevin
Post by Gurucharan Shetty
Post by McGarvey, Kevin
Where can I find that? I installed OVS from an rpm. I ran rpm -qlp on
the package and I didn't see the kernel module. Do I need to build it?
Where did you get your RPMS from? Did you build it?
RHEL comes with a default OVS kernel module. I think you are using it.
Each OVS version comes with 2 rpm packages. One includes user-space
utilities and another one includes the kernel module. You can read
INSTALL.RHEL for more details.
I got them from an in-house clone of the RDO repo. I looked for a
kmod-openvswitch rpm but didn't see one. I assume that redhat packages
the kernel module with the kernel.
I think so. So that would mean, you will have to build the kernel
module package manually for the version you are interested
in(INSTALL.RHEL gives the details.)
Post by McGarvey, Kevin
Post by Gurucharan Shetty
Post by McGarvey, Kevin
_______________________________________________
discuss mailing list
discuss at openvswitch.org
http://openvswitch.org/mailman/listinfo/discuss
Flavio Leitner
2014-01-24 13:36:32 UTC
Permalink
Post by Ben Pfaff
Post by Ben Pfaff
Post by McGarvey, Kevin
Post by Ben Pfaff
Post by McGarvey, Kevin
Post by Ben Pfaff
Post by McGarvey, Kevin
Post by McGarvey, Kevin
Post by Ben Pfaff
I'd expect a dramatic drop in CPU consumption in that case.
There
Post by Ben Pfaff
Post by McGarvey, Kevin
Post by Ben Pfaff
Post by McGarvey, Kevin
are
Post by McGarvey, Kevin
Post by Ben Pfaff
a few special cases where the upgrade wouldn't help. One is if
in-band control is in use, another is if NetFlow is turned on,
a
Post by Ben Pfaff
Post by McGarvey, Kevin
Post by Ben Pfaff
Post by McGarvey, Kevin
third
Post by McGarvey, Kevin
Post by Ben Pfaff
is if LACP bonds with L4 port based hashing are turned on, and
there
Post by Ben Pfaff
Post by McGarvey, Kevin
Post by McGarvey, Kevin
Post by Ben Pfaff
are probably a few others that don't come to mind immediately.
I plan to rerun the test to rule out some mistake on my part.
Could you provide more information about the nature of the
change
Post by Ben Pfaff
Post by McGarvey, Kevin
Post by Ben Pfaff
Post by McGarvey, Kevin
made in
Post by McGarvey, Kevin
1.11 that improves performance for this type of traffic? Is the
kernel
Post by McGarvey, Kevin
module able to forward UDP DNS packets without sending them to
userspace,
Post by McGarvey, Kevin
or was it an optimization of the userspace processing? What
roughly
Post by Ben Pfaff
Post by McGarvey, Kevin
is
Post by McGarvey, Kevin
the level of performance I should see?
In 1.11 and later, for simple OpenFlow tables (I don't think you
mentioned whether you are using a controller or which one), Open
vSwitch can set up only a single kernel flow that covers many
possible
Post by Ben Pfaff
Post by McGarvey, Kevin
flows, for example all possible UDP destination ports, rather than
setting up an individual kernel flow for each UDP packet. When
that
Post by Ben Pfaff
Post by McGarvey, Kevin
Post by Ben Pfaff
Post by McGarvey, Kevin
works, it eliminates most of the kernel/userspace traffic,
improving
Post by Ben Pfaff
Post by McGarvey, Kevin
Post by Ben Pfaff
Post by McGarvey, Kevin
performance. Version 2.0 is better at analyzing OpenFlow flow
tables
Post by Ben Pfaff
Post by McGarvey, Kevin
Post by Ben Pfaff
Post by McGarvey, Kevin
to see when this is possible, so it can better take advantage of
the
Post by Ben Pfaff
Post by McGarvey, Kevin
Post by Ben Pfaff
Post by McGarvey, Kevin
ability.
I see that I didn't answer your question about performance.
When this optimization kicks in fully, I guess that the performance
should be about the same as for traffic with long flows (like the
netperf TCP_STREAM test, for example) in terms of packets per
second.
Post by Ben Pfaff
Post by McGarvey, Kevin
Thanks. This is encouraging. The only question is why isn't the
optimization kicking in?
I repeated the test, and under a load of 10K DNS requests/responses
per
Post by Ben Pfaff
Post by McGarvey, Kevin
second ovs-vswitchd is using 82% of a core.
I wasn't sure whether in-band control was on or off by default, so I
disabled it with the command below and restarted openvswitch, but the cpu
ovs-vsctl set bridge <bridge> other-config:disable-in-band=true
I did not set up the configuration, but as far as I can tell Netflow
is
Post by Ben Pfaff
Post by McGarvey, Kevin
not turned on. The output of 'ovsdb-tool -show-log | grep -i
netflow'
Post by Ben Pfaff
Post by McGarvey, Kevin
is
empty.
There are no bonded interfaces. The 2 NICs used for DNS traffic are
associated with separate bridges.
We are not using a controller.
In your response you mentioned that for simple OpenFlow tables Open
vSwitch can set up a single kernel flow that covers many possible
flows.
Post by Ben Pfaff
Post by McGarvey, Kevin
I think this is exactly what I need. Do I need to add a flow using
ovs-ofctl?
No. With the settings you describe, it should kick in automatically.
Here is an experiment that might help. Take one of the flows that
"ovs-dpctl dump-flows" prints, then feed that flow back into
"ovs-appctl ofproto/trace", and show us the results. (You might have
to spend a few minutes reading the ovs-vswitchd manpage to learn the
ofproto/trace syntax, if you don't already know it.)
Below is the ofproto/trace output for an inbound request to bridge brsvr2.
One more piece of information is that the packets are going through a
load balancer.
It looks very much to me like you are using an OVS kernel module that
is too old to support this feature. Are you using the kernel module
that came with OVS 1.11, or a kernel module that came with your kernel
(which kernel version), or some other module? ("dmesg|grep Open" can
help find out.)
dmesg|grep Open
ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
openvswitch: Open vSwitch switching datapath
modinfo openvswitch
/lib/modules/2.6.32-358.123.4.openstack.el6.x86_64/kernel/net/openvswitch/o
penvswitch.ko
license: GPL
description: Open vSwitch switching datapath
srcversion: 19E48B3ED642482269914B5
depends: vxlan
vermagic: 2.6.32-358.123.4.openstack.el6.x86_64 SMP mod_unload
modversions
The ovs kernel module came with the kernel which is below. I upgraded to
this kernel on the recommendation of one of our engineers who works a lot
with OpenStack.
2.6.32-358.123.4.openstack.el6.x86_64 #1 SMP Wed Oct 30 13:52:57 EDT 2013
x86_64 x86_64 x86_64 GNU/Linux
Unfortunately this kernel doesn't include megaflow support.
fbl
McGarvey, Kevin
2014-01-24 18:41:33 UTC
Permalink
Post by Flavio Leitner
Post by McGarvey, Kevin
Post by Ben Pfaff
Post by McGarvey, Kevin
Post by Ben Pfaff
Post by McGarvey, Kevin
Post by Ben Pfaff
On Wed, Jan 22, 2014 at 05:35:40PM +0000, McGarvey, Kevin
Post by McGarvey, Kevin
Post by Ben Pfaff
I'd expect a dramatic drop in CPU consumption in that case.
There
Post by Ben Pfaff
Post by McGarvey, Kevin
Post by Ben Pfaff
are
Post by McGarvey, Kevin
Post by Ben Pfaff
a few special cases where the upgrade wouldn't help. One
is if
Post by Ben Pfaff
Post by McGarvey, Kevin
Post by Ben Pfaff
Post by McGarvey, Kevin
Post by Ben Pfaff
Post by McGarvey, Kevin
Post by Ben Pfaff
in-band control is in use, another is if NetFlow is turned
on,
Post by Ben Pfaff
Post by McGarvey, Kevin
a
Post by Ben Pfaff
Post by McGarvey, Kevin
Post by Ben Pfaff
third
Post by McGarvey, Kevin
Post by Ben Pfaff
is if LACP bonds with L4 port based hashing are turned on,
and
Post by Ben Pfaff
Post by McGarvey, Kevin
Post by Ben Pfaff
Post by McGarvey, Kevin
there
Post by Ben Pfaff
Post by McGarvey, Kevin
Post by Ben Pfaff
are probably a few others that don't come to mind
immediately.
Post by Ben Pfaff
Post by McGarvey, Kevin
Post by Ben Pfaff
Post by McGarvey, Kevin
Post by Ben Pfaff
Post by McGarvey, Kevin
I plan to rerun the test to rule out some mistake on my part.
Could you provide more information about the nature of the
change
Post by Ben Pfaff
Post by McGarvey, Kevin
Post by Ben Pfaff
made in
Post by McGarvey, Kevin
1.11 that improves performance for this type of traffic? Is
the
Post by Ben Pfaff
Post by McGarvey, Kevin
Post by Ben Pfaff
Post by McGarvey, Kevin
Post by Ben Pfaff
kernel
Post by McGarvey, Kevin
module able to forward UDP DNS packets without sending them
to
Post by Ben Pfaff
Post by McGarvey, Kevin
Post by Ben Pfaff
Post by McGarvey, Kevin
Post by Ben Pfaff
userspace,
Post by McGarvey, Kevin
or was it an optimization of the userspace processing? What
roughly
Post by Ben Pfaff
is
Post by McGarvey, Kevin
the level of performance I should see?
In 1.11 and later, for simple OpenFlow tables (I don't think
you
Post by Ben Pfaff
Post by McGarvey, Kevin
Post by Ben Pfaff
Post by McGarvey, Kevin
Post by Ben Pfaff
mentioned whether you are using a controller or which one),
Open
Post by Ben Pfaff
Post by McGarvey, Kevin
Post by Ben Pfaff
Post by McGarvey, Kevin
Post by Ben Pfaff
vSwitch can set up only a single kernel flow that covers many
possible
Post by Ben Pfaff
flows, for example all possible UDP destination ports, rather
than
Post by Ben Pfaff
Post by McGarvey, Kevin
Post by Ben Pfaff
Post by McGarvey, Kevin
Post by Ben Pfaff
setting up an individual kernel flow for each UDP packet. When
that
Post by Ben Pfaff
Post by McGarvey, Kevin
Post by Ben Pfaff
works, it eliminates most of the kernel/userspace traffic,
improving
Post by Ben Pfaff
Post by McGarvey, Kevin
Post by Ben Pfaff
performance. Version 2.0 is better at analyzing OpenFlow flow
tables
Post by Ben Pfaff
Post by McGarvey, Kevin
Post by Ben Pfaff
to see when this is possible, so it can better take advantage
of
Post by Ben Pfaff
Post by McGarvey, Kevin
the
Post by Ben Pfaff
Post by McGarvey, Kevin
Post by Ben Pfaff
ability.
I see that I didn't answer your question about performance.
When this optimization kicks in fully, I guess that the
performance
Post by Ben Pfaff
Post by McGarvey, Kevin
Post by Ben Pfaff
Post by McGarvey, Kevin
Post by Ben Pfaff
should be about the same as for traffic with long flows (like the
netperf TCP_STREAM test, for example) in terms of packets per
second.
Post by Ben Pfaff
Post by McGarvey, Kevin
Thanks. This is encouraging. The only question is why isn't the
optimization kicking in?
I repeated the test, and under a load of 10K DNS
requests/responses
Post by Ben Pfaff
Post by McGarvey, Kevin
per
Post by Ben Pfaff
Post by McGarvey, Kevin
second ovs-vswitchd is using 82% of a core.
I wasn't sure whether in-band control was on or off by default,
so I
Post by Ben Pfaff
Post by McGarvey, Kevin
Post by Ben Pfaff
Post by McGarvey, Kevin
disabled it with the command below and restarted openvswitch, but
the
Post by Ben Pfaff
Post by McGarvey, Kevin
Post by Ben Pfaff
Post by McGarvey, Kevin
cpu
ovs-vsctl set bridge <bridge> other-config:disable-in-band=true
I did not set up the configuration, but as far as I can tell
Netflow
Post by Ben Pfaff
Post by McGarvey, Kevin
is
Post by Ben Pfaff
Post by McGarvey, Kevin
not turned on. The output of 'ovsdb-tool -show-log | grep -i
netflow'
Post by Ben Pfaff
Post by McGarvey, Kevin
is
empty.
There are no bonded interfaces. The 2 NICs used for DNS traffic
are
Post by Ben Pfaff
Post by McGarvey, Kevin
Post by Ben Pfaff
Post by McGarvey, Kevin
associated with separate bridges.
We are not using a controller.
In your response you mentioned that for simple OpenFlow tables
Open
Post by Ben Pfaff
Post by McGarvey, Kevin
Post by Ben Pfaff
Post by McGarvey, Kevin
vSwitch can set up a single kernel flow that covers many possible
flows.
Post by Ben Pfaff
Post by McGarvey, Kevin
I think this is exactly what I need. Do I need to add a flow
using
Post by Ben Pfaff
Post by McGarvey, Kevin
Post by Ben Pfaff
Post by McGarvey, Kevin
ovs-ofctl?
No. With the settings you describe, it should kick in
automatically.
Post by Ben Pfaff
Post by McGarvey, Kevin
Post by Ben Pfaff
Here is an experiment that might help. Take one of the flows that
"ovs-dpctl dump-flows" prints, then feed that flow back into
"ovs-appctl ofproto/trace", and show us the results. (You might
have
Post by Ben Pfaff
Post by McGarvey, Kevin
Post by Ben Pfaff
to spend a few minutes reading the ovs-vswitchd manpage to learn the
ofproto/trace syntax, if you don't already know it.)
Below is the ofproto/trace output for an inbound request to bridge brsvr2.
One more piece of information is that the packets are going through
a
Post by Ben Pfaff
Post by McGarvey, Kevin
load balancer.
It looks very much to me like you are using an OVS kernel module that
is too old to support this feature. Are you using the kernel module
that came with OVS 1.11, or a kernel module that came with your kernel
(which kernel version), or some other module? ("dmesg|grep Open" can
help find out.)
dmesg|grep Open
ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
openvswitch: Open vSwitch switching datapath
modinfo openvswitch
/lib/modules/2.6.32-358.123.4.openstack.el6.x86_64/kernel/net/openvswitch
/o
penvswitch.ko
license: GPL
description: Open vSwitch switching datapath
srcversion: 19E48B3ED642482269914B5
depends: vxlan
vermagic: 2.6.32-358.123.4.openstack.el6.x86_64 SMP mod_unload
modversions
The ovs kernel module came with the kernel which is below. I upgraded to
this kernel on the recommendation of one of our engineers who works a lot
with OpenStack.
2.6.32-358.123.4.openstack.el6.x86_64 #1 SMP Wed Oct 30 13:52:57 EDT 2013
x86_64 x86_64 x86_64 GNU/Linux
Unfortunately this kernel doesn't include megaflow support.
fbl
I built 1.11 and installed the kernel module from the RPM and this
resolved the issue. The ovs-vswitchd process is now using very little CPU
and performance is in the range I was expecting.

Thanks for all your help!

Loading...