Skip to content

Nova/Neutron: Enable rootwrap daemon mode#26

Open
matelakat wants to merge 2 commits intosap-oc:stable/sap/3.0from
matelakat:rootwrap-fix
Open

Nova/Neutron: Enable rootwrap daemon mode#26
matelakat wants to merge 2 commits intosap-oc:stable/sap/3.0from
matelakat:rootwrap-fix

Conversation

@matelakat
Copy link

@matelakat matelakat commented Mar 13, 2017 •
edited by aspiers
Loading

Original commit message:

This provides a speedup (benchmarks that do a lot of neutron operations
go down from 24 minutes to 11 minutes, providing an almost twofold
improvement). The improvement is actually like eightfold on ARM.

(cherry picked from commit ef29f92)

Fixes #1

Sudoers File for Neutron

Specfile

install -D -m 440 %{SOURCE3} %{buildroot}%{_sysconfdir}/sudoers.d/openstack-neutron

Contents

The rootwrap-daemon line is new:

Defaults:neutron !syslog, !pam_session, !use_pty
neutron ALL = (root) NOPASSWD: /usr/bin/neutron-rootwrap-daemon /etc/neutron/rootwrap.conf
neutron ALL = (root) NOPASSWD: /usr/bin/neutron-rootwrap /etc/neutron/rootwrap.conf *

Sudoers File for Nova

Specfile

install -D -m 440 %{SOURCE5} %{buildroot}%{_sysconfdir}/sudoers.d/openstack-nova

Contents

The rootwrap-daemon line is new:

Defaults:nova !syslog, !pam_session, !use_pty
nova ALL = (root) NOPASSWD: /usr/bin/nova-rootwrap-daemon /etc/nova/rootwrap.conf
nova ALL = (root) NOPASSWD: /usr/bin/nova-rootwrap /etc/nova/rootwrap.conf *

@matelakat
Copy link
Author

matelakat commented Mar 13, 2017

I still need to verify this.

@rsalevsky rsalevsky changed the base branch from staging to stable/sap/3.0 March 13, 2017 13:03
@aspiers aspiers self-assigned this Mar 21, 2017
@aspiers
Copy link

aspiers commented Mar 21, 2017

I'm testing this now.

@aspiers aspiers self-requested a review March 21, 2017 21:24
aspiers
aspiers approved these changes Mar 21, 2017
View reviewed changes
Copy link

@aspiers aspiers left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have tested this now and it speeds up each rootwrap command by roughly a factor of 10. Unfortunately this only seems to double router migration speed though.

@aspiers
Copy link

aspiers commented Mar 21, 2017

Here's a sample of the speed difference. Individual rootwrap commands take roughly 10ms via the daemon:

2017-03-21 21:04:11.767 29740 DEBUG neutron.agent.linux.utils [...] Running command (rootwrap daemon): ['ip', 'link', 'set', 'qg-fccd10fb-6f', 'address', 'fa:16:3e:5d:d2:01'] execute_rootwrap_daemon /usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:101
2017-03-21 21:04:11.775 29740 DEBUG neutron.agent.linux.utils [...] Running command (rootwrap daemon): ['ip', '-o', 'netns', 'list'] execute_rootwrap_daemon /usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:101
2017-03-21 21:04:11.781 29740 DEBUG neutron.agent.linux.utils [...] Running command (rootwrap daemon): ['ip', 'link', 'set', 'qg-fccd10fb-6f', 'netns', 'qrouter-3fdf4656-d57d-42d5-acfb-787093d56b57'] execute_rootwrap_daemon /usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:101
2017-03-21 21:04:11.801 29740 DEBUG neutron.agent.linux.utils [...] Running command (rootwrap daemon): ['ip', 'netns', 'exec', 'qrouter-3fdf4656-d57d-42d5-acfb-787093d56b57', 'ip', 'link', 'set', 'qg-fccd10fb-6f', 'mtu', '1500'] execute_rootwrap_daemon /usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:101
2017-03-21 21:04:11.808 29740 DEBUG neutron.agent.linux.utils [...] Running command (rootwrap daemon): ['ip', 'netns', 'exec', 'qrouter-3fdf4656-d57d-42d5-acfb-787093d56b57', 'ip', 'link', 'set', 'qg-fccd10fb-6f', 'up'] execute_rootwrap_daemon /usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:101
2017-03-21 21:04:11.815 29740 DEBUG neutron.agent.linux.utils [...] Running command (rootwrap daemon): ['ip', 'netns', 'exec', 'qrouter-3fdf4656-d57d-42d5-acfb-787093d56b57', 'ip', '-o', 'link', 'show', 'qg-fccd10fb-6f'] execute_rootwrap_daemon /usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:101
2017-03-21 21:04:11.824 29740 DEBUG neutron.agent.linux.utils [...] Running command (rootwrap daemon): ['ip', 'netns', 'exec', 'qrouter-3fdf4656-d57d-42d5-acfb-787093d56b57', 'ip', 'addr', 'show', 'qg-fccd10fb-6f', 'permanent'] execute_rootwrap_daemon /usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:101
2017-03-21 21:04:11.834 29740 DEBUG neutron.agent.linux.utils [...] Running command (rootwrap daemon): ['ip', 'netns', 'exec', 'qrouter-3fdf4656-d57d-42d5-acfb-787093d56b57', 'ip', '-4', 'addr', 'add', '192.168.238.142/24', 'scope', 'global', 'dev', 'qg-fccd10fb-6f', 'brd', '192.168.238.255'] execute_rootwrap_daemon /usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:101
2017-03-21 21:04:11.844 29740 DEBUG neutron.agent.linux.utils [...] Running command (rootwrap daemon): ['ip', 'netns', 'exec', 'qrouter-3fdf4656-d57d-42d5-acfb-787093d56b57', 'ip', '-4', 'route', 'replace', 'default', 'via', '192.168.238.1', 'dev', 'qg-fccd10fb-6f'] execute_rootwrap_daemon /usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:101
2017-03-21 21:04:11.851 29740 DEBUG neutron.agent.linux.utils [...] Running command (rootwrap daemon): ['ip', 'netns', 'exec', 'qrouter-3fdf4656-d57d-42d5-acfb-787093d56b57', 'sysctl', '-w', 'net.ipv6.conf.qg-fccd10fb-6f.accept_ra=2'] execute_rootwrap_daemon /usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:101
2017-03-21 21:04:11.860 29740 DEBUG neutron.agent.linux.utils [...] Running command (rootwrap daemon): ['ip', 'netns', 'exec', 'qrouter-3fdf4656-d57d-42d5-acfb-787093d56b57', 'ip', '-4', 'route', 'list', 'dev', 'qg-fccd10fb-6f', 'scope', 'link'] execute_rootwrap_daemon /usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:101
2017-03-21 21:04:11.904 29740 DEBUG neutron.agent.linux.utils [...] Running command (rootwrap daemon): ['ip', 'netns', 'exec', 'qrouter-3fdf4656-d57d-42d5-acfb-787093d56b57', 'ip', '-6', 'route', 'list', 'dev', 'qg-fccd10fb-6f', 'scope', 'link'] execute_rootwrap_daemon /usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:101
2017-03-21 21:04:11.918 29740 DEBUG neutron.agent.linux.utils [...] Running command (rootwrap daemon): ['ip', 'netns', 'exec', 'qrouter-3fdf4656-d57d-42d5-acfb-787093d56b57', 'find', '/sys/class/net', '-maxdepth', '1', '-type', 'l', '-printf', '%f '] execute_rootwrap_daemon /usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:101

vs. 100ms without:

2017-03-21 21:20:51.867 350 DEBUG neutron.agent.linux.utils [...] Running command: ['sudo', 'neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'link', 'set', 'qr-3f68630e-b9', 'address', 'fa:16:3e:e6:41:75'] create_process /usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:85
2017-03-21 21:20:51.973 350 DEBUG neutron.agent.linux.utils [...] Running command: ['sudo', 'neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', '-o', 'netns', 'list'] create_process /usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:85
2017-03-21 21:20:52.085 350 DEBUG neutron.agent.linux.utils [...] Running command: ['sudo', 'neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'link', 'set', 'qr-3f68630e-b9', 'netns', 'qrouter-bdbbbfb7-ff13-4414-87f4-e8663331a28b'] create_process /usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:85
2017-03-21 21:20:52.257 350 DEBUG neutron.agent.linux.utils [...] Running command: ['sudo', 'neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'netns', 'exec', 'qrouter-bdbbbfb7-ff13-4414-87f4-e8663331a28b', 'ip', 'link', 'set', 'qr-3f68630e-b9', 'mtu', '1458'] create_process /usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:85
2017-03-21 21:20:52.418 350 DEBUG neutron.agent.linux.utils [...] Running command: ['sudo', 'neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'netns', 'exec', 'qrouter-bdbbbfb7-ff13-4414-87f4-e8663331a28b', 'ip', 'link', 'set', 'qr-3f68630e-b9', 'up'] create_process /usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:85
2017-03-21 21:20:52.606 350 DEBUG neutron.agent.linux.utils [...] Running command: ['sudo', 'neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'netns', 'exec', 'qrouter-bdbbbfb7-ff13-4414-87f4-e8663331a28b', 'ip', '-o', 'link', 'show', 'qr-3f68630e-b9'] create_process /usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:85
2017-03-21 21:20:52.721 350 DEBUG neutron.agent.linux.utils [...] Running command: ['sudo', 'neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'netns', 'exec', 'qrouter-bdbbbfb7-ff13-4414-87f4-e8663331a28b', 'ip', 'addr', 'show', 'qr-3f68630e-b9', 'permanent'] create_process /usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:85
2017-03-21 21:20:52.828 350 DEBUG neutron.agent.linux.utils [...] Running command: ['sudo', 'neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'netns', 'exec', 'qrouter-bdbbbfb7-ff13-4414-87f4-e8663331a28b', 'ip', '-4', 'addr', 'add', '192.168.123.1/24', 'scope', 'global', 'dev', 'qr-3f68630e-b9', 'brd', '192.168.123.255'] create_process /usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:85
2017-03-21 21:20:52.975 350 DEBUG neutron.agent.linux.utils [...] Running command: ['sudo', 'neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'netns', 'exec', 'qrouter-bdbbbfb7-ff13-4414-87f4-e8663331a28b', 'ip', '-4', 'route', 'list', 'dev', 'qr-3f68630e-b9', 'scope', 'link'] create_process /usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:85
2017-03-21 21:20:53.053 350 DEBUG neutron.agent.linux.utils [...] Running command: ['sudo', 'neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'netns', 'exec', 'qrouter-bdbbbfb7-ff13-4414-87f4-e8663331a28b', 'ip', '-6', 'route', 'list', 'dev', 'qr-3f68630e-b9', 'scope', 'link'] create_process /usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:85
2017-03-21 21:20:53.133 350 DEBUG neutron.agent.linux.utils [...] Running command: ['sudo', 'neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'netns', 'exec', 'qrouter-bdbbbfb7-ff13-4414-87f4-e8663331a28b', 'find', '/sys/class/net', '-maxdepth', '1', '-type', 'l', '-printf', '%f '] create_process /usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:85

Migration of 10 routers with the daemon takes ~43s:

root@controller1:~ # time neutron-ha-tool --l3-agent-evacuate d52-54-77-77-77-01
2017-03-21 21:00:00,613 requests.packages.urllib3.connectionpool INFO     Starting new HTTP connection (1): cluster-services.vp8.cloud.suse.de
2017-03-21 21:00:00,819 neutron-ha-tool INFO     Performing L3 Agent Evacuation from host d52-54-77-77-77-01
2017-03-21 21:00:01,004 neutron-ha-tool INFO     Querying agent_id=77b3edb6-9066-4447-af92-6b2f6fe69677 for routers to migrate away
2017-03-21 21:00:01,267 neutron-ha-tool INFO     Refreshing router count per agent cache
2017-03-21 21:00:01,290 neutron-ha-tool INFO     Migrating router=bdbbbfb7-ff13-4414-87f4-e8663331a28b from agent=77b3edb6-9066-4447-af92-6b2f6fe69677 to agent=fe1906c7-f8ea-47aa-b8c1-7fa6676a1c81
2017-03-21 21:00:01,662 neutron-ha-tool INFO     Wait for the ports and floating IPs of router_id=bdbbbfb7-ff13-4414-87f4-e8663331a28b to be ACTIVE
2017-03-21 21:00:05,153 neutron-ha-tool INFO     Migrating router=079bf0b9-22e6-48c8-829c-881e4ff8efe3 from agent=77b3edb6-9066-4447-af92-6b2f6fe69677 to agent=fe1906c7-f8ea-47aa-b8c1-7fa6676a1c81
2017-03-21 21:00:05,509 neutron-ha-tool INFO     Wait for the ports and floating IPs of router_id=079bf0b9-22e6-48c8-829c-881e4ff8efe3 to be ACTIVE
2017-03-21 21:00:08,882 neutron-ha-tool INFO     Migrating router=5cc10af6-b496-40ad-a805-c79fec4c3296 from agent=77b3edb6-9066-4447-af92-6b2f6fe69677 to agent=fe1906c7-f8ea-47aa-b8c1-7fa6676a1c81
2017-03-21 21:00:09,327 neutron-ha-tool INFO     Wait for the ports and floating IPs of router_id=5cc10af6-b496-40ad-a805-c79fec4c3296 to be ACTIVE
2017-03-21 21:00:12,682 neutron-ha-tool INFO     Migrating router=a4773172-27cd-40b8-a589-5e0b5a768a4d from agent=77b3edb6-9066-4447-af92-6b2f6fe69677 to agent=fe1906c7-f8ea-47aa-b8c1-7fa6676a1c81
2017-03-21 21:00:13,016 neutron-ha-tool INFO     Wait for the ports and floating IPs of router_id=a4773172-27cd-40b8-a589-5e0b5a768a4d to be ACTIVE
2017-03-21 21:00:16,359 neutron-ha-tool INFO     Migrating router=4a02ff92-3416-4b24-8f7f-3968968b9fb3 from agent=77b3edb6-9066-4447-af92-6b2f6fe69677 to agent=fe1906c7-f8ea-47aa-b8c1-7fa6676a1c81
2017-03-21 21:00:16,699 neutron-ha-tool INFO     Wait for the ports and floating IPs of router_id=4a02ff92-3416-4b24-8f7f-3968968b9fb3 to be ACTIVE
2017-03-21 21:00:18,958 neutron-ha-tool INFO     Migrating router=af899582-6d14-4584-82fd-f0bd2dced806 from agent=77b3edb6-9066-4447-af92-6b2f6fe69677 to agent=fe1906c7-f8ea-47aa-b8c1-7fa6676a1c81
2017-03-21 21:00:19,351 neutron-ha-tool INFO     Wait for the ports and floating IPs of router_id=af899582-6d14-4584-82fd-f0bd2dced806 to be ACTIVE
2017-03-21 21:00:23,224 neutron-ha-tool INFO     Migrating router=b041091b-fb0c-40b7-b0ff-82237b3afaf2 from agent=77b3edb6-9066-4447-af92-6b2f6fe69677 to agent=fe1906c7-f8ea-47aa-b8c1-7fa6676a1c81
2017-03-21 21:00:23,557 neutron-ha-tool INFO     Wait for the ports and floating IPs of router_id=b041091b-fb0c-40b7-b0ff-82237b3afaf2 to be ACTIVE
2017-03-21 21:00:26,878 neutron-ha-tool INFO     Migrating router=31621795-3b51-4299-b7d6-bc80462cccbf from agent=77b3edb6-9066-4447-af92-6b2f6fe69677 to agent=fe1906c7-f8ea-47aa-b8c1-7fa6676a1c81
2017-03-21 21:00:27,187 neutron-ha-tool INFO     Wait for the ports and floating IPs of router_id=31621795-3b51-4299-b7d6-bc80462cccbf to be ACTIVE
2017-03-21 21:00:31,077 neutron-ha-tool INFO     Migrating router=e11d1179-2e7a-4cf6-895c-0c660519f306 from agent=77b3edb6-9066-4447-af92-6b2f6fe69677 to agent=fe1906c7-f8ea-47aa-b8c1-7fa6676a1c81
2017-03-21 21:00:31,439 neutron-ha-tool INFO     Wait for the ports and floating IPs of router_id=e11d1179-2e7a-4cf6-895c-0c660519f306 to be ACTIVE
2017-03-21 21:00:34,886 neutron-ha-tool INFO     Migrating router=3f41c314-dd25-4a41-b507-6a2b2e4b78fe from agent=77b3edb6-9066-4447-af92-6b2f6fe69677 to agent=fe1906c7-f8ea-47aa-b8c1-7fa6676a1c81
2017-03-21 21:00:35,191 neutron-ha-tool INFO     Wait for the ports and floating IPs of router_id=3f41c314-dd25-4a41-b507-6a2b2e4b78fe to be ACTIVE
2017-03-21 21:00:39,187 neutron-ha-tool INFO     Migrating router=3fdf4656-d57d-42d5-acfb-787093d56b57 from agent=77b3edb6-9066-4447-af92-6b2f6fe69677 to agent=fe1906c7-f8ea-47aa-b8c1-7fa6676a1c81
2017-03-21 21:00:39,481 neutron-ha-tool INFO     Wait for the ports and floating IPs of router_id=3fdf4656-d57d-42d5-acfb-787093d56b57 to be ACTIVE
2017-03-21 21:00:43,337 neutron-ha-tool INFO     11 routers were evacuated from L3 agent d52-54-77-77-77-01

real    0m43.342s
user    0m0.812s
sys     0m0.104s

and ~84s without:

root@controller1:~ # time neutron-ha-tool --l3-agent-evacuate d52-54-77-77-77-01
2017-03-21 21:24:46,732 requests.packages.urllib3.connectionpool INFO     Starting new HTTP connection (1): cluster-services.vp8.cloud.suse.de
2017-03-21 21:24:46,917 neutron-ha-tool INFO     Performing L3 Agent Evacuation from host d52-54-77-77-77-01
2017-03-21 21:24:47,096 neutron-ha-tool INFO     Querying agent_id=77b3edb6-9066-4447-af92-6b2f6fe69677 for routers to migrate away
2017-03-21 21:24:47,276 neutron-ha-tool INFO     Refreshing router count per agent cache
2017-03-21 21:24:47,290 neutron-ha-tool INFO     Migrating router=bdbbbfb7-ff13-4414-87f4-e8663331a28b from agent=77b3edb6-9066-4447-af92-6b2f6fe69677 to agent=fe1906c7-f8ea-47aa-b8c1-7fa6676a1c81
2017-03-21 21:24:47,612 neutron-ha-tool INFO     Wait for the ports and floating IPs of router_id=bdbbbfb7-ff13-4414-87f4-e8663331a28b to be ACTIVE
2017-03-21 21:24:54,994 neutron-ha-tool INFO     Migrating router=079bf0b9-22e6-48c8-829c-881e4ff8efe3 from agent=77b3edb6-9066-4447-af92-6b2f6fe69677 to agent=fe1906c7-f8ea-47aa-b8c1-7fa6676a1c81
2017-03-21 21:24:55,476 neutron-ha-tool INFO     Wait for the ports and floating IPs of router_id=079bf0b9-22e6-48c8-829c-881e4ff8efe3 to be ACTIVE
2017-03-21 21:25:02,150 neutron-ha-tool INFO     Migrating router=5cc10af6-b496-40ad-a805-c79fec4c3296 from agent=77b3edb6-9066-4447-af92-6b2f6fe69677 to agent=fe1906c7-f8ea-47aa-b8c1-7fa6676a1c81
2017-03-21 21:25:02,534 neutron-ha-tool INFO     Wait for the ports and floating IPs of router_id=5cc10af6-b496-40ad-a805-c79fec4c3296 to be ACTIVE
2017-03-21 21:25:09,793 neutron-ha-tool INFO     Migrating router=a4773172-27cd-40b8-a589-5e0b5a768a4d from agent=77b3edb6-9066-4447-af92-6b2f6fe69677 to agent=fe1906c7-f8ea-47aa-b8c1-7fa6676a1c81
2017-03-21 21:25:10,132 neutron-ha-tool INFO     Wait for the ports and floating IPs of router_id=a4773172-27cd-40b8-a589-5e0b5a768a4d to be ACTIVE
2017-03-21 21:25:16,699 neutron-ha-tool INFO     Migrating router=4a02ff92-3416-4b24-8f7f-3968968b9fb3 from agent=77b3edb6-9066-4447-af92-6b2f6fe69677 to agent=fe1906c7-f8ea-47aa-b8c1-7fa6676a1c81
2017-03-21 21:25:17,002 neutron-ha-tool INFO     Wait for the ports and floating IPs of router_id=4a02ff92-3416-4b24-8f7f-3968968b9fb3 to be ACTIVE
2017-03-21 21:25:23,632 neutron-ha-tool INFO     Migrating router=af899582-6d14-4584-82fd-f0bd2dced806 from agent=77b3edb6-9066-4447-af92-6b2f6fe69677 to agent=fe1906c7-f8ea-47aa-b8c1-7fa6676a1c81
2017-03-21 21:25:24,018 neutron-ha-tool INFO     Wait for the ports and floating IPs of router_id=af899582-6d14-4584-82fd-f0bd2dced806 to be ACTIVE
2017-03-21 21:25:30,806 neutron-ha-tool INFO     Migrating router=b041091b-fb0c-40b7-b0ff-82237b3afaf2 from agent=77b3edb6-9066-4447-af92-6b2f6fe69677 to agent=fe1906c7-f8ea-47aa-b8c1-7fa6676a1c81
2017-03-21 21:25:31,557 neutron-ha-tool INFO     Wait for the ports and floating IPs of router_id=b041091b-fb0c-40b7-b0ff-82237b3afaf2 to be ACTIVE
2017-03-21 21:25:38,896 neutron-ha-tool INFO     Migrating router=31621795-3b51-4299-b7d6-bc80462cccbf from agent=77b3edb6-9066-4447-af92-6b2f6fe69677 to agent=fe1906c7-f8ea-47aa-b8c1-7fa6676a1c81
2017-03-21 21:25:39,297 neutron-ha-tool INFO     Wait for the ports and floating IPs of router_id=31621795-3b51-4299-b7d6-bc80462cccbf to be ACTIVE
2017-03-21 21:25:46,581 neutron-ha-tool INFO     Migrating router=e11d1179-2e7a-4cf6-895c-0c660519f306 from agent=77b3edb6-9066-4447-af92-6b2f6fe69677 to agent=fe1906c7-f8ea-47aa-b8c1-7fa6676a1c81
2017-03-21 21:25:46,949 neutron-ha-tool INFO     Wait for the ports and floating IPs of router_id=e11d1179-2e7a-4cf6-895c-0c660519f306 to be ACTIVE
2017-03-21 21:25:54,805 neutron-ha-tool INFO     Migrating router=3f41c314-dd25-4a41-b507-6a2b2e4b78fe from agent=77b3edb6-9066-4447-af92-6b2f6fe69677 to agent=fe1906c7-f8ea-47aa-b8c1-7fa6676a1c81
2017-03-21 21:25:55,220 neutron-ha-tool INFO     Wait for the ports and floating IPs of router_id=3f41c314-dd25-4a41-b507-6a2b2e4b78fe to be ACTIVE
2017-03-21 21:26:02,419 neutron-ha-tool INFO     Migrating router=3fdf4656-d57d-42d5-acfb-787093d56b57 from agent=77b3edb6-9066-4447-af92-6b2f6fe69677 to agent=fe1906c7-f8ea-47aa-b8c1-7fa6676a1c81
2017-03-21 21:26:02,771 neutron-ha-tool INFO     Wait for the ports and floating IPs of router_id=3fdf4656-d57d-42d5-acfb-787093d56b57 to be ACTIVE
2017-03-21 21:26:10,111 neutron-ha-tool INFO     11 routers were evacuated from L3 agent d52-54-77-77-77-01

real    1m23.947s
user    0m1.016s
sys     0m0.104s

So even though each rootwrap command is now roughly 10x faster, the router migration speed only doubled. It seems that the bottleneck now is waiting for the ports to become ACTIVE, although this seems to be mainly about waiting for rootwrap commands to complete, and to a lesser extent waiting for the neutron-server to notice the state changes.

@aspiers
Copy link

aspiers commented Mar 21, 2017

Patched oslo.rootwrap package is now in my OBS branch: https://build.opensuse.org/package/rdiff/home:aspiers:branches:Cloud:OpenStack:Liberty/python-oslo.rootwrap?opackage=python-oslo.rootwrap&oproject=Cloud%3AOpenStack%3ALiberty&rev=2

aspiers
aspiers suggested changes Mar 21, 2017
View reviewed changes
Copy link

@aspiers aspiers left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually, @matelakat please can you cherry-pick 5c7204f into this PR?

@matelakat
Copy link
Author

matelakat commented Mar 22, 2017

@aspiers done

@aspiers
Copy link

aspiers commented Mar 22, 2017

I've now submitted the necessary changes to the nova and neutron packages:

@aspiers aspiers requested a review from mkoderer March 22, 2017 11:54
@mkoderer
Copy link

mkoderer commented Mar 23, 2017

Changes LGTM - we should test all together

dirkmueller and others added 2 commits July 6, 2017 11:17
@dirkmueller
Nova/Neutron: Enable rootwrap daemon mode
316f9a7 
This provides a speedup (benchmarks that do a lot of neutron operations
go down from 24 minutes to 11 minutes, providing an almost twofold
improvement). The improvement is actually like eightfold on ARM.

(cherry picked from commit ef29f92)
@toabctl
nova: Use rootwrap daemon mode only for kvm (bsc#1010553)
23d45c2 
Using the daemon mode fails for Xen (see lp#1643457).
This is a workarround until the bug in oslo.rootwrap is fixed.

https://bugzilla.suse.com/show_bug.cgi?id=1010553
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@mkoderer mkoderer Awaiting requested review from mkoderer

1 more reviewer

@aspiers aspiers aspiers approved these changes

Reviewers whose approvals may not affect merge requirements

At least 1 approving review is required to merge this pull request.

Assignees

@aspiers aspiers

@gonzolino gonzolino

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Neutron rootwrap-daemon not active for neutron services

6 participants

@matelakat @aspiers @mkoderer @gonzolino @dirkmueller @toabctl