Wednesday, May 27, 2015

How VMs access metadata via qrouter-namespace in Openstack Kilo

It is actually an update for Neutron on Kilo of original blog entry
http://techbackground.blogspot.ie/2013/06/metadata-via-quantum-router.html 
considering  Quantum implementation on Grizzly.

From my standpoint understanding of core architecture of Neutron openstack flow in regards of nova-api metadata service access (and getting proper response from nova-api service) by VMs launching via nova causes a lot of problems due to leak of understanding of core concepts.

Neutron proxies metadata requests to Nova adding HTTP headers which Nova uses to identify the source instance. Neutron actually uses two proxies to do this: a namespace proxy and a metadata agent. This post shows how a metadata request gets from an instance to the Nova metadata service via a namespace proxy running in a Neutron router.

   


    Here both services openstack-nova-api && neutron-server are    running on Controller 192.169.142.127.

[root@ip-192-169-142-127 ~(keystone_admin)]# systemctl | grep nova-api
openstack-nova-api.service  loaded active running   OpenStack Nova API Server

[root@ip-192-169-142-127 ~(keystone_admin)]# systemctl | grep neutron-server
neutron-server.service         loaded active running   OpenStack Neutron Server

Regarding architecture in general,please,view http://lxer.com/module/newswire/view/214009/index.html


*************************************
1.Instance makes request
*************************************
[root@vf22rls ~]# curl http://169.254.169.254/latest/meta-data
ami-id
ami-launch-index
ami-manifest-path
block-device-mapping/
hostname
instance-action
instance-id
instance-type
kernel-id
local-hostname
local-ipv4
placement/
public-hostname
public-ipv4
public-keys/
ramdisk-id
reservation-id
security-groups

[root@vf22rls ~]# ip -4 address show dev eth0
2: eth0: mtu 1400 qdisc fq_codel state UP group default qlen 1000
    inet 50.0.0.15/24 brd 50.0.0.255 scope global dynamic eth0
       valid_lft 85770sec preferred_lft 85770sec

[root@vf22rls ~]#  ip route
default via 50.0.0.1 dev eth0  proto static  metric 100
50.0.0.0/24 dev eth0  proto kernel  scope link  src 50.0.0.15  metric 100


******************************************************************************
2.Namespace proxy receives request. The default gateway 50.0.0.1  exists within a Neutron router namespace on the network node.The Neutron-l3-agent started a namespace proxy in this namespace and added some iptables rules to redirect metadata requests to it. There are no special routes, so the request goes out the default gateway of course a Neutron router needs to have an interface on the subnet.
*******************************************************************************
Network Node 192.169.142.147
**********************************
[root@ip-192-169-142-147 ~(keystone_admin)]# ip netns
qdhcp-1bd1f3b8-8e4e-4193-8af0-023f0be4a0fb
qrouter-79801567-a0b5-4780-bfae-ac00e185a148

[root@ip-192-169-142-147 ~(keystone_admin)]# ip netns exec qdhcp-1bd1f3b8-8e4e-4193-8af0-023f0be4a0fb route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         50.0.0.1        0.0.0.0         UG    0      0        0 tapd6da9bb8-0e
50.0.0.0        0.0.0.0         255.255.255.0   U     0      0        0 tapd6da9bb8-0e

[root@ip-192-169-142-147 ~(keystone_admin)]# neutron router-list
+--------------------------------------+-----------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------+-------+
| id                                   | name      | external_gateway_info                                                                                                                                                                    | distributed | ha    |
+--------------------------------------+-----------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------+-------+
| 79801567-a0b5-4780-bfae-ac00e185a148 | RouerDemo | {"network_id": "1faee6ae-faea-4775-9c4e-abbf22c5815c", "enable_snat": true, "external_fixed_ips": [{"subnet_id": "35262e52-e288-4244-b107-dd093a2254d5", "ip_address": "172.24.4.227"}]} | False       | False |
+--------------------------------------+-----------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------+-------+


[root@ip-192-169-142-147 ~(keystone_admin)]# ip netns exec qrouter-79801567-a0b5-4780-bfae-ac00e185a148 ifconfig
lo: flags=73  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10
        loop  txqueuelen 0  (Local Loopback)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

qg-1feb35d8-b6: flags=4163  mtu 1500
        inet 172.24.4.227  netmask 255.255.255.240  broadcast 172.24.4.239
        inet6 fe80::f816:3eff:fe7b:7be0  prefixlen 64  scopeid 0x20
        ether fa:16:3e:7b:7b:e0  txqueuelen 0  (Ethernet)
        RX packets 868209  bytes 1181713676 (1.1 GiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 413610  bytes 32594119 (31.0 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

qr-6b8bf870-d4: flags=4163  mtu 1500
        inet 50.0.0.1  netmask 255.255.255.0  broadcast 50.0.0.255
        inet6 fe80::f816:3eff:feb3:30bf  prefixlen 64  scopeid 0x20
        ether fa:16:3e:b3:30:bf  txqueuelen 0  (Ethernet)
        RX packets 414032  bytes 32641578 (31.1 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 868416  bytes 1181753564 (1.1 GiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

[root@ip-192-169-142-147 ~(keystone_admin)]# ip netns exec qrouter-79801567-a0b5-4780-bfae-ac00e185a148 iptables-save| grep 9697
-A neutron-l3-agent-INPUT -p tcp -m tcp --dport 9697 -j DROP
-A neutron-l3-agent-PREROUTING -d 169.254.169.254/32 -i qr-+ -p tcp -m tcp --dport 80 -j REDIRECT --to-ports 9697

[root@ip-192-169-142-147 ~(keystone_admin)]# ip netns exec qrouter-79801567-a0b5-4780-bfae-ac00e185a148  netstat -anpt
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name   
tcp        0      0 0.0.0.0:9697            0.0.0.0:*               LISTEN      3210/python2       

[root@ip-192-169-142-147 ~(keystone_admin)]# ps -f --pid 3210 | fold -s -w 82
UID        PID  PPID  C STIME TTY          TIME CMD
neutron   3210     1  0 08:14 ?        00:00:00 /usr/bin/python2
/bin/neutron-ns-metadata-proxy
--pid_file=/var/lib/neutron/external/pids/79801567-a0b5-4780-bfae-ac00e185a148.pid
 --metadata_proxy_socket=/var/lib/neutron/metadata_proxy
--router_id=79801567-a0b5-4780-bfae-ac00e185a148 --state_path=/var/lib/neutron
--metadata_port=9697 --metadata_proxy_user=990 --metadata_proxy_group=988
--verbose
--log-file=neutron-ns-metadata-proxy-79801567-a0b5-4780-bfae-ac00e185a148.log
--log-dir=/var/log/neutron



The nameserver proxy adds two HTTP headers to the request:
    X-Forwarded-For: with the instance's IP address
    X-Neutron-Router-ID: with the uuid of the Neutron router
and proxies it to a Unix domain socket with name /var/lib/neutron/metadata_proxy.

***********************************************************************************
3. Metadata agent receives request and queries the Neutron service
The metadata agent listens on this Unix socket. It is a normal
Linux service that runs in the main operating system IP namespace,
and so it is able to reach the Neutron and Nova metadata services.
Its configuration file has all the information required to do so.
***********************************************************************************

[root@ip-192-169-142-147 ~(keystone_admin)]# netstat -lxp | grep metadata
unix  2      [ ACC ]     STREAM     LISTENING     36208    1291/python2         /var/lib/neutronmetadata_proxy

[root@ip-192-169-142-147 ~(keystone_admin)]# ps -f --pid 1291 | fold -w 80 -s
UID        PID  PPID  C STIME TTY          TIME CMD
neutron   1291     1  0 08:12 ?        00:00:06 /usr/bin/python2
/usr/bin/neutron-metadata-agent --config-file
/usr/share/neutron/neutron-dist.conf --config-file /etc/neutron/neutron.conf
--config-file /etc/neutron/metadata_agent.ini --config-dir
/etc/neutron/conf.d/neutron-metadata-agent --log-file
/var/log/neutron/metadata-agent.log

[root@ip-192-169-142-147 ~(keystone_admin)]# lsof /var/lib/neutron/metadata_proxy
lsof: WARNING: can't stat() fuse.gvfsd-fuse file system /run/user/1000/gvfs
      Output information may be incomplete.
COMMAND    PID    USER   FD   TYPE             DEVICE SIZE/OFF  NODE NAME
neutron-m 1291 neutron    5u  unix 0xffff8801375ecb40      0t0 36208 /var/lib/neutron/metadata_proxy
neutron-m 2764 neutron    5u  unix 0xffff8801375ecb40      0t0 36208 /var/lib/neutron/metadata_proxy
neutron-m 2765 neutron    5u  unix 0xffff8801375ecb40      0t0 36208 /var/lib/neutron/metadata_proxy

[root@ip-192-169-142-147 ~(keystone_admin)]# ps -f --pid 1291 | fold -w 80 -s
UID        PID  PPID  C STIME TTY          TIME CMD
neutron   1291     1  0 08:12 ?        00:00:06 /usr/bin/python2
/usr/bin/neutron-metadata-agent --config-file
/usr/share/neutron/neutron-dist.conf --config-file /etc/neutron/neutron.conf
--config-file /etc/neutron/metadata_agent.ini --config-dir
/etc/neutron/conf.d/neutron-metadata-agent --log-file
/var/log/neutron/metadata-agent.log

[root@ip-192-169-142-147 ~(keystone_admin)]# grep -v '^#\|^\s*$' /etc/neutron/metadata_agent.ini
[DEFAULT]
debug = False
auth_url = http://192.169.142.127:35357/v2.0
auth_region = RegionOne
auth_insecure = False
admin_tenant_name = services
admin_user = neutron
admin_password = 808e36e154bd4cee
nova_metadata_ip = 192.169.142.127
nova_metadata_port = 8775
nova_metadata_protocol = http
metadata_proxy_shared_secret =a965cd23ed2f4502
metadata_workers =2
metadata_backlog = 4096
cache_url = memory://?default_ttl=5

It reads the X-Forwarded-For and X-Neutron-Router-ID headers in the request and queries the Neutron service to find the ID of the instance that created the request.

***********************************************************************************
4.Metadata agent proxies request to Nova metadata service
It then adds these headers:
    X-Instance-ID: the instance ID returned from Neutron
    X-Instance-ID-Signature: instance ID signed with the shared-secret
    X-Forwarded-For: the instance's IP address
and proxies the request to the Nova metadata service.

5. Nova metadata service receives request
The metadata service was started by nova-api. The handler checks the X-Instance-ID-Signature with the shared key, looks up the data and returns the response which travels back via the two proxies to the instance.
************************************************************************************


*****************************
Controller 192.169.142.127
*****************************

[root@ip-192-169-142-127 ~(keystone_admin)]# grep metadata /etc/nova/nova.conf | grep -v ^# | grep -v ^$
enabled_apis=ec2,osapi_compute,metadata
metadata_listen=0.0.0.0
metadata_workers=2
metadata_host=192.169.142.127
service_metadata_proxy=True
metadata_proxy_shared_secret=a965cd23ed2f4502


[root@ip-192-169-142-127 ~(keystone_admin)]# grep metadata /etc/nova/nova.conf | grep -v ^# | grep -v ^$
enabled_apis=ec2,osapi_compute,metadata
metadata_listen=0.0.0.0
metadata_workers=2
metadata_host=192.169.142.127
service_metadata_proxy=True
metadata_proxy_shared_secret=a965cd23ed2f4502


[root@ip-192-169-142-127 ~(keystone_admin)]#  grep metadata /var/log/nova/nova-api.log | tail -15
2015-05-27 10:23:25.232 3987 INFO nova.metadata.wsgi.server [-] 50.0.0.15,192.169.142.147 "GET /latest/meta-data/local-ipv4 HTTP/1.1" status: 200 len: 125 time: 0.0006239
2015-05-27 10:23:25.271 3986 INFO nova.metadata.wsgi.server [-] 50.0.0.15,192.169.142.147 "GET /latest/meta-data/reservation-id HTTP/1.1" status: 200 len: 127 time: 0.0006211
2015-05-27 10:23:25.309 3986 INFO nova.metadata.wsgi.server [-] 50.0.0.15,192.169.142.147 "GET /latest/meta-data/local-hostname HTTP/1.1" status: 200 len: 134 time: 0.0006039
2015-05-27 10:23:25.348 3987 INFO nova.metadata.wsgi.server [-] 50.0.0.15,192.169.142.147 "GET /latest/meta-data/security-groups HTTP/1.1" status: 200 len: 116 time: 0.0006092
2015-05-27 10:23:25.386 3987 INFO nova.metadata.wsgi.server [-] 50.0.0.15,192.169.142.147 "GET /latest/meta-data/ami-launch-index HTTP/1.1" status: 200 len: 117 time: 0.0006170
2015-05-27 10:23:25.424 3987 INFO nova.metadata.wsgi.server [-] 50.0.0.15,192.169.142.147 "GET /latest/meta-data/ramdisk-id HTTP/1.1" status: 200 len: 120 time: 0.0006149
2015-05-27 10:23:25.463 3987 INFO nova.metadata.wsgi.server [-] 50.0.0.15,192.169.142.147 "GET /latest/meta-data/public-hostname HTTP/1.1" status: 200 len: 134 time: 0.0006301
2015-05-27 10:23:25.502 3987 INFO nova.metadata.wsgi.server [-] 50.0.0.15,192.169.142.147 "GET /latest/meta-data/hostname HTTP/1.1" status: 200 len: 134 time: 0.0006180
2015-05-27 10:23:25.541 3987 INFO nova.metadata.wsgi.server [-] 50.0.0.15,192.169.142.147 "GET /latest/meta-data/ami-id HTTP/1.1" status: 200 len: 129 time: 0.0006082
2015-05-27 10:23:25.581 3986 INFO nova.metadata.wsgi.server [-] 50.0.0.15,192.169.142.147 "GET /latest/meta-data/kernel-id HTTP/1.1" status: 200 len: 120 time: 0.0006080
2015-05-27 10:23:25.618 3986 INFO nova.metadata.wsgi.server [-] 50.0.0.15,192.169.142.147 "GET /latest/meta-data/instance-action HTTP/1.1" status: 200 len: 120 time: 0.0006869
2015-05-27 10:23:25.656 3987 INFO nova.metadata.wsgi.server [-] 50.0.0.15,192.169.142.147 "GET /latest/meta-data/public-ipv4 HTTP/1.1" status: 200 len: 129 time: 0.0006471
2015-05-27 10:23:25.696 3987 INFO nova.metadata.wsgi.server [-] 50.0.0.15,192.169.142.147 "GET /latest/meta-data/ami-manifest-path HTTP/1.1" status: 200 len: 121 time: 0.0007231
2015-05-27 10:23:25.735 3987 INFO nova.metadata.wsgi.server [-] 50.0.0.15,192.169.142.147 "GET /latest/meta-data/instance-type HTTP/1.1" status: 200 len: 124 time: 0.0006821
2015-05-27 10:23:25.775 3987 INFO nova.metadata.wsgi.server [-] 50.0.0.15,192.169.142.147 "GET /latest/meta-data/instance-id HTTP/1.1" status: 200 len: 127 time: 0.0007501