Thursday, March 27, 2014

Launching Instance via image and creating simultaneously bootable cinder volume (glusterfs) on Two Node GRE+OVS+Gluster F20 Cluster

Actually, post is about attempt to make working following  kind of `nova boot .. ` command via CLI or  Dashboard, launching Instance via image and creating simultaneously boot able cinder volume (on glusterfs). It follows up 
http://lxer.com/module/newswire/view/199524/index.html
http://lxer.com/module/newswire/view/199764/index.html

[root@dfw02 ~(keystone_admin)]$  nova boot --flavor 2 --key_name oskey25 --block-device source=image,id=de93ee44-4085-4111-b022-a7437da8feac,dest=volume,size=5,shutdown=preserve,bootindex=0 VF20RS025

where

 [root@dfw02 ~(keystone_boris)]$ glance image-list
+--------------------------------------+--------------------------+-------------+------------------+-----------+--------+
| ID                                   | Name                     | Disk Format | Container Format | Size      | Status |
+--------------------------------------+--------------------------+-------------+------------------+-----------+--------+
| a6e8ef59-e492-46e2-8147-fd8b1a65ed73 | CentOS 6.5 image         | qcow2       | bare             | 344457216 | active |
| dc992799-7831-4933-b6ee-7b81868f808b | CirrOS31                 | qcow2       | bare             | 13147648  | active |
| 03c9ad20-b0a3-4b71-aa08-2728ecb66210 | Fedora 19 x86_64         | qcow2       | bare             | 237371392 | active |
| de93ee44-4085-4111-b022-a7437da8feac | Fedora 20 image          | qcow2       | bare             | 214106112 | active |
| e70591fc-6905-4e57-84b7-4ffa7c001864 | Ubuntu Server 13.10      | qcow2       | bare             | 244514816 | active |
| 81734fc7-7300-48db-9b91-95e653e5ed5b | Ubuntu Trusty 03/25/2014 | qcow2       | bare             | 251920896 | active |
| 23ac0b82-9579-4491-8a12-9dfb566361a1 | Ubuntu Trusty 03/26/2014 | qcow2       | bare             | 251986432 | active |
+--------------------------------------+--------------------------+-------------+------------------+-----------+--------+



Controller is dfw02.localdomain (192.168.1.127)
Compute is  dfw01.localdomain (192.168.1.137)

Cluster Config

[root@dfw02 ~(keystone_admin)]$ openstack-status
== Nova services ==
openstack-nova-api:                     active
openstack-nova-cert:                    inactive  (disabled on boot)
openstack-nova-compute:                 inactive  (disabled on boot)
openstack-nova-network:                 inactive  (disabled on boot)
openstack-nova-scheduler:               active
openstack-nova-volume:                  inactive  (disabled on boot)
openstack-nova-conductor:               active

== Glance services ==
openstack-glance-api:                   active
openstack-glance-registry:              active

== Keystone service ==
openstack-keystone:                     active

== Horizon service ==
openstack-dashboard:                    active

== neutron services ==
neutron-server:                         active
neutron-dhcp-agent:                     active
neutron-l3-agent:                       active
neutron-metadata-agent:                 active
neutron-lbaas-agent:                    inactive  (disabled on boot)
neutron-openvswitch-agent:              active
neutron-linuxbridge-agent:              inactive  (disabled on boot)
neutron-ryu-agent:                      inactive  (disabled on boot)
neutron-nec-agent:                      inactive  (disabled on boot)
neutron-mlnx-agent:                     inactive  (disabled on boot)

== Cinder services ==
openstack-cinder-api:                   active
openstack-cinder-scheduler:             active
openstack-cinder-volume:                active

== Ceilometer services ==
openstack-ceilometer-api:               inactive  (disabled on boot)
openstack-ceilometer-central:           inactive  (disabled on boot)
openstack-ceilometer-compute:           active
openstack-ceilometer-collector:         inactive  (disabled on boot)
openstack-ceilometer-alarm-notifier:    inactive  (disabled on boot)
openstack-ceilometer-alarm-evaluator:   inactive  (disabled on boot)
== Support services ==
mysqld:                                 inactive  (disabled on boot)
libvirtd:                               active
openvswitch:                            active
dbus:                                   active
tgtd:                                   active
qpidd:                                  active
memcached:                              active

== Keystone users ==
+----------------------------------+---------+---------+-------+
|                id                |   name  | enabled | email |
+----------------------------------+---------+---------+-------+
| 970ed56ef7bc41d59c54f5ed8a1690dc |  admin  |   True  |       |
| 162021e787c54cac906ab3296a386006 |  boris  |   True  |       |
| 1beeaa4b20454048bf23f7d63a065137 |  cinder |   True  |       |
| 006c2728df9146bd82fab04232444abf |  glance |   True  |       |
| 5922aa93016344d5a5d49c0a2dab458c | neutron |   True  |       |
| af2f251586564b46a4f60cdf5ff6cf4f |   nova  |   True  |       |
+----------------------------------+---------+---------+-------+

== Glance images ==
+--------------------------------------+---------------------------------+-------------+------------------+-------------+--------+
| ID                                   | Name                            | Disk Format | Container Format | Size        | Status |
+--------------------------------------+---------------------------------+-------------+------------------+-------------+--------+
| a6e8ef59-e492-46e2-8147-fd8b1a65ed73 | CentOS 6.5 image                | qcow2       | bare             | 344457216   | active |
| dc992799-7831-4933-b6ee-7b81868f808b | CirrOS31                        | qcow2       | bare             | 13147648    | active |
| 03c9ad20-b0a3-4b71-aa08-2728ecb66210 | Fedora 19 x86_64                | qcow2       | bare             | 237371392   | active |
| de93ee44-4085-4111-b022-a7437da8feac | Fedora 20 image                 | qcow2       | bare             | 214106112   | active |
| e70591fc-6905-4e57-84b7-4ffa7c001864 | Ubuntu Server 13.10             | qcow2       | bare             | 244514816   | active |
| 2a95cb39-6e5f-422f-a113-49c3db8dbade | Ubuntu Trusty 03/27/2014        | qcow2       | bare             | 251986432   | active |
+--------------------------------------+---------------------------------+-------------+------------------+-------------+--------+

== Nova managed services ==
+------------------+-------------------+----------+---------+-------+----------------------------+-----------------+
| Binary           | Host              | Zone     | Status  | State | Updated_at                 | Disabled Reason |
+------------------+-------------------+----------+---------+-------+----------------------------+-----------------+
| nova-scheduler   | dfw02.localdomain | internal | enabled | up    | 2014-03-27T12:03:22.000000 | None            |
| nova-conductor   | dfw02.localdomain | internal | enabled | up    | 2014-03-27T12:03:20.000000 | None            |
| nova-compute     | dfw01.localdomain | nova     | enabled | up    | 2014-03-27T12:03:18.000000 | None            |
| nova-consoleauth | dfw02.localdomain | internal | enabled | up    | 2014-03-27T12:03:20.000000 | None            |
+------------------+-------------------+----------+---------+-------+----------------------------+-----------------+

== Nova networks ==
+--------------------------------------+-------+------+
| ID                                   | Label | Cidr |
+--------------------------------------+-------+------+
| 1eea88bb-4952-4aa4-9148-18b61c22d5b7 | int   | None |
| 426bb226-0ab9-440d-ba14-05634a17fb2b | int1  | None |
| 780ce2f3-2e6e-4881-bbac-857813f9a8e0 | ext   | None |
+--------------------------------------+-------+------+

== Nova instance flavors ==
+----+-----------+-----------+------+-----------+------+-------+-------------+-----------+
| ID | Name      | Memory_MB | Disk | Ephemeral | Swap | VCPUs | RXTX_Factor | Is_Public |
+----+-----------+-----------+------+-----------+------+-------+-------------+-----------+
| 1  | m1.tiny   | 512       | 1    | 0         |      | 1     | 1.0         | True      |
| 2  | m1.small  | 2048      | 20   | 0         |      | 1     | 1.0         | True      |
| 3  | m1.medium | 4096      | 40   | 0         |      | 2     | 1.0         | True      |
| 4  | m1.large  | 8192      | 80   | 0         |      | 4     | 1.0         | True      |
| 5  | m1.xlarge | 16384     | 160  | 0         |      | 8     | 1.0         | True      |
+----+-----------+-----------+------+-----------+------+-------+-------------+-----------+
== Nova instances ==
+----+------+--------+------------+-------------+----------+
| ID | Name | Status | Task State | Power State | Networks |
+----+------+--------+------------+-------------+----------+
+----+------+--------+------------+-------------+----------+


[root@dfw02 ~(keystone_admin)]$ nova-manage service list
Binary           Host                                 Zone             Status     State Updated_At
nova-scheduler   dfw02.localdomain                    internal         enabled    :-)   2014-03-27 12:13:12
nova-conductor   dfw02.localdomain                    internal         enabled    :-)   2014-03-27 12:13:10
nova-compute     dfw01.localdomain                    nova             enabled    :-)   2014-03-27 12:13:08
nova-consoleauth dfw02.localdomain                    internal         enabled    :-)   2014-03-27 12:13:10

[root@dfw02 ~(keystone_admin)]$ neutron agent-list
+--------------------------------------+--------------------+-------------------+-------+----------------+
| id                                   | agent_type         | host              | alive | admin_state_up |
+--------------------------------------+--------------------+-------------------+-------+----------------+
| 037b985d-7a7d-455b-8536-76eed40b0722 | L3 agent           | dfw02.localdomain | :-)   | True           |
| 22438ee9-b4ea-4316-9492-eb295288f61a | Open vSwitch agent | dfw02.localdomain | :-)   | True           |
| 76ed02e2-978f-40d0-879e-1a2c6d1f7915 | DHCP agent         | dfw02.localdomain | :-)   | True           |
| 951632a3-9744-4ff4-a835-c9f53957c617 | Open vSwitch agent | dfw01.localdomain | :-)   | True           |
+--------------------------------------+--------------------+-------------------+-------+----------------+



[root@dfw02 ~(keystone_admin)]$ df -h
Filesystem                      Size  Used Avail Use% Mounted on
/dev/mapper/fedora00-root        96G   40G   52G  44% /
devtmpfs                        3.9G     0  3.9G   0% /dev
tmpfs                           3.9G   92K  3.9G   1% /dev/shm
tmpfs                           3.9G  9.1M  3.9G   1% /run
tmpfs                           3.9G     0  3.9G   0% /sys/fs/cgroup
tmpfs                           3.9G   76K  3.9G   1% /tmp
/dev/sda5                       477M  122M  326M  28% /boot
/dev/mapper/fedora00-data1       77G  7.1G   66G  10% /data1
192.168.1.127:cinder-volumes05   77G  7.1G   66G  10% /var/lib/cinder/volumes/62f75cf6996a8a6bcc0d343be378c10a
tmpfs                           3.9G  9.1M  3.9G   1% /run/netns

Tenants environment

[root@dfw02 ~(keystone_boris)]$ nova list
+--------------------------------------+--------------+-----------+------------+-------------+------------------------------+
| ID                                   | Name         | Status    | Task State | Power State | Networks                     |
+--------------------------------------+--------------+-----------+------------+-------------+------------------------------+
| 389bd5d2-9dc9-4e73-9ee5-ad336bb3faaa | UbuntuTRS027 | SUSPENDED | None       | Shutdown    | int1=40.0.0.7, 192.168.1.113 |
| e237b214-d170-4c79-8fcb-a32a11863733 | VF20RS037    | SUSPENDED | None       | Shutdown    | int1=40.0.0.2, 192.168.1.115 |
| 97825e96-b4d1-4638-9212-480a37c61dd5 | VF20RS047    | SUSPENDED | None       | Shutdown    | int1=40.0.0.6, 192.168.1.107 |
| e896bf9f-e48b-4863-b30f-7bacde1abf4f | VF20RS050    | SUSPENDED | None       | Shutdown    | int1=40.0.0.4, 192.168.1.108 |
+--------------------------------------+--------------+-----------+------------+-------------+------------------------------+


[root@dfw02 ~(keystone_boris)]$ neutron subnet-list
+--------------------------------------+------+-------------+--------------------------------------------+
| id                                   | name | cidr        | allocation_pools                           |
+--------------------------------------+------+-------------+--------------------------------------------+
| 9e0d457b-c4c4-45cf-84e2-4ac7550f3b06 |      | 40.0.0.0/24 | {"start": "40.0.0.2", "end": "40.0.0.254"} |
+--------------------------------------+------+-------------+--------------------------------------------+



Check  file /usr/share/cinder/cinder-dist.conf

[root@dfw02 ~(keystone_admin)]$ cat /usr/share/cinder/cinder-dist.conf
[DEFAULT]
logdir = /var/log/cinder
state_path = /var/lib/cinder
lock_path = /var/lib/cinder/tmp
volumes_dir = /etc/cinder/volumes
iscsi_helper = tgtadm
sql_connection = mysql://cinder:cinder@localhost/cinder
rpc_backend = cinder.openstack.common.rpc.impl_qpid
rootwrap_config = /etc/cinder/rootwrap.conf
auth_strategy = keystone

[keystone_authtoken]
admin_tenant_name = %SERVICE_TENANT_NAME%
admin_user = %SERVICE_USER%
admin_password = %SERVICE_PASSWORD%
auth_host = 127.0.0.1
auth_port = 35357
auth_protocol = http

Hack bellow is similar https://ask.openstack.org/en/question/11672/ongoing-problem-with-starting-neutron-server-on-fedora-20-rdo-havana/

Log into MariaDB:-

MariaDB [(none)]> INSERT INTO mysql.user (User,Host,Password) VALUES('cinder','dfw02.localdomain',' ');

MariaDB [(none)]> UPDATE mysql.user SET Password = PASSWORD('cinder') WHERE User = 'cinder';

Query OK, 3 rows affected (0.00 sec)
Rows matched: 3  Changed: 3  Warnings: 0

MariaDB [(none)]> FLUSH PRIVILEGES ;
Query OK, 0 rows affected (0.00 sec)

MariaDB [(none)]> SELECT User, Host, Password FROM mysql.user;
+----------+-------------------+-------------------------------------------+
| User     | Host              | Password                                  |
+----------+-------------------+-------------------------------------------+
| root     | localhost         | *E0DC09146F1310B49A34199B04274A9EED6F9EC7 |
| root     | dfw02.localdomain | *E0DC09146F1310B49A34199B04274A9EED6F9EC7 |
| root     | 127.0.0.1         | *E0DC09146F1310B49A34199B04274A9EED6F9EC7 |
| root     | ::1               | *E0DC09146F1310B49A34199B04274A9EED6F9EC7 |
| keystone | localhost         | *936E8F7AB2E21B47F6C9A7E5D9FE14DBA2255E5A |
| keystone | %                 | *936E8F7AB2E21B47F6C9A7E5D9FE14DBA2255E5A |
| glance   | localhost         | *CC67CAF178CB9A07D756302E0BBFA3B0165DFD49 |
| glance   | %                 | *CC67CAF178CB9A07D756302E0BBFA3B0165DFD49 |
| cinder   | localhost         | *028F8298C041368BA08A280AA8D1EF895CB68D5C |
| cinder   | %                 | *028F8298C041368BA08A280AA8D1EF895CB68D5C
|
| neutron  | localhost         | *4DF421833991170108648F1103CD74FCB66BBE9E |
| neutron  | %                 | *03A31004769F9E4F94ECEEA61AA28D9649084839 |
| nova     | localhost         | *0BE3B501084D35F4C66DD3AC4569EAE5EA738212 |
| nova     | %                 | *0BE3B501084D35F4C66DD3AC4569EAE5EA738212 |
| nova     | dfw02.localdomain | *0BE3B501084D35F4C66DD3AC4569EAE5EA738212 |
| dash     | %                 | *C9E492EC67084E4255B200FD34BDF396E3CE1A36 |
| dash     | localhost         | *C9E492EC67084E4255B200FD34BDF396E3CE1A36 |
| cinder   | dfw02.localdomain | *028F8298C041368BA08A280AA8D1EF895CB68D5C |
+----------+-------------------+-------------------------------------------+
18 rows in set (0.00 sec)

[root@dfw02 ~(keystone_admin)]$ systemctl  restart  openstack-cinder-scheduler
 

[root@dfw02 ~(keystone_admin)]$ systemctl status  openstack-cinder-scheduler -l

openstack-cinder-scheduler.service - OpenStack Cinder Scheduler Server
   Loaded: loaded (/usr/lib/systemd/system/openstack-cinder-scheduler.service; enabled)
   Active: active (running) since Thu 2014-03-27 11:11:11 MSK; 25min ago
 Main PID: 4377 (cinder-schedule)
   CGroup: /system.slice/openstack-cinder-scheduler.service
           └─4377 /usr/bin/python /usr/bin/cinder-scheduler --config-file /usr/share/cinder/cinder-dist.conf   --config-file /etc/cinder/cinder.conf --logfile /var/log/cinder/scheduler.log

[root@dfw02 ~(keystone_admin)]$ systemctl restart  openstack-cinder-volume

[root@dfw02 ~(keystone_admin)]$ cinder service-list
+------------------+-------------------+------+---------+-------+----------------------------+
|      Binary      |        Host       | Zone |  Status | State |         Updated_at         |
+------------------+-------------------+------+---------+-------+----------------------------+
| cinder-scheduler | dfw02.localdomain | nova | enabled |   up  | 2014-03-28T09:42:51.000000 |
cinder-volume   | dfw02.localdomain | nova | enabled |   up  | 2014-03-28T09:42:48.000000 |
+------------------+-------------------+------+---------+-------+----------------------------+
 


[root@dfw02 ~(keystone_admin)]$ df -h
Filesystem                      Size  Used Avail Use% Mounted on
/dev/mapper/fedora00-root        96G   40G   53G  43% /
devtmpfs                        3.9G     0  3.9G   0% /dev
tmpfs                           3.9G  152K  3.9G   1% /dev/shm
tmpfs                           3.9G  9.1M  3.9G   1% /run
tmpfs                           3.9G     0  3.9G   0% /sys/fs/cgroup
tmpfs                           3.9G  140K  3.9G   1% /tmp
/dev/sda5                       477M  122M  326M  28% /boot
/dev/mapper/fedora00-data1       77G  4.7G   69G   7% /data1
192.168.1.127:cinder-volumes05   77G  4.7G   69G   7% /var/lib/cinder/volumes/62f75cf6996a8a6bcc0d343be378c10a
tmpfs                           3.9G  9.1M  3.9G   1% /run/netns
tmpfs                           3.9G  9.1M  3.9G   1% /run/netns
tmpfs                           3.9G  9.1M  3.9G   1% /run/netns


Afterwards create F20 instance via image loadable from volume via dashboard


  

 

 


[root@dfw02 ~(keystone_boris)]$ nova list

+--------------------------------------+--------------+--------+------------+-------------+------------------------------+
| ID                                   | Name         | Status | Task State | Power State | Networks                     |
+--------------------------------------+--------------+--------+------------+-------------+------------------------------+
| 389bd5d2-9dc9-4e73-9ee5-ad336bb3faaa | UbuntuTRS027 | ACTIVE | None       | Running     | int1=40.0.0.7, 192.168.1.113 |
| e237b214-d170-4c79-8fcb-a32a11863733 | VF20RS037    | ACTIVE | None       | Running     | int1=40.0.0.2, 192.168.1.115 |
| 97825e96-b4d1-4638-9212-480a37c61dd5 | VF20RS047    | ACTIVE | None       | Running     | int1=40.0.0.6, 192.168.1.107 |
| e896bf9f-e48b-4863-b30f-7bacde1abf4f | VF20RS050    | ACTIVE | None       | Running     | int1=40.0.0.4, 192.168.1.108 |
+--------------------------------------+--------------+--------+------------+-------------+------------------------------+

[root@dfw02 ~(keystone_boris)]$ nova show VF20RS047
+--------------------------------------+----------------------------------------------------------+
| Property                             | Value                                                    |
+--------------------------------------+----------------------------------------------------------+
| status                               | ACTIVE                                                   |
| updated                              | 2014-03-27T07:30:08Z                                     |
| OS-EXT-STS:task_state                | None                                                     |
| key_name                             | oskey25                                                  |
| image                                | Attempt to boot from volume - no image supplied          |
| hostId                               | 73ee4f5bd4da8ad7b39d768d0b167a03ac0471ea50d9ded6c6190fb1 |
| OS-EXT-STS:vm_state                  | active                                                   |
| OS-SRV-USG:launched_at               | 2014-03-27T07:30:08.000000                               |
| flavor                               | m1.small (2)                                             |
| id                                   | 97825e96-b4d1-4638-9212-480a37c61dd5                     |
| security_groups                      | [{u'name': u'default'}]                                  |
| OS-SRV-USG:terminated_at             | None                                                     |
| user_id                              | 162021e787c54cac906ab3296a386006                         |
| name                                 | VF20RS047                                                |
| created                              | 2014-03-27T07:28:57Z                                     |
| tenant_id                            | 4dacfff9e72c4245a48d648ee23468d5                         |
| OS-DCF:diskConfig                    | MANUAL                                                   |
| metadata                             | {}                                                       |
| os-extended-volumes:volumes_attached | [{u'id': u'd027a1bb-0739-4443-ab60-b00720bc3ba6'}]       |
| accessIPv4                           |                                                          |
| accessIPv6                           |                                                          |
| progress                             | 0                                                        |
| OS-EXT-STS:power_state               | 1                                                        |
| OS-EXT-AZ:availability_zone          | nova                                                     |
| int1 network                         | 40.0.0.6, 192.168.1.107                                  |
| config_drive                         |                                                          |
+--------------------------------------+----------------------------------------------------------+

[root@dfw02 ~(keystone_boris)]$ cinder list
+--------------------------------------+--------+---------------+------+-------------+----------+--------------------------------------+
|                  ID                  | Status |  Display Name | Size | Volume Type | Bootable |             Attached to              |
+--------------------------------------+--------+---------------+------+-------------+----------+--------------------------------------+
| 24f96221-c535-436c-832c-fe0130f43604 | in-use |               |  5   |     None    |   true   | e237b214-d170-4c79-8fcb-a32a11863733 |
| 2adb799d-079c-49ef-bcb3-ed13d1cb11a6 | in-use |   VF20VLG050  |  5   |  glusterfs  |   true   | e896bf9f-e48b-4863-b30f-7bacde1abf4f |
| 2eab2019-e80f-46f4-96fc-5afad10cfa27 | in-use | UbuntuTLVG027 |  5   |  glusterfs  |   true   | 389bd5d2-9dc9-4e73-9ee5-ad336bb3faaa |
| d027a1bb-0739-4443-ab60-b00720bc3ba6 | in-use |               |  5   |     None    |   true   | 97825e96-b4d1-4638-9212-480a37c61dd5 |
+--------------------------------------+--------+---------------+------+-------------+----------+--------------------------------------+

 However command like :

 [root@dfw02 ~(keystone_admin)]$  nova boot --flavor 2 --key_name oskey25 --block-device source=image,id=de93ee44-4085-4111-b022-a7437da8feac,dest=volume,size=5,shutdown=preserve,bootindex=0 VF20RS025

even ending up with "Error" status still  creates glusterfs cinder volume ( with system_id ) , which is quite healthy and may be utilized for building new instance of F20 or Ubuntu 14.04, whatever original image was,  via CLI or via Dashboard. It looks like a kind of bug in Nova&Neutron inter process communications. I would say synchronization at boot up.

I have to notice that procedure above is actually a hack , bug is serious and well known . View  :-
"Provide an API for external services to send defined events to the compute service for synchronization. This includes immediate needs for nova-neutron interaction around boot timing and network info updates"

https://blueprints.launchpad.net/nova/+spec/admin-event-callback-api  

      and bug report :-

https://bugs.launchpad.net/nova/+bug/1280357

As far as I can see target milestone is "Icehouse-rc1"