Saturday, October 19, 2013

Glusterfs Striped volumes based Havana 2013.2 instances on NFS-Like Standalone Storage Server With GlusterFS 3.4.1 Fedora 19

Please,  view first nice article: http://www.mirantis.com/blog/openstack-havana-glusterfs

and  https://wiki.openstack.org/wiki/CinderSupportMatrix

Here goes a sample of more complicated structure of  gluster volume for Havana's instances bootable cinder volumes storage  versus http://bderzhavets.blogspot.ru/2013/10/glusterfss-volume-based-havana-rc1

**************************************

Post install fix (as of 10/31/2013) : 

Also set  ALLOWED_HOSTS = ['*']  in /etc/openstack-dashboard/local_settings and restart  httpd
*********************************************
[root@localhost boris(keystone_admin)]# gluster volume create \

cinder-volumes stripe 3 \
192.168.1.142:/home/boris/node1 \
192.168.1.142:/home/boris/node2 \
192.168.1.142:/home/boris/node3

volume create: cinder-volumes: success: please start the volume to access data

[root@localhost boris(keystone_admin)]# gluster volume start cinder-volumes
volume start: cinder-volumes: success

[root@localhost boris(keystone_admin)]# gluster volume info

Volume Name: cinder-volume
Type: Distribute
Volume ID: be95ce0d-47ea-47f5-beee-66196c546d20
Status: Started
Number of Bricks: 1
Transport-type: tcp
Bricks:
Brick1: 192.168.1.142:/rhs/brick1/cinder-volume

Volume Name: cinder-volumes
Type: Stripe
Volume ID: 14b7de86-7b0e-4a60-b21b-4990b1222f43
Status: Started
Number of Bricks: 1 x 3 = 3
Transport-type: tcp
Bricks:
Brick1: 192.168.1.142:/home/boris/node1
Brick2: 192.168.1.142:/home/boris/node2
Brick3: 192.168.1.142:/home/boris/node3

[root@localhost boris(keystone_admin)]# systemctl status glusterd
glusterd.service - GlusterFS an clustered file-system server
   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled)
   Active: active (running) since Sun 2013-10-20 13:47:11 MSK; 2h 21min ago
  Process: 1350 ExecStart=/usr/sbin/glusterd -p /run/glusterd.pid (code=exited, status=0/SUCCESS)
 Main PID: 1380 (glusterd)
   CGroup: name=systemd:/system/glusterd.service
           ├─ 1380 /usr/sbin/glusterd -p /run/glusterd.pid
           ├─ 2217 /usr/sbin/glusterfsd -s 192.168.1.142 --volfile-id cinder-volume.192.168.1.142.rhs-brick1-cinder-volume ...
           ├─11156 /usr/sbin/glusterfsd -s 192.168.1.142 --volfile-id cinder-volumes.192.168.1.142.home-boris-node1 -p /var...
           ├─11165 /usr/sbin/glusterfsd -s 192.168.1.142 --volfile-id cinder-volumes.192.168.1.142.home-boris-node2 -p /var...
           ├─11174 /usr/sbin/glusterfsd -s 192.168.1.142 --volfile-id cinder-volumes.192.168.1.142.home-boris-node3 -p /var...
           ├─11190 /usr/sbin/glusterfs -s localhost --volfile-id gluster/nfs -p /var/lib/glusterd/nfs/run/nfs.pid -l /var/l...
           └─11198 /sbin/rpc.statd

Oct 20 13:47:11 localhost.localdomain rpc.statd[2261]: Version 1.2.7 starting
Oct 20 13:47:11 localhost.localdomain systemd[1]: Started GlusterFS an clustered file-system server.
Oct 20 13:58:50 localhost.localdomain rpc.statd[11198]: Version 1.2.7 starting
Oct 20 13:58:50 localhost.localdomain sm-notify[11199]: Version 1.2.7 starting

 Configuring Cinder to Add GlusterFS

 

If it has not been done

 

# openstack-config --set /etc/cinder/cinder.conf DEFAULT volume_driver cinder.volume.drivers.glusterfs.GlusterfsDriver

# openstack-config --set /etc/cinder/cinder.conf DEFAULT glusterfs_shares_config /etc/cinder/shares.conf

# openstack-config --set /etc/cinder/cinder.conf DEFAULT glusterfs_mount_point_base /var/lib/cinder/volumes

 

Update shares to point to another gluster volume 

 

 # vi /etc/cinder/shares.conf

    192.168.1.142:cinder-volumes

:wq

 

If it has not been done

 


# iptables-save > iptables.dump

Add to *filter section:

-A INPUT -m state --state NEW -m tcp -p tcp --dport 111 -j ACCEPT

-A INPUT -m state --state NEW -m tcp -p tcp --dport 24007 -j ACCEPT

-A INPUT -m state --state NEW -m tcp -p tcp --dport 24008 -j ACCEPT

-A INPUT -m state --state NEW -m tcp -p tcp --dport 24009 -j ACCEPT

-A INPUT -m state --state NEW -m tcp -p tcp --dport 24010 -j ACCEPT

-A INPUT -m state --state NEW -m tcp -p tcp --dport 24011 -j ACCEPT

-A INPUT -m state --state NEW -m tcp -p tcp --dport 38465:38469 -j ACCEPT

# iptables-restore < iptables.dump

# service iptables restart


[root@localhost boris(keystone_admin)]# df -h
Filesystem                   Size  Used Avail Use% Mounted on
/dev/mapper/fedora00-root    145G   60G   78G  44% /
devtmpfs                     3.9G     0  3.9G   0% /dev
tmpfs                        3.9G  648K  3.9G   1% /dev/shm
tmpfs                        3.9G 1020K  3.9G   1% /run
tmpfs                        3.9G     0  3.9G   0% /sys/fs/cgroup
tmpfs                        3.9G   96K  3.9G   1% /tmp
/dev/sdb3                    477M  104M  344M  24% /boot
/dev/loop0                   928M  1.4M  860M   1% /srv/node/device1
192.168.1.142:cinder-volume  145G   60G   78G  44% /var/lib/cinder/volumes/4b0a6960f94e0c2a28a479c06957e35a
tmpfs                        3.9G 1020K  3.9G   1% /run/netns

Ready to restart cinder services  

 

[root@localhost boris(keystone_admin)]# for i in api scheduler volume; do service openstack-cinder-${i} restart; done

Redirecting to /bin/systemctl restart  openstack-cinder-api.service
Redirecting to /bin/systemctl restart  openstack-cinder-scheduler.service
Redirecting to /bin/systemctl restart  openstack-cinder-volume.service


New directory mounted :



[root@localhost boris(keystone_admin)]# df -h
Filesystem                    Size  Used Avail Use% Mounted on
/dev/mapper/fedora00-root     145G   59G   78G  44% /
devtmpfs                      3.9G     0  3.9G   0% /dev
tmpfs                         3.9G  648K  3.9G   1% /dev/shm
tmpfs                         3.9G 1020K  3.9G   1% /run
tmpfs                         3.9G     0  3.9G   0% /sys/fs/cgroup
tmpfs                         3.9G  108K  3.9G   1% /tmp
/dev/sdb3                     477M  104M  344M  24% /boot
/dev/loop0                    928M  1.4M  860M   1% /srv/node/device1
tmpfs                         3.9G 1020K  3.9G   1% /run/netns
192.168.1.142:cinder-volumes  433G  177G  234G  44% /var/lib/cinder/volumes/76078cf31245409735587a509f24ab39

[root@localhost boris(keystone_admin)]# nova image-list
+--------------------------------------+----------------------------+--------+--------+
| ID                                   | Name                       | Status | Server |
+--------------------------------------+----------------------------+--------+--------+
| 807758c4-20fd-41b2-a943-27c80a651fc7 | Fedora19image              | ACTIVE |        |
| 35098ebe-f04d-4c26-bd8c-0bec0f0fb892 | UbuntuSaucyRelease         | ACTIVE |        |
| e8e426f8-8898-42a5-9b3f-0374d20e318b | UbuntuServer1310.image     | ACTIVE |        |
| 8fdc5c6e-1554-4ad2-bcaf-f4aefee7d690 | Windos Server2012 Std Eval | ACTIVE |        |
| 10295ec8-3d8d-4623-9490-f217f8498878 | cirros                     | ACTIVE |        |
+--------------------------------------+----------------------------+--------+--------+

[root@localhost boris(keystone_admin)]# cinder create --image-id 807758c4-20fd-41b2-a943-27c80a651fc7  --display_name Fedora19VL 5
+---------------------+--------------------------------------+
|       Property      |                Value                 |
+---------------------+--------------------------------------+
|     attachments     |                  []                  |
|  availability_zone  |                 nova                 |
|       bootable      |                false                 |
|      created_at     |      2013-10-20T10:05:22.586342      |
| display_description |                 None                 |
|     display_name    |              Fedora19VL              |
|          id         | d11f00de-e543-4ed5-9686-a0421639c5e3 |
|       image_id      | 807758c4-20fd-41b2-a943-27c80a651fc7 |
|       metadata      |                  {}                  |
|         size        |                  5                   |
|     snapshot_id     |                 None                 |
|     source_volid    |                 None                 |
|        status       |               creating               |
|     volume_type     |                 None                 |
+---------------------+--------------------------------------+

[root@localhost boris(keystone_admin)]# cinder list
+--------------------------------------+-----------+--------------+------+-------------+----------+-------------+
|                  ID                  |   Status  | Display Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+-----------+--------------+------+-------------+----------+-------------+
| d11f00de-e543-4ed5-9686-a0421639c5e3 | available |  Fedora19VL  |  5   |     None    |   true   |             |
+--------------------------------------+-----------+--------------+------+-------------+----------+-------------+

[root@localhost boris(keystone_admin)]# cinder create --image-id 35098ebe-f04d-4c26-bd8c-0bec0f0fb892 --display_name SalamanderVL 5
+---------------------+--------------------------------------+
|       Property      |                Value                 |
+---------------------+--------------------------------------+
|     attachments     |                  []                  |
|  availability_zone  |                 nova             |
|       bootable         |                false              |
|      created_at       |      2013-10-20T10:07:01.897362      |
| display_description |                 None        |
|     display_name  |             SalamanderVL  |
|          id                | a0102e10-52cb-49d6-bd58-f0365968f721 |
|       image_id       | 35098ebe-f04d-4c26-bd8c-0bec0f0fb892 |
|       metadata      |                  {}               |
|         size             |                  5                 |
|     snapshot_id   |                 None           |
|     source_volid   |                 None          |
|        status          |               creating       |
|     volume_type   |                 None          |
+---------------------+--------------------------------------+

[root@localhost boris(keystone_admin)]# cinder list
+--------------------------------------+-------------+--------------+------+-------------+----------+-------------+
|                  ID                  |    Status   | Display Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+-------------+--------------+------+-------------+----------+-------------+
| a0102e10-52cb-49d6-bd58-f0365968f721 | downloading | SalamanderVL |  5   |     None    |   true   |             |
| d11f00de-e543-4ed5-9686-a0421639c5e3 |  available  |  Fedora19VL  |  5   |     None    |   true   |             |
+--------------------------------------+-------------+--------------+------+-------------+----------+-------------+

[root@localhost boris(keystone_admin)]# cinder list
+--------------------------------------+-----------+--------------+------+-------------+----------+-------------+
|                  ID                  |   Status  | Display Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+-----------+--------------+------+-------------+----------+-------------+
| a0102e10-52cb-49d6-bd58-f0365968f721 | available | SalamanderVL |  5   |     None    |   true   |             |
| d11f00de-e543-4ed5-9686-a0421639c5e3 | available |  Fedora19VL  |  5   |     None    |   true   |             |
+--------------------------------------+-----------+--------------+------+-------------+----------+-------------+

[root@localhost boris(keystone_admin)]# ls -l /var/lib/cinder/volumes/76078cf31245409735587a509f24ab39
total 1496292
-rw-rw-rw-. 1 root root 5368709120 Oct 20 14:07 volume-a0102e10-52cb-49d6-bd58-f0365968f721
-rw-rw-rw-. 1 root root 5368709120 Oct 20 14:05 volume-d11f00de-e543-4ed5-9686-a0421639c5e3

[root@localhost boris(keystone_admin)]# nova list
+--------------------------------------+-------------------+--------+------------+-------------+--------------------------------+
| ID                                   | Name              | Status | Task State | Power State | Networks                       |
+--------------------------------------+-------------------+--------+------------+-------------+--------------------------------+
| 9754374c-bcf2-4f34-a931-649b417d939d | UbuntuSalamander1 | ACTIVE | None       | Running     | private=10.0.0.3, 192.168.1.61 |
+--------------------------------------+-------------------+--------+------------+-------------+--------------------------------+
[root@localhost boris(keystone_admin)]# nova show  9754374c-bcf2-4f34-a931-649b417d939d
+--------------------------------------+----------------------------------------------------------+
| Property                             | Value                                                    |
+--------------------------------------+----------------------------------------------------------+
| status                               | ACTIVE                                                   |
| updated                              | 2013-10-20T10:16:44Z                                     |
| OS-EXT-STS:task_state                | None                                                     |
| OS-EXT-SRV-ATTR:host                 | localhost.localdomain                                    |
| key_name                             | key2                                                     |
| image                                | Attempt to boot from volume - no image supplied          |
| private network                      | 10.0.0.3, 192.168.1.61                                   |
| hostId                               | 8a06ef818edcb74efba027817c5f14cb4d1d38c0fcd1dde73a9356d6 |
| OS-EXT-STS:vm_state                  | active                                                   |
| OS-EXT-SRV-ATTR:instance_name        | instance-00000013                                        |
| OS-SRV-USG:launched_at               | 2013-10-20T10:16:44.000000                               |
| OS-EXT-SRV-ATTR:hypervisor_hostname  | localhost.localdomain                                    |
| flavor                               | m1.small (2)                                             |
| id                                   | 9754374c-bcf2-4f34-a931-649b417d939d                     |
| security_groups                      | [{u'name': u'default'}]                                  |
| OS-SRV-USG:terminated_at             | None                                                     |
| user_id                              | 239b0f75c0514945acb5bb2041ccd89d                         |
| name                                 | UbuntuSalamander1                                        |
| created                              | 2013-10-20T10:16:35Z                                     |
| tenant_id                            | 644515d6bdf14f53a44f226fd7c9f69b                         |
| OS-DCF:diskConfig                    | MANUAL                                                   |
| metadata                             | {}                                                       |
| os-extended-volumes:volumes_attached | [{u'id': u'a0102e10-52cb-49d6-bd58-f0365968f721'}]       |
| accessIPv4                           |                                                          |
| accessIPv6                           |                                                          |
| progress                             | 0                                                        |
| OS-EXT-STS:power_state               | 1                                                        |
| OS-EXT-AZ:availability_zone          | nova                                                     |
| config_drive                         |                                                          |
+--------------------------------------+----------------------------------------------------------+
[root@localhost boris(keystone_admin)]# cinder list
+--------------------------------------+-----------+--------------+------+-------------+----------+--------------------------------------+
|                  ID                  |   Status  | Display Name | Size | Volume Type | Bootable |             Attached to              |
+--------------------------------------+-----------+--------------+------+-------------+----------+--------------------------------------+
| a0102e10-52cb-49d6-bd58-f0365968f721 |   in-use  | SalamanderVL |  5   |     None    |   true   | 9754374c-bcf2-4f34-a931-649b417d939d |
| d11f00de-e543-4ed5-9686-a0421639c5e3 | available |  Fedora19VL  |  5   |     None    |   true   |                                      |
+--------------------------------------+-----------+--------------+------+-------------+----------+--------------------------------------+

[root@localhost boris(keystone_admin)]# ls -lah /var/lib/cinder/volumes/76078cf31245409735587a509f24ab39
total 1.8G

drwxr-xr-x. 3 root   root   4.0K Oct 20 14:07 .
drwxr-xr-x. 4 cinder cinder 4.0K Oct 20 14:03 ..
-rw-rw-rw-. 1 qemu   qemu   5.0G Oct 20 14:28 volume-a0102e10-52cb-49d6-bd58-f0365968f721
-rw-rw-rw-. 1 root   root   5.0G Oct 20 14:05 volume-d11f00de-e543-4ed5-9686-a0421639c5e3

[root@localhost boris(keystone_admin)]# cd /home/boris
[root@localhost boris(keystone_admin)]# du | grep node.$
637972    ./node3
638368    ./node2
637964    ./node1




  

    Another snap shots series

   

  

 

  


   
  

Loading Windows 2012 Server via bootable cinder volume located on glusterfs stripe 3 volume.


[root@localhost (keystone_admin)]# nova image-list
+--------------------------------------+------------------------------+--------+--------+
| ID                                   | Name                         | Status | Server |
+--------------------------------------+------------------------------+--------+--------+
| 59758edc-da8d-444e-b0a0-d93d323fc026 | F19Image                     | ACTIVE |        |
| df912358-b227-43a5-94a3-edc874c577bc | UbuntuSalamander             | ACTIVE |        |
| 1e26928b-5df0-4097-bbc6-46832dc8361b | Windows Server 2012 Std Eval | ACTIVE |        |
| ae07d1ba-41de-44e9-877a-455f8956d86f | cirros                       | ACTIVE |        |
+--------------------------------------+------------------------------+--------+--------+
[root@localhost (keystone_admin)]# cinder create --image-id 1e26928b-5df0-4097-bbc6-46832dc8361b  --display_name WinSRV2012 20
+---------------------+--------------------------------------+
|       Property      |                Value                 |
+---------------------+--------------------------------------+
|     attachments     |                  []                  |
|  availability_zone  |                 nova                 |
|       bootable      |                false                 |
|      created_at     |      2013-10-22T06:03:05.019987      |
| display_description |                 None                 |
|     display_name    |              WinSRV2012              |
|          id         | 5812c7e5-d071-43cb-8a1c-81346167d72a |
|       image_id      | 1e26928b-5df0-4097-bbc6-46832dc8361b |
|       metadata      |                  {}                  |
|         size        |                  20                  |
|     snapshot_id     |                 None                 |
|     source_volid    |                 None                 |
|        status       |               creating               |
|     volume_type     |                 None                 |
+---------------------+--------------------------------------+

[root@localhost boris(keystone_admin)]# cinder list
+--------------------------------------+-------------+--------------+------+-------------+----------+--------------------------------------+
|                  ID                  |    Status   | Display Name | Size | Volume Type | Bootable |             Attached to              |
+--------------------------------------+-------------+--------------+------+-------------+----------+--------------------------------------+
| 5812c7e5-d071-43cb-8a1c-81346167d72a | downloading |  WinSRV2012  |  20  |     None    |  false   |                                      |
| be59b5ae-863b-41e2-abf6-de6756398722 |    in-use   | SalamanderVL |  5   |     None    |   true   | 82ba6ffe-dc01-4fcc-b247-7995a5ca7cb8 |
+--------------------------------------+-------------+--------------+------+-------------+----------+--------------------------------------+
[root@localhost boris(keystone_admin)]# ls -lah /var/lib/cinder/volumes/1d42fffa70f6647a45514f6fb5ce40ca
total 16G
drwxr-xr-x. 3 root   root   4.0K Oct 22 10:03 .
drwxr-xr-x. 5 cinder cinder 4.0K Oct 20 16:30 ..
-rw-rw-rw-. 1 root   root    16G Oct 22 10:21 volume-5812c7e5-d071-43cb-8a1c-81346167d72a
-rw-rw-rw-. 1 root   root   5.0G Oct 21 11:12 volume-be59b5ae-863b-41e2-abf6-de6756398722

[root@localhost boris(keystone_admin)]# du | grep node.$
5517480    ./node1
5516684    ./node3
5514680    ./node2
[root@localhost boris(keystone_admin)]# du | grep node.$
6551272    ./node1
6550168    ./node3
6549416    ./node2
[root@localhost boris(keystone_admin)]# cinder list
+--------------------------------------+-----------+--------------+------+-------------+----------+--------------------------------------+
|                  ID                  |   Status  | Display Name | Size | Volume Type | Bootable |             Attached to              |
+--------------------------------------+-----------+--------------+------+-------------+----------+--------------------------------------+
| 5812c7e5-d071-43cb-8a1c-81346167d72a | available |  WinSRV2012  |  20  |     None    |   true   |                                      |
| be59b5ae-863b-41e2-abf6-de6756398722 |   in-use  | SalamanderVL |  5   |     None    |   true   | 82ba6ffe-dc01-4fcc-b247-7995a5ca7cb8 |
+--------------------------------------+-----------+--------------+------+-------------+----------+--------------------------------------+
[root@localhost boris(keystone_admin)]# ls -lah /var/lib/cinder/volumes/1d42fffa70f6647a45514f6fb5ce40ca
total 19G
drwxr-xr-x. 3 root   root   4.0K Oct 22 10:03 .
drwxr-xr-x. 5 cinder cinder 4.0K Oct 20 16:30 ..
-rw-rw-rw-. 1 root   root    20G Oct 22 10:24 volume-5812c7e5-d071-43cb-8a1c-81346167d72a
-rw-rw-rw-. 1 root   root   5.0G Oct 21 11:12 volume-be59b5ae-863b-41e2-abf6-de6756398722
[root@localhost boris(keystone_admin)]# df -h
Filesystem                    Size  Used Avail Use% Mounted on
/dev/mapper/fedora-root       164G   70G   86G  45% /
devtmpfs                      3.9G     0  3.9G   0% /dev
tmpfs                         3.9G   92K  3.9G   1% /dev/shm
tmpfs                         3.9G  964K  3.9G   1% /run
tmpfs                         3.9G     0  3.9G   0% /sys/fs/cgroup
tmpfs                         3.9G  112K  3.9G   1% /tmp
/dev/sda1                     477M   87M  362M  20% /boot
/dev/loop0                    928M  1.4M  860M   1% /srv/node/device1
192.168.1.135:cinder-volumes  490G  209G  257G  45% /var/lib/cinder/volumes/1d42fffa70f6647a45514f6fb5ce40ca
tmpfs                         3.9G  964K  3.9G   1% /run/netns

[root@localhost boris(keystone_admin)]# cinder list
+--------------------------------------+--------+--------------+------+-------------+----------+--------------------------------------+
|                  ID                  | Status | Display Name | Size | Volume Type | Bootable |             Attached to              |
+--------------------------------------+--------+--------------+------+-------------+----------+--------------------------------------+
| 5812c7e5-d071-43cb-8a1c-81346167d72a | in-use |  WinSRV2012  |  20  |     None    |   true   | d6e702c3-16e6-4890-bbe7-22b19ed05263 |
| be59b5ae-863b-41e2-abf6-de6756398722 | in-use | SalamanderVL |  5   |     None    |   true   | 82ba6ffe-dc01-4fcc-b247-7995a5ca7cb8 |
+--------------------------------------+--------+--------------+------+-------------+----------+--------------------------------------+