Tuesday, August 23, 2016

Attempt to reproduce Deploying Kubernetes on Openstack using Heat by Ales Nosek

UPDATE 08/27/2016

I tested updated CentOS-7-x86_64-GenericCloud-1607.qcow2 with python2-boto 2.41 preinstalled.It eliminates "ERROR" during Master boot and provides the option to login into Master via ssh-keypair, exported in build environment. There is no any httpd daemon in SSL mode running into VM.Obviously https://Master-IP fails.

END UPDATE

Two Node Cluster Controller/Network/Compute  && Storage deployed on RDO
Mitaka.

====================================
Environment set up for kubernetes stack build via heat
====================================
[boris@CentOS72Server ~(keystone_build)]$ cat openrc.sh
unset OS_SERVICE_TOKEN
export OS_USERNAME=admin
export OS_PASSWORD=dda05d8fb4554e93
export OS_AUTH_URL=http://192.168.1.52:5000/v3
export PS1='[\u@\h \W(keystone_build)]\$ '

export OS_PROJECT_NAME=admin
export OS_USER_DOMAIN_NAME=Default
export OS_PROJECT_DOMAIN_NAME=Default
export OS_IDENTITY_API_VERSION=3
export OS_REGION_NAME=RegionOne
export OS_TENANT_ID=6e72c704971d4da3845f0ae9982bca6b

[boris@CentOS72Server ~(keystone_build)]$ cat openstack-heat.sh
export KUBERNETES_PROVIDER=openstack-heat
export STACK_NAME=kubernetes
export KUBERNETES_KEYPAIR_NAME=oskey082316
export NUMBER_OF_MINIONS=1
export MAX_NUMBER_OF_MINIONS=1
export EXTERNAL_NETWORK=public
export CREATE_IMAGE=false
export DOWNLOAD_IMAGE=false
export IMAGE_ID=7133dcf8-21a7-4beb-be1d-4a1f9d972cd8
export DNS_SERVER=83.221.202.254
export SWIFT_SERVER_URL=http://192.168.1.54:8080/v1/AUTH_6e72c704971d4da3845f0ae9982bca6b

1. Storage node separated during packstack deployment ( localhost:8080 causes issue on AIO box due to proxy-swift default endpoint )
2. SSL connection via horizon enabled in packstack deployment.
3. Security rules provide access to ports  443,80,22
========
Results
========
[root@CentOS72Server ~(keystone_admin)]# nova list
+--------------------------------------+--------------------------+--------+------------+-------------+---------------------------------------------------------------+
| ID                                   | Name                     | Status | Task State | Power State | Networks                                                      |
+--------------------------------------+--------------------------+--------+------------+-------------+---------------------------------------------------------------+
| f72bcec6-2def-4103-bb84-fcdc4a8af65e | CentOS72Devs01           | ACTIVE | -          | Running     | private=10.0.0.3, 192.168.1.150                               |
| 462e5122-fe5b-486e-8b1d-4379345271d6 | kubernetes-master        | ACTIVE | -          | Running     | kubernetes-fixed_network-htt6bujn7umv=10.0.0.3, 192.168.1.155 |
| 9c0f4e2c-1e9c-4370-8906-6b104b9bedbd | kubernetes-node-FhUQ6AJz | ACTIVE | -          | Running     | kubernetes-fixed_network-htt6bujn7umv=10.0.0.4, 192.168.1.156 |
+--------------------------------------+--------------------------+--------+------------+-------------+---------------------------------------------------------------+

[root@CentOS72Server ~(keystone_admin)]# openstack stack list
+------------------------+------------+-----------------+---------------------+--------------+
| ID                     | Stack Name | Stack Status    | Creation Time       | Updated Time |
+------------------------+------------+-----------------+---------------------+--------------+
| 57b4511f-d264-4a29     | kubernetes | CREATE_COMPLETE | 2016-08-23T14:29:43 | None         |
| -ab8c-9ce273a4d9bb     |            |                 |                     |              |
+------------------------+------------+-----------------+---------------------+--------------+
[root@CentOS72Server ~(keystone_admin)]# nova secgroup-list
+--------------------------------------+-----------------------------------------+------------------------+
| Id                                   | Name                                    | Description            |

+--------------------------------------+-----------------------------------------+------------------------+

| 9763cead-5816-40c5-a6e0-50a821347e52 | default                                 | Default security group |

| fc918814-db18-4be9-a319-4d8988b9060f | kubernetes-secgroup_base-7raauykt5owy   |                        |

| 29a1ff1d-be63-4bec-bac7-fdfa00a9c551 | kubernetes-secgroup_master-ztdnfr6paudu |                        |

| 08d5e1d7-0223-4acb-bf74-ed7230e98bf1 | kubernetes-secgroup_node-dt77fol3a7og   |                        |

+--------------------------------------+-----------------------------------------+------------------------+


[boris@CentOS72Server kubernetes(keystone_build)]$ ./cluster/kube-up.sh
... Starting cluster using provider: openstack-heat
... calling verify-prereqs
swift client installed
glance client installed
nova client installed
heat client installed
openstack client installed
... calling kube-up
kube-up for provider openstack-heat
[INFO] Execute commands to create Kubernetes cluster
[INFO] Uploading kubernetes-server-linux-amd64.tar.gz
kubernetes-server.tar.gz
[INFO] Uploading kubernetes-salt.tar.gz
kubernetes-salt.tar.gz
[INFO] Key pair already exists
Stack not found: kubernetes
[INFO] Create stack kubernetes
+---------------------+-------------------------------------------------------------------------+
| Field               | Value                                                                   |
+---------------------+-------------------------------------------------------------------------+
| id                  | 57b4511f-d264-4a29-ab8c-9ce273a4d9bb                                    |
| stack_name          | kubernetes                                                              |
| description         | Kubernetes cluster with one master and one or more worker nodes (as     |
|                     | specified by the number_of_minions parameter, which defaults to 3).     |
|                     |                                                                         |
| creation_time       | 2016-08-23T14:29:43                                                     |
| updated_time        | None                                                                    |
| stack_status        | CREATE_IN_PROGRESS                                                      |
| stack_status_reason |                                                                         |
+---------------------+-------------------------------------------------------------------------+

... calling validate-cluster
Cluster status CREATE_IN_PROGRESS
Cluster status CREATE_IN_PROGRESS
Cluster status CREATE_IN_PROGRESS
Cluster status CREATE_IN_PROGRESS
Cluster status CREATE_IN_PROGRESS
Cluster status CREATE_IN_PROGRESS
Cluster status CREATE_IN_PROGRESS
Cluster status CREATE_IN_PROGRESS
Cluster status CREATE_IN_PROGRESS
Cluster status CREATE_IN_PROGRESS
Cluster status CREATE_IN_PROGRESS
Cluster status CREATE_IN_PROGRESS
Cluster status CREATE_IN_PROGRESS
Cluster status CREATE_IN_PROGRESS
Cluster status CREATE_IN_PROGRESS
Cluster status CREATE_IN_PROGRESS
Cluster status CREATE_IN_PROGRESS
Cluster status CREATE_IN_PROGRESS
Cluster status CREATE_IN_PROGRESS
Cluster status CREATE_IN_PROGRESS
Cluster status CREATE_IN_PROGRESS
Cluster status CREATE_COMPLETE
cluster "openstack-kubernetes" set.
user "openstack-kubernetes" set.
context "openstack-kubernetes" set.
switched to context "openstack-kubernetes".
Wrote config for openstack-kubernetes to /home/boris/.kube/config
Done, listing cluster services:

The connection to the server 192.168.1.155 was refused - did you specify the right host or port?

[boris@CentOS72Server kubernetes(keystone_build)]$ cat  /home/boris/.kube/config
apiVersion: v1
clusters:
- cluster:
    insecure-skip-tls-verify: true
    server: https://192.168.1.155
  name: openstack-kubernetes
contexts:
- context:
    cluster: openstack-kubernetes
    user: openstack-kubernetes
  name: openstack-kubernetes
current-context: openstack-kubernetes
kind: Config
preferences: {}
users:
- name: openstack-kubernetes
  user:

  
 

=======
Finally
=======
[root@CentOS72Server ~(keystone_admin)]# neutron security-group-rule-create --protocol icmp  --direction ingress --remote-ip-prefix 0.0.0.0/0 fc918814-db18-4be9-a319-4d8988b9060f
Created a new security_group_rule:
+-------------------+--------------------------------------+
| Field             | Value                                |
+-------------------+--------------------------------------+
| description       |                                      |
| direction         | ingress                              |
| ethertype         | IPv4                                 |
| id                | 83e43587-1f6f-4f1b-b8b9-85e353b4d030 |
| port_range_max    |                                      |
| port_range_min    |                                      |
| protocol          | icmp                                 |
| remote_group_id   |                                      |
| remote_ip_prefix  | 0.0.0.0/0                            |
| security_group_id | fc918814-db18-4be9-a319-4d8988b9060f |
| tenant_id         | 6e72c704971d4da3845f0ae9982bca6b     |
+-------------------+--------------------------------------+

[root@CentOS72Server ~(keystone_admin)]# neutron security-group-rule-create --protocol icmp  --direction ingress --remote-ip-prefix 0.0.0.0/0 29a1ff1d-be63-4bec-bac7-fdfa00a9c551
Created a new security_group_rule:
+-------------------+--------------------------------------+
| Field             | Value                                |
+-------------------+--------------------------------------+
| description       |                                      |
| direction         | ingress                              |
| ethertype         | IPv4                                 |
| id                | 275f5b0b-4521-4b40-abb8-97bc1ab9566f |
| port_range_max    |                                      |
| port_range_min    |                                      |
| protocol          | icmp                                 |
| remote_group_id   |                                      |
| remote_ip_prefix  | 0.0.0.0/0                            |
| security_group_id | 29a1ff1d-be63-4bec-bac7-fdfa00a9c551 |
| tenant_id         | 6e72c704971d4da3845f0ae9982bca6b     |
+-------------------+--------------------------------------+

[root@CentOS72Server ~(keystone_admin)]# nova secgroup-list
+--------------------------------------+-----------------------------------------+------------------------+
| Id                                   | Name                                    | Description            |
+--------------------------------------+-----------------------------------------+------------------------+
| 9763cead-5816-40c5-a6e0-50a821347e52 | default                                 | Default security group |
| fc918814-db18-4be9-a319-4d8988b9060f | kubernetes-secgroup_base-7raauykt5owy   |                        |
| 29a1ff1d-be63-4bec-bac7-fdfa00a9c551 | kubernetes-secgroup_master-ztdnfr6paudu |                        |
| 08d5e1d7-0223-4acb-bf74-ed7230e98bf1 | kubernetes-secgroup_node-dt77fol3a7og   |                        |
+--------------------------------------+-----------------------------------------+------------------------+

[root@CentOS72Server ~(keystone_admin)]# neutron security-group-rule-create --protocol icmp  --direction ingress --remote-ip-prefix 0.0.0.0/0 08d5e1d7-0223-4acb-bf74-ed7230e98bf1
Created a new security_group_rule:
+-------------------+--------------------------------------+
| Field             | Value                                |
+-------------------+--------------------------------------+
| description       |                                      |
| direction         | ingress                              |
| ethertype         | IPv4                                 |
| id                | 8ef7ae78-42ff-4f82-baab-ce41e5e90cc8 |
| port_range_max    |                                      |
| port_range_min    |                                      |
| protocol          | icmp                                 |
| remote_group_id   |                                      |
| remote_ip_prefix  | 0.0.0.0/0                            |
| security_group_id | 08d5e1d7-0223-4acb-bf74-ed7230e98bf1 |
| tenant_id         | 6e72c704971d4da3845f0ae9982bca6b     |
+-------------------+--------------------------------------+

Can ping 192.168.1.155,192.168.1.156

Security rules for each kubernetes secgroup have ports 1 - 6535  open , however


==========================
 Kubernetes Master VM boot log contains
===========================

  
[[32m  OK  [0m] Started Update UTMP about System Runlevel Changes.
[  380.104758] cloud-init[4161]: [ERROR   ] boto_route53 requires at least boto 2.35.0.
[  455.439213] cloud-init[4161]: [ERROR   ] boto_route53 requires at least boto 2.35.0.
[  469.546079] cloud-init[4161]: [WARNING ] /usr/lib/python2.7/site-packages/salt/states/cmd.py:1041: DeprecationWarning: The legacy user/group arguments are deprecated. Replace them with runas. These arguments will be removed in Salt Oxygen.
[  521.559170] cloud-init[4161]: [WARNING ] State for file: /var/log/kube-apiserver.log - Neither 'source' nor 'contents' nor 'contents_pillar' nor 'contents_grains' was defined, yet 'replace' was set to 'True'. As there is no source to replace the file with, 'replace' has been set to 'False' to avoid reading the file unnecessarily.
[  521.723063] cloud-init[4161]: [ERROR   ] boto_route53 requires at least boto 2.35.0.

Even if I checkout branch :-

$ git clone https://github.com/kubernetes/kubernetes.git 
$ cd kubernetes 
$ git checkout origin/release-1.3.0
$ make quick-release
 
Same error in Master VM boot log.

I believe CentOS 7.2 image has to be updated up to python2-boto 2.41 via EPEL 7 during cloud-init run ( first boot )


  References
  http://alesnosek.com/blog/2016/06/26/deploying-kubernetes-on-openstack-using-heat/

Sunday, August 21, 2016

Emulation Triple0 QuickStart HA Controller's Cluster failover

Procedure bellow identify Controller which has RouterDSA in active state and
shutdown/startup  this Controller ( controller-1 in particular case).
Then  log into conntroller-1 and restart pcs cluster on particular  Controller,
afterwards  runs `pacemaker resource cleanup` for several resources what
results bringing back cluster nodes in proper status


 
[root@overcloud-controller-0 ~]# neutron l3-agent-list-hosting-router RouterDSA
+-----------------------------+-----------------------------+----------------+-------+----------+
| id                          | host                        | admin_state_up | alive | ha_state |
+-----------------------------+-----------------------------+----------------+-------+----------+
| 558fe2d4-a709-482f-         | overcloud-                  | True           | :-)   | active   |
| 85f2-9bb9835cf360           | controller-1.localdomain    |           |       |              |
| ae0f67ce-732b-                  | overcloud-                  | True           | :-)   | standby  |
| 4cb2-9b52-d15c22211972      | controller-0.localdomain    |          |       |          |
| fd9bfd34-9e36-4dac-a350-d18 | overcloud-         | True           | :-)   | standby  |
| fd1c3489b                      | controller-2.localdomain    |            |       |                |
+-----------------------------+-----------------------------+----------------+-------+----------+
[root@overcloud-controller-0 ~]# logout
[heat-admin@overcloud-controller-0 ~]$ logout
Connection to 192.0.2.16 closed.
[stack@undercloud ~]$ nova list
+--------------------------------------+-------------------------+--------+------------+-------------+---------------------+
| ID                                   | Name                    | Status | Task State | Power State | Networks            |
+--------------------------------------+-------------------------+--------+------------+-------------+---------------------+
| 5387385d-69a1-40ab-a77a-40d97949dc16 | overcloud-controller-0  | ACTIVE | -          | Running     | ctlplane=192.0.2.16 |
| 456031a7-21c4-497f-a7d8-baa3d403ee2f | overcloud-controller-1  | ACTIVE | -          | Running     | ctlplane=192.0.2.14 |
| 80b6ce3a-23a0-42d3-a1b3-fec22ca8f615 | overcloud-controller-2  | ACTIVE | -          | Running     | ctlplane=192.0.2.17 |
| b5a8c17c-e170-4f66-a5dd-846546afcfce | overcloud-novacompute-0 | ACTIVE | -          | Running     | ctlplane=192.0.2.13 |
| c10e25b3-6732-4afb-b51c-5d9f859bd7d6 | overcloud-novacompute-1 | ACTIVE | -          | Running     | ctlplane=192.0.2.15 |
+--------------------------------------+-------------------------+--------+------------+-------------+---------------------+
[stack@undercloud ~]$ nova stop overcloud-controller-1
Request to stop server overcloud-controller-1 has been accepted.
[stack@undercloud ~]$ nova list
+--------------------------------------+-------------------------+---------+------------+-------------+---------------------+
| ID                                   | Name                    | Status  | Task State | Power State | Networks            |
+--------------------------------------+-------------------------+---------+------------+-------------+---------------------+
| 5387385d-69a1-40ab-a77a-40d97949dc16 | overcloud-controller-0  | ACTIVE  | -          | Running     | ctlplane=192.0.2.16 |
| 456031a7-21c4-497f-a7d8-baa3d403ee2f | overcloud-controller-1  | SHUTOFF | -          | Shutdown    | ctlplane=192.0.2.14 |
| 80b6ce3a-23a0-42d3-a1b3-fec22ca8f615 | overcloud-controller-2  | ACTIVE  | -          | Running     | ctlplane=192.0.2.17 |
| b5a8c17c-e170-4f66-a5dd-846546afcfce | overcloud-novacompute-0 | ACTIVE  | -          | Running     | ctlplane=192.0.2.13 |
| c10e25b3-6732-4afb-b51c-5d9f859bd7d6 | overcloud-novacompute-1 | ACTIVE  | -          | Running     | ctlplane=192.0.2.15 |
+--------------------------------------+-------------------------+---------+------------+-------------+---------------------+
[stack@undercloud ~]$ nova start overcloud-controller-1
Request to start server overcloud-controller-1 has been accepted.
[stack@undercloud ~]$ nova list
+--------------------------------------+-------------------------+--------+------------+-------------+---------------------+
| ID                                   | Name                    | Status | Task State | Power State | Networks            |
+--------------------------------------+-------------------------+--------+------------+-------------+---------------------+
| 5387385d-69a1-40ab-a77a-40d97949dc16 | overcloud-controller-0  | ACTIVE | -          | Running     | ctlplane=192.0.2.16 |
| 456031a7-21c4-497f-a7d8-baa3d403ee2f | overcloud-controller-1  | ACTIVE | -          | Running     | ctlplane=192.0.2.14 |
| 80b6ce3a-23a0-42d3-a1b3-fec22ca8f615 | overcloud-controller-2  | ACTIVE | -          | Running     | ctlplane=192.0.2.17 |
| b5a8c17c-e170-4f66-a5dd-846546afcfce | overcloud-novacompute-0 | ACTIVE | -          | Running     | ctlplane=192.0.2.13 |
| c10e25b3-6732-4afb-b51c-5d9f859bd7d6 | overcloud-novacompute-1 | ACTIVE | -          | Running     | ctlplane=192.0.2.15 |
+--------------------------------------+-------------------------+--------+------------+-------------+---------------------+

[stack@undercloud ~]$ ssh heat-admin@192.0.2.14
The authenticity of host '192.0.2.14 (192.0.2.14)' can't be established.
ECDSA key fingerprint is a3:e6:de:2e:2b:45:e4:33:3d:d0:75:e5:b7:7f:da:0a.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '192.0.2.14' (ECDSA) to the list of known hosts.

[heat-admin@overcloud-controller-1 ~]$ sudo su -
[root@overcloud-controller-1 ~]# pcs status
Cluster name: tripleo_cluster
Last updated: Sun Aug 21 15:12:39 2016        Last change: Sun Aug 21 13:24:42 2016 by root via cibadmin on overcloud-controller-1
Stack: corosync
Current DC: overcloud-controller-0 (version 1.1.13-10.el7_2.4-44eb2dd) - partition with quorum
3 nodes and 127 resources configured

Online: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]

Full list of resources:

 ip-192.0.2.12    (ocf::heartbeat:IPaddr2):    Started overcloud-controller-0
 ip-172.16.2.5    (ocf::heartbeat:IPaddr2):    Started overcloud-controller-2
 ip-172.16.3.4    (ocf::heartbeat:IPaddr2):    Started overcloud-controller-2
 Clone Set: haproxy-clone [haproxy]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Master/Slave Set: galera-master [galera]
     Masters: [ overcloud-controller-0 overcloud-controller-2 ]
     Slaves: [ overcloud-controller-1 ]
 Clone Set: memcached-clone [memcached]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 ip-10.0.0.4    (ocf::heartbeat:IPaddr2):    Started overcloud-controller-0
 ip-172.16.2.4    (ocf::heartbeat:IPaddr2):    Started overcloud-controller-0
 ip-172.16.1.4    (ocf::heartbeat:IPaddr2):    Started overcloud-controller-2
 Clone Set: rabbitmq-clone [rabbitmq]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-core-clone [openstack-core]
     Started: [ overcloud-controller-0 overcloud-controller-2 ]
     Stopped: [ overcloud-controller-1 ]
 Master/Slave Set: redis-master [redis]
     Masters: [ overcloud-controller-0 ]
     Slaves: [ overcloud-controller-2 ]
     Stopped: [ overcloud-controller-1 ]
 Clone Set: mongod-clone [mongod]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-aodh-evaluator-clone [openstack-aodh-evaluator]
     Started: [ overcloud-controller-0 overcloud-controller-2 ]
     Stopped: [ overcloud-controller-1 ]
 Clone Set: openstack-nova-scheduler-clone [openstack-nova-scheduler]
     Started: [ overcloud-controller-0 overcloud-controller-2 ]
     Stopped: [ overcloud-controller-1 ]
 Clone Set: neutron-l3-agent-clone [neutron-l3-agent]
     Stopped: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: neutron-netns-cleanup-clone [neutron-netns-cleanup]
     Started: [ overcloud-controller-0 overcloud-controller-2 ]
     Stopped: [ overcloud-controller-1 ]
 Clone Set: neutron-ovs-cleanup-clone [neutron-ovs-cleanup]
     Started: [ overcloud-controller-0 overcloud-controller-2 ]
     Stopped: [ overcloud-controller-1 ]
 openstack-cinder-volume    (systemd:openstack-cinder-volume):    Stopped
 Clone Set: openstack-heat-engine-clone [openstack-heat-engine]
     Stopped: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-ceilometer-api-clone [openstack-ceilometer-api]
     Started: [ overcloud-controller-0 overcloud-controller-2 ]
     Stopped: [ overcloud-controller-1 ]
 Clone Set: openstack-aodh-listener-clone [openstack-aodh-listener]
     Stopped: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: neutron-metadata-agent-clone [neutron-metadata-agent]
     Stopped: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-gnocchi-metricd-clone [openstack-gnocchi-metricd]
     Started: [ overcloud-controller-0 overcloud-controller-2 ]
     Stopped: [ overcloud-controller-1 ]
 Clone Set: openstack-aodh-notifier-clone [openstack-aodh-notifier]
     Stopped: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-heat-api-clone [openstack-heat-api]
     Started: [ overcloud-controller-0 overcloud-controller-2 ]
     Stopped: [ overcloud-controller-1 ]
 Clone Set: openstack-ceilometer-collector-clone [openstack-ceilometer-collector]
     Started: [ overcloud-controller-0 overcloud-controller-2 ]
     Stopped: [ overcloud-controller-1 ]
 Clone Set: openstack-glance-api-clone [openstack-glance-api]
     Stopped: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-cinder-scheduler-clone [openstack-cinder-scheduler]
     Started: [ overcloud-controller-0 overcloud-controller-2 ]
     Stopped: [ overcloud-controller-1 ]
 Clone Set: openstack-nova-api-clone [openstack-nova-api]
     Started: [ overcloud-controller-0 overcloud-controller-2 ]
     Stopped: [ overcloud-controller-1 ]
 Clone Set: openstack-nova-consoleauth-clone [openstack-nova-consoleauth]
     Started: [ overcloud-controller-0 overcloud-controller-2 ]
     Stopped: [ overcloud-controller-1 ]
 Clone Set: openstack-sahara-api-clone [openstack-sahara-api]
     Started: [ overcloud-controller-0 overcloud-controller-2 ]
     Stopped: [ overcloud-controller-1 ]
 Clone Set: openstack-heat-api-cloudwatch-clone [openstack-heat-api-cloudwatch]
     Started: [ overcloud-controller-0 overcloud-controller-2 ]
     Stopped: [ overcloud-controller-1 ]
 Clone Set: openstack-sahara-engine-clone [openstack-sahara-engine]
     Started: [ overcloud-controller-0 ]
     Stopped: [ overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-glance-registry-clone [openstack-glance-registry]
     Started: [ overcloud-controller-0 overcloud-controller-2 ]
     Stopped: [ overcloud-controller-1 ]
 Clone Set: openstack-gnocchi-statsd-clone [openstack-gnocchi-statsd]
     Started: [ overcloud-controller-0 overcloud-controller-2 ]
     Stopped: [ overcloud-controller-1 ]
 Clone Set: openstack-ceilometer-notification-clone [openstack-ceilometer-notification]
     Started: [ overcloud-controller-0 overcloud-controller-2 ]
     Stopped: [ overcloud-controller-1 ]
 Clone Set: openstack-cinder-api-clone [openstack-cinder-api]
     Started: [ overcloud-controller-0 overcloud-controller-2 ]
     Stopped: [ overcloud-controller-1 ]
 Clone Set: neutron-dhcp-agent-clone [neutron-dhcp-agent]
     Started: [ overcloud-controller-0 overcloud-controller-2 ]
     Stopped: [ overcloud-controller-1 ]
 Clone Set: neutron-openvswitch-agent-clone [neutron-openvswitch-agent]
     Started: [ overcloud-controller-0 overcloud-controller-2 ]
     Stopped: [ overcloud-controller-1 ]
 Clone Set: openstack-nova-novncproxy-clone [openstack-nova-novncproxy]
     Started: [ overcloud-controller-0 overcloud-controller-2 ]
     Stopped: [ overcloud-controller-1 ]
 Clone Set: delay-clone [delay]
     Started: [ overcloud-controller-0 overcloud-controller-2 ]
     Stopped: [ overcloud-controller-1 ]
 Clone Set: neutron-server-clone [neutron-server]
     Started: [ overcloud-controller-0 overcloud-controller-2 ]
     Stopped: [ overcloud-controller-1 ]
 Clone Set: openstack-ceilometer-central-clone [openstack-ceilometer-central]
     Started: [ overcloud-controller-0 overcloud-controller-2 ]
     Stopped: [ overcloud-controller-1 ]
 Clone Set: httpd-clone [httpd]
     Started: [ overcloud-controller-0 overcloud-controller-2 ]
     Stopped: [ overcloud-controller-1 ]
 Clone Set: openstack-heat-api-cfn-clone [openstack-heat-api-cfn]
     Started: [ overcloud-controller-0 overcloud-controller-2 ]
     Stopped: [ overcloud-controller-1 ]
 Clone Set: openstack-nova-conductor-clone [openstack-nova-conductor]
     Started: [ overcloud-controller-0 overcloud-controller-2 ]
     Stopped: [ overcloud-controller-1 ]

Failed Actions:
* rabbitmq_monitor_10000 on overcloud-controller-0 'not running' (7): call=81, status=complete, exitreason='none',
    last-rc-change='Sun Aug 21 15:11:13 2016', queued=0ms, exec=0ms
* rabbitmq_monitor_10000 on overcloud-controller-2 'not running' (7): call=79, status=complete, exitreason='none',
    last-rc-change='Sun Aug 21 15:11:13 2016', queued=0ms, exec=0ms


PCSD Status:
  overcloud-controller-0: Online
  overcloud-controller-1: Online
  overcloud-controller-2: Online

Daemon Status:
  corosync: active/enabled
  pacemaker: active/enabled
  pcsd: active/enabled
[root@overcloud-controller-1 ~]# pcs cluster stop
Stopping Cluster (pacemaker)... Stopping Cluster (corosync)...
[root@overcloud-controller-1 ~]# pcs cluster start
Starting Cluster...
[root@overcloud-controller-1 ~]#
Broadcast message from systemd-journald@overcloud-controller-1.localdomain (Sun 2016-08-21 15:16:07 UTC):

haproxy[16997]: proxy nova_ec2 has no server available!

======================================
Script start.sh  [ 1 ]
======================================
#!/bin/bash -x
pcs resource cleanup rabbitmq-clone ;
sleep 10
pcs resource cleanup neutron-server-clone ;
sleep 10
pcs resource cleanup openstack-nova-api-clone ;
sleep 10
pcs resource cleanup openstack-nova-consoleauth-clone ;
sleep 10
pcs resource cleanup openstack-heat-engine-clone ;
sleep 10
pcs resource cleanup openstack-cinder-api-clone ;
sleep 10
pcs resource cleanup openstack-glance-registry-clone ;
sleep 10
pcs resource cleanup httpd-clone
=======================================


[root@overcloud-controller-1 ~]# . ./start.sh
Waiting for 3 replies from the CRMd... OK
Cleaning up rabbitmq:0 on overcloud-controller-0, removing fail-count-rabbitmq
Cleaning up rabbitmq:0 on overcloud-controller-1, removing fail-count-rabbitmq
Cleaning up rabbitmq:0 on overcloud-controller-2, removing fail-count-rabbitmq

Waiting for 3 replies from the CRMd... OK
Cleaning up neutron-server:0 on overcloud-controller-0, removing fail-count-neutron-server
Cleaning up neutron-server:0 on overcloud-controller-1, removing fail-count-neutron-server
Cleaning up neutron-server:0 on overcloud-controller-2, removing fail-count-neutron-server

Waiting for 3 replies from the CRMd... OK
Cleaning up openstack-nova-api:0 on overcloud-controller-0, removing fail-count-openstack-nova-api
Cleaning up openstack-nova-api:0 on overcloud-controller-1, removing fail-count-openstack-nova-api
Cleaning up openstack-nova-api:0 on overcloud-controller-2, removing fail-count-openstack-nova-api

Waiting for 3 replies from the CRMd... OK
Cleaning up openstack-nova-consoleauth:0 on overcloud-controller-0, removing fail-count-openstack-nova-consoleauth
Cleaning up openstack-nova-consoleauth:0 on overcloud-controller-1, removing fail-count-openstack-nova-consoleauth
Cleaning up openstack-nova-consoleauth:0 on overcloud-controller-2, removing fail-count-openstack-nova-consoleauth

Waiting for 3 replies from the CRMd... OK
Cleaning up openstack-heat-engine:0 on overcloud-controller-0, removing fail-count-openstack-heat-engine
Cleaning up openstack-heat-engine:0 on overcloud-controller-1, removing fail-count-openstack-heat-engine
Cleaning up openstack-heat-engine:0 on overcloud-controller-2, removing fail-count-openstack-heat-engine

Waiting for 3 replies from the CRMd... OK
Cleaning up openstack-cinder-api:0 on overcloud-controller-0, removing fail-count-openstack-cinder-api
Cleaning up openstack-cinder-api:0 on overcloud-controller-1, removing fail-count-openstack-cinder-api
Cleaning up openstack-cinder-api:0 on overcloud-controller-2, removing fail-count-openstack-cinder-api

Waiting for 3 replies from the CRMd... OK
Cleaning up openstack-glance-registry:0 on overcloud-controller-0, removing fail-count-openstack-glance-registry
Cleaning up openstack-glance-registry:0 on overcloud-controller-1, removing fail-count-openstack-glance-registry
Cleaning up openstack-glance-registry:0 on overcloud-controller-2, removing fail-count-openstack-glance-registry

Waiting for 3 replies from the CRMd... OK
Cleaning up httpd:0 on overcloud-controller-0, removing fail-count-httpd
Cleaning up httpd:0 on overcloud-controller-1, removing fail-count-httpd
Cleaning up httpd:0 on overcloud-controller-2, removing fail-count-httpd

[root@overcloud-controller-1 ~]# pcs status
Cluster name: tripleo_cluster
Last updated: Sun Aug 21 15:18:04 2016        Last change: Sun Aug 21 15:17:57 2016 by hacluster via crmd on overcloud-controller-0
Stack: corosync
Current DC: overcloud-controller-0 (version 1.1.13-10.el7_2.4-44eb2dd) - partition with quorum
3 nodes and 127 resources configured

Online: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]

Full list of resources:

 ip-192.0.2.12    (ocf::heartbeat:IPaddr2):    Started overcloud-controller-0
 ip-172.16.2.5    (ocf::heartbeat:IPaddr2):    Started overcloud-controller-2
 ip-172.16.3.4    (ocf::heartbeat:IPaddr2):    Started overcloud-controller-2
 Clone Set: haproxy-clone [haproxy]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Master/Slave Set: galera-master [galera]
     Masters: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: memcached-clone [memcached]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 ip-10.0.0.4    (ocf::heartbeat:IPaddr2):    Started overcloud-controller-0
 ip-172.16.2.4    (ocf::heartbeat:IPaddr2):    Started overcloud-controller-0
 ip-172.16.1.4    (ocf::heartbeat:IPaddr2):    Started overcloud-controller-2
 Clone Set: rabbitmq-clone [rabbitmq]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-core-clone [openstack-core]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Master/Slave Set: redis-master [redis]
     Masters: [ overcloud-controller-0 ]
     Slaves: [ overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: mongod-clone [mongod]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-aodh-evaluator-clone [openstack-aodh-evaluator]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-nova-scheduler-clone [openstack-nova-scheduler]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: neutron-l3-agent-clone [neutron-l3-agent]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: neutron-netns-cleanup-clone [neutron-netns-cleanup]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: neutron-ovs-cleanup-clone [neutron-ovs-cleanup]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 openstack-cinder-volume    (systemd:openstack-cinder-volume):    Started overcloud-controller-0
 Clone Set: openstack-heat-engine-clone [openstack-heat-engine]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-ceilometer-api-clone [openstack-ceilometer-api]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-aodh-listener-clone [openstack-aodh-listener]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: neutron-metadata-agent-clone [neutron-metadata-agent]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-gnocchi-metricd-clone [openstack-gnocchi-metricd]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-aodh-notifier-clone [openstack-aodh-notifier]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-heat-api-clone [openstack-heat-api]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-ceilometer-collector-clone [openstack-ceilometer-collector]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-glance-api-clone [openstack-glance-api]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-cinder-scheduler-clone [openstack-cinder-scheduler]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-nova-api-clone [openstack-nova-api]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-nova-consoleauth-clone [openstack-nova-consoleauth]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-sahara-api-clone [openstack-sahara-api]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-heat-api-cloudwatch-clone [openstack-heat-api-cloudwatch]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-sahara-engine-clone [openstack-sahara-engine]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-glance-registry-clone [openstack-glance-registry]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-gnocchi-statsd-clone [openstack-gnocchi-statsd]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-ceilometer-notification-clone [openstack-ceilometer-notification]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-cinder-api-clone [openstack-cinder-api]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: neutron-dhcp-agent-clone [neutron-dhcp-agent]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: neutron-openvswitch-agent-clone [neutron-openvswitch-agent]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-nova-novncproxy-clone [openstack-nova-novncproxy]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: delay-clone [delay]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: neutron-server-clone [neutron-server]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-ceilometer-central-clone [openstack-ceilometer-central]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: httpd-clone [httpd]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-heat-api-cfn-clone [openstack-heat-api-cfn]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-nova-conductor-clone [openstack-nova-conductor]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]

PCSD Status:
  overcloud-controller-0: Online
  overcloud-controller-1: Online
  overcloud-controller-2: Online

Daemon Status:
  corosync: active/enabled
  pacemaker: active/enabled
  pcsd: active/enabled
[root@overcloud-controller-1 ~]# logout
[heat-admin@overcloud-controller-1 ~]$ logout
Connection to 192.0.2.14 closed.
[stack@undercloud ~]$ ssh heat-admin@192.0.2.16
Last login: Sun Aug 21 15:08:18 2016 from 192.0.2.1
[heat-admin@overcloud-controller-0 ~]$ sudo su -
Last login: Sun Aug 21 15:08:24 UTC 2016 on pts/0
[root@overcloud-controller-0 ~]# .  keysstonerc_admin
[root@overcloud-controller-0 ~]# neutron l3-agent-list-hosting-router RouterDSA
+-----------------------------+-----------------------------+----------------+-------+----------+
| id                          | host                        | admin_state_up | alive | ha_state |
+-----------------------------+-----------------------------+----------------+-------+----------+
| 558fe2d4-a709-482f-         | overcloud-                  | True      | :-)   | standby  |
| 85f2-9bb9835cf360           | controller-1.localdomain    |      |       |                |
| ae0f67ce-732b-              | overcloud-                  | True           | :-)  | standby  |
| 4cb2-9b52-d15c22211972      | controller-0.localdomain    |          |       |       |
| fd9bfd34-9e36-4dac-a350-d18 | overcloud-                  | True   :-)   | active   |
| fd1c3489b                   | controller-2.localdomain        |                |     |          |
+-----------------------------+-----------------------------+----------------+-------+----------+

====================================
Verification Galera DB sync on Controllers
====================================

[root@overcloud-controller-0 ~]# clustercheck
HTTP/1.1 200 OK
Content-Type: text/plain
Connection: close
Content-Length: 32

Galera cluster node is synced.

[root@overcloud-controller-0 ~]# logout
[heat-admin@overcloud-controller-0 ~]$ logout
Connection to 192.0.2.16 closed.

[stack@undercloud ~]$ ssh heat-admin@192.0.2.14
Last login: Sun Aug 21 15:12:27 2016 from 192.0.2.1
[heat-admin@overcloud-controller-1 ~]$ sudo su -
Last login: Sun Aug 21 15:12:34 UTC 2016 on pts/0

[root@overcloud-controller-1 ~]# clustercheck
HTTP/1.1 200 OK
Content-Type: text/plain
Connection: close
Content-Length: 32

Galera cluster node is synced.


 ==================
 Setup details
==================

[boris@fedora24wks tripleo-quickstart]$ cat ./config/general_config/ha.yml
# Deploy an HA openstack environment.
#
# This will require (6144 * 4) == approx. 24GB for the overcloud
# nodes, plus another 8GB for the undercloud, for a total of around
# 32GB.
control_memory: 6144
compute_memory: 6144
default_vcpu: 2

undercloud_memory: 8192

# Giving the undercloud additional CPUs can greatly improve heat's
# performance (and result in a shorter deploy time).
undercloud_vcpu: 2

# Create three controller nodes and one compute node.
overcloud_nodes:
  - name: control_0
    flavor: control
  - name: control_1
    flavor: control
  - name: control_2
    flavor: control

  - name: compute_0
    flavor: compute
  - name: compute_1
    flavor: compute

# We don't need introspection in a virtual environment (because we are
# creating all the "hardware" we really know the necessary
# information).
step_introspect: true

# Tell tripleo about our environment.
network_isolation: true
extra_args: >-
  --control-scale 3 --compute-scale 2 --neutron-network-type vxlan
  --neutron-tunnel-types vxlan
  --ntp-server pool.ntp.org
test_tempest: false
test_ping: true
enable_pacemaker: true

##################################
Virtual Environment Setup Complete
##################################

Access the undercloud by:

    ssh -F /home/boris/.quickstart/ssh.config.ansible undercloud

There are scripts in the home directory to continue the deploy:

    overcloud-deploy.sh will deploy the overcloud
    overcloud-deploy-post.sh will do any post-deploy configuration
    overcloud-validate.sh will run post-deploy validation

Alternatively, you can ignore these scripts and follow the upstream docs,
starting from the overcloud deploy section:

    http://ow.ly/1Vc1301iBlb

##################################
Virtual Environment Setup Complete
##################################

[boris@fedora24wks tripleo-quickstart]$ ssh -F /home/boris/.quickstart/ssh.config.ansible undercloud
Warning: Permanently added '192.168.1.74' (ECDSA) to the list of known hosts.
Warning: Permanently added 'undercloud' (ECDSA) to the list of known hosts.
Last login: Wed Aug 24 12:13:16 2016 from gateway
[stack@undercloud ~]$ sudo su


[root@undercloud stack]# cd /etc/yum.repos.d
[root@undercloud yum.repos.d]# ls -l
total 40
-rw-r--r--. 1 root root 1664 Dec  9  2015 CentOS-Base.repo
-rw-r--r--. 1 root root 1057 Aug 24 02:58 CentOS-Ceph-Hammer.repo
-rw-r--r--. 1 root root 1309 Dec  9  2015 CentOS-CR.repo
-rw-r--r--. 1 root root  649 Dec  9  2015 CentOS-Debuginfo.repo
-rw-r--r--. 1 root root  290 Dec  9  2015 CentOS-fasttrack.repo
-rw-r--r--. 1 root root  630 Dec  9  2015 CentOS-Media.repo
-rw-r--r--. 1 root root 1331 Dec  9  2015 CentOS-Sources.repo
-rw-r--r--. 1 root root 1952 Dec  9  2015 CentOS-Vault.repo
-rw-r--r--. 1 root root  162 Aug 24 02:58 delorean-deps.repo
-rw-r--r--. 1 root root  220 Aug 24 02:58 delorean.repo


====================================================
Delorean repos file been installed via quickstart on undercloud
====================================================
[root@undercloud yum.repos.d]# cat delorean-deps.repo
[delorean-mitaka-testing]
name=dlrn-mitaka-testing
baseurl=http://buildlogs.centos.org/centos/7/cloud/$basearch/openstack-mitaka/
enabled=1
gpgcheck=0
priority=2

[root@undercloud yum.repos.d]# cat delorean.repo
[delorean]
name=delorean-openstack-rally-3909299306233247d547bad265a1adb78adfb3d4
baseurl=http://trunk.rdoproject.org/centos7-mitaka/39/09/3909299306233247d547bad265a1adb78adfb3d4_4e6dfa3c
enabled=1
gpgcheck=0

Wednesday, August 17, 2016

TripleO QuickStart HA Setup && Keeping undercloud persistent between cold reboots ( newly polished )

UPDATE 08/21/216


In case when virt tools (virsh,virt-manger ) stop to recognise running
qemu-kvm of undercloud as VM  issue `sudo shutdown -P now` via connection
 `ssh -F /home/boris/.quickstart/ssh.config.ansible undercloud`.
It will result graceful shutdown of undercloud's qemu-kvm process on VIRTHOST.


END UPDATE 



This post follows up http://lxer.com/module/newswire/view/230814/index.html
and might work as timer saver unless status undecloud.qcow2 per
http://artifacts.ci.centos.org/artifacts/rdo/images/mitaka/delorean/stable/
requires fresh installation to be done from scratch.
Current update allows to automate procedure via /etc/rc.d/rc.local and exports
in stack's shell variables which allow to start virt-manager right away , presuming that xhost+ was issued in root's shell.

Thus, we intend to survive VIRTHOST cold reboot (downtime) and keep previous version of undercloud VM been able to bring it up avoiding build via quickstart.sh and restart procedure from logging into undercloud and immediately run overcloud deployment. Proceed as follows :-

1. System shutdown
    Cleanly commit :-
    [stack@undercloud~] $ openstack stack delete overcloud
2. Login into VIRTHOST as stack and gracefully shutdown undercloud
    [stack@ServerCentOS72 ~]$ virsh shutdown undercloud


=====================
 Make following updates
=====================

[root@ServerTQS72 ~]# cat /etc/rc.d/rc.local
#!/bin/bash
# Please note that you must run 'chmod +x /etc/rc.d/rc.local' to ensure
# that this script will be executed during boot.
mkdir -p /run/user/1001
chown -R stack /run/user/1001

if [ $? -ne 0 ]
then
       exit 0   
fi
chgrp -R stack /run/user/1001

touch /var/lock/subsys/local

========================
In stack's .bashrc
========================

[stack@ServerTQS72 ~]$ cat .bashrc
# .bashrc

# Source global definitions
if [ -f /etc/bashrc ]; then
    . /etc/bashrc
fi

# Uncomment the following line if you don't like systemctl's auto-paging feature:
# export SYSTEMD_PAGER=

# User specific aliases and functions
# BEGIN ANSIBLE MANAGED BLOCK
# Make sure XDG_RUNTIME_DIR is set (used by libvirt
# for creating config and sockets for qemu:///session
# connections)
: ${XDG_RUNTIME_DIR:=/run/user/$(id -u)}
export XDG_RUNTIME_DIR
export DISPLAY=:0.0
export NO_AT_BRIDGE=1

# END ANSIBLE MANAGED BLOCK

=================
REBOOT  VIRTHOST
=================

$ sudo su -
# xhost +
# su - stack

[stack@ServerTQS72 ~]$ virt-manager --connect qemu:///session

Start VM undercloud
 


  Virt-tools misbehavior (UPDATE 08/21/16) . Six qemu-kvm are up and running

   1. Undercloud
   2. 3 Node HA Controller (Pacemaker/Corosync) cluster
   3. 2 Compute Nodes (nested KVM enabled )

 

=====================================
 Log into undercloud from Ansible Server via  :-
=====================================
[boris@fedora24wks tripleo-quickstart]$ ssh -F /home/boris/.quickstart/ssh.config.ansible undercloud

Deploy overcloud using old overcloud-deploy.sh

# Deploy the overcloud!
openstack overcloud deploy \
 --templates /usr/share/openstack-tripleo-heat-templates \
 --libvirt-type qemu --control-flavor oooq_control --compute-flavor oooq_compute \
 --ceph-storage-flavor oooq_ceph --timeout 90 -\
 -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \
 -e /usr/share/openstack-tripleo-heat-templates/environments/net-single-nic-with-vlans.yaml \
 -e $HOME/network-environment.yaml \
 -e /usr/share/openstack-tripleo-heat-templates/environments/puppet-pacemaker.yaml \
 --control-scale 3 --compute-scale 2 \
 --neutron-network-type vxlan --neutron-tunnel-types vxlan --ntp-server pool.ntp.org \
    ${DEPLOY_ENV_YAML:+-e $DEPLOY_ENV_YAML} "$@" || true

# We don't always get a useful error code from the openstack deploy command,
# so check `heat stack-list` for a CREATE_FAILED status.
if heat stack-list | grep -q 'CREATE_FAILED'; then

    # get the failures list
    openstack stack failures list overcloud > failed_deployment_list.log || true

    # get any puppet related errors
    for failed in $(heat resource-list \
        --nested-depth 5 overcloud | grep FAILED |
        grep 'StructuredDeployment ' | cut -d '|' -f3)
    do
       echo "heat deployment-show out put for deployment: $failed" >> failed_deployments.log
       echo "######################################################" >> failed_deployments.log
       heat deployment-show $failed >> failed_deployments.log
       echo "######################################################" >> failed_deployments.log
       echo "puppet standard error for deployment: $failed" >> failed_deployments.log
       echo "######################################################" >> failed_deployments.log
       # the sed part removes color codes from the text
       heat deployment-show $failed |
           jq -r .output_values.deploy_stderr |
           sed -r "s:\x1B\[[0-9;]*[mK]::g" >> failed_deployments.log
       echo "######################################################" >> failed_deployments.log
    done
fi

[stack@undercloud ~]$ . stackrc
[stack@undercloud ~]$ nova list
+--------------------------------------+-------------------------+--------+------------+-------------+---------------------+
| ID                                   | Name                    | Status | Task State | Power State | Networks            |
+--------------------------------------+-------------------------+--------+------------+-------------+---------------------+
| b6f105e8-3854-4939-99d9-73c16cf233fd | overcloud-controller-0  | ACTIVE | -          | Running     | ctlplane=192.0.2.23 |
| 30979d6e-773b-4d79-9446-1cd25bade373 | overcloud-controller-1  | ACTIVE | -          | Running     | ctlplane=192.0.2.20 |
| 256627a8-2202-4986-86b6-8cd6e46c21db | overcloud-controller-2  | ACTIVE | -          | Running     | ctlplane=192.0.2.22 |
| 9dc029bf-b096-4be6-b5a3-14b39ac098a4 | overcloud-novacompute-0 | ACTIVE | -          | Running     | ctlplane=192.0.2.21 |
| 16d0e195-c6a2-4286-a368-6fe9851ccd82 | overcloud-novacompute-1 | ACTIVE | -          | Running     | ctlplane=192.0.2.19 |
+--------------------------------------+-------------------------+--------+------------+-------------+---------------------+

Sunday, August 14, 2016

Access to TripleO QuickStart overcloud via sshuttle running on F24 WorkStation

Sshutle may be installed on Fedora 24 via straight forward `dnf -y install sshutle` [Fedora 24 Update: sshuttle-0.78.0-2.fc24]. So, when F24 has been set up as WKS for TripleO QuickStart deployment to VIRTHOST , there is no need to install add-on FoxyProxy and tune it on firefox as well as connect from ansible wks to undercloud via  $ ssh -F ~/.quickstart/ssh.config.ansible undercloud -D 9090

What is sshuttle? It’s a Python app that uses SSH to create a quick and dirty VPN between your Linux, BSD, or Mac OS X machine and a remote system that has SSH access and Python. Been licensed under the GPLv2, sshuttle is a transparent proxy server that lets users fake a VPN with minimal hassle.

========================================
First install and start sshutle on Fedora 24 :-
========================================
boris@fedora24wks ~] dnf -y install sshutle
[root@fedora24wks ~]# rpm -qa \*sshuttle\*
sshuttle-0.78.0-2.fc24.noarch

========================================================
Now start sshutle via ssh.config.ansible, where 10.0.0.0/24 has been installed
as external network for OverCloud already been set up on VIRTHOST
========================================================
[boris@fedora24wks ~]$ sshuttle -e "ssh -F $HOME/.quickstart/ssh.config.ansible" -r undercloud  -v 10.0.0.0/24 &
[3] 16385
[boris@fedora24wks ~]$ Starting sshuttle proxy.
firewall manager: Starting firewall with Python version 3.5.1
firewall manager: ready method name nat.
IPv6 enabled: False
UDP enabled: False
DNS enabled: False
TCP redirector listening on ('127.0.0.1', 12299).
Starting client with Python version 3.5.1
c : connecting to server...
Warning: Permanently added '192.168.1.74' (ECDSA) to the list of known hosts.
Warning: Permanently added 'undercloud' (ECDSA) to the list of known hosts.
Starting server with Python version 2.7.5
 s: latency control setting = True
 s: available routes:
 s:   2/10.0.0.0/24
 s:   2/192.0.2.0/24
 s:   2/192.168.23.0/24
 s:   2/192.168.122.0/24
c : Connected.
firewall manager: setting up.
>> iptables -t nat -N sshuttle-12299
>> iptables -t nat -F sshuttle-12299
>> iptables -t nat -I OUTPUT 1 -j sshuttle-12299
>> iptables -t nat -I PREROUTING 1 -j sshuttle-12299
>> iptables -t nat -A sshuttle-12299 -j REDIRECT --dest 10.0.0.0/24 -p tcp --to-ports 12299 -m ttl ! --ttl 42
>> iptables -t nat -A sshuttle-12299 -j RETURN --dest 127.0.0.1/8 -p tcp
c : Accept TCP: 192.168.1.13:53068 -> 10.0.0.4:80.
c : warning: closed channel 1 got cmd=TCP_STOP_SENDING len=0
c : Accept TCP: 192.168.1.13:53072 -> 10.0.0.4:80.
 s: SW'unknown':Mux#1: deleting (3 remain)
 s: SW#6:10.0.0.4:80: deleting (2 remain)
c : warning: closed channel 2 got cmd=TCP_STOP_SENDING len=0
c : Accept TCP: 192.168.1.13:53074 -> 10.0.0.4:80.
 s: SW'unknown':Mux#2: deleting (3 remain)
 s: SW#7:10.0.0.4:80: deleting (2 remain)
c : Accept TCP: 192.168.1.13:58210 -> 10.0.0.4:6080.
c : Accept TCP: 192.168.1.13:58212 -> 10.0.0.4:6080.
c : SW'unknown':Mux#2: deleting (9 remain)
c : SW#11:192.168.1.13:53072: deleting (8 remain)
c : SW'unknown':Mux#1: deleting (7 remain)
c : SW#9:192.168.1.13:53068: deleting (6 remain)
c : Accept TCP: 192.168.1.13:58214 -> 10.0.0.4:6080.
c : Accept TCP: 192.168.1.13:58216 -> 10.0.0.4:6080.
c : warning: closed channel 4 got cmd=TCP_STOP_SENDING len=0
 s: warning: closed channel 4 got cmd=TCP_STOP_SENDING len=0

Complete log may be seen here



This creates a transparent proxy server on your local machine for all IP addresses that match 10.0.0.0/24. Any TCP session you initiate to one of the proxied IP addresses will be captured by sshuttle and sent over an ssh session to the remote copy of sshuttle, which will then regenerate the connection on that end, and funnel the data back and forth through ssh. There is no need to install sshuttle on the remote server; the remote server just needs to have python available. sshuttle will automatically upload and run its source code to the remote python.

So,disable/remove  FoxyProxy add-on from firefox ( if it has been set up ); interrupt connection from work station to undercloud via `ssh -F ~/.quickstart/ssh.config.ansible undercloud -D 9090`. Restart firefox and launch browser to http://10.0.0.4/dashboard

 

  

   References
   1. http://g33kinfo.com/info/archives/5388

Sunday, July 31, 2016

Stable Mitaka HA instack-virt-setup on CentOS 7.2 VIRTHOST

Following is step by step self sufficient instruction performing
Mitaka HA instack-virt-setup on CentOS 7.2 VIRTHOST based on delorean
repos  :-
   http://trunk.rdoproject.org/centos7-mitaka/current/delorean.repo
   http://trunk.rdoproject.org/centos7-mitaka/delorean-deps.repo
It follows official guide lines and updates undercloud with OVSIntPort vlan10
for br-ctlplane OVS bridge making posible HA and/or Ceph overcloud deployments with "Network Isolation" enabled. See also an upstream commit https://review.openstack.org/#/c/329438/  been done by Marios Andreou on 06/14/2016

=========================================
VIRTHOST - stack's  .bashrc configuration
=========================================
# curl -o /etc/yum.repos.d/delorean-mitaka.repo \
    http://trunk.rdoproject.org/centos7-mitaka/current/delorean.repo
#  curl -o /etc/yum.repos.d/delorean-deps-mitaka.repo \
   http://trunk.rdoproject.org/centos7-mitaka/delorean-deps.repo
#  yum -y install yum-plugin-priorities
#  yum -y install epel-release

[stack@ServerCentOS7$ ~]$ env | grep NODE
NODE_MEM=6000
NODE_COUNT=5
UNDERCLOUD_NODE_CPU=2
NODE_CPU=2
NODE_DIST=centos7
UNDERCLOUD_NODE_MEM=7500
$ sudo yum install instack-undercloud
$ instack-virt-setup

===========================
INSTACK
===========================
[stack@instack ~]$  sudo curl -o /etc/yum.repos.d/delorean-mitaka.repo \
       http://trunk.rdoproject.org/centos7-mitaka/current/delorean.repo
[stack@instack ~]$  sudo curl -o /etc/yum.repos.d/delorean-deps-mitaka.repo \
       http://trunk.rdoproject.org/centos7-mitaka/delorean-deps.repo
[stack@instack ~]$  sudo yum -y install yum-plugin-priorities
[stack@instack ~]$ cat .bashrc
     # .bashrc
     # Source global definitions
     if [ -f /etc/bashrc ]; then
    . /etc/bashrc
     fi

     # Uncomment the following line if you don't like systemctl's auto-paging feature:
     # export SYSTEMD_PAGER=
     export NODE_DIST=centos7
     export USE_DELOREAN_TRUNK=1
     export DELOREAN_TRUNK_REPO="http://trunk.rdoproject.org/centos7-mitaka/current/"
     export DELOREAN_REPO_FILE="delorean.repo"
     # User specific aliases and functions

    $  sudo yum install -y python-tripleoclient
    $  openstack undercloud install
    $   source stackrc
    $   env | grep DEL
    $   source stackrc
    $  openstack overcloud image build --all
    $  openstack overcloud image upload
    $  openstack baremetal import instackenv.json
    $  openstack baremetal configure boot
    $  openstack baremetal introspection bulk start
    $  neutron subnet-list
    $  neutron subnet-update 1b7d82e5-0bf1-4ba5-8008-4aa402598065 --dns-nameserver 8.8.8.8
    $  sudo ovs-vsctl show
    $  sudo vi /etc/sysconfig/network-scripts/ifcfg-vlan10
       DEVICE=vlan10
       ONBOOT=yes
       DEVICETYPE=ovs
       TYPE=OVSIntPort
       BOOTPROTO=static
       IPADDR=10.0.0.1
       NETMASK=255.255.255.0
       OVS_BRIDGE=br-ctlplane
       OVS_OPTIONS="tag=10"

    $  sudo ifup vlan10
   [stack@instack ~]$ cat network_env.yaml
    {
    "parameter_defaults": {
        "ControlPlaneDefaultRoute": "192.0.2.1",
        "ControlPlaneSubnetCidr": "24",
        "DnsServers": [
            "192.168.122.43"
        ],
        "EC2MetadataIp": "192.0.2.1",
        "ExternalAllocationPools": [
            {
                "end": "10.0.0.250",
                "start": "10.0.0.4"
            }
        ],
        "ExternalNetCidr": "10.0.0.1/24",
        "NeutronExternalNetworkBridge": ""
    }
   }

Where 192.168.122.43 is instack VM Ip.

   $  sudo ifup ifcfg-vlan10
   $  sudo iptables -A BOOTSTACK_MASQ -s 10.0.0.0/24 ! -d 10.0.0.0/24 -j MASQUERADE -t nat

   $  sudo ovs-vsctl show
   $ [stack@instack ~]$ sudo ovs-vsctl show
    Bridge br-ctlplane
        Port "vlan10"
            tag: 10
            Interface "vlan10"
                type: internal
        Port "eth1"
            Interface "eth1"
        Port phy-br-ctlplane
            Interface phy-br-ctlplane
                type: patch
                options: {peer=int-br-ctlplane}
        Port br-ctlplane
            Interface br-ctlplane
                type: internal
    Bridge br-int
        fail_mode: secure
        Port int-br-ctlplane
            Interface int-br-ctlplane
                type: patch
                options: {peer=phy-br-ctlplane}
        Port br-int
            Interface br-int
                type: internal
        Port "tape8042136-81"
            tag: 1
            Interface "tape8042136-81"
                type: internal
    ovs_version: "2.5.0"
[stack@instack ~]$ ifconfig
br-ctlplane: flags=4163  mtu 1500
        inet 192.0.2.1  netmask 255.255.255.0  broadcast 192.0.2.255
        inet6 fe80::28c:73ff:fee7:a0c7  prefixlen 64  scopeid 0x20
        ether 00:8c:73:e7:a0:c7  txqueuelen 0  (Ethernet)
        RX packets 969827  bytes 64170341 (61.1 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 24905  bytes 1403706189 (1.3 GiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

eth0: flags=4163  mtu 1500
        inet 192.168.122.175  netmask 255.255.255.0  broadcast 192.168.122.255
        inet6 fe80::5054:ff:fee0:88ae  prefixlen 64  scopeid 0x20
        ether 52:54:00:e0:88:ae  txqueuelen 1000  (Ethernet)
        RX packets 792414  bytes 1147953477 (1.0 GiB)
        RX errors 0  dropped 2  overruns 0  frame 0
        TX packets 504124  bytes 41553838 (39.6 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

eth1: flags=4163  mtu 1500
        inet6 fe80::28c:73ff:fee7:a0c7  prefixlen 64  scopeid 0x20
        ether 00:8c:73:e7:a0:c7  txqueuelen 1000  (Ethernet)
        RX packets 969821  bytes 64170033 (61.1 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 24917  bytes 1403707133 (1.3 GiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10
        loop  txqueuelen 0  (Local Loopback)
        RX packets 345762  bytes 4297080640 (4.0 GiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 345762  bytes 4297080640 (4.0 GiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

vlan10: flags=4163  mtu 1500
        inet 10.0.0.1  netmask 255.255.255.0  broadcast 10.0.0.255
        inet6 fe80::b0f8:92ff:feed:99bb  prefixlen 64  scopeid 0x20
        ether b2:f8:92:ed:99:bb  txqueuelen 0  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 12  bytes 816 (816.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

$  vi overcloud-deploy.sh


     #!/bin/bash -x
     source /home/stack/stackrc    
     openstack overcloud deploy --templates --control-scale 3 \
     --compute-scale 2 \
     --libvirt-type qemu \
     --ntp-server pool.ntp.org  \
     -e  /usr/share/openstack-tripleo-heat-templates/environments/puppet-pacemaker.yaml \
     -e  /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \
     -e  /usr/share/openstack-tripleo-heat-templates/environments/net-single-nic-with-vlans.yaml \
     -e  $HOME/network_env.yaml
$  chmod a+x overcloud-deploy.sh
[stack@instack ~]$ ./overcloud-deploy.sh
Deploying templates in the directory /usr/share/openstack-tripleo-heat-templates
2016-07-31 05:39:20 [overcloud]: CREATE_IN_PROGRESS Stack CREATE started
2016-07-31 05:39:20 [Networks]: CREATE_IN_PROGRESS state changed
2016-07-31 05:39:20 [PcsdPassword]: CREATE_IN_PROGRESS state changed
2016-07-31 05:39:20 [HeatAuthEncryptionKey]: CREATE_IN_PROGRESS state changed
2016-07-31 05:39:20 [MysqlRootPassword]: CREATE_IN_PROGRESS state changed
2016-07-31 05:39:20 [MysqlClusterUniquePart]: CREATE_IN_PROGRESS state changed
2016-07-31 05:39:20 [RabbitCookie]: CREATE_IN_PROGRESS state changed
2016-07-31 05:39:20 [VipConfig]: CREATE_IN_PROGRESS state changed
2016-07-31 05:39:20 [HorizonSecret]: CREATE_IN_PROGRESS state changed
2016-07-31 05:39:20 [PcsdPassword]: CREATE_COMPLETE state changed
2016-07-31 05:39:20 [HeatAuthEncryptionKey]: CREATE_COMPLETE state changed
2016-07-31 05:39:20 [MysqlRootPassword]: CREATE_COMPLETE state changed
2016-07-31 05:39:20 [MysqlClusterUniquePart]: CREATE_COMPLETE state changed
2016-07-31 05:39:20 [overcloud-VipConfig-7kalyksojixl]: CREATE_IN_PROGRESS Stack CREATE started
2016-07-31 05:39:20 [VipConfigImpl]: CREATE_IN_PROGRESS state changed
2016-07-31 05:39:20 [overcloud-Networks-rto6netaoodk]: CREATE_IN_PROGRESS Stack CREATE started
2016-07-31 05:39:20 [ManagementNetwork]: CREATE_IN_PROGRESS state changed
2016-07-31 05:39:21 [RabbitCookie]: CREATE_COMPLETE state changed
2016-07-31 05:39:21 [HorizonSecret]: CREATE_COMPLETE state changed
2016-07-31 05:39:21 [VipConfigImpl]: CREATE_COMPLETE state changed
2016-07-31 05:39:21 [overcloud-VipConfig-7kalyksojixl]: CREATE_COMPLETE Stack CREATE completed successfully
2016-07-31 05:39:21 [ExternalNetwork]: CREATE_IN_PROGRESS state changed
2016-07-31 05:39:21 [InternalNetwork]: CREATE_IN_PROGRESS state changed
2016-07-31 05:39:21 [StorageNetwork]: CREATE_IN_PROGRESS state changed
2016-07-31 05:39:21 [StorageMgmtNetwork]: CREATE_IN_PROGRESS state changed
2016-07-31 05:39:21 [TenantNetwork]: CREATE_IN_PROGRESS state changed
2016-07-31 05:39:21 [overcloud-Networks-rto6netaoodk-ManagementNetwork-fbn47jr7bptq]: CREATE_IN_PROGRESS Stack CREATE started
2016-07-31 05:39:21 [overcloud-Networks-rto6netaoodk-ManagementNetwork-fbn47jr7bptq]: CREATE_COMPLETE Stack CREATE completed successfully
2016-07-31 05:39:21 [overcloud-Networks-rto6netaoodk-InternalNetwork-rwmlbfr5dzdk]: CREATE_IN_PROGRESS Stack CREATE started
2016-07-31 05:39:21 [InternalApiNetwork]: CREATE_IN_PROGRESS state changed
2016-07-31 05:39:21 [overcloud-Networks-rto6netaoodk-StorageMgmtNetwork-4e5xbdltuz7q]: CREATE_IN_PROGRESS Stack CREATE started
2016-07-31 05:39:21 [StorageMgmtNetwork]: CREATE_IN_PROGRESS state changed
2016-07-31 05:39:21 [overcloud-Networks-rto6netaoodk-StorageNetwork-e3tr6nhcwdss]: CREATE_IN_PROGRESS Stack CREATE started
2016-07-31 05:39:21 [StorageNetwork]: CREATE_IN_PROGRESS state changed
2016-07-31 05:39:21 [StorageNetwork]: CREATE_COMPLETE state changed
2016-07-31 05:39:21 [StorageSubnet]: CREATE_IN_PROGRESS state changed
2016-07-31 05:39:21 [overcloud-Networks-rto6netaoodk-ExternalNetwork-ge5f4ydo2yw6]: CREATE_IN_PROGRESS Stack CREATE started
2016-07-31 05:39:21 [ExternalNetwork]: CREATE_IN_PROGRESS state changed
2016-07-31 05:39:21 [ExternalNetwork]: CREATE_COMPLETE state changed
2016-07-31 05:39:21 [ExternalSubnet]: CREATE_IN_PROGRESS state changed
2016-07-31 05:39:22 [VipConfig]: CREATE_COMPLETE state changed
2016-07-31 05:39:22 [ManagementNetwork]: CREATE_COMPLETE state changed
2016-07-31 05:39:22 [InternalApiNetwork]: CREATE_COMPLETE state changed
2016-07-31 05:39:22 [InternalApiSubnet]: CREATE_IN_PROGRESS state changed
2016-07-31 05:39:22 [StorageMgmtNetwork]: CREATE_COMPLETE state changed
2016-07-31 05:39:22 [StorageMgmtSubnet]: CREATE_IN_PROGRESS state changed
2016-07-31 05:39:22 [overcloud-Networks-rto6netaoodk-TenantNetwork-xpwjehudwk6n]: CREATE_IN_PROGRESS Stack CREATE started
2016-07-31 05:39:22 [TenantNetwork]: CREATE_IN_PROGRESS state changed
2016-07-31 05:39:22 [TenantNetwork]: CREATE_COMPLETE state changed
2016-07-31 05:39:22 [TenantSubnet]: CREATE_IN_PROGRESS state changed
2016-07-31 05:39:23 [InternalApiSubnet]: CREATE_COMPLETE state changed
2016-07-31 05:39:23 [overcloud-Networks-rto6netaoodk-InternalNetwork-rwmlbfr5dzdk]: CREATE_COMPLETE Stack CREATE completed successfully
2016-07-31 05:39:23 [StorageMgmtSubnet]: CREATE_COMPLETE state changed
2016-07-31 05:39:23 [overcloud-Networks-rto6netaoodk-StorageMgmtNetwork-4e5xbdltuz7q]: CREATE_COMPLETE Stack CREATE completed successfully
2016-07-31 05:39:23 [StorageSubnet]: CREATE_COMPLETE state changed
2016-07-31 05:39:23 [overcloud-Networks-rto6netaoodk-StorageNetwork-e3tr6nhcwdss]: CREATE_COMPLETE Stack CREATE completed successfully
2016-07-31 05:39:23 [TenantSubnet]: CREATE_COMPLETE state changed
2016-07-31 05:39:23 [overcloud-Networks-rto6netaoodk-TenantNetwork-xpwjehudwk6n]: CREATE_COMPLETE Stack CREATE completed successfully
2016-07-31 05:39:23 [ExternalSubnet]: CREATE_COMPLETE state changed
2016-07-31 05:39:23 [overcloud-Networks-rto6netaoodk-ExternalNetwork-ge5f4ydo2yw6]: CREATE_COMPLETE Stack CREATE completed successfully
2016-07-31 05:39:24 [ExternalNetwork]: CREATE_COMPLETE state changed
2016-07-31 05:39:24 [InternalNetwork]: CREATE_COMPLETE state changed
2016-07-31 05:39:24 [StorageNetwork]: CREATE_COMPLETE state changed
2016-07-31 05:39:24 [StorageMgmtNetwork]: CREATE_COMPLETE state changed
2016-07-31 05:39:24 [TenantNetwork]: CREATE_COMPLETE state changed
2016-07-31 05:39:24 [overcloud-Networks-rto6netaoodk]: CREATE_COMPLETE Stack CREATE completed successfully
2016-07-31 05:39:25 [Networks]: CREATE_COMPLETE state changed
2016-07-31 05:39:25 [CephStorage]: CREATE_IN_PROGRESS state changed
2016-07-31 05:39:25 [ObjectStorage]: CREATE_IN_PROGRESS state changed
2016-07-31 05:39:25 [ControlVirtualIP]: CREATE_IN_PROGRESS state changed
2016-07-31 05:39:25 [overcloud-CephStorage-szz57nuogpix]: CREATE_IN_PROGRESS Stack CREATE started
2016-07-31 05:39:25 [overcloud-CephStorage-szz57nuogpix]: CREATE_COMPLETE Stack CREATE completed successfully
2016-07-31 05:39:25 [overcloud-ObjectStorage-gtvrpkjraesp]: CREATE_IN_PROGRESS Stack CREATE started
2016-07-31 05:39:25 [overcloud-ObjectStorage-gtvrpkjraesp]: CREATE_COMPLETE Stack CREATE completed successfully
2016-07-31 05:39:26 [ObjectStorage]: CREATE_COMPLETE state changed
2016-07-31 05:39:26 [CephStorage]: CREATE_COMPLETE state changed
2016-07-31 05:39:26 [ControlVirtualIP]: CREATE_COMPLETE state changed
2016-07-31 05:39:26 [StorageMgmtVirtualIP]: CREATE_IN_PROGRESS state changed
2016-07-31 05:39:26 [InternalApiVirtualIP]: CREATE_IN_PROGRESS state changed
2016-07-31 05:39:26 [overcloud-StorageMgmtVirtualIP-livasae7af57]: CREATE_IN_PROGRESS Stack CREATE started
2016-07-31 05:39:27 [RedisVirtualIP]: CREATE_IN_PROGRESS state changed
2016-07-31 05:39:27 [StorageVirtualIP]: CREATE_IN_PROGRESS state changed
2016-07-31 05:39:27 [PublicVirtualIP]: CREATE_IN_PROGRESS state changed
2016-07-31 05:39:27 [overcloud-RedisVirtualIP-t7iyxhta3pno]: CREATE_IN_PROGRESS Stack CREATE started
2016-07-31 05:39:27 [VipPort]: CREATE_IN_PROGRESS state changed
2016-07-31 05:39:27 [overcloud-InternalApiVirtualIP-4pgssemrxleu]: CREATE_IN_PROGRESS Stack CREATE started
2016-07-31 05:39:27 [InternalApiPort]: CREATE_IN_PROGRESS state changed
2016-07-31 05:39:27 [InternalApiPort]: CREATE_COMPLETE state changed
2016-07-31 05:39:27 [overcloud-StorageVirtualIP-jzamyndluhjn]: CREATE_IN_PROGRESS Stack CREATE started
2016-07-31 05:39:27 [StorageMgmtPort]: CREATE_IN_PROGRESS state changed
2016-07-31 05:39:27 [StorageMgmtPort]: CREATE_COMPLETE state changed
2016-07-31 05:39:27 [overcloud-StorageMgmtVirtualIP-livasae7af57]: CREATE_COMPLETE Stack CREATE completed successfully
2016-07-31 05:39:28 [VipPort]: CREATE_COMPLETE state changed
2016-07-31 05:39:28 [overcloud-RedisVirtualIP-t7iyxhta3pno]: CREATE_COMPLETE Stack CREATE completed successfully
2016-07-31 05:39:28 [overcloud-InternalApiVirtualIP-4pgssemrxleu]: CREATE_COMPLETE Stack CREATE completed successfully
2016-07-31 05:39:28 [StoragePort]: CREATE_IN_PROGRESS state changed
2016-07-31 05:39:28 [StoragePort]: CREATE_COMPLETE state changed
2016-07-31 05:39:28 [overcloud-StorageVirtualIP-jzamyndluhjn]: CREATE_COMPLETE Stack CREATE completed successfully
2016-07-31 05:39:28 [overcloud-PublicVirtualIP-fauhkcufypps]: CREATE_IN_PROGRESS Stack CREATE started
2016-07-31 05:39:28 [ExternalPort]: CREATE_IN_PROGRESS state changed
2016-07-31 05:39:28 [ExternalPort]: CREATE_COMPLETE state changed
2016-07-31 05:39:28 [overcloud-PublicVirtualIP-fauhkcufypps]: CREATE_COMPLETE Stack CREATE completed successfully
2016-07-31 05:39:29 [InternalApiVirtualIP]: CREATE_COMPLETE state changed
2016-07-31 05:39:29 [RedisVirtualIP]: CREATE_COMPLETE state changed
2016-07-31 05:39:29 [StorageMgmtVirtualIP]: CREATE_COMPLETE state changed
2016-07-31 05:39:29 [StorageVirtualIP]: CREATE_COMPLETE state changed
2016-07-31 05:39:29 [PublicVirtualIP]: CREATE_COMPLETE state changed
2016-07-31 05:39:29 [VipMap]: CREATE_IN_PROGRESS state changed
2016-07-31 05:39:29 [overcloud-VipMap-zu6i4tigbvs6]: CREATE_IN_PROGRESS Stack CREATE started
2016-07-31 05:39:29 [overcloud-VipMap-zu6i4tigbvs6]: CREATE_COMPLETE Stack CREATE completed successfully
2016-07-31 05:39:30 [VipMap]: CREATE_COMPLETE state changed
2016-07-31 05:39:30 [EndpointMap]: CREATE_IN_PROGRESS state changed
2016-07-31 05:39:30 [overcloud-EndpointMap-up3wygzq76fu]: CREATE_IN_PROGRESS Stack CREATE started
2016-07-31 05:39:30 [overcloud-EndpointMap-up3wygzq76fu]: CREATE_COMPLETE Stack CREATE completed successfully
2016-07-31 05:39:31 [EndpointMap]: CREATE_COMPLETE state changed
2016-07-31 05:39:31 [Compute]: CREATE_IN_PROGRESS state changed
2016-07-31 05:39:33 [BlockStorage]: CREATE_IN_PROGRESS state changed
2016-07-31 05:39:33 [Controller]: CREATE_IN_PROGRESS state changed
2016-07-31 05:39:33 [overcloud-BlockStorage-bo6qcyunjb45]: CREATE_IN_PROGRESS Stack CREATE started
2016-07-31 05:39:33 [overcloud-BlockStorage-bo6qcyunjb45]: CREATE_COMPLETE Stack CREATE completed successfully
2016-07-31 05:39:33 [overcloud-Compute-rbxr7ncefffr]: CREATE_IN_PROGRESS Stack CREATE started
2016-07-31 05:39:33 [1]: CREATE_IN_PROGRESS state changed
2016-07-31 05:39:34 [0]: CREATE_IN_PROGRESS state changed
2016-07-31 05:39:35 [overcloud-Compute-rbxr7ncefffr-1-6piplzp6ryef]: CREATE_IN_PROGRESS Stack CREATE started
2016-07-31 05:39:35 [NovaComputeConfig]: CREATE_IN_PROGRESS state changed
2016-07-31 05:39:35 [NodeUserData]: CREATE_IN_PROGRESS state changed
2016-07-31 05:39:35 [UpdateConfig]: CREATE_IN_PROGRESS state changed
2016-07-31 05:39:36 [overcloud-Controller-ksfwi2vsa5cj]: CREATE_IN_PROGRESS Stack CREATE started
2016-07-31 05:39:36 [1]: CREATE_IN_PROGRESS state changed
2016-07-31 05:39:36 [NodeAdminUserData]: CREATE_IN_PROGRESS state changed
2016-07-31 05:39:36 [NodeUserData]: CREATE_COMPLETE state changed
2016-07-31 05:39:36 [NovaComputeConfig]: CREATE_COMPLETE state changed
2016-07-31 05:39:36 [UpdateConfig]: CREATE_COMPLETE state changed
2016-07-31 05:39:36 [overcloud-Compute-rbxr7ncefffr-0-wyatkkjwlwtz]: CREATE_IN_PROGRESS Stack CREATE started
2016-07-31 05:39:36 [NodeAdminUserData]: CREATE_IN_PROGRESS state changed
2016-07-31 05:39:36 [UpdateConfig]: CREATE_IN_PROGRESS state changed
2016-07-31 05:39:37 [BlockStorage]: CREATE_COMPLETE state changed
2016-07-31 05:39:37 [0]: CREATE_IN_PROGRESS state changed
2016-07-31 05:39:37 [overcloud-Controller-ksfwi2vsa5cj-1-4zb4t5yuz3ex]: CREATE_IN_PROGRESS Stack CREATE started
2016-07-31 05:39:37 [NodeAdminUserData]: CREATE_IN_PROGRESS state changed
2016-07-31 05:39:37 [NodeUserData]: CREATE_IN_PROGRESS state changed
2016-07-31 05:39:37 [NovaComputeConfig]: CREATE_IN_PROGRESS state changed
2016-07-31 05:39:37 [UpdateConfig]: CREATE_COMPLETE state changed
2016-07-31 05:39:37 [NovaComputeConfig]: CREATE_COMPLETE state changed
2016-07-31 05:39:38 [2]: CREATE_IN_PROGRESS state changed
2016-07-31 05:39:38 [NodeUserData]: CREATE_IN_PROGRESS state changed
2016-07-31 05:39:38 [UpdateConfig]: CREATE_IN_PROGRESS state changed
2016-07-31 05:39:38 [NodeAdminUserData]: CREATE_COMPLETE state changed
2016-07-31 05:39:38 [UserData]: CREATE_IN_PROGRESS state changed
2016-07-31 05:39:38 [NodeUserData]: CREATE_COMPLETE state changed
2016-07-31 05:39:39 [NodeUserData]: CREATE_COMPLETE state changed
2016-07-31 05:39:39 [NodeAdminUserData]: CREATE_COMPLETE state changed
2016-07-31 05:39:39 [UpdateConfig]: CREATE_COMPLETE state changed
2016-07-31 05:39:39 [UserData]: CREATE_IN_PROGRESS state changed
2016-07-31 05:39:39 [overcloud-Controller-ksfwi2vsa5cj-0-rvldqd37mq4u]: CREATE_IN_PROGRESS Stack CREATE started
2016-07-31 05:39:39 [NodeAdminUserData]: CREATE_IN_PROGRESS state changed
2016-07-31 05:39:39 [UpdateConfig]: CREATE_IN_PROGRESS state changed
2016-07-31 05:39:39 [NodeUserData]: CREATE_IN_PROGRESS state changed
2016-07-31 05:39:39 [overcloud-Controller-ksfwi2vsa5cj-2-ohnilfuws65s]: CREATE_IN_PROGRESS Stack CREATE started
2016-07-31 05:39:39 [UpdateConfig]: CREATE_IN_PROGRESS state changed
2016-07-31 05:39:39 [UserData]: CREATE_COMPLETE state changed
2016-07-31 05:39:39 [NovaCompute]: CREATE_IN_PROGRESS state changed
2016-07-31 05:39:39 [NodeAdminUserData]: CREATE_COMPLETE state changed
2016-07-31 05:39:39 [UserData]: CREATE_IN_PROGRESS state changed
2016-07-31 05:39:40 [UserData]: CREATE_COMPLETE state changed
2016-07-31 05:39:40 [NodeUserData]: CREATE_COMPLETE state changed
2016-07-31 05:39:40 [NodeAdminUserData]: CREATE_COMPLETE state changed
2016-07-31 05:39:40 [UpdateConfig]: CREATE_COMPLETE state changed
2016-07-31 05:39:40 [UserData]: CREATE_IN_PROGRESS state changed
2016-07-31 05:39:40 [NodeAdminUserData]: CREATE_IN_PROGRESS state changed
2016-07-31 05:39:40 [NodeUserData]: CREATE_IN_PROGRESS state changed
2016-07-31 05:39:40 [UpdateConfig]: CREATE_COMPLETE state changed
2016-07-31 05:39:40 [UserData]: CREATE_COMPLETE state changed
2016-07-31 05:39:40 [NovaCompute]: CREATE_IN_PROGRESS state changed
2016-07-31 05:39:41 [Controller]: CREATE_IN_PROGRESS state changed
2016-07-31 05:39:41 [UserData]: CREATE_COMPLETE state changed
2016-07-31 05:39:41 [NodeUserData]: CREATE_COMPLETE state changed
2016-07-31 05:39:42 [Controller]: CREATE_IN_PROGRESS state changed
2016-07-31 05:39:42 [NodeAdminUserData]: CREATE_COMPLETE state changed
2016-07-31 05:39:42 [UserData]: CREATE_IN_PROGRESS state changed
2016-07-31 05:39:44 [UserData]: CREATE_COMPLETE state changed
2016-07-31 05:39:44 [Controller]: CREATE_IN_PROGRESS state changed
2016-07-31 05:50:55 [NovaCompute]: CREATE_COMPLETE state changed
2016-07-31 05:50:56 [Controller]: CREATE_COMPLETE state changed
2016-07-31 05:50:56 [Controller]: CREATE_COMPLETE state changed
2016-07-31 05:50:57 [InternalApiPort]: CREATE_IN_PROGRESS state changed
2016-07-31 05:50:57 [ExternalPort]: CREATE_IN_PROGRESS state changed
2016-07-31 05:50:58 [ManagementPort]: CREATE_IN_PROGRESS state changed
2016-07-31 05:50:59 [TenantPort]: CREATE_IN_PROGRESS state changed
2016-07-31 05:50:59 [ExternalPort]: CREATE_IN_PROGRESS state changed
2016-07-31 05:51:00 [StorageMgmtPort]: CREATE_IN_PROGRESS state changed
2016-07-31 05:51:00 [Controller]: CREATE_COMPLETE state changed
2016-07-31 05:51:00 [UpdateDeployment]: CREATE_IN_PROGRESS state changed
2016-07-31 05:51:00 [UpdateDeployment]: CREATE_IN_PROGRESS state changed
2016-07-31 05:51:02 [StorageMgmtPort]: CREATE_IN_PROGRESS state changed
2016-07-31 05:51:03 [ExternalPort]: CREATE_IN_PROGRESS state changed
2016-07-31 05:51:03 [NovaCompute]: CREATE_COMPLETE state changed
2016-07-31 05:51:03 [InternalApiPort]: CREATE_IN_PROGRESS state changed
2016-07-31 05:51:04 [ManagementPort]: CREATE_IN_PROGRESS state changed
2016-07-31 05:51:04 [InternalApiPort]: CREATE_IN_PROGRESS state changed
2016-07-31 05:51:05 [UpdateDeployment]: CREATE_IN_PROGRESS state changed
2016-07-31 05:51:05 [TenantPort]: CREATE_IN_PROGRESS state changed
2016-07-31 05:51:05 [TenantPort]: CREATE_IN_PROGRESS state changed
2016-07-31 05:51:05 [StoragePort]: CREATE_IN_PROGRESS state changed
2016-07-31 05:51:06 [InternalApiPort]: CREATE_IN_PROGRESS state changed
2016-07-31 05:51:06 [ManagementPort]: CREATE_IN_PROGRESS state changed
2016-07-31 05:51:07 [InternalApiPort]: CREATE_IN_PROGRESS state changed
2016-07-31 05:51:07 [StorageMgmtPort]: CREATE_IN_PROGRESS state changed
2016-07-31 05:51:09 [UpdateDeployment]: CREATE_IN_PROGRESS state changed
2016-07-31 05:51:09 [ExternalPort]: CREATE_IN_PROGRESS state changed
2016-07-31 05:51:09 [StorageMgmtPort]: CREATE_IN_PROGRESS state changed
2016-07-31 05:51:09 [TenantPort]: CREATE_IN_PROGRESS state changed
2016-07-31 05:51:10 [ManagementPort]: CREATE_IN_PROGRESS state changed
2016-07-31 05:51:11 [StoragePort]: CREATE_IN_PROGRESS state changed
2016-07-31 05:51:11 [TenantPort]: CREATE_IN_PROGRESS state changed
2016-07-31 05:51:12 [StoragePort]: CREATE_IN_PROGRESS state changed
2016-07-31 05:51:12 [UpdateDeployment]: CREATE_IN_PROGRESS state changed
2016-07-31 05:51:12 [ManagementPort]: CREATE_IN_PROGRESS state changed
2016-07-31 05:51:14 [ExternalPort]: CREATE_COMPLETE state changed
2016-07-31 05:51:14 [StoragePort]: CREATE_IN_PROGRESS state changed
2016-07-31 05:51:15 [StorageMgmtPort]: CREATE_COMPLETE state changed
2016-07-31 05:51:16 [InternalApiPort]: CREATE_COMPLETE state changed
2016-07-31 05:51:16 [ExternalPort]: CREATE_COMPLETE state changed
2016-07-31 05:51:16 [StorageMgmtPort]: CREATE_IN_PROGRESS state changed
2016-07-31 05:51:16 [StorageMgmtPort]: CREATE_COMPLETE state changed
2016-07-31 05:51:16 [InternalApiPort]: CREATE_COMPLETE state changed
2016-07-31 05:51:17 [ManagementPort]: CREATE_COMPLETE state changed
2016-07-31 05:51:17 [StorageMgmtPort]: CREATE_COMPLETE state changed
2016-07-31 05:51:17 [TenantPort]: CREATE_COMPLETE state changed
2016-07-31 05:51:17 [InternalApiPort]: CREATE_COMPLETE state changed
2016-07-31 05:51:17 [ExternalPort]: CREATE_IN_PROGRESS state changed
2016-07-31 05:51:17 [ExternalPort]: CREATE_COMPLETE state changed
2016-07-31 05:51:18 [TenantPort]: CREATE_COMPLETE state changed
2016-07-31 05:51:18 [ManagementPort]: CREATE_COMPLETE state changed
2016-07-31 05:51:18 [ManagementPort]: CREATE_COMPLETE state changed
2016-07-31 05:51:19 [TenantPort]: CREATE_COMPLETE state changed
2016-07-31 05:51:19 [TenantPort]: CREATE_COMPLETE state changed
2016-07-31 05:51:21 [StoragePort]: CREATE_IN_PROGRESS state changed
2016-07-31 05:51:22 [StoragePort]: CREATE_COMPLETE state changed
2016-07-31 05:51:22 [StoragePort]: CREATE_COMPLETE state changed
2016-07-31 05:51:22 [InternalApiPort]: CREATE_COMPLETE state changed
2016-07-31 05:51:22 [StoragePort]: CREATE_COMPLETE state changed
2016-07-31 05:51:22 [StoragePort]: CREATE_COMPLETE state changed
2016-07-31 05:51:23 [ManagementPort]: CREATE_COMPLETE state changed
2016-07-31 05:51:23 [NetIpMap]: CREATE_IN_PROGRESS state changed
2016-07-31 05:51:24 [NetworkConfig]: CREATE_IN_PROGRESS state changed
2016-07-31 05:51:24 [ExternalPort]: CREATE_COMPLETE state changed
2016-07-31 05:51:24 [ExternalPort]: CREATE_COMPLETE state changed
2016-07-31 05:51:25 [NetIpSubnetMap]: CREATE_IN_PROGRESS state changed
2016-07-31 05:51:25 [NetIpMap]: CREATE_IN_PROGRESS state changed
2016-07-31 05:51:25 [StorageMgmtPort]: CREATE_COMPLETE state changed
2016-07-31 05:51:25 [TenantPort]: CREATE_COMPLETE state changed
2016-07-31 05:51:25 [ManagementPort]: CREATE_COMPLETE state changed
2016-07-31 05:51:25 [NetworkConfig]: CREATE_IN_PROGRESS state changed
2016-07-31 05:51:26 [NetIpMap]: CREATE_IN_PROGRESS state changed
2016-07-31 05:51:26 [InternalApiPort]: CREATE_COMPLETE state changed
2016-07-31 05:51:27 [NetIpMap]: CREATE_IN_PROGRESS state changed
2016-07-31 05:51:27 [NetIpSubnetMap]: CREATE_IN_PROGRESS state changed
2016-07-31 05:51:27 [StorageMgmtPort]: CREATE_COMPLETE state changed
2016-07-31 05:51:28 [NetworkConfig]: CREATE_IN_PROGRESS state changed
2016-07-31 05:51:28 [NetworkConfig]: CREATE_IN_PROGRESS state changed
2016-07-31 05:51:28 [NetIpMap]: CREATE_COMPLETE state changed
2016-07-31 05:51:28 [NetworkConfig]: CREATE_COMPLETE state changed
2016-07-31 05:51:28 [NetworkDeployment]: CREATE_IN_PROGRESS state changed
2016-07-31 05:51:29 [NetworkConfig]: CREATE_COMPLETE state changed
2016-07-31 05:51:29 [NetIpMap]: CREATE_COMPLETE state changed
2016-07-31 05:51:29 [NetIpSubnetMap]: CREATE_IN_PROGRESS state changed
2016-07-31 05:51:29 [StoragePort]: CREATE_COMPLETE state changed
2016-07-31 05:51:30 [NetIpSubnetMap]: CREATE_COMPLETE state changed
2016-07-31 05:51:30 [NetIpMap]: CREATE_COMPLETE state changed
2016-07-31 05:51:30 [NetworkConfig]: CREATE_COMPLETE state changed
2016-07-31 05:51:30 [NetworkDeployment]: CREATE_IN_PROGRESS state changed
2016-07-31 05:51:30 [NetIpSubnetMap]: CREATE_COMPLETE state changed
2016-07-31 05:51:30 [NetworkDeployment]: CREATE_IN_PROGRESS state changed
2016-07-31 05:51:30 [NetworkConfig]: CREATE_IN_PROGRESS state changed
2016-07-31 05:51:30 [NetIpMap]: CREATE_IN_PROGRESS state changed
2016-07-31 05:51:31 [NetworkConfig]: CREATE_COMPLETE state changed
2016-07-31 05:51:31 [NetIpMap]: CREATE_COMPLETE state changed
2016-07-31 05:51:31 [NetIpSubnetMap]: CREATE_COMPLETE state changed
2016-07-31 05:51:32 [NetworkDeployment]: CREATE_IN_PROGRESS state changed
2016-07-31 05:51:32 [NetIpMap]: CREATE_COMPLETE state changed
2016-07-31 05:51:32 [NetworkConfig]: CREATE_COMPLETE state changed
2016-07-31 05:51:32 [NetworkDeployment]: CREATE_IN_PROGRESS state changed
2016-07-31 05:53:23 [UpdateDeployment]: SIGNAL_IN_PROGRESS Signal: deployment succeeded
2016-07-31 05:53:24 [UpdateDeployment]: CREATE_COMPLETE state changed
2016-07-31 05:53:26 [NetworkDeployment]: SIGNAL_IN_PROGRESS Signal: deployment succeeded
2016-07-31 05:53:26 [NetworkDeployment]: CREATE_COMPLETE state changed
2016-07-31 05:53:26 [NovaComputeDeployment]: CREATE_IN_PROGRESS state changed
2016-07-31 05:53:30 [UpdateDeployment]: SIGNAL_IN_PROGRESS Signal: deployment succeeded
2016-07-31 05:53:30 [UpdateDeployment]: CREATE_COMPLETE state changed
2016-07-31 05:53:31 [UpdateDeployment]: SIGNAL_IN_PROGRESS Signal: deployment succeeded
2016-07-31 05:53:32 [UpdateDeployment]: CREATE_COMPLETE state changed
2016-07-31 05:53:32 [UpdateDeployment]: SIGNAL_IN_PROGRESS Signal: deployment succeeded
2016-07-31 05:53:32 [NetworkDeployment]: SIGNAL_IN_PROGRESS Signal: deployment succeeded
2016-07-31 05:53:33 [UpdateDeployment]: CREATE_COMPLETE state changed
2016-07-31 05:53:33 [NetworkDeployment]: CREATE_COMPLETE state changed
2016-07-31 05:53:33 [NodeTLSCAData]: CREATE_IN_PROGRESS state changed
2016-07-31 05:53:33 [UpdateDeployment]: SIGNAL_IN_PROGRESS Signal: deployment succeeded
2016-07-31 05:53:33 [UpdateDeployment]: CREATE_COMPLETE state changed
2016-07-31 05:53:34 [NetworkDeployment]: SIGNAL_IN_PROGRESS Signal: deployment succeeded
2016-07-31 05:53:34 [NetworkDeployment]: CREATE_COMPLETE state changed
2016-07-31 05:53:34 [NodeTLSCAData]: CREATE_IN_PROGRESS state changed
2016-07-31 05:53:35 [NetworkDeployment]: SIGNAL_IN_PROGRESS Signal: deployment succeeded
2016-07-31 05:53:35 [NodeTLSCAData]: CREATE_COMPLETE state changed
2016-07-31 05:53:35 [NodeTLSData]: CREATE_IN_PROGRESS state changed
2016-07-31 05:53:35 [NetworkDeployment]: SIGNAL_IN_PROGRESS Signal: deployment succeeded
2016-07-31 05:53:36 [NetworkDeployment]: CREATE_COMPLETE state changed
2016-07-31 05:53:36 [NodeTLSCAData]: CREATE_IN_PROGRESS state changed
2016-07-31 05:53:36 [NetworkDeployment]: CREATE_COMPLETE state changed
2016-07-31 05:53:36 [NovaComputeDeployment]: CREATE_IN_PROGRESS state changed
2016-07-31 05:53:37 [NovaComputeDeployment]: SIGNAL_IN_PROGRESS Signal: deployment succeeded
2016-07-31 05:53:37 [NovaComputeDeployment]: CREATE_COMPLETE state changed
2016-07-31 05:53:37 [ComputeExtraConfigPre]: CREATE_IN_PROGRESS state changed
2016-07-31 05:53:37 [NodeTLSCAData]: CREATE_COMPLETE state changed
2016-07-31 05:53:37 [NodeTLSData]: CREATE_IN_PROGRESS state changed
2016-07-31 05:53:37 [NodeTLSCAData]: CREATE_IN_PROGRESS state changed
2016-07-31 05:53:38 [NodeTLSData]: CREATE_COMPLETE state changed
2016-07-31 05:53:38 [ControllerConfig]: CREATE_IN_PROGRESS state changed
2016-07-31 05:53:38 [NodeTLSData]: CREATE_COMPLETE state changed
2016-07-31 05:53:38 [ControllerConfig]: CREATE_IN_PROGRESS state changed
2016-07-31 05:53:38 [ControllerConfig]: CREATE_COMPLETE state changed
2016-07-31 05:53:38 [ControllerDeployment]: CREATE_IN_PROGRESS state changed
2016-07-31 05:53:38 [NodeTLSCAData]: CREATE_COMPLETE state changed
2016-07-31 05:53:38 [ComputeExtraConfigPre]: CREATE_COMPLETE state changed
2016-07-31 05:53:38 [NodeExtraConfig]: CREATE_IN_PROGRESS state changed
2016-07-31 05:53:39 [ControllerConfig]: CREATE_COMPLETE state changed
2016-07-31 05:53:39 [ControllerDeployment]: CREATE_IN_PROGRESS state changed
2016-07-31 05:53:39 [NodeExtraConfig]: CREATE_COMPLETE state changed
2016-07-31 05:53:39 [overcloud-Compute-rbxr7ncefffr-0-wyatkkjwlwtz]: CREATE_COMPLETE Stack CREATE completed successfully
2016-07-31 05:53:40 [ControllerConfig]: CREATE_COMPLETE state changed
2016-07-31 05:53:40 [ControllerDeployment]: CREATE_IN_PROGRESS state changed
2016-07-31 05:53:40 [0]: CREATE_COMPLETE state changed
2016-07-31 05:53:44 [NovaComputeDeployment]: SIGNAL_IN_PROGRESS Signal: deployment succeeded
2016-07-31 05:53:44 [NovaComputeDeployment]: CREATE_COMPLETE state changed
2016-07-31 05:53:44 [ComputeExtraConfigPre]: CREATE_IN_PROGRESS state changed
2016-07-31 05:53:44 [NodeTLSCAData]: CREATE_IN_PROGRESS state changed
2016-07-31 05:53:46 [NodeTLSCAData]: CREATE_COMPLETE state changed
2016-07-31 05:53:46 [ComputeExtraConfigPre]: CREATE_COMPLETE state changed
2016-07-31 05:53:46 [NodeExtraConfig]: CREATE_IN_PROGRESS state changed
2016-07-31 05:53:47 [1]: CREATE_COMPLETE state changed
2016-07-31 05:53:47 [overcloud-Compute-rbxr7ncefffr]: CREATE_COMPLETE Stack CREATE completed successfully
2016-07-31 05:53:47 [NodeExtraConfig]: CREATE_COMPLETE state changed
2016-07-31 05:53:47 [overcloud-Compute-rbxr7ncefffr-1-6piplzp6ryef]: CREATE_COMPLETE Stack CREATE completed successfully
2016-07-31 05:53:48 [Compute]: CREATE_COMPLETE state changed
2016-07-31 05:53:56 [ControllerDeployment]: SIGNAL_IN_PROGRESS Signal: deployment succeeded
2016-07-31 05:53:56 [ControllerDeployment]: CREATE_COMPLETE state changed
2016-07-31 05:53:56 [ControllerExtraConfigPre]: CREATE_IN_PROGRESS state changed
2016-07-31 05:53:57 [ControllerExtraConfigPre]: CREATE_COMPLETE state changed
2016-07-31 05:53:57 [NodeExtraConfig]: CREATE_IN_PROGRESS state changed
2016-07-31 05:53:58 [ControllerDeployment]: SIGNAL_IN_PROGRESS Signal: deployment succeeded
2016-07-31 05:53:58 [ControllerDeployment]: CREATE_COMPLETE state changed
2016-07-31 05:53:58 [ControllerExtraConfigPre]: CREATE_IN_PROGRESS state changed
2016-07-31 05:53:58 [NodeExtraConfig]: CREATE_COMPLETE state changed
2016-07-31 05:53:59 [2]: CREATE_COMPLETE state changed
2016-07-31 05:53:59 [overcloud-Controller-ksfwi2vsa5cj-2-ohnilfuws65s]: CREATE_COMPLETE Stack CREATE completed successfully
2016-07-31 05:54:00 [ControllerDeployment]: SIGNAL_IN_PROGRESS Signal: deployment succeeded
2016-07-31 05:54:00 [ControllerExtraConfigPre]: CREATE_COMPLETE state changed
2016-07-31 05:54:00 [NodeExtraConfig]: CREATE_IN_PROGRESS state changed
2016-07-31 05:54:01 [0]: CREATE_COMPLETE state changed
2016-07-31 05:54:01 [ControllerDeployment]: CREATE_COMPLETE state changed
2016-07-31 05:54:01 [ControllerExtraConfigPre]: CREATE_IN_PROGRESS state changed
2016-07-31 05:54:01 [NodeExtraConfig]: CREATE_COMPLETE state changed
2016-07-31 05:54:01 [overcloud-Controller-ksfwi2vsa5cj-0-rvldqd37mq4u]: CREATE_COMPLETE Stack CREATE completed successfully
2016-07-31 05:54:02 [ControllerExtraConfigPre]: CREATE_COMPLETE state changed
2016-07-31 05:54:02 [NodeExtraConfig]: CREATE_IN_PROGRESS state changed
2016-07-31 05:54:03 [NodeExtraConfig]: CREATE_COMPLETE state changed
2016-07-31 05:54:03 [overcloud-Controller-ksfwi2vsa5cj-1-4zb4t5yuz3ex]: CREATE_COMPLETE Stack CREATE completed successfully
2016-07-31 05:54:04 [Controller]: CREATE_COMPLETE state changed
2016-07-31 05:54:04 [1]: CREATE_COMPLETE state changed
2016-07-31 05:54:04 [overcloud-Controller-ksfwi2vsa5cj]: CREATE_COMPLETE Stack CREATE completed successfully
2016-07-31 05:54:05 [UpdateWorkflow]: CREATE_IN_PROGRESS state changed
2016-07-31 05:54:05 [ControllerBootstrapNodeConfig]: CREATE_IN_PROGRESS state changed
2016-07-31 05:54:05 [SwiftDevicesAndProxyConfig]: CREATE_IN_PROGRESS state changed
2016-07-31 05:54:05 [VipDeployment]: CREATE_IN_PROGRESS state changed
2016-07-31 05:54:05 [overcloud-UpdateWorkflow-4n5h2wcxblb3]: CREATE_IN_PROGRESS Stack CREATE started
2016-07-31 05:54:05 [overcloud-UpdateWorkflow-4n5h2wcxblb3]: CREATE_COMPLETE Stack CREATE completed successfully
2016-07-31 05:54:05 [overcloud-SwiftDevicesAndProxyConfig-4oamyzvbqcrb]: CREATE_IN_PROGRESS Stack CREATE started
2016-07-31 05:54:05 [SwiftDevicesAndProxyConfigImpl]: CREATE_IN_PROGRESS state changed
2016-07-31 05:54:05 [SwiftDevicesAndProxyConfigImpl]: CREATE_COMPLETE state changed
2016-07-31 05:54:05 [overcloud-SwiftDevicesAndProxyConfig-4oamyzvbqcrb]: CREATE_COMPLETE Stack CREATE completed successfully
2016-07-31 05:54:05 [overcloud-VipDeployment-wo6kjtzdoqzl]: CREATE_IN_PROGRESS Stack CREATE started
2016-07-31 05:54:05 [overcloud-ControllerBootstrapNodeConfig-idpjh3rkfxkk]: CREATE_IN_PROGRESS Stack CREATE started
2016-07-31 05:54:05 [BootstrapNodeConfigImpl]: CREATE_IN_PROGRESS state changed
2016-07-31 05:54:05 [BootstrapNodeConfigImpl]: CREATE_COMPLETE state changed
2016-07-31 05:54:05 [overcloud-ControllerBootstrapNodeConfig-idpjh3rkfxkk]: CREATE_COMPLETE Stack CREATE completed successfully
2016-07-31 05:54:06 [AllNodesValidationConfig]: CREATE_IN_PROGRESS state changed
2016-07-31 05:54:06 [ControllerClusterConfig]: CREATE_IN_PROGRESS state changed
2016-07-31 05:54:06 [1]: CREATE_IN_PROGRESS state changed
2016-07-31 05:54:06 [0]: CREATE_IN_PROGRESS state changed
2016-07-31 05:54:06 [2]: CREATE_IN_PROGRESS state changed
2016-07-31 05:54:06 [overcloud-AllNodesValidationConfig-uyaykmk5udka]: CREATE_IN_PROGRESS Stack CREATE started
2016-07-31 05:54:06 [AllNodesValidationsImpl]: CREATE_IN_PROGRESS state changed
2016-07-31 05:54:06 [AllNodesValidationsImpl]: CREATE_COMPLETE state changed
2016-07-31 05:54:06 [overcloud-AllNodesValidationConfig-uyaykmk5udka]: CREATE_COMPLETE Stack CREATE completed successfully
2016-07-31 05:54:07 [ControllerIpListMap]: CREATE_IN_PROGRESS state changed
2016-07-31 05:54:07 [overcloud-ControllerIpListMap-mx33gmf6yaml]: CREATE_IN_PROGRESS Stack CREATE started
2016-07-31 05:54:07 [overcloud-ControllerIpListMap-mx33gmf6yaml]: CREATE_COMPLETE Stack CREATE completed successfully
2016-07-31 05:54:08 [UpdateWorkflow]: CREATE_COMPLETE state changed
2016-07-31 05:54:08 [ControllerBootstrapNodeConfig]: CREATE_COMPLETE state changed
2016-07-31 05:54:08 [SwiftDevicesAndProxyConfig]: CREATE_COMPLETE state changed
2016-07-31 05:54:08 [AllNodesValidationConfig]: CREATE_COMPLETE state changed
2016-07-31 05:54:08 [ControllerClusterConfig]: CREATE_COMPLETE state changed
2016-07-31 05:54:08 [ControllerIpListMap]: CREATE_COMPLETE state changed
2016-07-31 05:54:08 [ControllerBootstrapNodeDeployment]: CREATE_IN_PROGRESS state changed
2016-07-31 05:54:08 [ControllerSwiftDeployment]: CREATE_IN_PROGRESS state changed
2016-07-31 05:54:08 [overcloud-ControllerBootstrapNodeDeployment-rb5tuhata5zw]: CREATE_IN_PROGRESS Stack CREATE started
2016-07-31 05:54:08 [1]: CREATE_IN_PROGRESS state changed
2016-07-31 05:54:09 [ControllerClusterDeployment]: CREATE_IN_PROGRESS state changed
2016-07-31 05:54:09 [overcloud-ControllerSwiftDeployment-2fhzhyjsqtox]: CREATE_IN_PROGRESS Stack CREATE started
2016-07-31 05:54:09 [1]: CREATE_IN_PROGRESS state changed
2016-07-31 05:54:09 [0]: CREATE_IN_PROGRESS state changed
2016-07-31 05:54:09 [overcloud-ControllerClusterDeployment-n47rsmrrjhm3]: CREATE_IN_PROGRESS Stack CREATE started
2016-07-31 05:54:09 [1]: CREATE_IN_PROGRESS state changed
2016-07-31 05:54:10 [0]: CREATE_IN_PROGRESS state changed
2016-07-31 05:54:10 [2]: CREATE_IN_PROGRESS state changed
2016-07-31 05:54:10 [0]: CREATE_IN_PROGRESS state changed
2016-07-31 05:54:11 [2]: CREATE_IN_PROGRESS state changed
2016-07-31 05:54:11 [CephClusterConfig]: CREATE_IN_PROGRESS state changed
2016-07-31 05:54:11 [overcloud-allNodesConfig-dlvmzfcjczrv]: CREATE_IN_PROGRESS Stack CREATE started
2016-07-31 05:54:11 [allNodesConfigImpl]: CREATE_IN_PROGRESS state changed
2016-07-31 05:54:12 [allNodesConfigImpl]: CREATE_COMPLETE state changed
2016-07-31 05:54:12 [overcloud-allNodesConfig-dlvmzfcjczrv]: CREATE_COMPLETE Stack CREATE completed successfully
2016-07-31 05:54:12 [overcloud-CephClusterConfig-mas6ebaob76u]: CREATE_IN_PROGRESS Stack CREATE started
2016-07-31 05:54:12 [CephClusterConfigImpl]: CREATE_IN_PROGRESS state changed
2016-07-31 05:54:12 [CephClusterConfigImpl]: CREATE_COMPLETE state changed
2016-07-31 05:54:12 [overcloud-CephClusterConfig-mas6ebaob76u]: CREATE_COMPLETE Stack CREATE completed successfully
2016-07-31 05:54:13 [allNodesConfig]: CREATE_COMPLETE state changed
2016-07-31 05:54:13 [ObjectStorageSwiftDeployment]: CREATE_COMPLETE state changed
2016-07-31 05:54:13 [CephClusterConfig]: CREATE_COMPLETE state changed
2016-07-31 05:54:13 [BlockStorageAllNodesDeployment]: CREATE_IN_PROGRESS state changed
2016-07-31 05:54:13 [ObjectStorageAllNodesDeployment]: CREATE_IN_PROGRESS state changed
2016-07-31 05:54:13 [CephStorageAllNodesDeployment]: CREATE_IN_PROGRESS state changed
2016-07-31 05:54:13 [overcloud-BlockStorageAllNodesDeployment-diu37hed5rd4]: CREATE_IN_PROGRESS Stack CREATE started
2016-07-31 05:54:13 [overcloud-BlockStorageAllNodesDeployment-diu37hed5rd4]: CREATE_COMPLETE Stack CREATE completed successfully
2016-07-31 05:54:13 [overcloud-ObjectStorageAllNodesDeployment-tz2t245d276l]: CREATE_IN_PROGRESS Stack CREATE started
2016-07-31 05:54:14 [ComputeAllNodesDeployment]: CREATE_IN_PROGRESS state changed
2016-07-31 05:54:14 [ControllerAllNodesDeployment]: CREATE_IN_PROGRESS state changed
2016-07-31 05:54:14 [CephStorageCephDeployment]: CREATE_IN_PROGRESS state changed
2016-07-31 05:54:14 [overcloud-CephStorageAllNodesDeployment-7evcwwskk2um]: CREATE_IN_PROGRESS Stack CREATE started
2016-07-31 05:54:14 [overcloud-CephStorageAllNodesDeployment-7evcwwskk2um]: CREATE_COMPLETE Stack CREATE completed successfully
2016-07-31 05:54:14 [overcloud-ComputeAllNodesDeployment-nwmxao5py32t]: CREATE_IN_PROGRESS Stack CREATE started
2016-07-31 05:54:14 [1]: CREATE_IN_PROGRESS state changed
2016-07-31 05:54:14 [overcloud-ObjectStorageAllNodesDeployment-tz2t245d276l]: CREATE_COMPLETE Stack CREATE completed successfully
2016-07-31 05:54:15 [ControllerCephDeployment]: CREATE_IN_PROGRESS state changed
2016-07-31 05:54:15 [overcloud-CephStorageCephDeployment-z3cay5ank3lf]: CREATE_IN_PROGRESS Stack CREATE started
2016-07-31 05:54:15 [overcloud-CephStorageCephDeployment-z3cay5ank3lf]: CREATE_COMPLETE Stack CREATE completed successfully
2016-07-31 05:54:15 [overcloud-ControllerAllNodesDeployment-mqgisqdiqojq]: CREATE_IN_PROGRESS Stack CREATE started
2016-07-31 05:54:15 [1]: CREATE_IN_PROGRESS state changed
2016-07-31 05:54:16 [ComputeCephDeployment]: CREATE_IN_PROGRESS state changed
2016-07-31 05:54:16 [overcloud-ComputeCephDeployment-56zf3zkhbhtq]: CREATE_IN_PROGRESS Stack CREATE started
2016-07-31 05:54:16 [1]: CREATE_IN_PROGRESS state changed
2016-07-31 05:54:16 [overcloud-ControllerCephDeployment-2t5ix3hth3ty]: CREATE_IN_PROGRESS Stack CREATE started
2016-07-31 05:54:16 [1]: CREATE_IN_PROGRESS state changed
2016-07-31 05:54:16 [0]: CREATE_IN_PROGRESS state changed
2016-07-31 05:54:16 [0]: CREATE_IN_PROGRESS state changed
2016-07-31 05:54:17 [BlockStorageAllNodesDeployment]: CREATE_COMPLETE state changed
2016-07-31 05:54:17 [CephStorageCephDeployment]: CREATE_COMPLETE state changed
2016-07-31 05:54:17 [ObjectStorageAllNodesDeployment]: CREATE_COMPLETE state changed
2016-07-31 05:54:17 [0]: CREATE_IN_PROGRESS state changed
2016-07-31 05:54:17 [2]: CREATE_IN_PROGRESS state changed
2016-07-31 05:54:17 [2]: CREATE_IN_PROGRESS state changed
2016-07-31 05:54:18 [CephStorageAllNodesDeployment]: CREATE_COMPLETE state changed
2016-07-31 05:54:18 [ObjectStorageAllNodesValidationDeployment]: CREATE_IN_PROGRESS state changed
2016-07-31 05:54:18 [BlockStorageAllNodesValidationDeployment]: CREATE_IN_PROGRESS state changed
2016-07-31 05:54:18 [CephStorageAllNodesValidationDeployment]: CREATE_IN_PROGRESS state changed
2016-07-31 05:54:18 [overcloud-ObjectStorageAllNodesValidationDeployment-cinldvvf4rii]: CREATE_IN_PROGRESS Stack CREATE started
2016-07-31 05:54:18 [overcloud-ObjectStorageAllNodesValidationDeployment-cinldvvf4rii]: CREATE_COMPLETE Stack CREATE completed successfully
2016-07-31 05:54:18 [0]: CREATE_IN_PROGRESS state changed
2016-07-31 05:54:18 [overcloud-BlockStorageAllNodesValidationDeployment-mmwxw7p7w7ps]: CREATE_IN_PROGRESS Stack CREATE started
2016-07-31 05:54:19 [overcloud-BlockStorageAllNodesValidationDeployment-mmwxw7p7w7ps]: CREATE_COMPLETE Stack CREATE completed successfully
2016-07-31 05:54:19 [overcloud-CephStorageAllNodesValidationDeployment-cwpp4jh7qzzc]: CREATE_IN_PROGRESS Stack CREATE started
2016-07-31 05:54:19 [overcloud-CephStorageAllNodesValidationDeployment-cwpp4jh7qzzc]: CREATE_COMPLETE Stack CREATE completed successfully
2016-07-31 05:54:20 [ObjectStorageAllNodesValidationDeployment]: CREATE_COMPLETE state changed
2016-07-31 05:54:20 [BlockStorageAllNodesValidationDeployment]: CREATE_COMPLETE state changed
2016-07-31 05:54:20 [CephStorageAllNodesValidationDeployment]: CREATE_COMPLETE state changed
2016-07-31 05:54:23 [1]: SIGNAL_IN_PROGRESS Signal: deployment succeeded
2016-07-31 05:54:24 [1]: CREATE_COMPLETE state changed
2016-07-31 05:54:30 [1]: SIGNAL_IN_PROGRESS Signal: deployment succeeded
2016-07-31 05:54:30 [1]: CREATE_COMPLETE state changed
2016-07-31 05:54:31 [1]: SIGNAL_IN_PROGRESS Signal: deployment succeeded
2016-07-31 05:54:31 [1]: CREATE_COMPLETE state changed
2016-07-31 05:54:33 [2]: SIGNAL_IN_PROGRESS Signal: deployment succeeded
2016-07-31 05:54:33 [2]: SIGNAL_IN_PROGRESS Signal: deployment succeeded
2016-07-31 05:54:33 [2]: SIGNAL_IN_PROGRESS Signal: deployment succeeded
2016-07-31 05:54:33 [2]: CREATE_COMPLETE state changed
2016-07-31 05:54:34 [2]: CREATE_COMPLETE state changed
2016-07-31 05:54:34 [2]: CREATE_COMPLETE state changed
2016-07-31 05:54:34 [0]: SIGNAL_IN_PROGRESS Signal: deployment succeeded
2016-07-31 05:54:34 [2]: SIGNAL_IN_PROGRESS Signal: deployment succeeded
2016-07-31 05:54:34 [2]: CREATE_COMPLETE state changed
2016-07-31 05:54:34 [0]: SIGNAL_IN_PROGRESS Signal: deployment succeeded
2016-07-31 05:54:35 [0]: SIGNAL_IN_PROGRESS Signal: deployment succeeded
2016-07-31 05:54:35 [0]: CREATE_COMPLETE state changed
2016-07-31 05:54:35 [0]: CREATE_COMPLETE state changed
2016-07-31 05:54:35 [0]: SIGNAL_IN_PROGRESS Signal: deployment succeeded
2016-07-31 05:54:35 [0]: CREATE_COMPLETE state changed
2016-07-31 05:54:36 [0]: CREATE_COMPLETE state changed
2016-07-31 05:54:36 [overcloud-VipDeployment-wo6kjtzdoqzl]: CREATE_COMPLETE Stack CREATE completed successfully
2016-07-31 05:54:37 [VipDeployment]: CREATE_COMPLETE state changed
2016-07-31 05:54:47 [1]: SIGNAL_IN_PROGRESS Signal: deployment succeeded
2016-07-31 05:54:47 [1]: CREATE_COMPLETE state changed
2016-07-31 05:54:48 [1]: SIGNAL_IN_PROGRESS Signal: deployment succeeded
2016-07-31 05:54:48 [1]: CREATE_COMPLETE state changed
2016-07-31 05:54:48 [overcloud-ControllerBootstrapNodeDeployment-rb5tuhata5zw]: CREATE_COMPLETE Stack CREATE completed successfully
2016-07-31 05:54:48 [1]: SIGNAL_IN_PROGRESS Signal: deployment succeeded
2016-07-31 05:54:49 [1]: CREATE_COMPLETE state changed
2016-07-31 05:54:49 [1]: SIGNAL_IN_PROGRESS Signal: deployment succeeded
2016-07-31 05:54:50 [1]: CREATE_COMPLETE state changed
2016-07-31 05:54:50 [overcloud-ControllerClusterDeployment-n47rsmrrjhm3]: CREATE_COMPLETE Stack CREATE completed successfully
2016-07-31 05:54:52 [0]: SIGNAL_IN_PROGRESS Signal: deployment succeeded
2016-07-31 05:54:52 [0]: CREATE_COMPLETE state changed
2016-07-31 05:54:52 [overcloud-ComputeAllNodesDeployment-nwmxao5py32t]: CREATE_COMPLETE Stack CREATE completed successfully
2016-07-31 05:54:53 [ComputeAllNodesDeployment]: CREATE_COMPLETE state changed
2016-07-31 05:54:53 [ComputeAllNodesValidationDeployment]: CREATE_IN_PROGRESS state changed
2016-07-31 05:54:53 [0]: SIGNAL_IN_PROGRESS Signal: deployment succeeded
2016-07-31 05:54:53 [0]: CREATE_COMPLETE state changed
2016-07-31 05:54:53 [overcloud-ComputeCephDeployment-56zf3zkhbhtq]: CREATE_COMPLETE Stack CREATE completed successfully
2016-07-31 05:54:53 [overcloud-ComputeAllNodesValidationDeployment-p3xhswchrxq2]: CREATE_IN_PROGRESS Stack CREATE started
2016-07-31 05:54:53 [1]: CREATE_IN_PROGRESS state changed
2016-07-31 05:54:54 [ComputeCephDeployment]: CREATE_COMPLETE state changed
2016-07-31 05:54:54 [0]: CREATE_IN_PROGRESS state changed
2016-07-31 05:54:58 [2]: SIGNAL_IN_PROGRESS Signal: deployment succeeded
2016-07-31 05:54:58 [2]: CREATE_COMPLETE state changed
2016-07-31 05:54:58 [2]: SIGNAL_IN_PROGRESS Signal: deployment succeeded
2016-07-31 05:54:59 [ControllerAllNodesDeployment]: CREATE_COMPLETE state changed
2016-07-31 05:54:59 [ControllerAllNodesValidationDeployment]: CREATE_IN_PROGRESS state changed
2016-07-31 05:54:59 [0]: SIGNAL_IN_PROGRESS Signal: deployment succeeded
2016-07-31 05:54:59 [0]: CREATE_COMPLETE state changed
2016-07-31 05:54:59 [overcloud-ControllerCephDeployment-2t5ix3hth3ty]: CREATE_COMPLETE Stack CREATE completed successfully
2016-07-31 05:54:59 [0]: SIGNAL_IN_PROGRESS Signal: deployment succeeded
2016-07-31 05:54:59 [0]: CREATE_COMPLETE state changed
2016-07-31 05:54:59 [2]: CREATE_COMPLETE state changed
2016-07-31 05:54:59 [overcloud-ControllerAllNodesDeployment-mqgisqdiqojq]: CREATE_COMPLETE Stack CREATE completed successfully
2016-07-31 05:55:00 [overcloud-ControllerAllNodesValidationDeployment-6ak7t5gcquby]: CREATE_IN_PROGRESS Stack CREATE started
2016-07-31 05:55:00 [1]: CREATE_IN_PROGRESS state changed
2016-07-31 05:55:00 [0]: CREATE_IN_PROGRESS state changed
2016-07-31 05:55:01 [2]: CREATE_IN_PROGRESS state changed
2016-07-31 05:55:05 [0]: SIGNAL_IN_PROGRESS Signal: deployment succeeded
2016-07-31 05:55:06 [0]: CREATE_COMPLETE state changed
2016-07-31 05:55:16 [1]: SIGNAL_IN_PROGRESS Signal: deployment succeeded
2016-07-31 05:55:17 [1]: CREATE_COMPLETE state changed
2016-07-31 05:55:17 [overcloud-ComputeAllNodesValidationDeployment-p3xhswchrxq2]: CREATE_COMPLETE Stack CREATE completed successfully
2016-07-31 05:55:18 [ComputeAllNodesValidationDeployment]: CREATE_COMPLETE state changed
2016-07-31 05:55:25 [0]: SIGNAL_IN_PROGRESS Signal: deployment succeeded
2016-07-31 05:55:26 [0]: CREATE_COMPLETE state changed
2016-07-31 05:55:26 [2]: SIGNAL_IN_PROGRESS Signal: deployment succeeded
2016-07-31 05:55:27 [2]: CREATE_COMPLETE state changed
2016-07-31 05:55:30 [1]: SIGNAL_IN_PROGRESS Signal: deployment succeeded
2016-07-31 05:55:31 [1]: CREATE_COMPLETE state changed
2016-07-31 05:55:31 [overcloud-ControllerAllNodesValidationDeployment-6ak7t5gcquby]: CREATE_COMPLETE Stack CREATE completed successfully
2016-07-31 05:55:32 [ControllerAllNodesValidationDeployment]: CREATE_COMPLETE state changed
2016-07-31 05:55:32 [AllNodesExtraConfig]: CREATE_IN_PROGRESS state changed
2016-07-31 05:55:32 [overcloud-AllNodesExtraConfig-yejvpdp543x4]: CREATE_IN_PROGRESS Stack CREATE started
2016-07-31 05:55:32 [overcloud-AllNodesExtraConfig-yejvpdp543x4]: CREATE_COMPLETE Stack CREATE completed successfully
2016-07-31 05:55:33 [AllNodesExtraConfig]: CREATE_COMPLETE state changed
2016-07-31 05:55:33 [ComputeNodesPostDeployment]: CREATE_IN_PROGRESS state changed
2016-07-31 05:55:34 [ObjectStorageNodesPostDeployment]: CREATE_IN_PROGRESS state changed
2016-07-31 05:55:34 [ControllerNodesPostDeployment]: CREATE_IN_PROGRESS state changed
2016-07-31 05:55:34 [overcloud-ComputeNodesPostDeployment-unvzoeo2d6h4]: CREATE_IN_PROGRESS Stack CREATE started
2016-07-31 05:55:34 [ComputeArtifactsConfig]: CREATE_IN_PROGRESS state changed
2016-07-31 05:55:34 [ComputePuppetConfig]: CREATE_IN_PROGRESS state changed
2016-07-31 05:55:34 [ComputePuppetConfig]: CREATE_COMPLETE state changed
2016-07-31 05:55:34 [overcloud-ComputeNodesPostDeployment-unvzoeo2d6h4-ComputeArtifactsConfig-g7rb4jq3zzo5]: CREATE_IN_PROGRESS Stack CREATE started
2016-07-31 05:55:34 [DeployArtifacts]: CREATE_IN_PROGRESS state changed
2016-07-31 05:55:34 [DeployArtifacts]: CREATE_COMPLETE state changed
2016-07-31 05:55:34 [overcloud-ComputeNodesPostDeployment-unvzoeo2d6h4-ComputeArtifactsConfig-g7rb4jq3zzo5]: CREATE_COMPLETE Stack CREATE completed successfully
2016-07-31 05:55:34 [overcloud-ObjectStorageNodesPostDeployment-64keujysbe5f]: CREATE_IN_PROGRESS Stack CREATE started
2016-07-31 05:55:34 [StorageRingbuilderPuppetConfig]: CREATE_IN_PROGRESS state changed
2016-07-31 05:55:34 [StorageArtifactsConfig]: CREATE_IN_PROGRESS state changed
2016-07-31 05:55:34 [StoragePuppetConfig]: CREATE_IN_PROGRESS state changed
2016-07-31 05:55:34 [StorageRingbuilderPuppetConfig]: CREATE_COMPLETE state changed
2016-07-31 05:55:34 [StoragePuppetConfig]: CREATE_COMPLETE state changed
2016-07-31 05:55:34 [overcloud-ObjectStorageNodesPostDeployment-64keujysbe5f-StorageArtifactsConfig-mqi2izssk2xf]: CREATE_IN_PROGRESS Stack CREATE started
2016-07-31 05:55:34 [DeployArtifacts]: CREATE_IN_PROGRESS state changed
2016-07-31 05:55:34 [DeployArtifacts]: CREATE_COMPLETE state changed
2016-07-31 05:55:34 [overcloud-ObjectStorageNodesPostDeployment-64keujysbe5f-StorageArtifactsConfig-mqi2izssk2xf]: CREATE_COMPLETE Stack CREATE completed successfully
2016-07-31 05:55:35 [ComputeArtifactsConfig]: CREATE_COMPLETE state changed
2016-07-31 05:55:35 [ComputeArtifactsDeploy]: CREATE_IN_PROGRESS state changed
2016-07-31 05:55:35 [StorageArtifactsConfig]: CREATE_COMPLETE state changed
2016-07-31 05:55:36 [overcloud-ComputeNodesPostDeployment-unvzoeo2d6h4-ComputeArtifactsDeploy-dtfanvkrdx3p]: CREATE_IN_PROGRESS Stack CREATE started
2016-07-31 05:55:36 [1]: CREATE_IN_PROGRESS state changed
2016-07-31 05:55:36 [StorageArtifactsDeploy]: CREATE_IN_PROGRESS state changed
2016-07-31 05:55:36 [overcloud-ObjectStorageNodesPostDeployment-64keujysbe5f-StorageArtifactsDeploy-fklf72gbmbum]: CREATE_IN_PROGRESS Stack CREATE started
2016-07-31 05:55:36 [overcloud-ObjectStorageNodesPostDeployment-64keujysbe5f-StorageArtifactsDeploy-fklf72gbmbum]: CREATE_COMPLETE Stack CREATE completed successfully
2016-07-31 05:55:37 [0]: CREATE_IN_PROGRESS state changed
2016-07-31 05:55:37 [StorageArtifactsDeploy]: CREATE_COMPLETE state changed
2016-07-31 05:55:37 [StorageDeployment_Step1]: CREATE_IN_PROGRESS state changed
2016-07-31 05:55:37 [overcloud-ObjectStorageNodesPostDeployment-64keujysbe5f-StorageDeployment_Step1-oerm7md7ixbe]: CREATE_IN_PROGRESS Stack CREATE started
2016-07-31 05:55:38 [StorageDeployment_Step1]: CREATE_COMPLETE state changed
2016-07-31 05:55:38 [StorageRingbuilderDeployment_Step2]: CREATE_IN_PROGRESS state changed
2016-07-31 05:55:38 [overcloud-ObjectStorageNodesPostDeployment-64keujysbe5f-StorageDeployment_Step1-oerm7md7ixbe]: CREATE_COMPLETE Stack CREATE completed successfully
2016-07-31 05:55:39 [overcloud-ObjectStorageNodesPostDeployment-64keujysbe5f-StorageRingbuilderDeployment_Step2-pgyombae2wtv]: CREATE_IN_PROGRESS Stack CREATE started
2016-07-31 05:55:39 [overcloud-ObjectStorageNodesPostDeployment-64keujysbe5f-StorageRingbuilderDeployment_Step2-pgyombae2wtv]: CREATE_COMPLETE Stack CREATE completed successfully
2016-07-31 05:55:40 [StorageRingbuilderDeployment_Step2]: CREATE_COMPLETE state changed
2016-07-31 05:55:40 [ExtraConfig]: CREATE_IN_PROGRESS state changed
2016-07-31 05:55:40 [overcloud-ObjectStorageNodesPostDeployment-64keujysbe5f-ExtraConfig-mqgvwxccn6r5]: CREATE_IN_PROGRESS Stack CREATE started
2016-07-31 05:55:40 [overcloud-ObjectStorageNodesPostDeployment-64keujysbe5f-ExtraConfig-mqgvwxccn6r5]: CREATE_COMPLETE Stack CREATE completed successfully
2016-07-31 05:55:41 [ExtraConfig]: CREATE_COMPLETE state changed
2016-07-31 05:55:41 [overcloud-ObjectStorageNodesPostDeployment-64keujysbe5f]: CREATE_COMPLETE Stack CREATE completed successfully
2016-07-31 05:55:42 [ObjectStorageNodesPostDeployment]: CREATE_COMPLETE state changed
2016-07-31 05:55:50 [0]: SIGNAL_IN_PROGRESS Signal: deployment succeeded
2016-07-31 05:55:50 [0]: CREATE_COMPLETE state changed
2016-07-31 05:56:04 [1]: SIGNAL_IN_PROGRESS Signal: deployment succeeded
2016-07-31 05:56:04 [1]: SIGNAL_IN_PROGRESS Signal: deployment succeeded
2016-07-31 05:56:05 [1]: CREATE_COMPLETE state changed
2016-07-31 05:56:05 [1]: CREATE_COMPLETE state changed
2016-07-31 05:56:05 [overcloud-ComputeNodesPostDeployment-unvzoeo2d6h4-ComputeArtifactsDeploy-dtfanvkrdx3p]: CREATE_COMPLETE Stack CREATE completed successfully
2016-07-31 05:56:06 [ComputeArtifactsDeploy]: CREATE_COMPLETE state changed
2016-07-31 05:56:06 [ComputePuppetDeployment]: CREATE_IN_PROGRESS state changed
2016-07-31 05:56:07 [overcloud-ComputeNodesPostDeployment-unvzoeo2d6h4-ComputePuppetDeployment-6iey3yhrflle]: CREATE_IN_PROGRESS Stack CREATE started
2016-07-31 05:56:07 [1]: CREATE_IN_PROGRESS state changed
2016-07-31 05:56:08 [0]: CREATE_IN_PROGRESS state changed
2016-07-31 05:56:09 [0]: SIGNAL_IN_PROGRESS Signal: deployment succeeded
2016-07-31 05:56:10 [0]: CREATE_COMPLETE state changed
2016-07-31 05:56:11 [2]: SIGNAL_IN_PROGRESS Signal: deployment succeeded
2016-07-31 05:56:12 [ControllerArtifactsDeploy]: CREATE_COMPLETE state changed
2016-07-31 05:56:12 [2]: CREATE_COMPLETE state changed
2016-07-31 05:56:12 [overcloud-ControllerNodesPostDeployment-jd3rmihnzq6a-ControllerArtifactsDeploy-rng3m5gi7vkn]: CREATE_COMPLETE Stack CREATE completed successfully
2016-07-31 05:56:13 [ControllerPrePuppet]: CREATE_COMPLETE state changed
2016-07-31 05:56:13 [ControllerLoadBalancerDeployment_Step1]: CREATE_IN_PROGRESS state changed
2016-07-31 05:56:13 [ControllerPrePuppetMaintenanceModeDeployment]: CREATE_COMPLETE state changed
2016-07-31 05:56:13 [overcloud-ControllerNodesPostDeployment-jd3rmihnzq6a-ControllerPrePuppet-sswsnxvmedf6]: CREATE_COMPLETE Stack CREATE completed successfully
2016-07-31 05:56:14 [overcloud-ControllerNodesPostDeployment-jd3rmihnzq6a-ControllerLoadBalancerDeployment_Step1-emrhcpofvvcc]: CREATE_IN_PROGRESS Stack CREATE started
2016-07-31 05:56:14 [1]: CREATE_IN_PROGRESS state changed
2016-07-31 05:56:14 [0]: CREATE_IN_PROGRESS state changed
2016-07-31 05:56:14 [2]: CREATE_IN_PROGRESS state changed
2016-07-31 05:58:04 [0]: SIGNAL_IN_PROGRESS Signal: deployment succeeded
2016-07-31 05:58:05 [0]: CREATE_COMPLETE state changed
2016-07-31 05:58:09 [1]: SIGNAL_IN_PROGRESS Signal: deployment succeeded
2016-07-31 05:58:09 [2]: SIGNAL_IN_PROGRESS Signal: deployment succeeded
2016-07-31 05:58:10 [ControllerLoadBalancerDeployment_Step1]: CREATE_COMPLETE state changed
2016-07-31 05:58:10 [ControllerServicesBaseDeployment_Step2]: CREATE_IN_PROGRESS state changed
2016-07-31 05:58:10 [1]: CREATE_COMPLETE state changed
2016-07-31 05:58:10 [2]: CREATE_COMPLETE state changed
2016-07-31 05:58:10 [overcloud-ControllerNodesPostDeployment-jd3rmihnzq6a-ControllerLoadBalancerDeployment_Step1-emrhcpofvvcc]: CREATE_COMPLETE Stack CREATE completed successfully
2016-07-31 05:58:11 [overcloud-ControllerNodesPostDeployment-jd3rmihnzq6a-ControllerServicesBaseDeployment_Step2-ofxpiumqvk2t]: CREATE_IN_PROGRESS Stack CREATE started
2016-07-31 05:58:11 [1]: CREATE_IN_PROGRESS state changed
2016-07-31 05:58:11 [0]: CREATE_IN_PROGRESS state changed
2016-07-31 05:58:11 [2]: CREATE_IN_PROGRESS state changed
2016-07-31 05:58:42 [1]: SIGNAL_IN_PROGRESS Signal: deployment succeeded
2016-07-31 05:58:43 [1]: CREATE_COMPLETE state changed
2016-07-31 05:58:43 [2]: SIGNAL_IN_PROGRESS Signal: deployment succeeded
2016-07-31 05:58:44 [2]: CREATE_COMPLETE state changed
2016-07-31 06:00:00 [0]: SIGNAL_IN_PROGRESS Signal: deployment succeeded
2016-07-31 06:00:01 [0]: CREATE_COMPLETE state changed
2016-07-31 06:00:01 [overcloud-ControllerNodesPostDeployment-jd3rmihnzq6a-ControllerServicesBaseDeployment_Step2-ofxpiumqvk2t]: CREATE_COMPLETE Stack CREATE completed successfully
2016-07-31 06:00:02 [ControllerServicesBaseDeployment_Step2]: CREATE_COMPLETE state changed
2016-07-31 06:00:02 [ControllerRingbuilderDeployment_Step3]: CREATE_IN_PROGRESS state changed
2016-07-31 06:00:03 [overcloud-ControllerNodesPostDeployment-jd3rmihnzq6a-ControllerRingbuilderDeployment_Step3-sekrzrwigsll]: CREATE_IN_PROGRESS Stack CREATE started
2016-07-31 06:00:03 [1]: CREATE_IN_PROGRESS state changed
2016-07-31 06:00:03 [0]: CREATE_IN_PROGRESS state changed
2016-07-31 06:00:04 [2]: CREATE_IN_PROGRESS state changed
2016-07-31 06:00:35 [0]: SIGNAL_IN_PROGRESS Signal: deployment succeeded
2016-07-31 06:00:35 [0]: CREATE_COMPLETE state changed
2016-07-31 06:00:49 [1]: SIGNAL_IN_PROGRESS Signal: deployment succeeded
2016-07-31 06:00:50 [1]: CREATE_COMPLETE state changed
2016-07-31 06:00:50 [2]: SIGNAL_IN_PROGRESS Signal: deployment succeeded
2016-07-31 06:00:51 [2]: CREATE_COMPLETE state changed
2016-07-31 06:00:51 [overcloud-ControllerNodesPostDeployment-jd3rmihnzq6a-ControllerRingbuilderDeployment_Step3-sekrzrwigsll]: CREATE_COMPLETE Stack CREATE completed successfully
2016-07-31 06:00:52 [ControllerRingbuilderDeployment_Step3]: CREATE_COMPLETE state changed
2016-07-31 06:00:52 [ControllerOvercloudServicesDeployment_Step4]: CREATE_IN_PROGRESS state changed
2016-07-31 06:00:52 [overcloud-ControllerNodesPostDeployment-jd3rmihnzq6a-ControllerOvercloudServicesDeployment_Step4-xanhnrjlhlck]: CREATE_IN_PROGRESS Stack CREATE started
2016-07-31 06:00:52 [1]: CREATE_IN_PROGRESS state changed
2016-07-31 06:00:53 [0]: CREATE_IN_PROGRESS state changed
2016-07-31 06:00:53 [2]: CREATE_IN_PROGRESS state changed
2016-07-31 06:03:09 [2]: SIGNAL_IN_PROGRESS Signal: deployment succeeded
2016-07-31 06:03:10 [2]: CREATE_COMPLETE state changed
2016-07-31 06:03:13 [1]: SIGNAL_IN_PROGRESS Signal: deployment succeeded
2016-07-31 06:03:14 [1]: CREATE_COMPLETE state changed
2016-07-31 06:04:12 [0]: SIGNAL_IN_PROGRESS Signal: deployment succeeded
2016-07-31 06:04:13 [ControllerOvercloudServicesDeployment_Step4]: CREATE_COMPLETE state changed
2016-07-31 06:04:13 [ControllerOvercloudServicesDeployment_Step5]: CREATE_IN_PROGRESS state changed
2016-07-31 06:04:13 [0]: CREATE_COMPLETE state changed
2016-07-31 06:04:13 [overcloud-ControllerNodesPostDeployment-jd3rmihnzq6a-ControllerOvercloudServicesDeployment_Step4-xanhnrjlhlck]: CREATE_COMPLETE Stack CREATE completed successfully
2016-07-31 06:04:14 [overcloud-ControllerNodesPostDeployment-jd3rmihnzq6a-ControllerOvercloudServicesDeployment_Step5-hebgu4yslwh2]: CREATE_IN_PROGRESS Stack CREATE started
2016-07-31 06:04:14 [1]: CREATE_IN_PROGRESS state changed
2016-07-31 06:04:14 [0]: CREATE_IN_PROGRESS state changed
2016-07-31 06:04:15 [2]: CREATE_IN_PROGRESS state changed
2016-07-31 06:05:29 [1]: SIGNAL_IN_PROGRESS Signal: deployment succeeded
2016-07-31 06:05:29 [1]: CREATE_COMPLETE state changed
2016-07-31 06:06:14 [2]: SIGNAL_IN_PROGRESS Signal: deployment succeeded
2016-07-31 06:06:14 [2]: CREATE_COMPLETE state changed
2016-07-31 06:06:33 [0]: SIGNAL_IN_PROGRESS Signal: deployment succeeded
2016-07-31 06:06:34 [ControllerOvercloudServicesDeployment_Step5]: CREATE_COMPLETE state changed
2016-07-31 06:06:34 [ControllerOvercloudServicesDeployment_Step6]: CREATE_IN_PROGRESS state changed
2016-07-31 06:06:34 [0]: CREATE_COMPLETE state changed
2016-07-31 06:06:34 [overcloud-ControllerNodesPostDeployment-jd3rmihnzq6a-ControllerOvercloudServicesDeployment_Step5-hebgu4yslwh2]: CREATE_COMPLETE Stack CREATE completed successfully
2016-07-31 06:06:34 [overcloud-ControllerNodesPostDeployment-jd3rmihnzq6a-ControllerOvercloudServicesDeployment_Step6-2mzciuctoola]: CREATE_IN_PROGRESS Stack CREATE started
2016-07-31 06:06:34 [1]: CREATE_IN_PROGRESS state changed
2016-07-31 06:06:35 [0]: CREATE_IN_PROGRESS state changed
2016-07-31 06:06:35 [2]: CREATE_IN_PROGRESS state changed
2016-07-31 06:08:09 [1]: SIGNAL_IN_PROGRESS Signal: deployment succeeded
2016-07-31 06:08:10 [1]: CREATE_COMPLETE state changed
2016-07-31 06:08:25 [2]: SIGNAL_IN_PROGRESS Signal: deployment succeeded
2016-07-31 06:08:25 [2]: CREATE_COMPLETE state changed
2016-07-31 06:09:29 [0]: SIGNAL_IN_PROGRESS Signal: deployment succeeded
2016-07-31 06:09:30 [0]: CREATE_COMPLETE state changed
2016-07-31 06:09:30 [overcloud-ControllerNodesPostDeployment-jd3rmihnzq6a-ControllerOvercloudServicesDeployment_Step6-2mzciuctoola]: CREATE_COMPLETE Stack CREATE completed successfully
2016-07-31 06:09:31 [ControllerOvercloudServicesDeployment_Step6]: CREATE_COMPLETE state changed
2016-07-31 06:09:31 [ControllerPostPuppet]: CREATE_IN_PROGRESS state changed
2016-07-31 06:09:31 [overcloud-ControllerNodesPostDeployment-jd3rmihnzq6a-ControllerPostPuppet-bceb6374n3yu]: CREATE_IN_PROGRESS Stack CREATE started
2016-07-31 06:09:31 [ControllerPostPuppetMaintenanceModeConfig]: CREATE_IN_PROGRESS state changed
2016-07-31 06:09:31 [ControllerPostPuppetRestartConfig]: CREATE_IN_PROGRESS state changed
2016-07-31 06:09:31 [ControllerPostPuppetMaintenanceModeConfig]: CREATE_COMPLETE state changed
2016-07-31 06:09:31 [ControllerPostPuppetRestartConfig]: CREATE_COMPLETE state changed
2016-07-31 06:09:31 [ControllerPostPuppetMaintenanceModeDeployment]: CREATE_IN_PROGRESS state changed
2016-07-31 06:10:36 [ControllerPostPuppetMaintenanceModeDeployment]: CREATE_COMPLETE state changed
2016-07-31 06:10:36 [ControllerPostPuppetRestartDeployment]: CREATE_IN_PROGRESS state changed
2016-07-31 06:10:54 [1]: SIGNAL_IN_PROGRESS Signal: deployment succeeded
2016-07-31 06:10:54 [0]: SIGNAL_IN_PROGRESS Signal: deployment succeeded
2016-07-31 06:10:54 [0]: CREATE_COMPLETE state changed
2016-07-31 06:10:59 [1]: CREATE_COMPLETE state changed
2016-07-31 06:11:00 [overcloud-ComputeNodesPostDeployment-unvzoeo2d6h4-ComputePuppetDeployment-6iey3yhrflle]: CREATE_COMPLETE Stack CREATE completed successfully
2016-07-31 06:11:01 [ComputePuppetDeployment]: CREATE_COMPLETE state changed
2016-07-31 06:11:01 [ExtraConfig]: CREATE_IN_PROGRESS state changed
2016-07-31 06:11:02 [overcloud-ComputeNodesPostDeployment-unvzoeo2d6h4-ExtraConfig-bq5rbyg2xnor]: CREATE_IN_PROGRESS Stack CREATE started
2016-07-31 06:11:02 [overcloud-ComputeNodesPostDeployment-unvzoeo2d6h4-ExtraConfig-bq5rbyg2xnor]: CREATE_COMPLETE Stack CREATE completed successfully
2016-07-31 06:11:03 [ExtraConfig]: CREATE_COMPLETE state changed
2016-07-31 06:11:03 [overcloud-ComputeNodesPostDeployment-unvzoeo2d6h4]: CREATE_COMPLETE Stack CREATE completed successfully
2016-07-31 06:11:04 [ComputeNodesPostDeployment]: CREATE_COMPLETE state changed
2016-07-31 06:11:26 [ControllerPostPuppetRestartDeployment]: CREATE_COMPLETE state changed
2016-07-31 06:11:26 [overcloud-ControllerNodesPostDeployment-jd3rmihnzq6a-ControllerPostPuppet-bceb6374n3yu]: CREATE_COMPLETE Stack CREATE completed successfully
2016-07-31 06:11:27 [ControllerPostPuppet]: CREATE_COMPLETE state changed
2016-07-31 06:11:27 [ExtraConfig]: CREATE_IN_PROGRESS state changed
2016-07-31 06:11:28 [overcloud-ControllerNodesPostDeployment-jd3rmihnzq6a-ExtraConfig-jhzhhys3nwsd]: CREATE_IN_PROGRESS Stack CREATE started
2016-07-31 06:11:28 [overcloud-ControllerNodesPostDeployment-jd3rmihnzq6a-ExtraConfig-jhzhhys3nwsd]: CREATE_COMPLETE Stack CREATE completed successfully
2016-07-31 06:11:29 [ExtraConfig]: CREATE_COMPLETE state changed
2016-07-31 06:11:29 [overcloud-ControllerNodesPostDeployment-jd3rmihnzq6a]: CREATE_COMPLETE Stack CREATE completed successfully
2016-07-31 06:11:29 [ControllerNodesPostDeployment]: CREATE_COMPLETE state changed
2016-07-31 06:11:30 [CephStorageNodesPostDeployment]: CREATE_IN_PROGRESS state changed
2016-07-31 06:11:30 [BlockStorageNodesPostDeployment]: CREATE_IN_PROGRESS state changed
2016-07-31 06:11:30 [overcloud-CephStorageNodesPostDeployment-g42ajs2fb2wz]: CREATE_IN_PROGRESS Stack CREATE started
2016-07-31 06:11:30 [CephStorageArtifactsConfig]: CREATE_IN_PROGRESS state changed
2016-07-31 06:11:31 [overcloud-BlockStorageNodesPostDeployment-dxs3ionfumox]: CREATE_IN_PROGRESS Stack CREATE started
2016-07-31 06:11:31 [VolumeArtifactsConfig]: CREATE_IN_PROGRESS state changed
2016-07-31 06:11:31 [CephStoragePuppetConfig]: CREATE_IN_PROGRESS state changed
2016-07-31 06:11:31 [CephStoragePuppetConfig]: CREATE_COMPLETE state changed
2016-07-31 06:11:31 [overcloud-CephStorageNodesPostDeployment-g42ajs2fb2wz-CephStorageArtifactsConfig-6v5avhacddmk]: CREATE_IN_PROGRESS Stack CREATE started
2016-07-31 06:11:31 [DeployArtifacts]: CREATE_IN_PROGRESS state changed
2016-07-31 06:11:31 [DeployArtifacts]: CREATE_COMPLETE state changed
2016-07-31 06:11:31 [overcloud-CephStorageNodesPostDeployment-g42ajs2fb2wz-CephStorageArtifactsConfig-6v5avhacddmk]: CREATE_COMPLETE Stack CREATE completed successfully
2016-07-31 06:11:32 [overcloud-BlockStorageNodesPostDeployment-dxs3ionfumox-VolumeArtifactsConfig-ecc3vkpmyh5b]: CREATE_IN_PROGRESS Stack CREATE started
2016-07-31 06:11:32 [DeployArtifacts]: CREATE_IN_PROGRESS state changed
2016-07-31 06:11:32 [DeployArtifacts]: CREATE_COMPLETE state changed
2016-07-31 06:11:32 [overcloud-BlockStorageNodesPostDeployment-dxs3ionfumox-VolumeArtifactsConfig-ecc3vkpmyh5b]: CREATE_COMPLETE Stack CREATE completed successfully
2016-07-31 06:11:32 [CephStorageArtifactsConfig]: CREATE_COMPLETE state changed
2016-07-31 06:11:32 [CephStorageArtifactsDeploy]: CREATE_IN_PROGRESS state changed
2016-07-31 06:11:32 [overcloud-CephStorageNodesPostDeployment-g42ajs2fb2wz-CephStorageArtifactsDeploy-agdpcpxpmmg5]: CREATE_IN_PROGRESS Stack CREATE started
2016-07-31 06:11:32 [overcloud-CephStorageNodesPostDeployment-g42ajs2fb2wz-CephStorageArtifactsDeploy-agdpcpxpmmg5]: CREATE_COMPLETE Stack CREATE completed successfully
2016-07-31 06:11:33 [VolumeArtifactsConfig]: CREATE_COMPLETE state changed
2016-07-31 06:11:33 [VolumeArtifactsDeploy]: CREATE_IN_PROGRESS state changed
2016-07-31 06:11:33 [overcloud-BlockStorageNodesPostDeployment-dxs3ionfumox-VolumeArtifactsDeploy-uaa5xotz2wxt]: CREATE_IN_PROGRESS Stack CREATE started
2016-07-31 06:11:33 [overcloud-BlockStorageNodesPostDeployment-dxs3ionfumox-VolumeArtifactsDeploy-uaa5xotz2wxt]: CREATE_COMPLETE Stack CREATE completed successfully
2016-07-31 06:11:33 [CephStorageArtifactsDeploy]: CREATE_COMPLETE state changed
2016-07-31 06:11:33 [CephStorageDeployment_Step1]: CREATE_IN_PROGRESS state changed
2016-07-31 06:11:34 [VolumeArtifactsDeploy]: CREATE_COMPLETE state changed
2016-07-31 06:11:34 [VolumePuppetConfig]: CREATE_IN_PROGRESS state changed
2016-07-31 06:11:34 [overcloud-CephStorageNodesPostDeployment-g42ajs2fb2wz-CephStorageDeployment_Step1-s677wxrlxtvk]: CREATE_IN_PROGRESS Stack CREATE started
2016-07-31 06:11:34 [overcloud-CephStorageNodesPostDeployment-g42ajs2fb2wz-CephStorageDeployment_Step1-s677wxrlxtvk]: CREATE_COMPLETE Stack CREATE completed successfully
2016-07-31 06:11:35 [VolumePuppetConfig]: CREATE_COMPLETE state changed
2016-07-31 06:11:35 [VolumeDeployment_Step1]: CREATE_IN_PROGRESS state changed
2016-07-31 06:11:35 [overcloud-BlockStorageNodesPostDeployment-dxs3ionfumox-VolumeDeployment_Step1-bympimwcluwg]: CREATE_IN_PROGRESS Stack CREATE started
2016-07-31 06:11:35 [CephStorageDeployment_Step1]: CREATE_COMPLETE state changed
2016-07-31 06:11:35 [ExtraConfig]: CREATE_IN_PROGRESS state changed
2016-07-31 06:11:35 [overcloud-CephStorageNodesPostDeployment-g42ajs2fb2wz-ExtraConfig-ej46y7xnajub]: CREATE_IN_PROGRESS Stack CREATE started
2016-07-31 06:11:35 [overcloud-CephStorageNodesPostDeployment-g42ajs2fb2wz-ExtraConfig-ej46y7xnajub]: CREATE_COMPLETE Stack CREATE completed successfully
2016-07-31 06:11:36 [VolumeDeployment_Step1]: CREATE_COMPLETE state changed
2016-07-31 06:11:36 [ExtraConfig]: CREATE_IN_PROGRESS state changed
2016-07-31 06:11:36 [overcloud-BlockStorageNodesPostDeployment-dxs3ionfumox-VolumeDeployment_Step1-bympimwcluwg]: CREATE_COMPLETE Stack CREATE completed successfully
2016-07-31 06:11:36 [ExtraConfig]: CREATE_COMPLETE state changed
2016-07-31 06:11:36 [overcloud-CephStorageNodesPostDeployment-g42ajs2fb2wz]: CREATE_COMPLETE Stack CREATE completed successfully
2016-07-31 06:11:37 [CephStorageNodesPostDeployment]: CREATE_COMPLETE state changed
2016-07-31 06:11:37 [overcloud-BlockStorageNodesPostDeployment-dxs3ionfumox-ExtraConfig-on2vflovtnsn]: CREATE_IN_PROGRESS Stack CREATE started
2016-07-31 06:11:37 [overcloud-BlockStorageNodesPostDeployment-dxs3ionfumox-ExtraConfig-on2vflovtnsn]: CREATE_COMPLETE Stack CREATE completed successfully
2016-07-31 06:11:38 [ExtraConfig]: CREATE_COMPLETE state changed
2016-07-31 06:11:38 [overcloud-BlockStorageNodesPostDeployment-dxs3ionfumox]: CREATE_COMPLETE Stack CREATE completed successfully
2016-07-31 06:11:39 [BlockStorageNodesPostDeployment]: CREATE_COMPLETE state changed
2016-07-31 06:11:39 [overcloud]: CREATE_COMPLETE Stack CREATE completed successfully
Stack overcloud CREATE_COMPLETE
Skipping "horizon" postconfig because it wasn't found in the endpoint map output
PKI initialization in init-keystone is deprecated and will be removed.
Warning: Permanently added '192.0.2.6' (ECDSA) to the list of known hosts.
The following cert files already exist, use --rebuild to remove the existing files before regenerating:
/etc/keystone/ssl/certs/ca.pem already exists
/etc/keystone/ssl/private/signing_key.pem already exists
/etc/keystone/ssl/certs/signing_cert.pem already exists
Connection to 192.0.2.6 closed.
Overcloud Endpoint: http://10.0.0.4:5000/v2.0
Overcloud Deployed
[stack@instack ~]$ nova list
+--------------------------------------+-------------------------+--------+------------+-------------+---------------------+
| ID                                   | Name                    | Status | Task State | Power State | Networks            |
+--------------------------------------+-------------------------+--------+------------+-------------+---------------------+
| 9055adbe-bbfa-4a4c-b7f9-5570b6da03a7 | overcloud-controller-0  | ACTIVE | -          | Running     | ctlplane=192.0.2.9  |
| 4285ddee-0368-461a-859c-f80a1b29b9a1 | overcloud-controller-1  | ACTIVE | -          | Running     | ctlplane=192.0.2.8  |
| 579f7e9f-aa4d-4415-82ad-ec48c86553b9 | overcloud-controller-2  | ACTIVE | -          | Running     | ctlplane=192.0.2.10 |
| 39eb184e-8ff1-4267-82a8-b7f5f1ffa1e7 | overcloud-novacompute-0 | ACTIVE | -          | Running     | ctlplane=192.0.2.11 |
| d5084553-b32d-47ac-93b6-d3188d1c3bfb | overcloud-novacompute-1 | ACTIVE | -          | Running     | ctlplane=192.0.2.7  |
+--------------------------------------+-------------------------+--------+------------+-------------+---------------------+
[stack@instack ~]$ neutron net-list
+--------------------------------------+--------------+--------------------------------------------+
| id                                   | name         | subnets                                    |
+--------------------------------------+--------------+--------------------------------------------+
| 67212060-3712-4ac7-b321-7f11ef2f24c2 | tenant       | 82b41d3a-25ef-4593-a0f8-dcc3f4a23036       |
|                                      |              | 172.16.0.0/24                              |
| 72368be6-9bb1-4dbc-b823-4247599a29f2 | storage      | 138fbd64-a741-452e-8e79-84690f7b4f1c       |
|                                      |              | 172.16.1.0/24                              |
| cdf13bb1-5c69-410d-b262-6f2251f0aa1b | storage_mgmt | adf3fc93-1f14-43f4-ac59-9643aa0e8854       |
|                                      |              | 172.16.3.0/24                              |
| 4b80d4bf-f3b0-4d53-901d-f65bb089d335 | external     | e58d6ef8-88ea-4737-9732-ae79b7b889ee       |
|                                      |              | 10.0.0.0/24                                |
| 600ad88d-efe4-432b-83e8-d85c4f9ade81 | internal_api | a85b388e-478f-4db6-93f6-6a0d7e3edca5       |
|                                      |              | 172.16.2.0/24                              |
| f53a98b2-18f8-4e92-8047-779651615a49 | ctlplane     | 3b7d83e5-0bf3-4ba5-8008-4aa401598065       |
|                                      |              | 192.0.2.0/24                               |
+--------------------------------------+--------------+--------------------------------------------+