Friday, July 31, 2015

CPU Pinning and NUMA Topology on RDO Kilo on Fedora Server 22

Posting bellow follows up http://redhatstackblog.redhat.com/2015/05/05/cpu-pinning-and-numa-topology-awareness-in-openstack-compute/
on RDO Kilo installed on Fedora 22 . After upgrade to upstream version
of openstack-puppet-modules-2015.1.9 procedure of RDO Kilo install on F22
significantly changed. Details follow bellow :-

*****************************************************************************************
RDO Kilo set up on Fedora ( openstack-puppet-modules-2015.1.9-4.fc23.noarch)
*****************************************************************************************
Follow RDO Quick Start page https://www.rdoproject.org/Quickstart
Generate answer-file and make update :-
# packstack  --gen-answer-file answer-file-aio.txt
   and set CONFIG_KEYSTONE_SERVICE_NAME=httpd

****************************************************************************
I also commented out second line in  /etc/httpd/conf.d/mod_dnssd.conf
****************************************************************************
 # packstack  --answer-file=./answer-file-aio.txt
Workaround ( at the point on neutron puppet failure run ) :-

# chown neutron:neutron /etc/neutron/plugings/ml2/ml2_conf.ini 
Rerun packstack :-
# packstack  --answer-file=./answer-file-aio.txt 

Final target is to reproduce mentioned article on i7 4790 Haswell CPU box, perform launching nova instance with CPU pinning.

 [root@fedora22server ~(keystone_admin)]# uname -a
Linux fedora22server.localdomain 4.1.3-200.fc22.x86_64 #1 SMP Wed Jul 22 19:51:58 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

[root@fedora22server ~(keystone_admin)]# rpm -qa \*qemu\*
qemu-system-x86-2.3.0-6.fc22.x86_64
qemu-img-2.3.0-6.fc22.x86_64
qemu-guest-agent-2.3.0-6.fc22.x86_64
qemu-kvm-2.3.0-6.fc22.x86_64
ipxe-roms-qemu-20150407-1.gitdc795b9f.fc22.noarch
qemu-common-2.3.0-6.fc22.x86_64
libvirt-daemon-driver-qemu-1.2.13.1-2.fc22.x86_64


[root@fedora22server ~(keystone_admin)]# numactl --hardware
available: 1 nodes (0)
node 0 cpus: 0 1 2 3 4 5 6 7
node 0 size: 15991 MB
node 0 free: 4399 MB
node distances:
node   0
  0:  10

[root@fedora22server ~(keystone_admin)]# virsh capabilities
<capabilities>
<host>
    <uuid>00fd5d2c-dad7-dd11-ad7e-7824af431b53</uuid>
    <cpu>
      <arch>x86_64</arch>
      <model>Haswell-noTSX</model>
      <vendor>Intel</vendor>
      <topology sockets='1' cores='4' threads='2'/>
      <feature name='invtsc'/>
      <feature name='abm'/>
      <feature name='pdpe1gb'/>
      <feature name='rdrand'/>
      <feature name='f16c'/>
      <feature name='osxsave'/>
      <feature name='pdcm'/>
      <feature name='xtpr'/>
      <feature name='tm2'/>
      <feature name='est'/>
      <feature name='smx'/>
      <feature name='vmx'/>
      <feature name='ds_cpl'/>
      <feature name='monitor'/>
      <feature name='dtes64'/>
      <feature name='pbe'/>
      <feature name='tm'/>
      <feature name='ht'/>
      <feature name='ss'/>
      <feature name='acpi'/>
      <feature name='ds'/>
      <feature name='vme'/>
      <pages unit='KiB' size='4'/>
      <pages unit='KiB' size='2048'/>
    </cpu>
    <power_management>
      <suspend_mem/>
      <suspend_disk/>
      <suspend_hybrid/>
    </power_management>
    <migration_features>
      <live/>
      <uri_transports>
        <uri_transport>tcp</uri_transport>
        <uri_transport>rdma</uri_transport>
      </uri_transports>
    </migration_features>
    <topology>
      <cells num='1'>
        <cell id='0'>
          <memory unit='KiB'>16374824</memory>
          <pages unit='KiB' size='4'>4093706</pages>
          <pages unit='KiB' size='2048'>0</pages>
          <distances>
            <sibling id='0' value='10'/>
          </distances>
          <cpus num='8'>
            <cpu id='0' socket_id='0' core_id='0' siblings='0,4'/>
            <cpu id='1' socket_id='0' core_id='1' siblings='1,5'/>
            <cpu id='2' socket_id='0' core_id='2' siblings='2,6'/>
            <cpu id='3' socket_id='0' core_id='3' siblings='3,7'/>
            <cpu id='4' socket_id='0' core_id='0' siblings='0,4'/>
            <cpu id='5' socket_id='0' core_id='1' siblings='1,5'/>
            <cpu id='6' socket_id='0' core_id='2' siblings='2,6'/>
            <cpu id='7' socket_id='0' core_id='3' siblings='3,7'/>
          </cpus>
        </cell>
      </cells>
    </topology>

On each Compute node that pinning of virtual machines will be permitted on open the /etc/nova/nova.conf file and make the following modifications:
  • Set the vcpu_pin_set value to a list or range of logical CPU cores  to reserve for virtual machine processes. OpenStack Compute will ensure guest virtual machine instances are pinned to these virtual CPU cores. 
  • vcpu_pin_set=2,3,6,7
  • Set the reserved_host_memory_mb to reserve RAM for host processes. For the purposes of testing used  the default of 512 MB: 
  • reserved_host_memory_mb=512 
# systemctl restart openstack-nova-compute.service

************************************
SCHEDULER CONFIGURATION
************************************
Update /etc/nova/nova.conf

scheduler_default_filters=RetryFilter,AvailabilityZoneFilter,RamFilter,
ComputeFilter,ComputeCapabilitiesFilter,ImagePropertiesFilter,CoreFilter,
NUMATopologyFilter,AggregateInstanceExtraSpecsFilter

# systemctl restart openstack-nova-scheduler.service


 At this point if  creating  a guest you may see some changes to appear in the XML, pinning the guest vCPU(s) to the cores listed in vcpu_pin_set:

<vcpu placement='static' cpuset='2-3,6-7'>1</vcpu>

Add to vmlinuz grub2 command line at the end 
isolcpus=2,3,6,7

***************
REBOOT
***************

[root@fedora22server ~(keystone_admin)]# nova aggregate-create performance
+----+-------------+-------------------+-------+----------+
| Id | Name        | Availability Zone | Hosts | Metadata |
+----+-------------+-------------------+-------+----------+
| 1  | performance | -                 |       |          |
+----+-------------+-------------------+-------+----------+

[root@fedora22server ~(keystone_admin)]# nova aggregate-set-metadata 1 pinned=true
Metadata has been successfully updated for aggregate 1.
+----+-------------+-------------------+-------+---------------+
| Id | Name        | Availability Zone | Hosts | Metadata      |
+----+-------------+-------------------+-------+---------------+
| 1  | performance | -                 |       | 'pinned=true' |
+----+-------------+-------------------+-------+---------------+

[root@fedora22server ~(keystone_admin)]# nova flavor-create m1.small.performance 6 4096 20 4
+----+----------------------+-----------+------+-----------+------+-------+-------------+-----------+
| ID | Name                 | Memory_MB | Disk | Ephemeral | Swap | VCPUs | RXTX_Factor | Is_Public |
+----+----------------------+-----------+------+-----------+------+-------+-------------+-----------+
| 6  | m1.small.performance | 4096      | 20   | 0         |      | 4     | 1.0         | True      |
+----+----------------------+-----------+------+-----------+------+-------+-------------+-----------+
[root@fedora22server ~(keystone_admin)]# nova flavor-key 6 set hw:cpu_policy=dedicated

[root@fedora22server ~(keystone_admin)]# nova flavor-key 6 set aggregate_instance_extra_specs:pinned=true

[root@fedora22server ~(keystone_admin)]# hostname
fedora22server.localdomain

[root@fedora22server ~(keystone_admin)]# nova aggregate-add-host 1 fedora22server.localdomain
Host fedora22server.localdomain has been successfully added for aggregate 1
+----+-------------+-------------------+------------------------------+---------------+
| Id | Name        | Availability Zone | Hosts                        | Metadata      |
+----+-------------+-------------------+------------------------------+---------------+
| 1  | performance | -                 | 'fedora22server.localdomain' | 'pinned=true' |
+----+-------------+-------------------+------------------------------+---------------+

[root@fedora22server ~(keystone_admin)]# . keystonerc_demo
[root@fedora22server ~(keystone_demo)]# glance image-list
+--------------------------------------+---------------------------------+-------------+------------------+-------------+--------+
| ID                                   | Name                            | Disk Format | Container Format | Size        | Status |
+--------------------------------------+---------------------------------+-------------+------------------+-------------+--------+
| bf6f5272-ae26-49ae-b0f9-3c4fcba350f6 | CentOS71Image                   | qcow2       | bare             | 1004994560  | active |
| 05ac955e-3503-4bcf-8413-6a1b3c98aefa | cirros                          | qcow2       | bare             | 13200896    | active |
| 7b2085b8-4fe7-4d32-a5c9-5eadaf8bfc52 | VF22Image                       | qcow2       | bare             | 228599296   | active |
| c695e7fa-a69f-4220-abd8-2269b75af827 | Windows Server 2012 R2 Std Eval | qcow2       | bare             | 17182752768 | active |
+--------------------------------------+---------------------------------+-------------+------------------+-------------+--------+

[root@fedora22server ~(keystone_demo)]#neutron net-list
+--------------------------------------+----------+-----------------------------------------------------+
| id                                   | name     | subnets                                             |
+--------------------------------------+----------+-----------------------------------------------------+
| 0daa3a02-c598-4c46-b1ac-368da5542927 | public   | 8303b2f3-2de2-44c2-bd5e-fc0966daec53 192.168.1.0/24 |
| c85a4215-1558-4a95-886d-a2f75500e052 | demo_net | 0cab6cbc-dd80-42c6-8512-74d7b2cbf730 50.0.0.0/24    |
+--------------------------------------+----------+-----------------------------------------------------+

*************************************************************************
At this point attempt to launch F22 Cloud instance with created flavor
m1.small.performance
*************************************************************************

[root@fedora22server ~(keystone_demo)]# nova boot --image  7b2085b8-4fe7-4d32-a5c9-5eadaf8bfc52 --key-name oskeydev --flavor  m1.small.performance --nic net-id=c85a4215-1558-4a95-886d-a2f75500e052 vf22-instance

+--------------------------------------+--------------------------------------------------+
| Property                             | Value                                            |
+--------------------------------------+--------------------------------------------------+
| OS-DCF:diskConfig                    | MANUAL                                           |
| OS-EXT-AZ:availability_zone          | nova                                             |
| OS-EXT-STS:power_state               | 0                                                |
| OS-EXT-STS:task_state                | scheduling                                       |
| OS-EXT-STS:vm_state                  | building                                         |
| OS-SRV-USG:launched_at               | -                                                |
| OS-SRV-USG:terminated_at             | -                                                |
| accessIPv4                           |                                                  |
| accessIPv6                           |                                                  |
| adminPass                            | XsGr87ZLGX8P                                     |
| config_drive                         |                                                  |
| created                              | 2015-07-31T08:03:49Z                             |
| flavor                               | m1.small.performance (6)                         |
| hostId                               |                                                  |
| id                                   | 4b99f3cf-3126-48f3-9e00-94787f040e43             |
| image                                | VF22Image (7b2085b8-4fe7-4d32-a5c9-5eadaf8bfc52) |
| key_name                             | oskeydev                                         |
| metadata                             | {}                                               |
| name                                 | vf22-instance                                    |
| os-extended-volumes:volumes_attached | []                                               |
| progress                             | 0                                                |
| security_groups                      | default                                          |
| status                               | BUILD                                            |
| tenant_id                            | 14f736e6952644b584b2006353ca51be                 |
| updated                              | 2015-07-31T08:03:50Z                             |
| user_id                              | 4ece2385b17a4490b6fc5a01ff53350c                 |
+--------------------------------------+--------------------------------------------------+
[root@fedora22server ~(keystone_demo)]#nova list
+--------------------------------------+---------------+---------+------------+-------------+-----------------------------------+
| ID                                   | Name          | Status  | Task State | Power State | Networks                          |
+--------------------------------------+---------------+---------+------------+-------------+-----------------------------------+
| 93906a61-ec0b-481d-b964-2bb99d095646 | CentOS71RLX   | SHUTOFF | -          | Shutdown    | demo_net=50.0.0.21, 192.168.1.159 |
| ac7e9be5-d2dc-4ec0-b0a1-4096b552e578 | VF22Devpin    | ACTIVE  | -          | Running     | demo_net=50.0.0.22                |
| b93c9526-ded5-4b7a-ae3a-106b34317744 | VF22Devs      | SHUTOFF | -          | Shutdown    | demo_net=50.0.0.19, 192.168.1.157 |
| bef20a1e-3faa-4726-a301-73ca49666fa6 | WinSrv2012    | SHUTOFF | -          | Shutdown    | demo_net=50.0.0.16                |
| 4b99f3cf-3126-48f3-9e00-94787f040e43 | vf22-instance | ACTIVE  | -          | Running     | demo_net=50.0.0.23, 192.168.1.160                |
+--------------------------------------+---------------+---------+------------+-------------+-----------------------------------+
[root@fedora22server ~(keystone_demo)]#virsh list
 Id    Name                           State
----------------------------------------------------
 2     instance-0000000c              running
 3     instance-0000000d              running

Please, see http://redhatstackblog.redhat.com/2015/05/05/cpu-pinning-and-numa-topology-awareness-in-openstack-compute/
regarding detailed explanation of highlighted blocks, keeping in mind that pinning is done to logical CPU cores ( not physical due to 4 Core CPU with HT enabled ). Multiple cells are also absent, due limitations of i7 47XX Haswell CPU architecture

[root@fedora22server ~(keystone_demo)]#virsh dumpxml instance-0000000d > vf22-instance.xml
<domain type='kvm' id='3'>
  <name>instance-0000000d</name>
  <uuid>4b99f3cf-3126-48f3-9e00-94787f040e43</uuid>
  <metadata>
    <nova:instance xmlns:nova="http://openstack.org/xmlns/libvirt/nova/1.0">
      <nova:package version="2015.1.0-3.fc23"/>
      <nova:name>vf22-instance</nova:name>
      <nova:creationTime>2015-07-31 08:03:54</nova:creationTime>
      <nova:flavor name="m1.small.performance">
        <nova:memory>4096</nova:memory>
        <nova:disk>20</nova:disk>
        <nova:swap>0</nova:swap>
        <nova:ephemeral>0</nova:ephemeral>
        <nova:vcpus>4</nova:vcpus>
      </nova:flavor>
      <nova:owner>
        <nova:user uuid="4ece2385b17a4490b6fc5a01ff53350c">demo</nova:user>
        <nova:project uuid="14f736e6952644b584b2006353ca51be">demo</nova:project>
      </nova:owner>
      <nova:root type="image" uuid="7b2085b8-4fe7-4d32-a5c9-5eadaf8bfc52"/>
    </nova:instance>
  </metadata>
  <memory unit='KiB'>4194304</memory>
  <currentMemory unit='KiB'>4194304</currentMemory>
  <vcpu placement='static'>4</vcpu>
  <cputune>
    <shares>4096</shares>
    <vcpupin vcpu='0' cpuset='2'/>
    <vcpupin vcpu='1' cpuset='6'/>
    <vcpupin vcpu='2' cpuset='3'/>
    <vcpupin vcpu='3' cpuset='7'/>
    <emulatorpin cpuset='2-3,6-7'/>
  </cputune>
  <numatune>
    <memory mode='strict' nodeset='0'/>
    <memnode cellid='0' mode='strict' nodeset='0'/>
  </numatune>

  <resource>
    <partition>/machine</partition>
  </resource>
  <sysinfo type='smbios'>
    <system>
      <entry name='manufacturer'>Fedora Project</entry>
      <entry name='product'>OpenStack Nova</entry>
      <entry name='version'>2015.1.0-3.fc23</entry>
      <entry name='serial'>f1b336b1-6abf-4180-865a-b6be5670352e</entry>
      <entry name='uuid'>4b99f3cf-3126-48f3-9e00-94787f040e43</entry>
    </system>
  </sysinfo>
  <os>
    <type arch='x86_64' machine='pc-i440fx-2.3'>hvm</type>
    <boot dev='hd'/>
    <smbios mode='sysinfo'/>
  </os>
  <features>
    <acpi/>
    <apic/>
  </features>
  <cpu mode='host-model'>
    <model fallback='allow'/>
    <topology sockets='2' cores='1' threads='2'/>
    <numa>
      <cell id='0' cpus='0-3' memory='4194304' unit='KiB'/>
    </numa>
  </cpu>

  <clock offset='utc'>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <devices>
    <emulator>/usr/bin/qemu-kvm</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2' cache='none'/>
      <source file='/var/lib/nova/instances/4b99f3cf-3126-48f3-9e00-94787f040e43/disk'/>
      <backingStore type='file' index='1'>
        <format type='raw'/>
        <source file='/var/lib/nova/instances/_base/6c60a5ed1b3037bbdb2bed198dac944f4c0d09cb'/>
        <backingStore/>
      </backingStore>
      <target dev='vda' bus='virtio'/>
      <alias name='virtio-disk0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </disk>
    <controller type='usb' index='0'>
      <alias name='usb0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'>
      <alias name='pci.0'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <alias name='virtio-serial0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </controller>
    <interface type='bridge'>
      <mac address='fa:16:3e:4f:25:03'/>
      <source bridge='qbr567b21fe-52'/>
      <target dev='tap567b21fe-52'/>
      <model type='virtio'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
    <serial type='file'>
      <source path='/var/lib/nova/instances/4b99f3cf-3126-48f3-9e00-94787f040e43/console.log'/>
      <target port='0'/>
      <alias name='serial0'/>
    </serial>
    <serial type='pty'>
      <source path='/dev/pts/2'/>
      <target port='1'/>
      <alias name='serial1'/>
    </serial>
    <console type='file'>
      <source path='/var/lib/nova/instances/4b99f3cf-3126-48f3-9e00-94787f040e43/console.log'/>
      <target type='serial' port='0'/>
      <alias name='serial0'/>
    </console>
    <channel type='spicevmc'>
      <target type='virtio' name='com.redhat.spice.0' state='disconnected'/>
      <alias name='channel0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <graphics type='spice' port='5901' autoport='yes' listen='0.0.0.0' keymap='en-us'>
      <listen type='address' address='0.0.0.0'/>
    </graphics>
    <sound model='ich6'>
      <alias name='sound0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </sound>
    <video>
      <model type='qxl' ram='65536' vram='65536' vgamem='16384' heads='1'/>
      <alias name='video0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </video>
    <memballoon model='virtio'>
      <alias name='balloon0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
      <stats period='10'/>
    </memballoon>
  </devices>
  <seclabel type='dynamic' model='selinux' relabel='yes'>
    <label>system_u:system_r:svirt_t:s0:c359,c706</label>
    <imagelabel>system_u:object_r:svirt_image_t:s0:c359,c706</imagelabel>
  </seclabel>
</domain>

 

Tuesday, July 28, 2015

CPU Pinning and NUMA Topology on RDO Kilo && Hypervisor Upgrade up to qemu-kvm-ev-2.1.2-23.el7.1 on CentOS 7.1

Posting bellow follows up http://redhatstackblog.redhat.com/2015/05/05/cpu-pinning-and-numa-topology-awareness-in-openstack-compute/
on RDO Kilo upgraded via  qemu-kvm-ev-2.1.2-23.el7.1  on CentOS 7.1
Recent build on CentOS 7.X qemu-kvm-ev-2.1.2-23.el7.1 
enables CPU Pinning and NUMA Topology for RDO Kilo on CentOS 7.1
Qemu-kvm upgrade is supposed to be done as post installation procedure,
i.e. after RDO Kilo deployment on the system. Final target is to reproduce mentioned article on i7 4790 Haswell CPU box, perform launching nova instance with CPU pinning.

See also CPU Pinning and NUMA Topology on RDO Kilo on Fedora Server 22

[root@Centos71 x86_64]# numactl --hardware
available: 1 nodes (0)
node 0 cpus: 0 1 2 3 4 5 6 7
node 0 size: 16326 MB
node 0 free: 4695 MB
node distances:
node   0
0:  10

# virsh capabilities
   .  .  .  .  .

 <topology>
      <cells num='1'>
        <cell id='0'>
          <memory unit='KiB'>16718464</memory>
          <pages unit='KiB' size='4'>4179616</pages>
          <pages unit='KiB' size='2048'>0</pages>
          <distances>
            <sibling id='0' value='10'/>
          </distances>
          <cpus num='8'>
            <cpu id='0' socket_id='0' core_id='0' siblings='0,4'/>
            <cpu id='1' socket_id='0' core_id='1' siblings='1,5'/>
            <cpu id='2' socket_id='0' core_id='2' siblings='2,6'/>
            <cpu id='3' socket_id='0' core_id='3' siblings='3,7'/>
            <cpu id='4' socket_id='0' core_id='0' siblings='0,4'/>
            <cpu id='5' socket_id='0' core_id='1' siblings='1,5'/>
            <cpu id='6' socket_id='0' core_id='2' siblings='2,6'/>
            <cpu id='7' socket_id='0' core_id='3' siblings='3,7'/>
          </cpus>
        </cell>
      </cells>
</topology>



On each Compute node that pinning of virtual machines will be permitted on open the /etc/nova/nova.conf file and make the following modifications:
  • Set the vcpu_pin_set value to a list or range of logical CPU cores  to reserve for virtual machine processes. OpenStack Compute will ensure guest virtual machine instances are pinned to these virtual CPU cores. 
  • vcpu_pin_set=2,3,6,7
  • Set the reserved_host_memory_mb to reserve RAM for host processes. For the purposes of testing used  the default of 512 MB: 
  • reserved_host_memory_mb=512 
# systemctl restart openstack-nova-compute.service

************************************
SCHEDULER CONFIGURATION
************************************
Update /etc/nova/nova.conf

scheduler_default_filters=RetryFilter,AvailabilityZoneFilter,RamFilter,
ComputeFilter,ComputeCapabilitiesFilter,ImagePropertiesFilter,CoreFilter,
NUMATopologyFilter,AggregateInstanceExtraSpecsFilter

# systemctl restart openstack-nova-scheduler.service


 At this point if  creating  a guest you may see some changes to appear in the XML, pinning the guest vCPU(s) to the cores listed in vcpu_pin_set:

<vcpu placement='static' cpuset='2-3,6-7'>1</vcpu>

Add to vmlinuz grub2 command line at the end 


isolcpus=2,3,6,7

***************************
REBOOT SYSTEM
***************************

[root@Centos71 ~(keystone_admin)]# nova aggregate-create performance
+----+-------------+-------------------+-------+----------+
| Id | Name        | Availability Zone | Hosts | Metadata |
+----+-------------+-------------------+-------+----------+
| 1  | performance | -                 |       |          |
+----+-------------+-------------------+-------+----------+

[root@Centos71 ~(keystone_admin)]# nova aggregate-set-metadata 1
pinned=true
Metadata has been successfully updated for aggregate 1.
+----+-------------+-------------------+-------+---------------+
| Id | Name        | Availability Zone | Hosts | Metadata      |
+----+-------------+-------------------+-------+---------------+
| 1  | performance | -                 |       | 'pinned=true' |
+----+-------------+-------------------+-------+---------------+


Create a new flavor for performance intensive instances. Here is created the m1.small.performance flavor, based on the values used in the existing m1.small flavor. The differences in behaviour between the two will be the result of the metadata to be  added  to the new flavor.


[root@Centos71 ~(keystone_admin)]# nova flavor-create m1.small.performance 6 4096 20 4
+----+----------------------+-----------+------+-----------+------+-------+-------------+-----------+
| ID | Name                 | Memory_MB | Disk | Ephemeral | Swap | VCPUs | RXTX_Factor | Is_Public |
+----+----------------------+-----------+------+-----------+------+-------+-------------+-----------+
| 6  | m1.small.performance | 4096      | 20   | 0                |          | 4     | 1.0  | True      |
+----+----------------------+-----------+------+-----------+------+-------+-------------+-----------+

[root@Centos71 ~(keystone_admin)]# nova flavor-key 6 set hw:cpu_policy=dedicated

[root@Centos71 ~(keystone_admin)]# nova flavor-key 6 set aggregate_instance_extra_specs:pinned=true


[root@Centos71 ~(keystone_admin)]# nova aggregate-add-host 1 Centos71.localdomain

Host Centos71.localdomain has been successfully added for aggregate 1
+----+-------------+-------------------+------------------------+---------------+
| Id | Name        | Availability Zone | Hosts                  | Metadata      |
+----+-------------+-------------------+------------------------+---------------+
| 1  | performance | -                 | 'Centos71.localdomain' | 'pinned=true' |
+----+-------------+-------------------+------------------------+---------------+



[root@Centos71 ~(keystone_admin)]# .   keystonerc_demo

[root@Centos71 ~(keystone_demo)]# glance image-list
+--------------------------------------+-----------------+-------------+------------------+------------+--------+
| ID                                   | Name            | Disk Format | Container Format | Size       | Status |
+--------------------------------------+-----------------+-------------+------------------+------------+--------+
| 4a2d708c-7624-439f-9e7e-6e133062e23a | CentOS71Image   | qcow2       | bare             | 1004994560 | active |
| fae94d4b-e810-46a9-8a8f-94dfb812e098 | cirros          | qcow2       | bare             | 13200896   | active |
| f823f0a0-bcdf-416d-915a-8d7cc0278ed7 | Fedora20image   | qcow2       | bare             | 210829312  | active |
| 2198786d-e77f-47ec-959f-fcf7435d5e78 | Fedora21image   | qcow2       | bare             | 158443520  | active |
| 5f1ca33e-d5cc-43fb-9d88-5a7ef0f75959 | VF22Image       | qcow2       | bare             | 228599296  | active |
+--------------------------------------+-----------------+-------------+------------------+------------+--------+

[root@Centos71 ~(keystone_demo)]#  neutron net-list
+--------------------------------------+----------+-----------------------------------------------------+
| id                                   | name     | subnets                                             |
+--------------------------------------+----------+-----------------------------------------------------+
| ab4bd4f8-22b7-43c3-ac60-c1a917a230d7 | public   | 4cbcf377-3742-4385-8362-c071f499ad9c 192.168.1.0/24 |
| 93e0b4be-7900-4a34-adef-758578f75774 | demo_net | 17a5e4f7-fd1b-45fa-b84f-c3af1378c42c 50.0.0.0/24    |
+--------------------------------------+----------+-----------------------------------------------------+

[root@Centos71 ~(keystone_demo)]# nova boot --image 5f1ca33e-d5cc-43fb-9d88-5a7ef0f75959 --key-name oskeydev --flavor m1.small.performance --nic net-id=93e0b4be-7900-4a34-adef-758578f75774 vf22-instance

+--------------------------------------+--------------------------------------------------+
| Property                             | Value                                            |
+--------------------------------------+--------------------------------------------------+
| OS-DCF:diskConfig                    | MANUAL                                           |
| OS-EXT-AZ:availability_zone          | nova                                             |
| OS-EXT-STS:power_state               | 0                                                |
| OS-EXT-STS:task_state                | scheduling                                       |
| OS-EXT-STS:vm_state                  | building                                         |
| OS-SRV-USG:launched_at               | -                                                |
| OS-SRV-USG:terminated_at             | -                                                |
| accessIPv4                           |                                                  |
| accessIPv6                           |                                                  |
| adminPass                            | trBdDQWd75ck                                     |
| config_drive                         |                                                  |
| created                              | 2015-07-29T07:59:23Z                             |
| flavor                               | m1.small.performance (6)                         |
| hostId                               |                                                  |
| id                                   | d7fda7ca-7124-4c8b-a085-1da784d57348             |
| image                                | VF22Image (5f1ca33e-d5cc-43fb-9d88-5a7ef0f75959) |
| key_name                             | oskeydev                                         |
| metadata                             | {}                                               |
| name                                 | vf22-instance                                    |
| os-extended-volumes:volumes_attached | []                                               |
| progress                             | 0                                                |
| security_groups                      | default                                          |
| status                               | BUILD                                            |
| tenant_id                            | 8c9defac20a74633af4bb4773e45f11e                 |
| updated                              | 2015-07-29T07:59:23Z                             |
| user_id                              | da79d2c66db747eab942bdbe20bb3f44                 |
+--------------------------------------+--------------------------------------------------+

[root@Centos71 ~(keystone_demo)]# nova list
+--------------------------------------+---------------+-----------+------------+-------------+-----------------------------------+
| ID                                   | Name          | Status    | Task State | Power State | Networks                          |
+--------------------------------------+---------------+-----------+------------+-------------+-----------------------------------+
| 455877f2-7070-48a7-bb24-e0702be2fbc5 | CentOS7RSX05  | SUSPENDED | -          | Shutdown    | demo_net=50.0.0.13, 192.168.1.153 |
| 44645495-a158-4b99-b96b-26f8178fa28f | VF22Devs      | ACTIVE    | -          | Running     | demo_net=50.0.0.23, 192.168.1.163 |
| d7fda7ca-7124-4c8b-a085-1da784d57348 | vf22-instance | ACTIVE    | -          | Running     | demo_net=50.0.0.24, 192.168.1.164 |
+--------------------------------------+---------------+-----------+------------+-------------+-----------------------------------+

[root@Centos71 ~(keystone_demo)]# nova show d7fda7ca-7124-4c8b-a085-1da784d57348
+--------------------------------------+----------------------------------------------------------+
| Property                             | Value                                                    |
+--------------------------------------+----------------------------------------------------------+
| OS-DCF:diskConfig                    | MANUAL                                                   |
| OS-EXT-AZ:availability_zone          | nova                                                     |
| OS-EXT-STS:power_state               | 1                                                        |
| OS-EXT-STS:task_state                | -                                                        |
| OS-EXT-STS:vm_state                  | active                                                   |
| OS-SRV-USG:launched_at               | 2015-07-29T08:00:12.000000                               |
| OS-SRV-USG:terminated_at             | -                                                        |
| accessIPv4                           |                                                          |
| accessIPv6                           |                                                          |
| config_drive                         |                                                          |
| created                              | 2015-07-29T07:59:23Z                                     |
| demo_net network                     | 50.0.0.24, 192.168.1.164                                 |
| flavor                               | m1.small.performance (6)                                 |
| hostId                               | de84a0c94e3271ef0f4620e113814fa69132fea1be65e7ad33edde7d |
| id                                   | d7fda7ca-7124-4c8b-a085-1da784d57348                     |
| image                                | VF22Image (5f1ca33e-d5cc-43fb-9d88-5a7ef0f75959)         |
| key_name                             | oskeydev                                                 |
| metadata                             | {}                                                       |
| name                                 | vf22-instance                                            |
| os-extended-volumes:volumes_attached | []                                                       |
| progress                             | 0                                                        |
| security_groups                      | default                                                  |
| status                               | ACTIVE                                                   |
| tenant_id                            | 8c9defac20a74633af4bb4773e45f11e                         |
| updated                              | 2015-07-29T08:00:12Z                                     |
| user_id                              | da79d2c66db747eab942bdbe20bb3f44                         |
+--------------------------------------+---------------------------------------------------------


[root@Centos71 x86_64]# virsh list
 Id    Name                           State
----------------------------------------------------
 2     instance-0000000d              running
 3     instance-0000000e              running

[root@Centos71 x86_64]# virsh dumpxml instance-0000000e > vf22-02.xml

Please, see http://redhatstackblog.redhat.com/2015/05/05/cpu-pinning-and-numa-topology-awareness-in-openstack-compute/
regarding detailed explanation of highlighted blocks, keeping in mind that pinning is done to logical CPU cores ( not physical due to 4 Core CPU with HT enabled ). Multiple cells are also absent, due limitations of i7 47XX Haswell CPU architecture

[root@Centos71 x86_64]# cat  vf22-02.xml

<domain type='kvm' id='3'>
  <name>instance-0000000e</name>
  <uuid>d7fda7ca-7124-4c8b-a085-1da784d57348</uuid>
  <metadata>
    <nova:instance xmlns:nova="http://openstack.org/xmlns/libvirt/nova/1.0">
      <nova:package version="2015.1.0-3.el7"/>
      <nova:name>vf22-instance</nova:name>
      <nova:creationTime>2015-07-29 08:00:06</nova:creationTime>
      <nova:flavor name="m1.small.performance">
        <nova:memory>4096</nova:memory>
        <nova:disk>20</nova:disk>
        <nova:swap>0</nova:swap>
        <nova:ephemeral>0</nova:ephemeral>
        <nova:vcpus>4</nova:vcpus>
      </nova:flavor>
      <nova:owner>
        <nova:user uuid="da79d2c66db747eab942bdbe20bb3f44">demo</nova:user>
        <nova:project uuid="8c9defac20a74633af4bb4773e45f11e">demo</nova:project>
      </nova:owner>
      <nova:root type="image" uuid="5f1ca33e-d5cc-43fb-9d88-5a7ef0f75959"/>
    </nova:instance>
  </metadata>
  <memory unit='KiB'>4194304</memory>
  <currentMemory unit='KiB'>4194304</currentMemory>
  <vcpu placement='static'>4</vcpu>
  <cputune>
    <shares>4096</shares>
    <vcpupin vcpu='0' cpuset='2'/>
    <vcpupin vcpu='1' cpuset='6'/>
    <vcpupin vcpu='2' cpuset='3'/>
    <vcpupin vcpu='3' cpuset='7'/>
    <emulatorpin cpuset='2-3,6-7'/>
  </cputune>
  <numatune>
    <memory mode='strict' nodeset='0'/>
    <memnode cellid='0' mode='strict' nodeset='0'/>
  </numatune>

  <resource>
    <partition>/machine</partition>
  </resource>
    <sysinfo type='smbios'>
      <system>
        <entry name='manufacturer'>Fedora Project</entry>
        <entry name='product'>OpenStack Nova</entry>
        <entry name='version'>2015.1.0-3.el7</entry>
        <entry name='serial'>b3fae7c3-10bd-455b-88b7-95e586342203</entry>
        <entry name='uuid'>d7fda7ca-7124-4c8b-a085-1da784d57348</entry>
      </system>
    </sysinfo>
  <os>
    <type arch='x86_64' machine='pc-i440fx-rhel7.1.0'>hvm</type>
    <boot dev='hd'/>
    <smbios mode='sysinfo'/>
  </os>
  <features>
    <acpi/>
    <apic/>
  </features>
  <cpu mode='host-model'>
    <model fallback='allow'/>
    <topology sockets='2' cores='1' threads='2'/>
    <numa>
      <cell id='0' cpus='0-3' memory='4194304'/>
    </numa>
  </cpu>

 <clock offset='utc'>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <devices>
    <emulator>/usr/libexec/qemu-kvm</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2' cache='none'/>
      <source file='/var/lib/nova/instances/d7fda7ca-7124-4c8b-a085-1da784d57348/disk'/>
      <backingStore type='file' index='1'>
        <format type='raw'/>
        <source file='/var/lib/nova/instances/_base/99f1a80be14fc2563a2af39e944ee1c305ed8c34'/>
        <backingStore/>
      </backingStore>
      <target dev='vda' bus='virtio'/>
      <alias name='virtio-disk0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </disk>
    <controller type='usb' index='0'>
      <alias name='usb0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'>
      <alias name='pci.0'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <alias name='virtio-serial0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </controller>
    <interface type='bridge'>
      <mac address='fa:16:3e:94:77:fd'/>
      <source bridge='qbr2d20d535-5c'/>
      <target dev='tap2d20d535-5c'/>
      <model type='virtio'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
    <serial type='file'>
      <source path='/var/lib/nova/instances/d7fda7ca-7124-4c8b-a085-1da784d57348/console.log'/>
      <target port='0'/>
      <alias name='serial0'/>
    </serial>
    <serial type='pty'>
      <source path='/dev/pts/3'/>
      <target port='1'/>
      <alias name='serial1'/>
    </serial>
    <console type='file'>
      <source path='/var/lib/nova/instances/d7fda7ca-7124-4c8b-a085-1da784d57348/console.log'/>
      <target type='serial' port='0'/>
      <alias name='serial0'/>
    </console>
    <channel type='spicevmc'>
      <target type='virtio' name='com.redhat.spice.0' state='disconnected'/>
      <alias name='channel0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <graphics type='spice' port='5901' autoport='yes' listen='0.0.0.0 ( only  Compute )' keymap='en-us'>
      <listen type='address' address='0.0.0.0 ( only  Compute )'/>
    </graphics>
    <sound model='ich6'>
      <alias name='sound0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </sound>
    <video>
      <model type='qxl' ram='65536' vram='65536' vgamem='16384' heads='1'/>
      <alias name='video0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </video>
    <memballoon model='virtio'>
      <alias name='balloon0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
      <stats period='10'/>
    </memballoon>
  </devices>
  <seclabel type='dynamic' model='selinux' relabel='yes'>
    <label>system_u:system_r:svirt_t:s0:c670,c918</label>
    <imagelabel>system_u:object_r:svirt_image_t:s0:c670,c918</imagelabel>
  </seclabel>
</domain>

  

Wednesday, July 22, 2015

Setup NovaDocker Container via supertaihei02/docker-centos-lamp:lamp with RDO Kilo on CentOS 7.1

[root@ServerCentOS71 ~(keystone_admin)]# docker pull "supertaihei02/docker-centos-lamp:lamp"
Trying to pull repository docker.io/supertaihei02/docker-centos-lamp ...
fd8dbcf7c7a8: Download complete 
511136ea3c5a: Download complete 
34e94e67e63a: Download complete 
0c752394b855: Download complete 
313bc652d65e: Download complete 
7bca8e6c58d0: Download complete 
02c7dd4b7b17: Download complete 
1ccb205c0045: Download complete 
67c95f15098c: Download complete 
265c1884886d: Download complete 
397ee04ea830: Download complete 
d57e43613b5c: Download complete 
2614d5f8ae73: Download complete 
bcd82bfaa75d: Download complete 
2513dc5a7634: Download complete 
99c99df8dd45: Download complete 
2c7af5e056c6: Download complete 
474020a06892: Download complete 
72ca693b56ec: Download complete 
32538e0f2a30: Download complete 
67d8a91c730d: Download complete 
a3f485a396f4: Download complete 
ad78608b9dec: Download complete 
08fbdbd00577: Download complete 
1fa3dcbd7265: Download complete 
789146ed1568: Download complete 
297a56c9f440: Download complete 
1b1962c27933: Download complete 
de86771fcddf: Download complete 
2fc57b1bb564: Download complete 
b9979c770dbd: Download complete 
dfa087672ba1: Download complete 
108d43aa66f4: Download complete 
4573e6861576: Download complete 
Status: Downloaded newer image for docker.io/supertaihei02/docker-centos-lamp:lamp
 
[root@ServerCentOS71 ~(keystone_admin)]# docker save supertaihei02/docker-centos-lamp:lamp | \
glance image-create --is-public=True   --container-format=docker \
--disk-format=raw --name supertaihei02/docker-centos-lamp:lamp
 
+------------------+---------------------------------------+
| Property         | Value                                 |
+------------------+---------------------------------------+
| checksum         | 24169bb5bf7b8e218af565b9105d231f      |
| container_format | docker                                |
| created_at       | 2015-07-22T13:15:49.000000            |
| deleted          | False                                 |
| deleted_at       | None                                  |
| disk_format      | raw                                   |
| id               | b317fed0-4618-406b-925f-954eb2aa35f7  |
| is_public        | True                                  |
| min_disk         | 0                                     |
| min_ram          | 0                                     |
| name             | supertaihei02/docker-centos-lamp:lamp |
| owner            | 616a4800e9a541b78e4aba836314883d      |
| protected        | False                                 |
| size             | 899017728                             |
| status           | active                                |
| updated_at       | 2015-07-22T13:17:13.000000            |
| virtual_size     | None                                  |
+------------------+---------------------------------------+
Next
[root@ServerCentOS71 ~(keystone_admin)]# docker images |grep lamp
docker.io/supertaihei02/docker-centos-lamp   lamp                fd8dbcf7c7a8      13 months ago       883.8 MB
 
Nova-Doker container started

[root@ServerCentOS71 ~(keystone_admin)]# docker ps
 
CONTAINER ID        IMAGE                                   COMMAND               CREATED             STATUS              PORTS               NAMES
6a1e0a782a7d        supertaihei02/docker-centos-lamp:lamp   "/usr/bin/monit -I"   4 minutes ago       Up 4 minutes                            nova-8ade872c-45dc-40b0-be38-a5ab61d7de0f 
 
[root@ServerCentOS71 ~(keystone_admin)]# docker logs 6a1e0a782a7d
monit: generated unique Monit id 78e9d47e7325f30d9ba76b908278921b and stored to '/var/monit/
 
    
 
  
 
[root@ServerCentOS71 ~]# ssh -p 22 -l guest  192.168.1.165
guest@192.168.1.165's password: 
Last login: Thu Jul 23 04:39:50 2015 from 192.168.1.87
[guest@instance-00000010 ~]$ cat /etc/issue
CentOS release 6.5 (Final)
Kernel \r on an \m

[guest@instance-00000010 ~]$ sudo su -[root@instance-00000010 ~]# ps -ef | grep -v ""

UID        PID  PPID  C STIME TTY          TIME CMD
root         1     0  0 01:22 ?        00:00:00 /usr/bin/monit -I
root        18     1  0 01:22 ?        00:00:00 /usr/sbin/sshd
root       300     1  0 01:22 ?        00:00:00 /usr/sbin/httpd
apache     302   300  0 01:22 ?        00:00:00 /usr/sbin/httpd
apache     303   300  0 01:22 ?        00:00:00 /usr/sbin/httpd
apache     304   300  0 01:22 ?        00:00:00 /usr/sbin/httpd
apache     305   300  0 01:22 ?        00:00:00 /usr/sbin/httpd
apache     306   300  0 01:22 ?        00:00:00 /usr/sbin/httpd
apache     307   300  0 01:22 ?        00:00:00 /usr/sbin/httpd
apache     308   300  0 01:22 ?        00:00:00 /usr/sbin/httpd
apache     309   300  0 01:22 ?        00:00:00 /usr/sbin/httpd
root      1079     1  0 02:08 ?        00:00:00 /bin/sh /usr/bin/mysqld_safe --datadir=/var/lib/mysql --socket=/var/lib/mysql/mysql.sock --pid-file=/var/run/mysqld/mysqld.pid --basedir=/usr --user=mysql
mysql     1286  1079  0 02:08 ?        00:00:03 /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib64/mysql/plugin --user=mysql --log-error=/var/log/mysqld.log --pid-file=/var/run/mysqld/mysqld.pid --socket=/var/lib/mysql/mysql.sock
boris     4270     1  0 04:25 ?        00:00:01 /usr/bin/Xvnc :1 -desktop instance-00000010:1 (boris) -auth /home/boris/.Xauthority -geometry 1024x768 -rfbwait 30000 -rfbauth /home/boris/.vnc/passwd -rfbport 5901 -fp catalogue:/etc/X11/fontpath.d -pn
boris     4275     1  0 04:25 ?        00:00:00 /usr/bin/gnome-session
boris     4276  4275  0 04:25 ?        00:00:00 vncconfig -iconic
boris     4284     1  0 04:25 ?        00:00:00 dbus-launch --sh-syntax --exit-with-session
boris     4285     1  0 04:25 ?        00:00:00 /bin/dbus-daemon --fork --print-pid 4 --print-address 6 --session
boris     4296     1  0 04:25 ?        00:00:00 /usr/libexec/gconfd-2
boris     4301     1  0 04:25 ?        00:00:00 gnome-keyring-daemon --start
boris     4304     1  0 04:25 ?        00:00:00 /usr/libexec/gnome-settings-daemon
boris     4309     1  0 04:25 ?        00:00:00 /usr/libexec/gvfsd
boris     4313  4275  0 04:25 ?        00:00:00 metacity
boris     4319  4275  0 04:25 ?        00:00:00 gnome-panel
boris     4321  4275  0 04:25 ?        00:00:00 nautilus
boris     4323     1  0 04:25 ?        00:00:00 /usr/libexec/bonobo-activation-server --ac-activate --ior-output-fd=18
boris     4345     1  0 04:25 ?        00:00:00 /usr/libexec/wnck-applet --oaf-activate-iid=OAFIID:GNOME_Wncklet_Factory --oaf-ior-fd=18
boris     4348     1  0 04:25 ?        00:00:00 /usr/libexec/trashapplet --oaf-activate-iid=OAFIID:GNOME_Panel_TrashApplet_Factory --oaf-ior-fd=24
boris     4356  4275  0 04:25 ?        00:00:00 gnome-volume-control-applet
boris     4360     1  0 04:25 ?        00:00:00 gnome-screensaver
boris     4366     1  0 04:25 ?        00:00:00 /usr/libexec/gvfsd-trash --spawner :1.7 /org/gtk/gvfs/exec_spaw/0
boris     4383     1  0 04:25 ?        00:00:00 /usr/libexec/notification-area-applet --oaf-activate-iid=OAFIID:GNOME_NotificationAreaApplet_Factory --oaf-ior-fd=28
boris     4384     1  0 04:25 ?        00:00:00 /usr/libexec/clock-applet --oaf-activate-iid=OAFIID:GNOME_ClockApplet_Factory --oaf-ior-fd=34
boris     4470     1  0 04:25 ?        00:00:05 /usr/lib64/firefox/firefox
root      6879    18  0 04:39 ?        00:00:00 sshd: guest [priv]
guest     6895  6879  0 04:39 ?        00:00:00 sshd: guest@pts/0
guest     6896  6895  0 04:39 pts/0    00:00:00 -bash
root      6925  6896  0 04:39 pts/0    00:00:00 sudo su -
root      6926  6925  0 04:39 pts/0    00:00:00 su -
root      6927  6926  0 04:39 pts/0    00:00:00 -bash
root      7082  6927  3 04:40 pts/0    00:00:00 ps -ef