[----] I, [2018-03-11T16:12:10.941498 #3004:4f7130] INFO -- : MIQ(ManageIQ::Providers::Amazon::CloudManager::Vm#raw_scan) NAME [Scan from Vm vmtosmartstate] SCAN [nil] [NilClass] [----] I, [2018-03-11T16:12:11.068516 #3004:4f7130] INFO -- : Job created: guid: [b0c7c318-2844-436c-bc44-21a29dfa3cce], userid: [admin], name: [Scan from Vm vmtosmartstate], target class: [VmOrTemplate], target id: [99000000000166], process type: [VmScan], server id: [], zone: [default] [----] I, [2018-03-11T16:12:15.868847 #3014:4f7130] INFO -- : MIQ(MiqQueue.put) Message id: [99000000000427], id: [], Zone: [default], Role: [smartstate], Server: [], Ident: [generic], Target id: [], Instance id: [], Task id: [job_dispatcher], Command: [JobProxyDispatcher.dispatch], Timeout: [600], Priority: [20], State: [ready], Deliver On: [], Data: [], Args: [] [----] I, [2018-03-11T16:12:21.133242 #3004:4f7130] INFO -- : MIQ(MiqPriorityWorker::Runner#get_message_via_drb) Message id: [99000000000427], MiqWorker id: [99000000000004], Zone: [default], Role: [smartstate], Server: [], Ident: [generic], Target id: [], Instance id: [], Task id: [job_dispatcher], Command: [JobProxyDispatcher.dispatch], Timeout: [600], Priority: [20], State: [dequeue], Deliver On: [], Data: [], Args: [], Dequeued in: [5.266320951] seconds [----] I, [2018-03-11T16:13:05.374585 #3424:4f7130] INFO -- : MIQ(ManageIQ::Providers::Amazon::AgentCoordinator#find_or_create_keypair) KeyPair smartstate-2ae1670a-2fcc-4bbb-abcc-3fa962895096 will be created! [----] I, [2018-03-11T16:13:05.877668 #3424:4f7130] INFO -- : MIQ(ManageIQ::Providers::Amazon::AgentCoordinator#deploy_agent) Smartstate agent will be deployed in vpc: [vpc-076ad862], zone: [us-east-1a] subnet: [subnet-3cc6154b] [----] I, [2018-03-11T16:13:11.145451 #3424:4f7130] INFO -- : MIQ(ManageIQ::Providers::Amazon::AgentCoordinator#get_agent_image_id) AMI Image: RHEL-Atomic_7.4_HVM_GA-20180104-x86_64-1-Access2-GP2 [ami-d97120a3] is used to launch smartstate agent. [----] I, [2018-03-11T16:15:55.976224 #2716:4f7130] INFO -- : MiqServer: local=Y, master=Y, status= started, id=99000000000001, pid=02716, guid=c5b77187-7fc9-426e-b3cb-6f0ce7860c8e, name=EVM, zone=default, hostname=ip-172-31-44-62.ec2.internal, ipaddress=172.31.44.62, version=5.9.0.22, build=20180221205805_f93a675, active roles=automate:database_operations:datab
There are a few interesting parts here. What is the AgentCoordinator? The code can be found here:
/opt/rh/cfme-gemset/bundler/gems/manageiq-providers-amazon-9620e26f4381/app/models/manageiq/providers/amazon/agent_coordinator.rb
It’s out of scope to discuss the entire ruby code, but it will tell you, how we do it.
So when you look at the evm log, you see that it says “find_or_create_keypair KeyPair smartstate-2ae1670a-2fcc-4bbb-abcc-3fa962895096″
This means that the key used for the ssh login to the RHEL-Atomic_7.4_HVM_GA-20180104-x86_64-1-Access2-GP2 (ami-d97120a3) is generated on the fly.
Let’s explain how you can get that key so you can SSH into the atomic instance and see whats going on.
vmdb
bin/rails c
$evm = MiqAeMethodService::MiqAeService.new(MiqAeEngine::MiqAeWorkspaceRuntime.new)
$evm.vmdb(:ManageIQ_Providers_CloudManager_AuthKeyPair).all.first
$ssa=$evm.vmdb(:ManageIQ_Providers_CloudManager_AuthKeyPair).find_by_resource_id(99000000000001)
$ssa[:auth_key]
irb(main):007:0> $ssa[:auth_key]
=>
"v2:{UOjuc9OSrBm2AcOU5HmnLi1+7XVumlIlfX2SO7fbe1FyAwt9RGqEqyJD5VSRD4QG8RhoFt0yBe0fk5LCq0RGxgdiuIClv50mePsgTyx7f4cOCT94MKM5s3NkfeEq3BdNCzL+CWji+WSXZZn7zE9l/4ezFW7eesfJzsW6npMlH/SNUBdAgiVZzbsKrHZuP4n5agPBYSX+APf+1KvAOGLYaMREGfw4E7OytKGIfOAD/dSKXRx/bhEadCOvnf6sG6XKYGzi82aa+AMzgexszx/93Ce+Pp6fbODxxXJmoDNyzJEoKyuavDiWBEK2SHMV5t2G8U/pb0mjYvavNCl86gtqOGJQ26bZNuUkJlRoSyErOMyrOOhreq0/2RMJvI4Uy6ekCBggyPFS3iPDduQTNM1tPhZPVAW2H+RB8UKZOtf+Oo+eqfkGvQYlnsJt2a4XET+alDxpDT6TEVRCrMpAUUiJt2B1AG3uQ8wL3RDRMvj54uR/uRsvlFMkO9HYrlcKKrQ4myBa3UbVETNcwT844sYwXEP8thmIVMFwIp03oY5ZQOXoPikD1wJJSHQON2bmIbh/xZ3e2ggPwuLkc9ROSBUxkRK4zXhdezwjI3UfP3LVwujHJVmsIBpUDR7oXw2CUwr0be72MFKoWpmvbXf7E8KZaJz1FNZUoOGpMPz5j3eDetlCEwgjJF2UVElT8+2dLosxN2NeLpFY6vpZXm3erOG6WVLM/v5kiX3V2S2JHaUOZdVRrHvk9yoSF5YhQgn72LlpG+Y4DhT3QOf9HMwzg4A4D6EgkRmZfWdr8ABMMoEFonIkPX08ZwJnr3iNAtam8HCvnGGcs49eTPrAzNG/9zjn6ViS4tvGk9EaYYUuHcWgmOySl2WsezmEdE9oHEC7NkX/MbPdSlr1HxfNAkphjdsvmwSABLhOzFeE25zrftTW3Hf1ui7+tsJztDDio00qtXcT+NY8ySupb2siZcywFZFf3WgRzbOX9yjZv4QG8xPzQ80v4+EbKUuDJJkAHd85kAxZYryHQFi/dQO8O39lhJWog6T58gCteMBnYCvLqnYxr40qwXJ6lp44yoi+EBSH7TpLmsY/AM+YvQH1rqUTbi/1o9PvzwkY3C52WpMFeRlh9XDdg/hWWwcHCJc07dNP2kOCtsR1Te5dhTejsJ9ul2MzAGdRnPwuUz9RrMj0Xj/5X8we+4RPgF+/Z3EpjMOFFzrBbykj003n/pRJE0651a5FbdUkO9QeE3jjj4QGaWAb+3dTbVH3OGVIQ0Ebm31po6G1+9SPyfUSVOmmlHAK4XZybPGNQzsW7uorOAtet9++GAR11mBvuO5g9GSOBGw6EEqN76X1CJ3WFaKwfpoQzDXDC3hOMjahB9cZaWu7zXebJPlKUiZPuIfbB93WCOVjS5TU01McrBsjt0rgsSAgPwVzEAyGVInx1knlQ34SiPvgOZYZuvlr9ln2dAdzfmecSR4na4LwjXJd4B3LhBCDofEtTu21ArdxLu5nHjYg1JiaLm1jL2MJeElFkMtJjH8dKFO/e7rbF6Gtq5aMMP1GNFle2EqW53jJsOF7FLZAa5rTNP2qGKBJyWw6dmOPnszIWrMHrSLEJPTpIr4F0V27rwyknCijIDZG/XENDMRtJHLfpjqajVfnlMDp0+1gHX7GhepBN+1b7l9LaRGavJ9iyyBaiw6kHgp7mseRvceDbTkkOUQHEcNL1CYzxt2QHap7noKtfYa0K3GPCa41pXHwaMrG6jeM3Oi0KaoXTBmlfqeDKtqJ1YOzqQ5+qoVd/Um8nWq/HWvjhtoEK7K4zXaEsaVBlPeqrDqUY+sppbvKENXRRfotim9QG+b5ujINRCLl95HSNIQ5J6zDjYPqDToQuRF72L2SdoDHNSotcSzMDf3m91Idoec5oV8M+jPwqjm7BgzCK7RxW8aAAgh0Ht6j+QHYr8Isg00sCDFBrMG9P6nxbZsipeIklTjyCUWFA8rFPBMDfETFhNCkcpZ7TQqZsQtzBgc3gbgIemXw2ORSKvO6RB++b3s81Xj4OPgwG6cH/rvNGvf1ZsE3ONQFXCoVx0ZCXhdiLcvqovSrMctJKFzZdAVTSLKBARc7nc2JQx3Tam+29nn3cTvHH++zx52roTmpO2eI3gWkqpM8TEYAWQVLK8KaK33WCbu7Q6/iCOEKuHPz9v2L1oJBt/7zpU8wxbfafcxOzYuUTbw74EykGzUVaDUuSCbh14oZO9Z1vuBxWO8U}"
In my case its the first key as I did not create another one yet.
Then go ahead and decrypt that key.
MiqPassword.decrypt("v2:{UOjuc9OSrBm2AcOU5HmnLi1+7XVumlIlfX2SO7fbe1FyAwt9RGqEqyJD5VSRD4QG8RhoFt0yBe0fk5LCq0RGxgdiuIClv50mePsgTyx7f4cOCT94MKM5s3NkfeEq3BdNCzL+CWji+WSXZZn7zE9l/4ezFW7eesfJzsW6npMlH/SNUBdAgiVZzbsKrHZuP4n5agPBYSX+APf+1KvAOGLYaMREGfw4E7OytKGIfOAD/dSKXRx/bhEadCOvnf6sG6XKYGzi82aa+AMzgexszx/93Ce+Pp6fbODxxXJmoDNyzJEoKyuavDiWBEK2SHMV5t2G8U/pb0mjYvavNCl86gtqOGJQ26bZNuUkJlRoSyErOMyrOOhreq0/2RMJvI4Uy6ekCBggyPFS3iPDduQTNM1tPhZPVAW2H+RB8UKZOtf+Oo+eqfkGvQYlnsJt2a4XET+alDxpDT6TEVRCrMpAUUiJt2B1AG3uQ8wL3RDRMvj54uR/uRsvlFMkO9HYrlcKKrQ4myBa3UbVETNcwT844sYwXEP8thmIVMFwIp03oY5ZQOXoPikD1wJJSHQON2bmIbh/xZ3e2ggPwuLkc9ROSBUxkRK4zXhdezwjI3UfP3LVwujHJVmsIBpUDR7oXw2CUwr0be72MFKoWpmvbXf7E8KZaJz1FNZUoOGpMPz5j3eDetlCEwgjJF2UVElT8+2dLosxN2NeLpFY6vpZXm3erOG6WVLM/v5kiX3V2S2JHaUOZdVRrHvk9yoSF5YhQgn72LlpG+Y4DhT3QOf9HMwzg4A4D6EgkRmZfWdr8ABMMoEFonIkPX08ZwJnr3iNAtam8HCvnGGcs49eTPrAzNG/9zjn6ViS4tvGk9EaYYUuHcWgmOySl2WsezmEdE9oHEC7NkX/MbPdSlr1HxfNAkphjdsvmwSABLhOzFeE25zrftTW3Hf1ui7+tsJztDDio00qtXcT+NY8ySupb2siZcywFZFf3WgRzbOX9yjZv4QG8xPzQ80v4+EbKUuDJJkAHd85kAxZYryHQFi/dQO8O39lhJWog6T58gCteMBnYCvLqnYxr40qwXJ6lp44yoi+EBSH7TpLmsY/AM+YvQH1rqUTbi/1o9PvzwkY3C52WpMFeRlh9XDdg/hWWwcHCJc07dNP2kOCtsR1Te5dhTejsJ9ul2MzAGdRnPwuUz9RrMj0Xj/5X8we+4RPgF+/Z3EpjMOFFzrBbykj003n/pRJE0651a5FbdUkO9QeE3jjj4QGaWAb+3dTbVH3OGVIQ0Ebm31po6G1+9SPyfUSVOmmlHAK4XZybPGNQzsW7uorOAtet9++GAR11mBvuO5g9GSOBGw6EEqN76X1CJ3WFaKwfpoQzDXDC3hOMjahB9cZaWu7zXebJPlKUiZPuIfbB93WCOVjS5TU01McrBsjt0rgsSAgPwVzEAyGVInx1knlQ34SiPvgOZYZuvlr9ln2dAdzfmecSR4na4LwjXJd4B3LhBCDofEtTu21ArdxLu5nHjYg1JiaLm1jL2MJeElFkMtJjH8dKFO/e7rbF6Gtq5aMMP1GNFle2EqW53jJsOF7FLZAa5rTNP2qGKBJyWw6dmOPnszIWrMHrSLEJPTpIr4F0V27rwyknCijIDZG/XENDMRtJHLfpjqajVfnlMDp0+1gHX7GhepBN+1b7l9LaRGavJ9iyyBaiw6kHgp7mseRvceDbTkkOUQHEcNL1CYzxt2QHap7noKtfYa0K3GPCa41pXHwaMrG6jeM3Oi0KaoXTBmlfqeDKtqJ1YOzqQ5+qoVd/Um8nWq/HWvjhtoEK7K4zXaEsaVBlPeqrDqUY+sppbvKENXRRfotim9QG+b5ujINRCLl95HSNIQ5J6zDjYPqDToQuRF72L2SdoDHNSotcSzMDf3m91Idoec5oV8M+jPwqjm7BgzCK7RxW8aAAgh0Ht6j+QHYr8Isg00sCDFBrMG9P6nxbZsipeIklTjyCUWFA8rFPBMDfETFhNCkcpZ7TQqZsQtzBgc3gbgIemXw2ORSKvO6RB++b3s81Xj4OPgwG6cH/rvNGvf1ZsE3ONQFXCoVx0ZCXhdiLcvqovSrMctJKFzZdAVTSLKBARc7nc2JQx3Tam+29nn3cTvHH++zx52roTmpO2eI3gWkqpM8TEYAWQVLK8KaK33WCbu7Q6/iCOEKuHPz9v2L1oJBt/7zpU8wxbfafcxOzYuUTbw74EykGzUVaDUuSCbh14oZO9Z1vuBxWO8U}") => "-----BEGIN RSA PRIVATE KEY-----\nMIIEpAIBAAKCAQEA4VajXvftedVzvG .......
Copy your RSA private key to cfme.pem and format it correctly:
sed -i 's/\\n/\n/g' cfme.pem
Next you need to fix perms
chmod 400 cfme.pem
If you want you can also compare fingerprints
openssl pkcs8 -in smartstate-1c6df722-bf49-4e32-ad6c-e0aea84a842a.pem -nocrypt -topk8 -outform DER | openssl sha1 -c (stdin)= 70:6f:cc:40:10:ea:97:43:26:6e:f5:73:4c:e8:0d:15:e8:59:5a:c1
This should match what you see in the Compute> Cloud -> KeyPairs.
As I assume that you already launched the SmartState analysis . 3 SQS queues were created in the meantime for you as well as an s3 bucket for the heartbeat where it will store the instance id for the agent. If you go to SQS in the AWS console you will see.
Here is the s3 bucket
When running SmartState you will actually have a message in the request queue. This will the CFME to launch the SmartState instance.
As you probably noticed a security group named smartstate was created for you as well.
Now I was wondering if everything on the atomic host is actually kosher. So that’s where our ssh-key comes in
Lets ssh into the atomic host
ssh -i "cfme.pem" ec2-user@ec2-57-88-166-140.compute-1.amazonaws.com
If everything is working correctly you should see the that the cfme-amazon-smartstate image gets pulled.
[root@ip-172-31-22-123 ~]# atomic images list REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE TYPE >registry.access.redhat.com/cloudforms46/cfme-amazon-smartstate latest 53c8509ae76a 2018-02-22 16:20 311.44 MB docker
The next step, of course, is that CFME will launch it.
[root@ip-172-31-22-123 ~]# atomic containers list CONTAINER ID IMAGE NAME COMMAND CREATED STATE BACKEND RUNTIME 60cb45e8e0d4 registry.access.redh silly_nort /bin/sh -c 2018-03-11 20:18 running docker docker
If the scan completes successfully you will see vm_scan_complete
[----] I, [2018-03-11T16:20:46.085250 #2978:4f7130] INFO -- : Q-task_id([b0c7c318-2844-436c-bc44-21a29dfa3cce]) MIQ(MiqEvent#process_evm_event) target = [#<ManageIQ::Providers::Amazon::CloudManager::Vm id: 99000000000166, vendor: "amazon", format: nil, version: nil, name: "vmtosmartstate", description: nil, location: "ec2-54-160-15-44.compute-1.amazonaws.com", config_xml: nil, autostart: nil, host_id: nil, last_sync_on: "2018-03-11 20:20:08", created_on: "2018-03-11 20:08:19", updated_on: "2018-03-11 20:20:42", storage_id: nil, guid: "13f6cdd0-69db-4ce6-b7eb-3f04f38ce739", ems_id: 99000000000001, last_scan_on: "2018-03-11 20:18:45", last_scan_attempt_on: "2018-03-11 20:12:10", uid_ems: "i-0de0a2338f41c40a4", retires_on: nil, retired: nil, boot_time: "2018-03-11 20:03:10", tools_status: nil, standby_action: nil, power_state: "on", state_changed_on: "2018-03-11 20:08:19", previous_state: nil, connection_state: nil, last_perf_capture_on: nil, registered: nil, busy: nil, smart: nil, memory_reserve: nil, memory_reserve_expand: nil, memory_limit: nil, memory_shares: nil, memory_shares_level: nil, cpu_reserve: nil, cpu_reserve_expand: nil, cpu_limit: nil, cpu_shares: nil, cpu_shares_level: nil, cpu_affinity: nil, ems_created_on: nil, template: false, evm_owner_id: nil, ems_ref_obj: nil, miq_group_id: 99000000000001, linked_clone: nil, fault_tolerance: nil, type: "ManageIQ::Providers::Amazon::CloudManager::Vm", ems_ref: "i-0de0a2338f41c40a4", ems_cluster_id: nil, retirement_warn: nil, retirement_last_warn: nil, vnc_port: nil, flavor_id: 99000000000002, availability_zone_id: 99000000000002, cloud: true, retirement_state: nil, cloud_network_id: nil, cloud_subnet_id: nil, cloud_tenant_id: nil, raw_power_state: "running", publicly_available: nil, orchestration_stack_id: nil, retirement_requester: nil, tenant_id: 99000000000001, resource_group_id: nil, deprecated: nil, storage_profile_id: nil, cpu_hot_add_enabled: nil, cpu_hot_remove_enabled: nil, memory_hot_add_enabled: nil, memory_hot_add_limit: nil, memory_hot_add_increment: nil>] [----] I, [2018-03-11T16:20:46.098293 #2978:4f7130] INFO -- : Q-task_id([b0c7c318-2844-436c-bc44-21a29dfa3cce]) MIQ(MiqAlert.evaluate_alerts) [vm_scan_complete] Target: ManageIQ::Providers::Amazon::CloudManager::Vm Name: [vmtosmartstate], Id: [99000000000166] [----] I, [2018-03-11T16:21:01.555276 #2716:4f7130] INFO -- : MiqServer: local=Y, master=Y, status= started, id=99000000000001, pid=02716, guid=c5b77187-7fc9-426e-b3cb-6f0ce7860c8e, name=EVM, zone=default, hostname=ip-172-31-44-62.ec2.internal, ipaddress=172.31.44.62, version=5.9.0.22, build=20180221205805_f93a675, active roles=automate:database_operations:database_owner:ems_inventory:ems_operations:event:reporting:scheduler:smartproxy:smartstate:user_interface:web_services:websocket
Now you probably wondering, What happens during the SmartState analysis? I’ve created the following diagram for you to understand the magic:
Please, stay tuned for the next part of the series