ViRL/CML simulation issues "Failed loading nodes from Open Stack"

[softnet706]

Hello,

Currently testing ViRL ( not sure what the actual version is) on an ESX 5.5i server host. Running simulation on an Eclipse (VMaestro) client  PC  connected to the ViRL server results in the following:

1. Failed loading nodes from OpenStack: HTTP-status: "500", HTTP-reason: "Internal Server Error", response: "{"computeFault": {"message": "The server has either erred  or is incapable of performing the requested operation.", "code": 500}}"

2. The User Work space management console ( Admin mode), confirms that are no Open stack nodes found. All settings in this console display ACTIVE status for  nodes, networks, projects etc. 

3. All the web services on the Eclipse (VMaestro) client - Roster, Simulation Engine, OpenStack Autonetkit -  connected to the ViRL server show GREEN status as COMPATIBLE.

4. Executing Restack script from CLI (devstack directory) , results in an error and crashes the ViRL and the ViRL VM has to be reinstalled.

Thanks and Regards.

[bikasdas]

Hello,

Thanks for posting the issue.

Kindly share the output of $virl_health_status  and below logs from CML server to analyze the issue.
$var/local/virl/logs
$var/log/syslog

Also share the screenshot of the issue and generate problem report from CML client.

Steps for generating problem report: Go to CML client->Navigate to help->Generate Problem Report.

Thanks and Regards,

Bikash

[softnet706]

Hi Bikash,

Thank you for your quick response and here are the captures, logfiles and CML client problem reports.

Please let me know if you need any information.

Thanks and Regards,

Mohan

[bikasdas]

Hello,

From the $virl_health_status I could see all the openstack services are not available, which leads the issue.

OpenStack image service for STD is not available
OpenStack compute service for STD is not available
OpenStack network service for STD is not available
OpenStack identity service for STD is not available  

Please let us know you got this issue in newly installed CML or it was working earlier and now you are facing the issue.

If it is on newly installed server then there may be some installation problem.

If it was working earlier,please let me know whether you have made any changes in to it.

Also please share output of below commands on CML server.

$nova service-list

$keystone user-list

Thanks and Regards,

Bikash

[softnet706]

Hi Bikash,

Thanks very much for the update.

I noticed it too but the message on Workspace Management - "User "uwmadmin" has no STD endpoint and cannot run simulations." - i thought is only applicable to the uwmadmin user and not too other users like guest.

The ViRL was never working (meaning Open stack node Internal error 500 ) and this is a new install. but on the client side, all the services are working including connectivity to the Server.

virl@virldu:/var$ nova service-list
ERROR: Cannot connect to proxy. Socket error: [Errno -2] Name or service not known.
virl@virldu:/var$ keystone user-list
WARNING: Bypassing authentication using a token & endpoint (authentication credentials are being ignored).
Unable to establish connection to http://localhost:35357/v2.0/users
virl@virldu:/var$ 

installation wise, i just installed the ViRL OVA on ESX 5.5 host and there were two other files i think one was with *. mf and *.vmdk extensions.

[softnet706]

Just to add to the above, prior to ViRL installation i had created two networks FLAT and SNAT on ESX, but during installation only Management Interfaces appear.

Thanks and Regards.

[bikasdas · Accepted solution]

Hello,

Could you please open a TAC case for this issue.

Thanks and Regards,

Bikash

[softnet706]

OK, Bikash, will do. I noticed that the OVA image size is only 11KB, whereas this should be around 4GB. So will try to download the image again and install and then raise TAC if required.  By the way, is it possible to get the procedure how to install CML *.vmdk  files on ESX.

Thanks and Regards,

Mohan

[bikasdas]

Hello Mohan,

Thanks for the update.

Please re install and let us know if you are facing any issues. Open a TAC case if the issue persists.

For uploading .vmdk files to CML server, please copy the image using SFTP to CML server and try to load the image from UWM with using local URL path  (where the image is copied ex: home/virl/<filename>).

Thanks and Regards,

Bikash

[softnet706]

Thanks will raise a TAC case as the fresh install from OVA appears to run into the same issue. I managed to convert ovf to ova using the vmware ovftool, but the problem still persists.

Also tried to install *.vmdk image on UWM, but get an error message if  the local URL path is provided ( the image was copied to /home/virl via sftp) and i have tried with the "server" subtype for this. Do you need a different subtype for uploading vmdk or can choose any of the available subtypes here?

Thanks again very much for your help.

Mohan

[bikasdas]

Hello Mohan,

Thanks for the update.

The download should appear as ESXi specific OVA file, so you don't need any conversion and directly deploy that OVA to ESXi.

Please let me know from where you are downloading the file and for uploading image the above mentioned process should work.

Thanks and Regards,

Bikash

[softnet706]

Hi Bikash,

Sorry for late reply. the image was provided by our clients and this contained *.mf, *.vmdk and *.ovf files. 

1. Installation was tried with deploying the ovf file on ESXi which worked fine but simulation issues due to OpenStack was the problem.

2. Converted ovf to ova using the vmware ovftool and installed OVA with the same results as above. 

3. I am not sure if the ViRL version is TAC supported, so have requested for a new OVA version .So will let you know how we go. 

Thanks and Regards

[bikasdas · Accepted solution]

Hi Mohan,

The VIRL version is not TAC supported.

Please get the new OVA , deploy  it and let us know if the issue persists.

Thanks and Regards,

Bikash

[softnet706]

Hi Bikas,

Thank you and will update as soon as i get and deploy the new OVA file for ViRL. 

In the meantime, i am trying to understand how the vIOS routers are orchestrated from UWM and interactions among AutonetKit, L2 Bridge and UWM orchestration. Please find attached a sketch below of the VirL  network that has been set up in our lab, brief description is as follows:

Virl Server is deployed within ESX5i server. The server has two physical interfaces eth0 and eth1. Two port groups one for (ViRL server access) and another for Flat network is created on ESX attached to the same vmnic0. The ViRL server is accessed from an external laptop running VMaestro and defined with new session profile to communicate to the four web services.

Going through the CML documentation, understand that two physical interfaces are required a) for Server access (in this case will be via VLAN 1) and b) FLAT network on VLAN 597 - I am assuming that this is for OOB access to the routers or Management network required for Orchestration of vIOS routers via UWM??

From the UWM it was observed that the vIOS routers receive                      IP addresses  from the 10.11.12.0/24 pool when Autonet kit is           instantiated from the ViRL client.

Now, what i fail is to understand (please correct me if i am wrong) is:

1) whether we require FLAT ( or SNAT ) for this Orchestration or is it purely related to OOB access to the routers bypassing VMaestro?

2) Does the IP pool for this FLAT network is assigned from an initialization file *.ini for the 172.16.1.0 pool.

3) Which is the Management network required for UWM orchestration as far as ESX port groups are concerned?

Thanks and Regards,

Mohan

[bikasdas · Accepted solution]

Hello Mohan,

Please find the answers below to your query.

1.Whether we require FLAT ( or SNAT ) for this Orchestration or is it purely related to OOB access to the routers bypassing VMaestro?

It is not purely related to OOB access, you can do external connectivity to the routers running in the simulation by using following ways:

Set Up a FLAT Network for Out of Band (OOB) Management Access
Connecting to external devices using FLAT Inband access
Connecting to external devices using SNAT Inband access

2.Does the IP pool for this FLAT network is assigned from an initialization file *.ini for the 172.16.1.0 pool?

Yes you are correct.

3.Which is the Management network required for UWM orchestration as far as ESX port groups are concerned?

It takes the eth0 IP as the management Network required for UWM orchestration.

Thanks and Regards,

Bikash