iSCSI boot for XenServer?

Unanswered Question
Jun 4th, 2012

Anyone tried this yet? I know a lot of VMWare people are probably doing this. I have Xenserver running when installed on the local storage (does not require iscsi vnics) however I can not get the iscsi boot working.

After reading all the documentation I can find on the subject, at the Xenserver install screen you are supposed to enter multipath mode but that's not actually a real option here since I'm not doing multipath to the backend storage for my dev Xen environment.

At any rate, when I'm booting up the blade I do see an iscsi initiator connection on our NetApp so I think the iscsi vnic is working correctly (my boot policy is set as CD-ROM then ISCSI boot).

So at this point, I'm not sure if it's a problem witt my iscsi vnic setup on the UCS or if it's a problem with XenServer itself. If I go through the normal install steps on Xenserver it will load ot the local disks, even though my local disk configuration is set as "No Disk Policy".

Can anyone please send me in the right direction? I figure if I can just get Xen loaded then it will boot correctly since I see the iscsi connection to the backend storage - I just can't tell what the problem is at this point.

Incidentally, I see the following message on the NetApp right before seeing the subsequent "Initialize error 1" message on the blade - SO, it appears the iscsi connection is being made to the NetApp, however, I still see the error on the blade.

OxNas1-F2> Mon Jun  4 10:12:34 EST [OxNas1-F2: iscsi.notice:notice]: ISCSI: New session from initiator iqn.2012-06.com.example:xenucs02 at IP addr

initerror.png

One other thing : running the following commands from the FI shows the output below. Not sure why the "State" of the target is "INVALID"

----------------------------------------------------------------

connect adapter x/y/z (chassis/blade/adapter#)

connect

attach-mcp

iscsi_get_config

-----------------------------------------------------------------

vnic_id: 8

          link_state: Up

       Initiator Cfg:

     initiator_state: ISCSI_INITIATOR_READY

initiator_error_code: ISCSI_BOOT_NIC_NO_ERROR

                vlan: 0

         dhcp status: false

                 IQN: iqn.2012-06.com.example:XENUCS02

             IP Addr: 2.2.2.239

         Subnet Mask: 255.255.255.0

             Gateway: 2.2.2.1

          Target Cfg:

          Target Idx: 0

               State: INVALID

          Prev State: ISCSI_TARGET_GET_LUN_INFO

        Target Error: ISCSI_TARGET_GET_HBT_ERROR

                 IQN: iqn.1992-08.com.netapp:sn.135113177

             IP Addr: 2.2.2.101

                Port: 3260

            Boot Lun: 0

          Ping Stats: Success (10.090ms)

I've already tried the following docs :

http://docs.vmd.citrix.com/XenServer/5.6.0fp1/1.0/en_gb/installation.html#boot_from_san

http://www.cisco.com/en/US/prod/collateral/ps10265/ps10276/whitepaper_c11-702584.html

http://www.cisco.com/en/US/docs/unified_computing/ucs/sw/gui/config/guide/2.0/b_UCSM_GUI_Configuration_Guide_2_0_chapter_011101.html#concept_FBB613095A774844ACA4DD7377EE102D

Doing Multipath iscsi boot is a whole other animal at this point - I'll tackle that one next

Thanks everyone

I have this problem too.
0 votes
  • 1
  • 2
  • 3
  • 4
  • 5
Average Rating: 0 (0 ratings)
randerson@oxfor... Mon, 06/04/2012 - 08:58

UPDATE - got the iscsi LUN to be recognized correctly during the blade boot (ie. no more "Initialize error 1" message). There was an issue in the LUN mapping on the netapp side.

Still can't get the Xenserver installer to recognize the remote LUN though - it always defaults to using the local disk, even though I've got the local disk policy set to not have disks.

BTW - the procedure I'm using for doing the Xenserver install is to hit F2 once the installer splash screen comes up, then type "multipath" at the prompt

gkumark Mon, 06/04/2012 - 23:47

Hi Ross,

May I know which blade moel and the adapter card being used here? Also make sure that you have given the proper LUN ID and make sure that the LUN is in the right iGroup from NetApp side.

Make sure that in the boot policy you have iSCSI as the first option set.

randerson@oxfor... Tue, 06/05/2012 - 05:52

Thanks for the reply Ganesh.

We have B200M2 blades with the M81KR adapter. I see the iscsi connection being made successfully so I know that is working (see attached screen shots).

1) I can see the iscsi LUN being connected to the blade via the Cisco VIC (iscsi vNIC)

2) I can see the iscsi LUN connection at the adapter via the FI

3) I can see the iscsi LUN connection at the NetApp backend storage

4) I'm choosing the recommended option at the Citrix Xenserver installer splash screen which is supposed to allow the presentation of a remote LUN to the installer

-----

-----

-----

So here is where I am stuck. As far as UCS is concerned, I think I have the iscsi LUN connected successfully - the trick, how do I get Xenserver (or any VMWare, or any other hypervisor for that matter) to recognize the remote iscsi LUN when installing??

One other side note, if I go the "shell" mode via the Xenserver installer, none of the ethernet NICs have any configuration, so something is definitely missing. In my case, the iscsi vNIC on the UCS is NIC #3 (eth3), but as you can see when looking at the linux config (via the Xen installer), the nic has no config so I'm not sure how Xen would ever see that iscsi connection to the backend storage - SO, does a UCS linux driver need to be installed for Xen to see the iscsi vNIC being offered up by UCS?  Argh - I don't know.

randerson@oxfor... Tue, 06/05/2012 - 06:39

Yes, I've read that first article but it is from 2010 (I've tried both Xen 5.6 SP2 and 6.0.2, btw) - based on the current cisco ucs/citrix compatibility matrix (from your third link), this should be supported as far as I can tell; however, I don't see iscsi boot explicitly mentioned in the compatibility matrix - cisco VIC is not mentioned anywhere in the storage controllers section either.

I'm going through that second article right now actually and trying to install the linux drivers manually via the Xen Installer "shell" but I'm having problems with reading the ISO (ie. mounting the b-series blade drivers). I see there is a xenserver folder on that ISO.

I have to imagine the procedure is very similiar for VMware iscsi boot as well, but I don't see a specific set of steps for that. This article http://www.linkedin.com/groups/Anyone-had-experience-putting-Citrix-109454.S.103881909
references Xenserver iscsi boot but there is no mention of manually installing linux drivers - only that you do the multipath option, which does not work

randerson@oxfor... Tue, 06/05/2012 - 10:19

UPDATE

At any rate, I've made a little more progress based on this article

http://www.krystalmods.com/index.php?title=xenserver-6-supports-iscsi-boot-undocumented-feature&more=1&c=1&tb=1&pb=1


If you go into "Shell" mode on the installer and issue "/opt/xensource/installer/init --use_ibft --mpath", the installer will be loaded using the multipath method with access to the remote LUN (I imagine this is the way the "multipath" option is supposed to work). I know this part is working correctly because Xen correctly recognizes the NIC I am using for iscsi boot (see pic 1). Unfortunately, I get an error message saying it can't access the LUN, even though it knows the correct NIC to use.


Stuck again .. anyone seen this message?  What a pain .. I thought iscsi boot would be easier ..

randerson@oxfor... Tue, 06/05/2012 - 11:56

UPDATE


I now see that once you specify the iscsi IBFT mode for the Xen installer (see the pics in my above post) that Xen is trying to connect to the backend LUN with a randomly generated open-scsi IQN - I was under the impression it would use the previously created IQN from the iscsi NIC (in this case, provided by UCS - Cisco VIC).

Red is for the UCS VIC IQN, Green is for the randomly generated Xen IQN.

Now I see two independent iscsi initiators on the NetApp, one for the UCS VIC and one for Xen itself - the trouble is that since the Xen IQN is randomly generated each time, I can't give permission to the LUN on the backend. I assumed Xen would use the previously generated iscsi connected (since the Xen installer actually ASKS WHICH ENTRY TO USE FROM THE IBFT) .. but no .. it's trying to connect to the backend LUN instead of using the already established connection.


I'm looking through the xensource installer files now to see if there is a way to hardcode the IQN to be the same as the IQN on the iscsi vnic provided by UCS. Once the installer fails, it puts you back out the linux (bash) prompt and you can see that an initiatorname.iscsi file was created with the random IQN. Million dollar question - how do you get Xen to use the previously created iscsi vnic and it's associated IQN instead

randerson@oxfor... Wed, 06/06/2012 - 06:10

UPDATE

I followed this article (http://www.rienbroekstra.nl/?q=node/19) and was able to map to the iscsi LUN manually. Here are the initial steps, as outlined in the article :


1) Insert relevant kernel modules

modprobe scsi_transport_iscsi

modprobe iscsi_tcp


2) Configure our initiator name, replace the name with whatever is correct for your organisati

on

echo "InitiatorName=iqn.2010-01.com.example:whatever" > /etc/iscsi/initiatorname.iscsi


3) Start the iscsi daemon

iscsid


4) Configure the network interface which is to be used to access the iscsi target, and bring it up

ip addr add 1.2.3.4/24 dev eth0

ifconfig eth0 up


5) Discover the available targets on your iscsi filer (This step is necessary,

even if you know what the correct target is)

iscsiadm -m discovery -t sendtargets -p 1.2.3.1


6) Connect to the target

iscsiadm -m node --targetname iqn.2006-01.com.openfiler:whatever -p 1.2.3.1 --login


7) Kill and restart iscsid. I'm not sure that this is necessary, but it might write the correct configuration to /etc/iscsi/, so I guess it can't hurt.

8) Exit the shell in order to start the installer.

exit


Once you do these steps, Xen will recognize the iscsi LUN and be able to install to that LUN (see attached pics).


The problem now (of course) is that once it boots up again, it hangs on the "Citrix XenServer" splash screen indefinitely. The article above says this is due to an error in the initd in Xenserver - its assuming you are using DHCP, so it does not get the correct IP settings for the iscsi NIC when it is rebooted. There are a set of secondary steps in the article above as well to rectify that but I can not get those to work - I cannot mount the sda1 drive as is indicated in the article.


Here is the thing though - we should not have to do any of these manual steps. I am presenting an iscsi NIC (from UCS) to Xenserver via iBFT. Xenserver in turn is supposed to be taking those NIC settings (ie. IP addr, IQN, etc) and connecting to the backend LUN automatically.


I'll try this other suggestion of running the "mkinitrd"command to hopefully update the installer after manually creating the initiatorname.iscsi file with the correct IQN - this may not work either because the IP address on the dedicated iscsi NIC may not get set.

Frustrated ..

randerson@oxfor... Wed, 06/06/2012 - 12:49

SOLVED

The intitiator name ended up being the problem when running the installer (ie. the Xen Installer reads all of the iBFT settings EXCEPT the IQN for some reason - instead it generates an IQN automatically, instead of linking to the existing IQN from the iBFT). Assuming you have setup the Cisco UCS iscsi boot steps as well as the back-end storage LUN and permissions, you can run the following steps in order to install Xenserver with iscsi boot.

1) At the Xenserver installer screen, press F2 to go into Advanced mode

2) At the prompt, type "shell"

3) Once the bash prompt appears, type in the following :

echo "InitiatorName=iqn.2010-01.com.example:name" > /etc/iscsi/initiatorname.iscsi

where "iqn.2010-01.com.example:name" is your IQN name from the UCS setup (or from your IQN from an iscsi boot agent - works the same way)

4) Now run the following command to initiate the installer :

/opt/xensource/installer/init --use_ibft

You can also use the option "--mpath" if you have multipath setup to the iscsi LUN - this probably would require an update to the multipath.conf file so I left it off for now.

5) The Xen installer will start now - if you have local disks installed then you will be prompted to install to the local disks or to the iscsi LUN; if you don't have local disks you won't get a selection screen at all.

The installer should complete normally now and Xenserver should boot from the iscsi lun. NOTE that Xen now has the iscsi NIC persistently mapped to your iscsi network, so I don't believe you can create another NIC within Xencenter on that same network - I tried and everything stopped working

I did a complete write-up if anyone needs more information.  Thanks all those that chimed in with advice.

Actions

Login or Register to take actions

This Discussion

Posted June 4, 2012 at 7:39 AM
Stats:
Replies:9 Avg. Rating:
Views:3601 Votes:0
Shares:0
Categories: General UCS Hardware
+

Related Content

Discussions Leaderboard