I understand that system vlan allows traffic flowing for those vlan even if VSM is not reachable, and system vlan should NOT be normal virtual machine traffic vlan. In my deployment of a normal vSphere environment with N1kv, I am setting these vlans as system vlan: ESXi Mgmt, N1kv mgmt, control & packet, VMotion, IP Storage.
I am setting those vlans as system vlan on both uplink port profiles and indivdual port profiles for each of the VLAN. Correct me if this is wrong.
what else should be system vlan, or what of those should NOT be system vlan? VMotion vlan? what are the cons for specifying all vlans as system vlan? Isn't it better because even if VSM is down for any reason, VEMs will still forward traffic for all VMs?
When you specify a VLAN to be system vlan it will ALWAYS be forwarded. This pretty much circumvents a profile "checking-in" with the VSM before forwarding traffic for VMs assigned to it.
The reason why your Mgmt VLAN is normally a system vlan will ensure you always have access. If you have a VM assigned to a VLAN that was a system vlan you're circumventing all programming & security that you may want to apply to that VM. Let's say for example you define QoS, ACLs and other security settings on your VMs port profile - they will not get enforced on your VM should you reboot a VEM host, and the VSM is not reachable. Since your VLAN is a system VLAN it will have open access to forward any/all traffic when you might not want that.
Comes down to enforcing settings & security on Port Profiles, vs. ensuring access to system interfaces for managing tasks.
Your understanding of system VLANs is not fully accurate. All VLANs will remain forwarding in the event your VSM is not reachable. Each VEM module will continue to pass both system and non-system vlan traffic if the VSM goes offline. EACH VEM will keep its current programming, but will not accept any changes until the VSM is back online. System VLANs behave differently in that they will always remain in a forwarding state. Systems VLANs will forward traffic even before a VEM is programmed by the VSM. This is why certain system profiles require them - ie. Control/Packet etc. These VLANs need to be forwarding IN ORDER for the VEM to talk to the VSM.
As for your list of "what should be system vlans" - remove VMotion. There's no reason your VMotion network would need to be defined as a system VLAN. All others are correct.
Also remember you can ONLY define a VLAN on one uplink port profile. So if you're using one uplink for "system" type traffic and another for "VM Data" type traffic, you would only have any single VLAN "allowed" on one uplink - not both. Allowing them on both will cause issues. The only case you need to keep in mind is that for a "system vlan" to be applied, it must be defined on both the vEthernet Port profile and an Uplink Port Profile.
Let's say my Service Console uses VLAN 10, and my VMs also use VLAN 10 for their data traffic. (Bad design, but just to illustrate a point).
Having to define the system VLAN in "two places" would allow you to treat ONLY your "Service Console" traffic as a system traffic, and still enforce programming/security for your "VLAN Data" traffic. Following a reboot, you Service Console traffic would flow immediately, but your VM Data would not until the VEM had pulled programming from the VSM.
port-profile type vethernet dvs_ServiceConsole
switchport mode access
switchport access vlan 10
system vlan 10 <== Defined as System VLAN
port-profile type vethernet dvs_VM_Data_VLAN10
switchport mode access
switchport access vlan 10 <== No System VLAN
port-profile type ethernet system-uplink
switchport mode trunk
switchport trunk allowed vlan 10,3001-3002
channel-group auto mode active
system vlan 10,3001-3002 <== System VLAN 10 Defined
Hope this clears up your understanding.