CISCO Nexus 1000V VEM v142 VUM Fix

We had an issue in our environment when upgrading Cisco Nexus 1000V VEM module on all our ESXi hosts from version 4.2.(1)SV1(5.1a), or v142 as found in the filesystem structure. The update just wouldn’t go through VMware Update Manager, either with “ERRNO 8” or error stating that “The host needs to be in the Maintenance Mode before the patch is installed”, although the host was in maintenance already. The same error was thrown when trying to make this update via esxcli.

Me and my colleague poked through this situation and finally came around the conclusion, that the upgrade function inside the vssnet-functions module didn’t properly terminate the vemdpa process. When inspecting the same module of the version we were upgrading to, the upgrade script had even a developer’s note that the vemdpa process needs to be killed with -9 switch, because SIGTERM does not do its job:image002

The modified lines vssnet-functions module will look like this:


To use this script, you will need the plink utility, be able to authenticate with Active Directory – ESX Admin privileges to the ESXi host itself and a bit of patience if the script wouldn’t want to work. Below is the screenshot of patching process:

Screenshot of VEM script

And the script code is below. Notice that it is highly modular and in general you could modify it so it allows sending commands via SSH via a non-root account. How to run commands via plink under as a root even when permit root login is off will be in a scope of a future Scripting Corner.

function ESXSvc ($command, $service, $inESX) { #Function starts here
# Check if the service is running...
$running = Get-VMHostService -VMHost $inESX | Where { $_.Key -eq "$service"}  | Select -ExpandProperty Running

If ($command -eq "start") {
	If ($running -eq $false) {
		Write-Host "Starting $service Service on $inESX"
		Get-VMHostService -VMHost $inESX | Where { $_.Key -eq "$service" } | Start-VMHostService -Confirm:$false | Out-Null
	Write-Host "The $service Service is already running on $inESX"
ElseIf ($command -eq "stop") {
	If ($running -eq $true) {
		Write-Host "Stopping $service Service on $inESX"
		Get-VMHostService -VMHost $inESX| Where { $_.Key -eq “$service”} | Stop-VMHostService -Confirm:$false | Out-Null
	Else {
	Write-Host "The $service Service is already stopped on $inESX"
} # Function ends here

Write-Host "--- n1k-vem fix script ---`n To be used when upgrading the module from version v142" -ForegroundColor Green

# Define username to be used for pushing the file via SCP, the user must be able to access the ESXi host with these credentials
$scpuser = Read-Host 'Enter your username in "username@domain" format'
$scppwd = Read-Host 'Enter your Account password' -AsSecureString

# Define domain here
$domain = "yourdomain.lab"

# set variables for SCP transfer
$scpapp = '.\pscp.exe'
$termapp = '.\plink.exe'
$tmp = '/tmp'
$vemfolder = '/opt/cisco/v142/nexus/vem-v142/shell/'
$vssfile = 'vssnet-functions'
$sshfile = '.\sshfeed.txt'

# Set variables for SSH and Shell Functions
$sshsvc = 'TSM-SSH'

<# Remove this multicomment if you want to estabilish a per-run, per-session connection on the script
# Define vCenter
$vcenter = Read-Host 'Enter the vCenter Server where the ESXi host resides: '
Write-Host "Connecting to vCenter $vcenter"`n
# Connect to vCenter
Connect-VIServer $vcenter | Out-Null

<# If you have an array of ESXi hosts, you can input it here and comment the single ESXi input below
$esxiArray = @()

<# If you have a single ESXi, comment this ForEach Cycle Definition
 orEach ($esxi in $esxiArray) {

# Read ESXi hostname - used when single ESXi needs to be remediated
$esxi = Read-Host 'Enter ESXi host name without domain '

Write-Host `n"--- Processing host $esxi ---"`n -ForegroundColor Cyan
# Convert to FQDN
$esxihost = "$esxi`.$domain"
$esxiget = Get-VMHost $esxihost

Write-Host "--- Starting Remote Management Services ---"`n -ForegroundColor Green
# Enable Remote management on the ESXi host
ESXSvc "start" $sshsvc $esxihost
Write-Host "Disabling Lockdown Mode"
(Get-VMhost $esxihost | Get-View).ExitLockdownMode() | Out-Null

# Convert the secure string so it can be used with pscp and plink
$BSTR = `
$PlainPassword = [System.Runtime.InteropServices.Marshal]::PtrToStringAuto($BSTR)

# Construct a command to copy the file from /tmp to designated location
Write-Host "Modifying vss-functions file on $esxi"

# If the host's key has not been cached before, send a "yes" before performing a blank function inside the session.
$runssh = "cmd /c echo y | $termapp -pw $PlainPassword $scpuser`@$esxi exit"
Invoke-Expression $runssh

Write-Host "---"

# Run the sshfeed.txt inside the session.
$runssh = "$termapp -m $sshfile -pw $PlainPassword $scpuser`@$esxi"
Invoke-Expression $runssh

#Reset the converted password string to NULL
$PlainPassword = $null

# Disable remote management
Write-Host "--- Stopping Remote Management Services ---"`n -ForegroundColor Green
ESXSvc "stop" $sshsvc $esxihost

Write-Host "Enabling Lockdown Mode"
(get-vmhost $esxihost | get-view).EnterLockdownMode() | Out-Null

Write-Host `n"Processing of $esxi completed."`n -ForegroundColor Cyan

# }

# Disconnect from vCenter Server if a per-run, per-session basis was used.
# Disconnect-ViServer $vcenter

Also, you will need the sshfeed.txt file which will tell the plink utility what to do:

cp /opt/cisco/v142/nexus/vem-v142/shell/vssnet-functions /tmp/vssnet-functions.orig
sed 's/doCommand kill \${DPAPID}/doCommand kill -9 \${DPAPID}/g' /tmp/vssnet-functions.orig >/tmp/vssnet-functions.tmp
sed 's/doCommand kill -TERM \${DPAPID}/doCommand kill -9 \${DPAPID}/g' /tmp/vssnet-functions.tmp > /tmp/vssnet-functions
mv /tmp/vssnet-functions /opt/cisco/v142/nexus/vem-v142/shell/
chmod +x /opt/cisco/v142/nexus/vem-v142/shell/vssnet-functions

Once you have both these files in directory, just run the script and patch the hosts’ vssnet-functions file that way.


Share your thoughts

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s