PSOD caused by LINT1 Motherboard Interrupt

One night we had a situation on our remote site that was running ESX 4.1.0 on a DELL PowerEdge T710 Server. It went to PSOD and then the RAID controller stated that it was unable  to boot. The screen captures we got were:


And after a reboot, an unwelcoming screen was shown:

Fortunately, after another reboot the system booted just fine, however it was pretty obvious that the hardware itself was in a pretty unstable state. On iDRAC, we have discovered that we got a critical warning on a component (unfortunately it was late at night and I didn’t think about screenshotting that) with Bus IDs 03:0:0. Listing components via lspci revealed that the following component was sitting on the given ID:

03:00.0 RAID bus controller: LSI Logic / Symbios Logic MegaRAID SAS GEN2 Controller (rev 05)

 Even if it was straightforward from the get-go which component might have been failing, it was double-confirmed by the very useful lspci command.

ESXi Boot Loop on Dell PowerEdge R720

We have faced quite a strange issue with one of our Dell PowerEdge servers on a remote site. When the branded image was deployed on the host, we kept getting bootloops. The system has just started unloading modules after they were seemingly loaded. After inspecting the vmkernel log at boot-time by pressing ALT-F11, I have noticed a few strange warnings:

2014-11-24T04:13:50.237Z cpu2:2631)WARNING: ScsiScan: 1485: Failed to add path vmhba1:C0:T0:L0 : Not found
2014-11-24T04:15:08.990Z cpu7:2792)WARNING: ScsiScan: 1485: Failed to add path vmhba1:C0:T0:L0 : Not found

I have poked around the settings in BIOS to find out what could have been causing the issue that were seemingly coming from the RAID controller itself. I have changed the SATA to report as RAID opposed to AHCI which was set previously, and the next boot was successful.

This didn’t have any effect on already present drives or data because the only device that used the on-board storage controller was the DVD-ROM.

HyperThreading: What is it and does it benefit ESXi?

Many times I come across the question of HyperThreading and its benefits – either in personal computing, but more importantly over the last few years, virtualization. I’d like to talk about what HyperThreading is for a moment, and show you if it benefits the virtualized environment.

What is HyperThreading?

Today, you see HyperThreadng (HT) technology is present on almost every Intel processor, be it Xeon or Core i3/i5/i7 Series. Basically, it splits one physical core to two logical cores, but the term splitting is somewhat inaccurate and confuses many consumers. Thinking that when they run a 2.5GHz 4-core, HyperThreaded CPU, they immediately have 8 effective cores carrying the full processing capability of 20 GHz. Mainly because when you say you split something, you think that this has been divided to two equal parts (or at least that’s what I think, anyways). Continue reading

1GbE Intel NIC Throttled to 100Mbit By SmartSpeed

We had a case on one of our ESXi hosts equipped with an Intel Corporation 82571EB Gigabit Ethernet Controller – although it was 1Gbit in speed, we were unable to achieve autonegotiation higher than 100 Mbit. When setting it manually to 1Gbit, the NIC disconnected itself from the network. Every other setting worked – 10 Mbit and 100Mbit both half and full duplex. We tried investigating with our Network Team, forcing 1Gbit on switch and that has also brought the NIC down.

I delved deeper into this issue and observed the VMkernel log via tail -f when I have forcibly disconnected the NIC and reconnected it again via esxcli. One line appeared that caught my attention:

vmnic6 NIC Link is Up 100 Mbps Full Duplex, Flow Control: None
e1000e 0000:07:00.0: vmnic6: Link Speed was downgraded by SmartSpeed
e1000e 0000:07:00.0: vmnic6: 10/100 speed: disabling TSO

I immediately caugt up on SmartSpeed and tried to find a way to disable it – that is until I have found out on many discussion threads later that SmartSpeed is an intelligent throtlling mechanism that is supposed to keep the connection running on various link speeds when an error somewhere on the link path is detected. The switches were working okay, the NIC didn’t detect any errors, so the next thing to be checked would be the cabling.

I arranged a cable check with the Data Center operators and what do you know – replacing cables for brand new ones eventually solved the issue! Sometimes the failing component causing you a headache for a good few hours can be a “mundane” piece of equipment such as patch cables.

PowerCLI Session 02: Pipes, Filtering, Logical Operators

About Pipes

In the last lesson we have laid a foundation for variables and arrays, and today we’ll learn how to search & filter them according to custom conditions. In order to do that and much more, we need to use Pipes. The Pipe character | is used after a command, and the data from the command left of the pipe gets presented to the succeeding command. If you remember the old MS-DOS days, you used to list contents of a directory containing many items with:

dir | more

and so you pressed Enter every time you wanted to see the next screen of content. You passed the output of dir command to more that makes reading large amount of text more pleasant.

Simple Example

Now in PowerCLI, let’s say you want to list all VMs on a single ESXi host and put them to a variable. Considering you are already connected to your vCenter via Connect-VIServer, you would  then do:

$allmyVMs = Get-VMHost | Get-VM

Let’s take piping even further – the invaluable Where clause, which can be replaced by a quick mnemonic, a question mark sign ? . If not explicitly stated, the variables that are output from the pipe are stored in the global variable $_ . Any condition that follows must be stored in curly brackets {}.

Filtering, Logical Operators

So if we’d like to further filter the VMs that contain test in their name, we’d use

$testVMs = $allmyVMs | ? {$_.Name -like "*test*"}

When you review your variable, you will see that all VMs containing the name test ale filled in it. You can also filter by more properties than just name, let’s say you only want the VMs that are online, so you exclude the valid powerstate property (you can get the properties list via piping the VM’s variable $allmyVMs | gm).

You will accomplish this by using Logical Operators -and (and) , -eq (equals). There are many more:

  • logical: -or, -not (!)
  • comparison-oriented (return $true or $false): -like, -notlike (both sides of the expression have to be the same, supports wildcards) -contains, -notcontains (for finding an exactly matching string), -ne (not equal), -match, -notmatch (returns true if it finds just a fraction in the compared expression)
  • math-oriented: -ge (greater or equal), -gt (greater than), -le (less or equal), -lt (less than),
$testVMsOnline = $allmyVMs | ? {$_.Name -like "*test*" -and $_.powerstate -eq "PoweredOn"}

Piping to a command

And for our last example, let’s say you actually want to find powered off VMs and then power them on. You will be passing the stored array into a command like this:

Get-VM | ? {$_.powerstate -eq "PoweredOff"} | Start-VM

Piping isn’t really that complicated once you get to know the drill. It is just passing the outputs “along the pipeline” to the commands you would like to. Just experiment with it (be sure to be in a lab environment though) and you will soon become a pipe master 🙂 See you around!

PowerCLI Session 01: Variables, Arrays

Variable Definitions

Welcome to our first lesson in PowerCLI. Today we’ll be getting our hands on  the second most-used thing after commandlets, and these are variables. You will be using them to temporarily store information that you will either use as an input for your subsequent commands (or store commands themselves!), or save them to a text file, maybe even a report.

The variables are always initialized, called, and saved with a dollar sign preceeding a string of your choice. For example $a, $var1, $myvariable, $esxihost etc.

Feeding a value to a variable is pretty straight forward – it is done with an equal sign.

$var1 = 9001

Now, a variable can contain more than just numbers and strings, for example objects, arrays, imported CSVs and hashtables. You can also force a certain data type to be used in your variable.

Variable Definition Code Examples

Since this is a first lesson, I’d like to say that commenting your code properly and clearly is the very fundamental thing to do. This helps you orient in the code, and if you decide to share it – provide some insight for other people who might be using or analyzing your scripts.

Commenting a single line is done via hashtag # and commenting multiple lines of code is used by closing the hashtags in less/greater than signs like this: <# #>

With all of the above, let me show you how variables are defined:

# Let's start with a simple string
$string1 = "Hello World!"
# And then writing your first Hello World to the console.
Write-Host $string1

# You can also force a variable type for your arrays
# Find more info on data types here:
[boolean]$istrue = false
[int]$volume = 5000

# This is a definition an empty array...
$arr1 = @()

# ...and a hashtable
$hash1 = @{}

<# Now, a single variable can contian many things,
like objects and commands. The below stores an
array of objects - all ESXi hosts inside a vCenter#>

$vmhosts = Get-VMHost 

$csv1 = Import-CSV .\myreport.csv

# You can store a command and then subsequently run it with invoke-expression!
$mycommand = "Get-VM"
Invoke-Expression $mycommand


Now that you know how to define variables, let’s take a look at Arrays. Hashtables will be covered in one of future sessions, they are quite advanced and I’d like to maintain a coherent way of showing you around.

Arrays are pretty self-explanatory -you can store variables and even objects inside them. One important thing to remember is, that if you have filled an array with objects via a command, you can not prune (remove objects from) the array manually. All the objects need to be explicitly defined by a filter when you push them into an array (we shall look at that in the next session), or you can create an empty array and then push your objects there.

After you store data in an array, you an list it – as you can do with all variables – by simply entering its variable name into the powershell prompt.01

Arrays are indexed with brackets [], starting from 0. So you can see ther “0th” entry like this:02

The individual members of the whole array can also be listed, so as you were shown in the first lesson, using the Get-Member, or gm command in a pipe (pipes will be a subject of our next session). You can also fill an array with a CSV file if you have more parameters you wish to extract from your array (which will become handy later)

This is for just a short overview of the arrays – you will get much more familiar with them in a more hands-on approach. See you around!


Threaded Report Generator: VMs without a vmxnet3 vmnic

Yesterday I worked on a nice-to-have script for our Team. Since I had some time on my hands, I decided to give it a take from a different angle and make it threaded. I was fiddling around and googling like crazy, but eventually after several failed attempts and debugging sessions it paid off. I did a huge step in scripting for myself, and I’d like to present the script to you! Threading is something I wanted to learn for a long time, so I am very happy I am able to share this piece of code.

What the script basically does is grab a list of vCenters either via an array (the line can be modified to get it via get-content instead), and then creates a thread (aka a Background Job) for each of them, using the ScriptBlock that is defined in $getnics variable. You can take a look at it and it somewhat closely resembles a function. With one exception – the ScriptBlock for this kind of job (having pristine CSV values) must be completely silent except the last step – returning the $csv value.

When all jobs finish via Get-Job | Wait-Job, all the variables are aggregated into one master variable $csv with Receive-Job commandlet. This retrieves the output contents of the job but also marks them in a way that you couldn’t retrieve these contents again (the variable will be blank), so extra caution is needed there. In the end it all gets outputted to C:\temp\ as a CSV with forced semicolon separator.

In the end you are shown the total time in seconds the script took to run – in our environment of roughly 4500 VMs across all vCenters, it took about 1s per VM non-threaded. The threaded run was completed in ~700 seconds, which is a nice speed boost 🙂 The sample of how the report looks is below:

Sample Report

A report generated with columns for vCenter, its VM, the NIC Type and Guest OS.


Write-Host "Threaded vmnic type Gatherer " -ForeGroundColor Cyan
$startedtime = Get-Date

# get date to present to the .csv file
$date = Get-Date -format ddMMyy-HHmm

# Initialize CSV Here
$csv = @()
$csv += "sep=;"
$csv += "vCenter;VM;NIC Type;Guest OS"

# List of vCenters where to gather data from, can be replaced with get-content.
$vcenterlist = @("vcenter1","vcenter2")

# Initialize ScriptBlock to be used in Threads
$getnics= {	# The curly bracket is needed to initiate ScriptBlock
	Param([String]$vc)  # A vCenter will be an input parameter
	Process {	# Start the ScriptBlock's work here:
	$WarningPreference = "SilentlyContinue" # Disable warnings
	[array]$csv = $null # Initialize empty CSV Array

	# Since PowerCLI SnapIn isn't present in Threading, we need to load it
	add-pssnapin VMware.VimAutomation.Core -ea 0 | out-null

	Connect-VIServer $vc -wa 0 | out-null # Connect to vCenter, Silently

	$AllVMs = get-vm # Gather Virtual Machines...

		# Do a ForEach Object (%) on each VM to find out wich VMnic they have
		$AllVMs | % {

		$ThisVM = $_
		$vmnic = (Get-NetworkAdapter -VM $ThisVM).Type

		# No vmxnet3 NICs, no NULL values and no dummy SMVI machines.
	If ($vmnic -ne 'vmxnet3' -and $vmnic -ne $null) # If vmnic isn't vmxnet3 and isn't null, then:
		$guestos = $ThisVM.Guest.OSFullName
		$csv += "$vc;$ThisVM;$vmnic;$guestos"

		return ,$csv

	# Disconnect from vCenter
	Disconnect-VIServer $vc -wa 0 -confirm:$false | out-null

# Start threads and impose a wait time so the CPU on the terminal doesn't go crazy.
forEach ($vc in $vcenterlist) {

Write-Host "Starting Thread for" $vc -NoNewLine
Start-Job -Scriptblock $getnics -ArgumentList $vc | out-null
Write-Host "...Done!" -ForegroundColor "Green"
Start-Sleep -s 15


Write-Host "All threads started! Waiting for them to finish." -Foregroundcolor Green

# List jobs and wait until they are all completed
Get-Job | Wait-Job 

# Gather data from Threads by aggregating each job into a master CSV variable.
Write-Host "Gathering data from the threads..."
$jobs = Get-Job | ? {$_.State -eq "Completed"}
ForEach ($job in $jobs) {
	$csv += Receive-job $job

# Write CSV File
$csvpath = "C:\temp\vmnicreport_$date.csv"
Write-Host "Writing CSV file to $csvpath"
$csv | Out-File $csvpath

# Final stats
$endedtime = Get-Date
$timetook = ($endedtime - $startedtime).TotalSeconds

# Cleanup
Get-Job | Remove-Job

Write-Host "Finished in" $timetook "seconds!" -ForegroundColor Green

I hope you will find this script useful one day when deciding which VMs to upgrade – but don’t forget to update VMware tools first before you get into it, so your VM has sustained connectivity after a reboot.

How much memory should be free for VMkernel?

Recently I have made a small research to see how much free RAM does VMkernel need to work without any hiccups due to:

  • Memory Reclamation Techniques
  • Memory Reservation for the VMkernel itself

I have gathered this data from live environment. However one very important metric is not included in the below measures and graphs, and that is the Virtual Machine overhead that is individual for each environment and is dependant on the VMs’ Memory and vCPU amount.

A quick explanation:

  • RAM [GB]: How many Gigabytes of RAM are installed in the Server.
  • VMKernel [MB]: How many MB are reserved for the VMkernel itself (you can find this value in Configuration -> System Resources Tab).
  • Reclamation [MB]: Calculated with a Memory Reclamation Formula (900 MB + 1% of memory above 24GB).
  • Total [MB]: Sum of VMKernel & Reclamation values. This should be the governing baseline value.
  • Free [%]: How much % of the total server’s memory should be free.
RAM [GB] VMKernel [MB] Reclamation [MB] Total [MB] Free [%]
8 1393 900.0 2293.0 28.0
16 1749 900.0 2649.0 16.2
24 2378 900.0 3278.0 13.3
48 2682 1145.8 3827.8 7.8
64 2745 1309.6 4054.6 6.2
96 3514.5 1637.3 5151.8 5.2
128 3612 1965.0 5577.0 4.3
192 5218.5 2620.3 7838.8 4.0
384 8220 4586.4 12806.4 3.3
512 9985 5897.1 15882.1 3.0

And a graph is below:

Memory Reservation Graph

A Graph representing the GB Installed vs. MB reserved memory.

I hope this table comes in useful when deciding how much RAM there is in your environment for the hosts to use.

PowerCLI Session 00: Introduction to PowerShell & PowerCLI

A few words to start with

Welcome to PowerCLI Sessions, where I’ll be showing you how VMware’s PowerShell Module PowerCLI works, along with examples so you can analyze, study, and most conveniently use the knowledge you find to your own good. I’d like to start with a “0th” session, to introduce people to PowerShell-based scripting on Windows Server systems.

Starting with Windows Server 2008, Microsoft has included an object-oriented shell to its systems called PowerShell. This was to quickly replace the previous, widely used scripting engine own to the Windows OS Family, Visual Basic. Whereas I have done several Visual Basic scripts in the past, it was frankly cumbersome to work with (and I guess many of you guys who have had the pleasure of scripting in VB will agree) – debugging was quite hard, the mnemonics were not that easy to remember – in short it did its job but you had to spend some time fiddling with the code as IDEs were next to none for this scripting language.

Introducing PowerShell & PowerCLI Module

With PowerShell, your weapons in the arsenal are the so-called commandlets (or cmdlets for short), and they are almost always based on a verb-noun basis, so they are pretty easy to remember. The outputs are almost always objects with various properties (or members) that are incredibly useful as I will be showing you throughout the lessons. You will need to wrap your mind around the fact that you are now working with objects in a shell environment – no more feeding plain strings everywhere. Sometimes you will need to input an object into your command else it will fail.

How does PowerCLI come into play here then? Simply by being a Module (we can also call it a plugin, extension, etc.) for PowerShell – supplying it with many new commandlets to be used exclusively within a VMware environment – all these commandlets operate by communicating with vCenter Server, or the ESXi host itself.


First things first – make sure your core PowerShell is updated to the most recent version possible (v3 for Windows 2008 R2 and v4 for Windows Server 2012 onwards) You can get the PowerCLI Version 5.5 in VMware’s repository – you will just need an account at The installation is pretty simple. Just download the executable, follow the instructions and then launch the console with a shortcut that has been created either in your start menu or on the Desktop.

Getting warmed up

The simplest command you will use in PowerCLI is Connect-VIServer (notice the Verb-Noun mnemonic?). This will establish either a connection to the vCenter or ESXi host. If you are unsure how to use the command, just try to Get-Help for it.gethelp

Now you are ready to run the command.


From there you can use Get-VMHost to list all your ESXi hosts connected to the vCenter Server, Get-VM to get all the Virtual Machines.getvmhost

To explore each object’s properties, a VERY useful tool is Get-Member – this will show you what else is hiding behind the values that were just listed. Let’s try it with Get-VMHostgetmember

As you have noticed I have used the pipe and a shortcut to this command which is gm – and you will be using a lot while learning about objects’ members. To explore a member of the first position in the array, you use a dot like this – let’s use it to check the build number of the first ESXi host:


To disconnect from the vCenter Server or the ESXi host, just type in Disconnect-VIServer servername and you are done.

Congratulations! You have just tapped in the awesome world of PowerCLI – we’ll continue with introduction to variables in the next Session.

Online ESXi Firmware and Driver Upgrade on HP Servers

When upgrading firmware and drivers on a huge amount of servers, it used to be time-consuming to perform a firmware upgrade after a reboot on each and every one of your ESXi hosts to match the standard. Not anymore – since Service Pack for ProLiant 2014.09.0, the NIC Firmware can be upgraded online as well since its 10.x version (a bump from the 3.x or 4.x versions that now share a unified firmware). A huge step forward – now all the applicable firmware can be upgraded in one go – and online! No need to wait to catch the boot menu and go through HP Smart Update Manager individually.

Here’s a step by step walkthrough:

  1. Download the HP Service Pack for ProLiant you wish to apply. You will need to have a HP account and a server under warranty linked to it in order to download the newest releases.
  2. Stage the .iso file to a server that has a good connection to all the ESXi hosts you plan to upgrade (preferrably a terminal server inside the Data Center) and unpack it to the location of your liking.
  3. Run \\spplocation\hp\swpackages\x64\hpsum_bin_x64.exe – the binary will depend on your OS flavor.
  4. The following console window will pop up, stating that the HP SUM Web Service has been launched and a default web browser will lanch on the machine, opening the address localhost:63001 and automatically logging you in by passing through your credentials. You can also connect to your terminal server from any other computer that can access its ports 63001 or 63002 (and it is more comfortable that way). I strongly suggest using Google Chrome.
  5. If you access the web interface, this is what you get.image003
  6.  Start by clicking on the drop-down arrow in the top left corner and select Baseline Libraryimage005
  7. You will need to manually initiate the inventory process for the selected baseline, so click on the already present one for the process to begin.
    After a few minutes, the inventory completes.
  8. Now we need to add our ESXi hosts, select VM hosts from the drop-down menu.image011
  9. Localhost is added automatically and unfortunately can’t be changed. Click on Add Node.image013
  10. You can either add a single node by its FQDN or a range of IP addresses separated by a dash. You need to specify the type of device you are adding and the package that is your baseline. Don’t forget to put in the root credentials else the initialization will fail.image015
  11. If you need to select specific nodes inside a range, the second entry in the “Select the type of add” has just what you need. You enter the range, and after a scan you select the nodes you desire. Shift+Click and CTRL+Click work here like a charm.image017
  12. After you have added the nodes via the “Node Range” method, select the baseline to apply to them and enter the root credentials. image019
  13. When you were successful and the hosts were added, you can select multiple hosts by shift+click or ctrl+click and the right frame will change to multiple selection operation.image021
  14. Here you will need to select the baseline again by clicking on Select Baselinesimage023
  15. Select the SPP and click on Add
  16. Back in the multi-select frame you enter root credentials in order to scan the hostsimage027
  17. You will see the inventarization progressimage029
  18. Once the SUM evaluates an update is needed, input the root credentials again and Deploy the components.image031image033
  19. You have reached the familiar deploy screen where you choose the components to upgrade. When you choose Deploy, it will initialize and you will see a gray wheel spinning beside the chosen hosts.image035

When the deployment is complete, you will have a green light next to your hosts you applied updates to, and the updates will be applied on the next reboot – which is ideal for combination with VMware Update Manager to apply patches & firmware in one take.