Monthly Archives: December 2014

PowerCLI Session 02: Pipes, Filtering, Logical Operators

About Pipes

In the last lesson we have laid a foundation for variables and arrays, and today we’ll learn how to search & filter them according to custom conditions. In order to do that and much more, we need to use Pipes. The Pipe character | is used after a command, and the data from the command left of the pipe gets presented to the succeeding command. If you remember the old MS-DOS days, you used to list contents of a directory containing many items with:

dir | more

and so you pressed Enter every time you wanted to see the next screen of content. You passed the output of dir command to more that makes reading large amount of text more pleasant.

Simple Example

Now in PowerCLI, let’s say you want to list all VMs on a single ESXi host and put them to a variable. Considering you are already connected to your vCenter via Connect-VIServer, you would  then do:

$allmyVMs = Get-VMHost esxihost1.fqdn.net | Get-VM

Let’s take piping even further – the invaluable Where clause, which can be replaced by a quick mnemonic, a question mark sign ? . If not explicitly stated, the variables that are output from the pipe are stored in the global variable $_ . Any condition that follows must be stored in curly brackets {}.

Filtering, Logical Operators

So if we’d like to further filter the VMs that contain test in their name, we’d use

$testVMs = $allmyVMs | ? {$_.Name -like "*test*"}

When you review your variable, you will see that all VMs containing the name test ale filled in it. You can also filter by more properties than just name, let’s say you only want the VMs that are online, so you exclude the valid powerstate property (you can get the properties list via piping the VM’s variable $allmyVMs | gm).

You will accomplish this by using Logical Operators -and (and) , -eq (equals). There are many more:

  • logical: -or, -not (!)
  • comparison-oriented (return $true or $false): -like, -notlike (both sides of the expression have to be the same, supports wildcards) -contains, -notcontains (for finding an exactly matching string), -ne (not equal), -match, -notmatch (returns true if it finds just a fraction in the compared expression)
  • math-oriented: -ge (greater or equal), -gt (greater than), -le (less or equal), -lt (less than),
$testVMsOnline = $allmyVMs | ? {$_.Name -like "*test*" -and $_.powerstate -eq "PoweredOn"}

Piping to a command

And for our last example, let’s say you actually want to find powered off VMs and then power them on. You will be passing the stored array into a command like this:

Get-VM | ? {$_.powerstate -eq "PoweredOff"} | Start-VM

Piping isn’t really that complicated once you get to know the drill. It is just passing the outputs “along the pipeline” to the commands you would like to. Just experiment with it (be sure to be in a lab environment though) and you will soon become a pipe master 🙂 See you around!

PowerCLI Session 01: Variables, Arrays

Variable Definitions

Welcome to our first lesson in PowerCLI. Today we’ll be getting our hands on  the second most-used thing after commandlets, and these are variables. You will be using them to temporarily store information that you will either use as an input for your subsequent commands (or store commands themselves!), or save them to a text file, maybe even a report.

The variables are always initialized, called, and saved with a dollar sign preceeding a string of your choice. For example $a, $var1, $myvariable, $esxihost etc.

Feeding a value to a variable is pretty straight forward – it is done with an equal sign.

$var1 = 9001

Now, a variable can contain more than just numbers and strings, for example objects, arrays, imported CSVs and hashtables. You can also force a certain data type to be used in your variable.

Variable Definition Code Examples

Since this is a first lesson, I’d like to say that commenting your code properly and clearly is the very fundamental thing to do. This helps you orient in the code, and if you decide to share it – provide some insight for other people who might be using or analyzing your scripts.

Commenting a single line is done via hashtag # and commenting multiple lines of code is used by closing the hashtags in less/greater than signs like this: <# #>

With all of the above, let me show you how variables are defined:


# Let's start with a simple string
$string1 = "Hello World!"
# And then writing your first Hello World to the console.
Write-Host $string1

# You can also force a variable type for your arrays
# Find more info on data types here: http://ss64.com/ps/syntax-datatypes.html
[boolean]$istrue = false
[int]$volume = 5000

# This is a definition an empty array...
$arr1 = @()

# ...and a hashtable
$hash1 = @{}

<# Now, a single variable can contian many things,
like objects and commands. The below stores an
array of objects - all ESXi hosts inside a vCenter#>

$vmhosts = Get-VMHost 

$csv1 = Import-CSV .\myreport.csv

# You can store a command and then subsequently run it with invoke-expression!
$mycommand = "Get-VM"
Invoke-Expression $mycommand

Arrays

Now that you know how to define variables, let’s take a look at Arrays. Hashtables will be covered in one of future sessions, they are quite advanced and I’d like to maintain a coherent way of showing you around.

Arrays are pretty self-explanatory -you can store variables and even objects inside them. One important thing to remember is, that if you have filled an array with objects via a command, you can not prune (remove objects from) the array manually. All the objects need to be explicitly defined by a filter when you push them into an array (we shall look at that in the next session), or you can create an empty array and then push your objects there.

After you store data in an array, you an list it – as you can do with all variables – by simply entering its variable name into the powershell prompt.01

Arrays are indexed with brackets [], starting from 0. So you can see ther “0th” entry like this:02

The individual members of the whole array can also be listed, so as you were shown in the first lesson, using the Get-Member, or gm command in a pipe (pipes will be a subject of our next session). You can also fill an array with a CSV file if you have more parameters you wish to extract from your array (which will become handy later)

This is for just a short overview of the arrays – you will get much more familiar with them in a more hands-on approach. See you around!

 

Threaded Report Generator: VMs without a vmxnet3 vmnic

Yesterday I worked on a nice-to-have script for our Team. Since I had some time on my hands, I decided to give it a take from a different angle and make it threaded. I was fiddling around and googling like crazy, but eventually after several failed attempts and debugging sessions it paid off. I did a huge step in scripting for myself, and I’d like to present the script to you! Threading is something I wanted to learn for a long time, so I am very happy I am able to share this piece of code.

What the script basically does is grab a list of vCenters either via an array (the line can be modified to get it via get-content instead), and then creates a thread (aka a Background Job) for each of them, using the ScriptBlock that is defined in $getnics variable. You can take a look at it and it somewhat closely resembles a function. With one exception – the ScriptBlock for this kind of job (having pristine CSV values) must be completely silent except the last step – returning the $csv value.

When all jobs finish via Get-Job | Wait-Job, all the variables are aggregated into one master variable $csv with Receive-Job commandlet. This retrieves the output contents of the job but also marks them in a way that you couldn’t retrieve these contents again (the variable will be blank), so extra caution is needed there. In the end it all gets outputted to C:\temp\ as a CSV with forced semicolon separator.

In the end you are shown the total time in seconds the script took to run – in our environment of roughly 4500 VMs across all vCenters, it took about 1s per VM non-threaded. The threaded run was completed in ~700 seconds, which is a nice speed boost 🙂 The sample of how the report looks is below:

Sample Report

A report generated with columns for vCenter, its VM, the NIC Type and Guest OS.

 

Write-Host "Threaded vmnic type Gatherer " -ForeGroundColor Cyan
$startedtime = Get-Date

# get date to present to the .csv file
$date = Get-Date -format ddMMyy-HHmm

# Initialize CSV Here
$csv = @()
$csv += "sep=;"
$csv += "vCenter;VM;NIC Type;Guest OS"

# List of vCenters where to gather data from, can be replaced with get-content.
$vcenterlist = @("vcenter1","vcenter2")

# Initialize ScriptBlock to be used in Threads
$getnics= {	# The curly bracket is needed to initiate ScriptBlock
	Param([String]$vc)  # A vCenter will be an input parameter
	Process {	# Start the ScriptBlock's work here:
	$WarningPreference = "SilentlyContinue" # Disable warnings
	[array]$csv = $null # Initialize empty CSV Array

	# Since PowerCLI SnapIn isn't present in Threading, we need to load it
	add-pssnapin VMware.VimAutomation.Core -ea 0 | out-null

	Connect-VIServer $vc -wa 0 | out-null # Connect to vCenter, Silently

	$AllVMs = get-vm # Gather Virtual Machines...

		# Do a ForEach Object (%) on each VM to find out wich VMnic they have
		$AllVMs | % {

		$ThisVM = $_
		$vmnic = (Get-NetworkAdapter -VM $ThisVM).Type

		# No vmxnet3 NICs, no NULL values and no dummy SMVI machines.
	If ($vmnic -ne 'vmxnet3' -and $vmnic -ne $null) # If vmnic isn't vmxnet3 and isn't null, then:
		{
		$guestos = $ThisVM.Guest.OSFullName
		$csv += "$vc;$ThisVM;$vmnic;$guestos"
		}

		}
		return ,$csv

	# Disconnect from vCenter
	Disconnect-VIServer $vc -wa 0 -confirm:$false | out-null
	}
}

# Start threads and impose a wait time so the CPU on the terminal doesn't go crazy.
forEach ($vc in $vcenterlist) {

Write-Host "Starting Thread for" $vc -NoNewLine
Start-Job -Scriptblock $getnics -ArgumentList $vc | out-null
Write-Host "...Done!" -ForegroundColor "Green"
Start-Sleep -s 15

}

Write-Host "All threads started! Waiting for them to finish." -Foregroundcolor Green

# List jobs and wait until they are all completed
Get-Job | Wait-Job 

# Gather data from Threads by aggregating each job into a master CSV variable.
Write-Host "Gathering data from the threads..."
$jobs = Get-Job | ? {$_.State -eq "Completed"}
ForEach ($job in $jobs) {
	$csv += Receive-job $job
}

# Write CSV File
$csvpath = "C:\temp\vmnicreport_$date.csv"
Write-Host "Writing CSV file to $csvpath"
$csv | Out-File $csvpath

# Final stats
$endedtime = Get-Date
$timetook = ($endedtime - $startedtime).TotalSeconds

# Cleanup
Get-Job | Remove-Job

Write-Host "Finished in" $timetook "seconds!" -ForegroundColor Green

I hope you will find this script useful one day when deciding which VMs to upgrade – but don’t forget to update VMware tools first before you get into it, so your VM has sustained connectivity after a reboot.

How much memory should be free for VMkernel?

Recently I have made a small research to see how much free RAM does VMkernel need to work without any hiccups due to:

  • Memory Reclamation Techniques
  • Memory Reservation for the VMkernel itself

I have gathered this data from live environment. However one very important metric is not included in the below measures and graphs, and that is the Virtual Machine overhead that is individual for each environment and is dependant on the VMs’ Memory and vCPU amount.

A quick explanation:

  • RAM [GB]: How many Gigabytes of RAM are installed in the Server.
  • VMKernel [MB]: How many MB are reserved for the VMkernel itself (you can find this value in Configuration -> System Resources Tab).
  • Reclamation [MB]: Calculated with a Memory Reclamation Formula (900 MB + 1% of memory above 24GB).
  • Total [MB]: Sum of VMKernel & Reclamation values. This should be the governing baseline value.
  • Free [%]: How much % of the total server’s memory should be free.
RAM [GB] VMKernel [MB] Reclamation [MB] Total [MB] Free [%]
8 1393 900.0 2293.0 28.0
16 1749 900.0 2649.0 16.2
24 2378 900.0 3278.0 13.3
48 2682 1145.8 3827.8 7.8
64 2745 1309.6 4054.6 6.2
96 3514.5 1637.3 5151.8 5.2
128 3612 1965.0 5577.0 4.3
192 5218.5 2620.3 7838.8 4.0
384 8220 4586.4 12806.4 3.3
512 9985 5897.1 15882.1 3.0

And a graph is below:

Memory Reservation Graph

A Graph representing the GB Installed vs. MB reserved memory.

I hope this table comes in useful when deciding how much RAM there is in your environment for the hosts to use.

PowerCLI Session 00: Introduction to PowerShell & PowerCLI

A few words to start with

Welcome to PowerCLI Sessions, where I’ll be showing you how VMware’s PowerShell Module PowerCLI works, along with examples so you can analyze, study, and most conveniently use the knowledge you find to your own good. I’d like to start with a “0th” session, to introduce people to PowerShell-based scripting on Windows Server systems.

Starting with Windows Server 2008, Microsoft has included an object-oriented shell to its systems called PowerShell. This was to quickly replace the previous, widely used scripting engine own to the Windows OS Family, Visual Basic. Whereas I have done several Visual Basic scripts in the past, it was frankly cumbersome to work with (and I guess many of you guys who have had the pleasure of scripting in VB will agree) – debugging was quite hard, the mnemonics were not that easy to remember – in short it did its job but you had to spend some time fiddling with the code as IDEs were next to none for this scripting language.

Introducing PowerShell & PowerCLI Module

With PowerShell, your weapons in the arsenal are the so-called commandlets (or cmdlets for short), and they are almost always based on a verb-noun basis, so they are pretty easy to remember. The outputs are almost always objects with various properties (or members) that are incredibly useful as I will be showing you throughout the lessons. You will need to wrap your mind around the fact that you are now working with objects in a shell environment – no more feeding plain strings everywhere. Sometimes you will need to input an object into your command else it will fail.

How does PowerCLI come into play here then? Simply by being a Module (we can also call it a plugin, extension, etc.) for PowerShell – supplying it with many new commandlets to be used exclusively within a VMware environment – all these commandlets operate by communicating with vCenter Server, or the ESXi host itself.

Installation

First things first – make sure your core PowerShell is updated to the most recent version possible (v3 for Windows 2008 R2 and v4 for Windows Server 2012 onwards) You can get the PowerCLI Version 5.5 in VMware’s repository – you will just need an account at my.vmware.com. The installation is pretty simple. Just download the executable, follow the instructions and then launch the console with a shortcut that has been created either in your start menu or on the Desktop.

Getting warmed up

The simplest command you will use in PowerCLI is Connect-VIServer (notice the Verb-Noun mnemonic?). This will establish either a connection to the vCenter or ESXi host. If you are unsure how to use the command, just try to Get-Help for it.gethelp

Now you are ready to run the command.

connectvcenter

From there you can use Get-VMHost to list all your ESXi hosts connected to the vCenter Server, Get-VM to get all the Virtual Machines.getvmhost

To explore each object’s properties, a VERY useful tool is Get-Member – this will show you what else is hiding behind the values that were just listed. Let’s try it with Get-VMHostgetmember

As you have noticed I have used the pipe and a shortcut to this command which is gm – and you will be using a lot while learning about objects’ members. To explore a member of the first position in the array, you use a dot like this – let’s use it to check the build number of the first ESXi host:

getmember-buld

To disconnect from the vCenter Server or the ESXi host, just type in Disconnect-VIServer servername and you are done.

Congratulations! You have just tapped in the awesome world of PowerCLI – we’ll continue with introduction to variables in the next Session.

Online ESXi Firmware and Driver Upgrade on HP Servers

When upgrading firmware and drivers on a huge amount of servers, it used to be time-consuming to perform a firmware upgrade after a reboot on each and every one of your ESXi hosts to match the standard. Not anymore – since Service Pack for ProLiant 2014.09.0, the NIC Firmware can be upgraded online as well since its 10.x version (a bump from the 3.x or 4.x versions that now share a unified firmware). A huge step forward – now all the applicable firmware can be upgraded in one go – and online! No need to wait to catch the boot menu and go through HP Smart Update Manager individually.

Here’s a step by step walkthrough:

  1. Download the HP Service Pack for ProLiant you wish to apply. You will need to have a HP account and a server under warranty linked to it in order to download the newest releases.
  2. Stage the .iso file to a server that has a good connection to all the ESXi hosts you plan to upgrade (preferrably a terminal server inside the Data Center) and unpack it to the location of your liking.
  3. Run \\spplocation\hp\swpackages\x64\hpsum_bin_x64.exe – the binary will depend on your OS flavor.
  4. The following console window will pop up, stating that the HP SUM Web Service has been launched and a default web browser will lanch on the machine, opening the address localhost:63001 and automatically logging you in by passing through your credentials. You can also connect to your terminal server from any other computer that can access its ports 63001 or 63002 (and it is more comfortable that way). I strongly suggest using Google Chrome.
    image001
  5. If you access the web interface, this is what you get.image003
  6.  Start by clicking on the drop-down arrow in the top left corner and select Baseline Libraryimage005
  7. You will need to manually initiate the inventory process for the selected baseline, so click on the already present one for the process to begin.
    image007
    After a few minutes, the inventory completes.
    image009
  8. Now we need to add our ESXi hosts, select VM hosts from the drop-down menu.image011
  9. Localhost is added automatically and unfortunately can’t be changed. Click on Add Node.image013
  10. You can either add a single node by its FQDN or a range of IP addresses separated by a dash. You need to specify the type of device you are adding and the package that is your baseline. Don’t forget to put in the root credentials else the initialization will fail.image015
  11. If you need to select specific nodes inside a range, the second entry in the “Select the type of add” has just what you need. You enter the range, and after a scan you select the nodes you desire. Shift+Click and CTRL+Click work here like a charm.image017
  12. After you have added the nodes via the “Node Range” method, select the baseline to apply to them and enter the root credentials. image019
  13. When you were successful and the hosts were added, you can select multiple hosts by shift+click or ctrl+click and the right frame will change to multiple selection operation.image021
  14. Here you will need to select the baseline again by clicking on Select Baselinesimage023
  15. Select the SPP and click on Add
    image025
  16. Back in the multi-select frame you enter root credentials in order to scan the hostsimage027
  17. You will see the inventarization progressimage029
  18. Once the SUM evaluates an update is needed, input the root credentials again and Deploy the components.image031image033
  19. You have reached the familiar deploy screen where you choose the components to upgrade. When you choose Deploy, it will initialize and you will see a gray wheel spinning beside the chosen hosts.image035

When the deployment is complete, you will have a green light next to your hosts you applied updates to, and the updates will be applied on the next reboot – which is ideal for combination with VMware Update Manager to apply patches & firmware in one take.