Hyper-V specified node is not an owner or possible owner

Recently I added a 3rd not to our Hyper-V cluster which uses cluster shared volumes. After this I started seeing errors when trying to live migrate or backup with BackupExec some of the VM's. The error was:

"The operation failed because either the specified cluster node is not the owner of the group, or the node is not a possible owner of the group"

This error is from BackupExec but the error from live migration is almost exactly the same. Looking at the VM's they all appeared to inherit the new cluster node correctly so it was not obvious what the issue was. After a bit of digging I finally figured out the problem was with the cluster shared volumes they had not added the new node as a possible owner. The normal volumes had though. to determine if this is the problem you can run the following PowerShell command:

cluster.exe res "<cluster disk name>" /listowners
e.g.   cluster.exe res "test disk" /listowners

To add owners use the following command

 cluster.exe res "<cluster disk name>" /addowner:<new owner>
e.g.  cluster.exe res "test disk" /addowner:node3

There is no GUI option for viewing or setting these that I have found.

Failover Cluster migrations fail

We just added a 3rd node to our Hyper-V failover cluster and everything went great right up until I tried to do a test live migration. I got the ever so helpful

live migration did not succeed at the source.

Ok so going on past experience I know this has usually been the binding order to the adapters not being the same or name of the network adapter being different. So I go and check and everything looks great. The cluster passes verification and the cluster resources list all my network adapters as online and good but still a no go on the migration.

Ok for fun lets try to migrate just the disks and see if i get a more helpful error:

The operation failed because either the specified cluster node is not the owner of the group or the node is not possible owner of the group

Well that error SEEMS more helpful. Unfortunately it led me on a wild goosechase.  I do some more digging and find a more helpful error on the destination machine:

live migration did not succeed at the destination.

Configuration setup for live migration failed on the destination node. Make sure that name of the virtual network is the same on the source and destination nodes, and try the live migration again.

Ok this was interesting since the source machine said the problem was at the source. Long story short after a quick google I found  http://support.microsoft.com/kb/2475761 it turns out I was on the right track from the beginning and although all the names looked good in the network connections area it actually cares about the names in Hyper-V Manager/Virtual Network Manager. Once I fixed those it is back to migrating.

Monitoring Cluster Shared Volumes without SCOM

Recently we had one of our SAN volumes attached to our VM cluster run out of space. This was a bit of a surprise since I had thought our SAN would warn us when a volume was getting low but it turns out it won't. Luckily all that happened was our virtual guest paused itself so there was no data loss but this was a production server so we needed to make sure this would not continue to happen. Hunting for a way to monitor Cluster Shared Volume (CSV) status I really could not find a way other than Microsoft System Center Operations Manager (SCOM) which is not an option for us. So instead I started looking at scripting something in Powershell.  I knew I could get the available disk space via the VMM cluster console so a script should be possible.

I was able to put this script together thanks to several great examples from others. It will check the CSV status and if it falls below a certain threshold it will send an email to the address specified. You will need to modify a few of the lines below for email server, to address, etc.

    #Load the FailoverClusters module
    Import-Module FailoverClusters

    $warninglevel = 15  # The percent to send warning at
    $objs = @()
    $nl = [Environment]::NewLine

    $csv_status = Get-ClusterSharedVolume
    foreach ( $csv in $csv_status )
    {
       $expanded_csv_info = $csv | select -Property Name -ExpandProperty SharedVolumeInfo
       foreach ( $csvinfo in $expanded_csv_info )
       {
          $obj = New-Object PSObject -Property @{
             Name        = $csvinfo.Name
             Path        = $csvinfo.FriendlyVolumeName
             Size        = $csvinfo.Partition.Size
             FreeSpace   = $csvinfo.Partition.FreeSpace
             UsedSpace   = $csvinfo.Partition.UsedSpace
             PercentFree = $csvinfo.Partition.PercentFree
          }
          if ($csvinfo.Partition.PercentFree -lt $warninglevel) { $objs += $obj }
       }
    }


    if ($objs.count -gt 0) {
        $smtpServer = "mailserver"
        $msg = new-object Net.Mail.MailMessage
        $smtp = new-object Net.Mail.SmtpClient($smtpServer)
        $msg.From = "from address"
        $msg.To.Add("to address")
        $msg.Priority = "high"
        $msg.Subject = "Warning Cluster Volume low on space"
        #Preamble text
        $msg.body = "Enter any explanatory message here or delete this line"              
        #Next line is what puts in the volume information
        $msg.body += $objs | ft -auto Name,Path,@{ Label = "Size(GB)" ; Expression = { "{0:N2}" -f ($_.Size/1024/1024/1024) } },@{ Label = "Free(GB)" ; Expression = { "{0:N2}" -f ($_.FreeSpace/1024/1024/1024) } },@{ Label = "Used(GB)" ; Expression = { "{0:N2}" -f ($_.UsedSpace/1024/1024/1024) } },@{ Label = "PercentFree" ; Expression = { "{0:N2}" -f ($_.PercentFree) } } | Out-String
        $smtp.Send($msg)

    }

To make this work create the script in a ps1 file. Modify the settings that need to be modified and adjust your warning threshold to a percent you like. Create a scheduled job to run it at whatever interval you wish. I run mine every 15 minutes. When creating the job make sure the program you run is powershell.exe then give it a command line option of your script name.