XenApp Auto Scaling Script

I wrote a similar script for a customer running XenApp 6.5 in AWS. The script below has been heavily updated for XenApp 7.x and is much more advanced than the XenApp 6.5 script. It uses the Hypervisior Power Management commands from XenApp to turn machines on and off so will work in any environment that has a Hypervisior connection configured, not just AWS.

I have only tested this script in my lab environment so I am looking for some beta testers.

The script is designed to run in a infinite loop which is broken down into a Scale Up period and a Scale Down period.

During the Scale Up period the script monitors the average load across each Delivery Group and powers on a new server once the load reaches the value set in ScaleUpLoad parameter. If there are no servers powered on from a Delivery Group when the script starts it will power 1 server on to get the ball rolling, as such.

The script disables logons to a percentage of servers during the Scale Down period, and then shuts down a server when there are no more sessions remaining on the server. Only servers that are powered on, registered, and not in maintenance mode will be touched by the script during scale down. If 50% of servers in a Delivery Group are in maintenance mode then a percentage of the remaining 50% will be kept online and the rest will be shutdown by the script (when / if no users sessions remain).

The ReverseScaleLoad parameter defines at what load servers should be taken out of Maintenance Mode or powered back on if the remaining servers get overloaded during the Scale Down period.

If you want a server to be ignored by the script, tag it with NoAutoScaling.

The script logs information to the XenApp configuration logging database as can be seen in the screenshot below.

Auto Scaling Logging Auto Scaling Logging 2

 

I have built the following fail-safes into the script to try to prevent a production outage where servers don’t get powered on or powered off due to the script having failed.

  • The script can be run multiple times on multiple controllers. The script will only ever stay running on 1 controller at a time (Use a scheduled task to run the script every 10 minutes)
  • The script periodically checks the list of controllers it is talking to and only selects controllers which are active. By default it will talk to the controller it is running on if it is active, if not active it will try a remote controller.
  • If the script starts on a controller where the broker services is down (and therefore cannot retrieve the list of other controllers) it will exit to allow another controller to run the script
  • The script saves key information to XenApp tags, allowing another controllers / script process to pick up where another script failed.
  • A kill switch has been built in to allow for emergencies where maximum capacity may be required. When a flag file is placed in the $Path directory, all servers are turned on and taken out of Maintenance Mode.

A few other notes.

  • Due to the way the script runs in an infinite loop, when you first run the script it may consume high CPU until it hits the first loop. So if your Scale Up time is going to start at 07:00, schedule the script to start at 06:50 the first time and repeat every 10 minutes. Once it enters the first loop, CPU will go down.
  • Don’t rename the script file otherwise the check to see if the script is already running locally or on other controllers will fail.

Desktop Groups must be tagged with AutoScaling to be included in the script, this allows Desktop Groups to be added and removed from the script without having to change or stop the scheduled task on all the controllers. Desktop Groups tagged with NoAutoScaling will be ignored, even if tagged with AutoScaling as well.

Use the following command to create the tag

New-BrokerTag -Name 'AutoScaling'

And this command to add the tag to the desired desktop group, in this example the Engineering Desktop Group

Add-BrokerTag -Name 'AutoScaling' -DesktopGroup 'Engineering'

We don’t have a code signing certificate yet so both script are available as .txt and .ps1 unsigned.

[download id=”2778″]

[download id=”2773″]

 

<# .SYNOPSIS AutoScales XenApp 7,x environments based on server load and users sessions .DESCRIPTION Script runs in a continuous loop which is broken down into a Scale Up period and a Scale Down period. Only Desktop Groups that are tagged with the AutoScaling tag will touched by the script. This tag must be added manually to the Desktop Group. During the Scale Up period the average load across the available servers is monitored. When the average load goes above the specified load (ScaleUpLoad) a number of servers (PowerOnQuantity) are turned on. During the Scale Down period a percentage of servers are kept online (KeepOnline), the rest of the servers are placed into Maintenance Mode. When a server has 0 sessions remaining it will be turned off. If the average load of the servers kept online during the Scale Down period goes above the specified load (ReverseScaleLoad) the script will first try to take servers out of Maintenance Mode, until the average load goes below ReverseScaleLoad). If no servers are turned on and in Maintenance Mode the script will turn on number of servers as specified by PowerOnQuantity. Set the ReverseScaleLoad to be suitably high so as not to prevent servers being needlessly taken out of Maintenance Mode. Recommend a value between 8000 and 9500. Only Desktop Groups tagged with AutoScaling and NOT tagged with NoAutoScaling will be processed by the script. Only servers which are NOT tagged with NoAutoScaling will be processed during the Scale Up period and only servers which are On, Registered, NOT in Maintenance Mode and NOT tagged with NoAutoScaling will be dealt with by the script during the Scale Down period. A kill switch functionality has been built into the script. If a fill called killscaling.txt is placed in the location specified by the Path parameter, all servers will be turned on and taken out of maintenance mode. The script will then exit and all instances of the script will not run until the killscaling.txt file is removed. Servers are tagged with AutoScaling tag during Scale Down period to keep track of which servers the script is processing. This allows another instance of the script running on the same server or another server to take over the AutoScaling task. Other key information is written to the tags to allow the continuity in the case of a script crash. The script is designed to be run on XenApp controllers. When the script starts it gets a list of controllers from the farm. If it cannot get a list of controllers the script will exit, allowing another controller to run the script. By default the script will use the local controller to make PowerShell calls to the XenApp farm. The script periodically checks if the local controller is Active and will revert to talking to a remote controller if it is not. The script also checks another instances of itself is already running locally or on any other controller in the farm. If it finds another instance it will exit. This allows the script to be run on multiple controllers at multiple times throughout the day and only 1 instance of the script will ever be active at any one time. Use the "Repeat Task Every" functionality of the Scheduled Task trigger and set to suitably low value such as 10 minutes. Stagger the Scheduled Task start times across your controllers so that the script does not start on every controller at the same time. Do not rename the script file as this will break the functionality described above. Key information is logged to the XenApp Configuration Logging database. .PARAMETER ScaleUpTime Time of the day that the Scale Up period should start. Should be specified using 24 clock, E.G for 7:45am it would be 07:45 .PARAMETER ScaleUpLoad The average load at which a new server is turned on during the Scale Up period. This should be a number between 0 and 10000, 10000 being maximum load The script will not turn servers on instantly so plan for this. When a server starts up it's load is at 10000 for some time. The script therefore does not check load continuously otherwise all servers would be turned on as soon as the ScaleUpLoad was exceeded. .PARAMETER PowerOnQuantity The number of servers powered on at a time when ScaleUpLoad or ReverseScaleLoad are exceeded .PARAMETER ScaleDownTime Time of the day that the Scale Up period should start. Should be specified using 24 clock, E.G for 5:30pm it would be 17:30 .PARAMETER KeepOnline The percentage of servers to be kept online during the Scale Down period. If less than 1, script will round up to 1. If 10 servers "available" in the delivery group, and this is set to 20, 8 servers will be placed into Maintenance Mode during the Scale Down period. .PARAMETER ReverseScaleLoad The average load across the servers kept online above which new servers are taken out of Maintenance Mode or powered on. .PARAMETER Path The path to the share or directory where the killscaling.txt file will be looked for. .EXAMPLE PS C:\Scripts > .\XenApp_Auto_Scaling_Script.ps1 -ScaleUpTime 06:00 -ScaleUpLoad 7000 -PowerOnQuantity 2 -ScaleDownTime 17:00 -KeepOnline 20 -ReverseScaleLoad 9000
    -Path \\IREALVFIL001\XenAppScaling
    
    Scale Up period will start at 6am, new servers will be turned on when the average load of availabe servers goes above 7000, 2 servers will be turned on at a time.
    Scale Down period will start at 5pm, 20% of servers will be kept online. New servers will be taken out of Maintenance Mode or powered back on when the average
    load of the available servers goes above 9000

    The script will look in the share \\IREALVFIL001\XenAppScaling for the killscaling.txt flag file.
.INPUTS
	None. You cannot pipe objects to this script.
.OUTPUTS
	No objects are output from this script.  This script creates a its own logs files
.LINK
	
.NOTES
    NAME: XenApp_Auto_Scaling.ps1
    VERSION: 1.00
    AUTHOR: Shaun Ritchie
    shaun.ritchie@euc.consulting
    https://merazuppdwebwordpress01.azurewebsites.net	
    
    The script must be executed using an account which has permissions to query the XenApp farm and place servers
    into Maintenance Mode, edit Tags, and power manage machines.
#>

[CmdletBinding(SupportsShouldProcess = $false, ConfirmImpact = "None", DefaultParameterSetName = "") ]

Param ([parameter(Mandatory=$true)] 
	   [Alias("SU")]
	   [ValidateNotNullOrEmpty()]
	   [String]$ScaleUpTime="",

       [parameter(Mandatory=$true)] 
	   [Alias("D1")]
	   [ValidateNotNullOrEmpty()]
	   [Int]$ScaleUpLoad="5000",

       [parameter(Mandatory=$true)] 
	   [Alias("MH")]
	   [ValidateNotNullOrEmpty()]
	   [Int]$PowerOnQuantity="",
       
       [parameter(Mandatory=$true)] 
	   [Alias("D2")]
	   [ValidateNotNullOrEmpty()]
	   [String]$ScaleDownTime="",

       [parameter(Mandatory=$true)] 
	   [Alias("RT")]
	   [ValidateNotNullOrEmpty()]
	   [Int]$KeepOnline="10",
       
       [parameter(Mandatory=$true)] 
	   [Alias("D3")]
	   [ValidateNotNullOrEmpty()]
	   [Int]$ReverseScaleLoad="",
       
       [parameter(Mandatory=$true)]
	   [Alias("MT")]
	   [ValidateNotNullOrEmpty()]
	   [String]$Path="")

$modules = 'Citrix.ConfigurationLogging.Admin.V1','Citrix.Broker.Admin.V2'

foreach ($module in $modules)
{
        If (!(Get-PSSnapin $module -ErrorAction SilentlyContinue))
        {
                Add-PSSnapin $module -ErrorAction SilentlyContinue
        }
}

$scalingModeMetaData      = 'ScalingMode'
$scaleUpEpochMetaData     = 'ScaleUpTimeEpoch'
$scaleDownEpochMetaData   = 'ScaleDownTimeEpoch'
$startUpTimeMetaData      = 'StartUpTime'
$shutdownTimeMetaData     = 'ShutdownTime'
$maintenanceModeMetaData  = 'MaintenanceMode'
$scaleUpTimeEpoch         = $null
$scaleDownTimeEpoch       = $null

Function GetActiveController
{
       $AdminAddresses = $controllers
       $fail = @()
       Try
       {
            Get-BrokerSite -ErrorAction Stop -AdminAddress 'localhost' | Out-Null
       }
       Catch [system.exception]
       {
       
            $fail = $true
       }
       Finally
       {
               If ($fail)
               {
                    (($AdminAddresses | Where-Object {$_.State -eq 'Active' -and $_.DNSName -ne ($env:COMPUTERNAME + "." + $env:USERDNSDOMAIN) } | Select-Object DNSName)[0]).DNSName
               }
               Else
               {
                    ($AdminAddresses | Where-Object {$_.DNSName -eq ($env:COMPUTERNAME + "." + $env:USERDNSDOMAIN)}).DNSName
               }
        }
}

Function KillSwitch
{
        If (Test-Path "$path\killscaling.txt")
        {
                $highLevelLogOp = Start-LogHighLevelOperation  -AdminAddress $adminAddress -Source "AutoScaling" -StartTime (Get-Date -Format G) -Text "EUC Consulting Auto Scaling: KillSwitch has been engaged by flag file."
                Stop-LogHighLevelOperation -EndTime (Get-Date -Format G) -HighLevelOperationId $highLevelLogOp.Id -IsSuccessful $True -AdminAddress $adminAddress
                ForEach ($desktopGroup in $desktopGroups)
                {
                        $brokerMachines = Get-BrokerMachine -DesktopGroupName $desktopGroup.Name {Tag -ne 'NoAutoScaling' -and Tag -eq 'AutoScaling'} -AdminAddress $adminAddress
                        ForEach ($machine in $brokerMachines)
                        {
                                Get-LogSite -AdminAddress $adminAddress | Out-Null
                                $highLevelLogOp = Start-LogHighLevelOperation  -AdminAddress $adminAddress -Source "AutoScaling" -StartTime (Get-Date -Format G) -Text "EUC Consulting Auto Scaling: Kill Switch engaged. Turning on $($machine.MachineName) and taking out of Maintenance Mode"
                                New-BrokerHostingPowerAction -Action TurnOn -MachineName $machine.MachineName -AdminAddress $adminAddress | Out-Null
                                Set-BrokerMachine -MachineName $machine.MachineName -InMaintenanceMode $false -AdminAddress $adminAddress | Out-Null
                                Stop-LogHighLevelOperation -EndTime (Get-Date -Format G) -HighLevelOperationId $highLevelLogOp.Id -IsSuccessful $True -AdminAddress $adminAddress
                        }
                }
                Exit
        }
}

Function TagBrokerMachines
{
        Param([object]$machine)
        $tag = Get-BrokerTag -Name 'AutoScaling' -DesktopUid $machine.Uid -ErrorAction SilentlyContinue
        If (!($tag))
        {
                Add-BrokerTag -Name 'AutoScaling' -Desktop $machine.Uid -AdminAddress $adminAddress
        }
}

# Get all controllers in the farm. If no controllers captured, local Broker Services is down.

$controllers = Get-BrokerController
If ($controllers.Length -eq 0)
{
        Exit
}

# Check if script is already running on local controller or any other controller in the farm, exit if running

foreach ($controller in $controllers)
{
        $currentProcess =  [System.Diagnostics.Process]::GetCurrentProcess()
        $process = Get-WMIObject -ComputerName $controller.DNSName -Class Win32_Process -Filter "Name='PowerShell.exe'" | Where {$_.CommandLine -Like "*XenApp_Auto_Scaling*"}
        If ($process -ne $null)
        {
                If ($process.CSName -ne $env:COMPUTERNAME)
                {
                    Exit
                }
                If ($process.CSName -eq $env:COMPUTERNAME -and $process.ProcessId -ne $currentProcess.Id)
                {
                    Exit
                }
        }
}

# Check if Kill Switch is in place

If (test-path "$path\killscaling.txt")
{
        Exit
}

While ($true)
{
        [string]$adminAddress = GetActiveController
        Get-LogSite -AdminAddress $adminAddress
             
        # Get broker machines
        
        $brokermachines = Get-BrokerMachine -DesktopGroupUid $desktopGroup.Uid -AdminAddress $adminAddress -Filter {Tag -ne 'NoAutoScaling'}

        # If ScaleDownTime is less than ScaleUpTime, ScaleDownTime is tomorrow. Convert all times to Epoch time

        If ($scaleDownTime -lt $scaleUpTime)
        {
                $scaleDownTimeEpoch = (Get-Date $scaleDownTime -UFormat "%s")
                $scaleDownTimeEpoch = [int]$scaleDownTimeEpoch + 86400
        }
        Else
        {
                $scaleDownTimeEpoch = (Get-Date $scaleDownTime -UFormat "%s")    
        }
        $scaleUpTimeEpoch   = (Get-Date $scaleUpTime -UFormat "%s")

        # Clear Tag MetaData if Epoch times are invalid

        foreach ($desktopGroup in $desktopGroups)
        {
                $tag = Get-BrokerTag -Name 'AutoScaling' -DesktopGroupUid $desktopGroup.Uid
                If ($tag.MetaDataMap[$scalingModeMetaData] -eq 'Up' -and $tag.MetadataMap[$scaleDownEpochMetaData] -lt (Get-Date $scaleDownTime -UFormat "%s"))
                {
                        $tag | Set-BrokerTagMetadata -Name $scaleDownEpochMetaData -Value 'null' -AdminAddress $adminAddress
                }

                If ($tag.MetaDataMap[$scalingModeMetaData] -eq 'Down' -and $tag.MetadataMap[$scaleUpEpochMetaData] -lt (Get-Date $scaleDownTime -UFormat "%s"))
                {
                        $tag | Set-BrokerTagMetadata -Name $scaleUpEpochMetaData -Value 'null' -AdminAddress $adminAddress
                }
        }
        
        # Read / write Epoch times from / to Desktop Group tag
        
        foreach ($desktopGroup in $desktopGroups)
        {
                $tag = Get-BrokerTag -Name 'AutoScaling' -DesktopGroupUid $desktopGroup.Uid
                If ($tag.MetadataMap.Keys -notcontains $scalingModeMetaData)
                {
                        $tag| Set-BrokerTagMetadata -Name $scalingModeMetaData -Value 'null' -AdminAddress $adminAddress
                }
                If ($tag.MetadataMap.Keys -notcontains ($DesktopGroup.Name + $maintenanceModeMetaData))
                {
                        $tag | Set-BrokerTagMetadata -Name ($DesktopGroup.Name + $maintenanceModeMetaData) -Value 'null' -AdminAddress $adminAddress
                }
                If ($tag.MetaDataMap.Keys -notcontains $scaleUpEpochMetaData)
                {
                        $tag | Set-BrokerTagMetadata -Name $scaleUpEpochMetaData -Value 'null' -AdminAddress $adminAddress
                }
                If ($tag.MetadataMap[$scaleUpEpochMetaData] -eq 'null')
                {
                        $tag | Set-BrokerTagMetadata -Name $scaleUpEpochMetaData -Value $scaleUpTimeEpoch -AdminAddress $adminAddress
                }
                Else
                {
                        $scaleUpTimeEpoch = $tag.MetadataMap[$scaleUpEpochMetaData]
                }
                If ($tag.MetaDataMap.Keys -notcontains $scaleDownEpochMetaData)
                {
                        $tag | Set-BrokerTagMetadata -Name $scaleDownEpochMetaData -Value 'null' -AdminAddress $adminAddress
                }
                If ($tag.MetaDataMap[$scaleDownEpochMetaData] -eq 'null')
                {
                        $tag | Set-BrokerTagMetadata -Name $scaleDownEpochMetaData -Value $scaleDownTimeEpoch -AdminAddress $adminAddress
                }
                Else
                {
                        $scaleDownTimeEpoch = $tag.MetadataMap[$scaleDownEpochMetaData]
                }
        }

        # Check if there are enough servers to power on in the Desktop Group

        foreach ($desktopGroup in $desktopGroups)
        {
                If ((Get-BrokerMachine -DesktopGroupName $desktopGroup.Name -Filter {Tag -ne 'NoAutoScaling'} -AdminAddress $adminAddress).count -lt $powerOnQuantity)
                {
                        
                        $highLevelLogOp = Start-LogHighLevelOperation  -AdminAddress $adminAddress -Source "AutoScaling" -StartTime (Get-Date -Format G) -Text "EUC Consulting Auto Scaling: There are not enough servers to power on in the Desktop Group $($desktopGroup.Name)"
                        Stop-LogHighLevelOperation  -AdminAddress $adminAddress -EndTime (Get-Date -Format G) -HighLevelOperationId $highLevelLogOp.Id -IsSuccessful $True
                }
        }

        $desktopGroups =  Get-BrokerDesktopGroup -AdminAddress $adminAddress -Filter {Tag -eq 'AutoScaling' -and Tag -ne 'NoAutoScaling'}
        
        # Scaling up loop

        While ((Get-Date -UFormat "%s") -gt $scaleUpTimeEpoch -and (Get-Date -UFormat "%s") -lt $scaleDownTimeEpoch)
        {                                
                [string]$adminAddress = GetActiveController
                Get-LogSite -AdminAddress $adminAddress | Out-Null

                # Log controller running script
                                
                $highLevelLogOp = Start-LogHighLevelOperation  -AdminAddress $adminAddress -Source "AutoScaling" -StartTime (Get-Date -Format G) -Text "EUC Consulting Auto Scaling: Script is running on $($env:COMPUTERNAME + "." + $env:USERDNSDOMAIN)"
                Stop-LogHighLevelOperation -EndTime (Get-Date -Format G) -HighLevelOperationId $highLevelLogOp.Id -IsSuccessful $True -AdminAddress $adminAddress
                $desktopGroups =  Get-BrokerDesktopGroup -AdminAddress $adminAddress -Filter {Tag -eq 'AutoScaling' -and Tag -ne 'NoAutoScaling'}

                # Log active controller

                $highLevelLogOp = Start-LogHighLevelOperation  -AdminAddress $adminAddress -Source "AutoScaling" -StartTime (Get-Date -Format G) -Text "EUC Consulting Auto Scaling: Script is using $($adminAddress) as the active controller"
                Stop-LogHighLevelOperation -EndTime (Get-Date -Format G) -HighLevelOperationId $highLevelLogOp.Id -IsSuccessful $True -AdminAddress $adminAddress
                $desktopGroups =  Get-BrokerDesktopGroup -AdminAddress $adminAddress -Filter {Tag -eq 'AutoScaling' -and Tag -ne 'NoAutoScaling'}

                foreach ($desktopGroup in $desktopGroups)
                {
                        $tag = Get-BrokerTag -Name 'AutoScaling' -DesktopGroupUid $desktopGroup.Uid -AdminAddress $adminAddress
                        If ($tag.MetadataMap[$scalingModeMetaData] -notcontains 'Up')
                        {
                                $highLevelLogOp = Start-LogHighLevelOperation  -AdminAddress $adminAddress -Source "AutoScaling" -StartTime (Get-Date -Format G) -Text "EUC Consulting Auto Scaling: Scaling up has commenced for the Desktop Group $($desktopGroup.Name)"
                                $tag | Set-BrokerTagMetadata -Name $scalingModeMetaData -Value 'Up'
                                Stop-LogHighLevelOperation  -AdminAddress $adminAddress -EndTime (Get-Date -Format G) -HighLevelOperationId $highLevelLogOp.Id -IsSuccessful $True
                        }
                }
                
                # If no servers running in Desktop Group, turn ($powerOnQuantity) number of servers on.

                ForEach ($desktopGroup in $desktopGroups)
                {
                        $brokerMachines = Get-BrokerMachine -DesktopGroupName $desktopGroup.Name -Filter {Tag -ne 'NoAutoScaling'} -AdminAddress $adminAddress
                        If ((($brokerMachines | Select-Object -ExpandProperty PowerState) -notcontains "On") -eq $true -and ($brokerMachines | Measure-Object).Count -ge $powerOnQuantity)
                                {
                                        If ($brokermachines.Count -gt 0)
                                        {
                                                $x = 0
                                                Do 
                                                {
                                                        $highLevelLogOp = Start-LogHighLevelOperation  -AdminAddress $adminAddress -Source "AutoScaling" -StartTime (Get-Date -Format G) -Text "EUC Consulting Auto Scaling: $($brokerMachines[$x].MachineName) turned on"
                                                        New-BrokerHostingPowerAction -Action TurnOn -MachineName $brokerMachines[$x].MachineName -AdminAddress $adminAddress | Out-Null
                                                        Stop-LogHighLevelOperation  -AdminAddress $adminAddress -EndTime (Get-Date -Format G) -HighLevelOperationId $highLevelLogOp.Id -IsSuccessful $True
                                                        $x ++
                                                } Until ($x -eq $powerOnQuantity)
                                        }
                                }
                }
                
                # Check average load accross servers in Desktop Group, if load above specified amount ($scaleUpLoad), turn on X number of servers ($powerOnQuantity)
                
                ForEach ($desktopGroup in $desktopGroups)
                {
                        $brokerMachines = Get-BrokerMachine -DesktopGroupName $desktopGroup.Name -Filter {Tag -ne 'NoAutoScaling'}
                        If ((($brokerMachines | Select-Object -ExpandProperty PowerState) -contains "On") -and (($brokerMachines | Select-Object -ExpandProperty PowerState) -contains "Off" | Measure-Object).Count -ge $powerOnQuantity)
                        {
                                $averageLoad = (($brokerMachines | Where-Object {$_.PowerState -eq 'On'}) | Measure-Object LoadIndex -Average).Average
                                $averageLoad = [math]::Round($averageLoad)
                                $highLevelLogOp = Start-LogHighLevelOperation  -AdminAddress $adminAddress -Source "AutoScaling" -StartTime (Get-Date -Format G) -Text "EUC Consulting Auto Scaling: Average load accross the Desktop Group $($desktopGroup.Name) is $($averageLoad)"
                                Stop-LogHighLevelOperation  -AdminAddress $adminAddress -EndTime (Get-Date -Format G) -HighLevelOperationId $highLevelLogOp.Id -IsSuccessful $True
                                                                
                                If ($averageLoad -ge $scaleUpLoad)
                                {
                                        $highLevelLogOp = Start-LogHighLevelOperation  -AdminAddress $adminAddress -Source "AutoScaling" -StartTime (Get-Date -Format G) -Text "EUC Consulting Auto Scaling: Average load accross the Desktop Group $($desktopGroup.Name) has exceeded the Scale Up Load of $($scaleUpLoad)"
                                        Stop-LogHighLevelOperation  -AdminAddress $adminAddress -EndTime (Get-Date -Format G) -HighLevelOperationId $highLevelLogOp.Id -IsSuccessful $True

                                        $stoppedMachines = $brokerMachines | Where-Object {$_.DesktopGroupName -eq $desktopGroup.Name -and $_.PowerState -eq "Off"}
                                        If ($stoppedMachines.Count -gt 0)
                                        {
                                                $x=0
                                                Do
                                                {
                                                        $highLevelLogOp = Start-LogHighLevelOperation  -AdminAddress $adminAddress -Source "AutoScaling" -StartTime (Get-Date -Format G) -Text "EUC Consulting Auto Scaling: $($stoppedMachines[$x].MachineName) turned on"
                                                        New-BrokerHostingPowerAction -Action TurnOn -MachineName $stoppedMachines[$x].MachineName -AdminAddress $adminAddress | Out-Null
                                                        Stop-LogHighLevelOperation  -AdminAddress $adminAddress -EndTime (Get-Date -Format G) -HighLevelOperationId $highLevelLogOp.Id -IsSuccessful $True
                                                        $x ++
                                                } Until ($x -eq $powerOnQuantity)
                                        }
                                }
                        }   
                }

                # If ScaleDown loop gets to last 380 seconds, clear tag metadata

                If ((Get-Date -UFormat "%s") -gt ($scaleDownTimeEpoch -440) -and (Get-Date -UFormat "%s") -lt $scaleDownTimeEpoch)
                {
                        foreach ($desktopGroup in $desktopGroups)
                        {        
                                $tag = Get-BrokerTag -Name 'AutoScaling' -DesktopGroupUid $desktopGroup.Uid -AdminAddress $adminAddress
                                $tag | Set-BrokerTagMetadata -Name $scalingModeMetaData -Value 'null' -AdminAddress $adminAddress
                                $tag | Set-BrokerTagMetadata -Name $scaleUpEpochMetaData -Value 'null' -AdminAddress $adminAddress
                        }

                        $highLevelLogOp = Start-LogHighLevelOperation  -AdminAddress $adminAddress -Source "AutoScaling" -StartTime (Get-Date -Format G) -Text "EUC Consulting Auto Scaling: Scaling up has finished, tags have been cleared"
                        Stop-LogHighLevelOperation  -AdminAddress $adminAddress -EndTime (Get-Date -Format G) -HighLevelOperationId $highLevelLogOp.Id -IsSuccessful $True
              
                }
                Start-Sleep -Seconds 420
                KillSwitch               
        }
      
        # If ScaleUpTime is less than ScaleDownTime,  ScaleDownTime is tomorrow. Convert all times to Epoch time

        If ($scaleUpTime -lt $scaleDownTime)
        {
                $scaleUpTimeEpoch = (Get-Date $scaleUpTime -UFormat "%s")
                $scaleUpTimeEpoch = [int]$scaleUpTimeEpoch + 86400
        }
        Else
        {
                $scaleUpTimeEpoch = (Get-Date $scaleUpTime -UFormat "%s")    
        }

        # Check if Scale Up metadata occurs in the past, clear if true
        
        foreach ($desktopGroup in $desktopGroups)
        {
                $tag = Get-BrokerTag -Name 'AutoScaling' -DesktopGroupUid $desktopGroup.Uid
                If ($tag.MetadataMap[$scaleUpEpochMetaData] -lt (Get-Date -UFormat "%s"))
                {
                        $tag | Set-BrokerTagMetadata -Name $scaleUpEpochMetaData -Value $scaleUpTimeEpoch -AdminAddress $adminAddress
                }
                Else
                {
                        $scaleUpTimeEpoch = $tag.MetadataMap[$scaleUpEpochMetaData]
                }
        }
        
        # Get Broker Machines, if servers are tagged with AutoScaling, assume script started after a crash. If servers are not Tagged, Tags are added
        
        $brokerMachines = Get-BrokerMachine -Filter {Tag -ne 'NoAutoScaling' -and Tag -eq 'AutoScaling'} -AdminAddress $adminAddress
        If (!($brokermachines))
        {
                $brokermachines = Get-BrokerMachine -AdminAddress $adminAddress -Filter {Tag -ne 'NoAutoScaling' -and PowerState -eq 'On' -and RegistrationState -eq 'Registered' -and InMaintenanceMode -eq $false}
                foreach ($machine in $brokerMachines)
                {
                        Add-BrokerTag -Name 'AutoScaling' -Desktop $machine.Uid -AdminAddress $adminAddress
                }
        }

                
        # Scaling down loop

        While ((Get-Date -UFormat "%s") -gt $scaleDownTimeEpoch -and (Get-Date -UFormat "%s") -lt ($scaleUpTimeEpoch -60))
        {
                [string]$adminAddress = GetActiveController
                Get-LogSite -AdminAddress $adminAddress | Out-Null

                # Log controller running script
                                
                $highLevelLogOp = Start-LogHighLevelOperation  -AdminAddress $adminAddress -Source "AutoScaling" -StartTime (Get-Date -Format G) -Text "EUC Consulting Auto Scaling: Script is running on $($env:COMPUTERNAME + "." + $env:USERDNSDOMAIN)"
                Stop-LogHighLevelOperation -EndTime (Get-Date -Format G) -HighLevelOperationId $highLevelLogOp.Id -IsSuccessful $True -AdminAddress $adminAddress
                $desktopGroups =  Get-BrokerDesktopGroup -AdminAddress $adminAddress -Filter {Tag -eq 'AutoScaling' -and Tag -ne 'NoAutoScaling'}

                # Log active controller

                $highLevelLogOp = Start-LogHighLevelOperation  -AdminAddress $adminAddress -Source "AutoScaling" -StartTime (Get-Date -Format G) -Text "EUC Consulting Auto Scaling: Script is using $($adminAddress) as the active controller"
                Stop-LogHighLevelOperation -EndTime (Get-Date -Format G) -HighLevelOperationId $highLevelLogOp.Id -IsSuccessful $True -AdminAddress $adminAddress
                $desktopGroups =  Get-BrokerDesktopGroup -AdminAddress $adminAddress -Filter {Tag -eq 'AutoScaling' -and Tag -ne 'NoAutoScaling'}

                # Set scaling mode Metadata

                foreach ($desktopGroup in $desktopGroups)
                {
                        $tag = Get-BrokerTag -Name 'AutoScaling' -DesktopGroupUid $desktopGroup.Uid -AdminAddress $adminAddress
                        If ($tag.MetadataMap[$scalingModeMetaData] -notcontains 'Down')
                        {
                                $highLevelLogOp = Start-LogHighLevelOperation  -AdminAddress $adminAddress -Source "AutoScaling" -StartTime (Get-Date -Format G) -Text "EUC Consulting Auto Scaling: Scaling down has commenced for the Desktop Group $($desktopGroup.Name)"
                                $tag | Set-BrokerTagMetadata -Name $scalingModeMetaData -Value 'Down'
                                Stop-LogHighLevelOperation  -AdminAddress $adminAddress -EndTime (Get-Date -Format G) -HighLevelOperationId $highLevelLogOp.Id -IsSuccessful $True
                        }
                }

                # Put servers into Maintenance Mode, percentage of servers determined by $keepOnline are left out of Maintenance Mode, servers with least load are kept online

                ForEach ($desktopGroup in $desktopGroups)
                {
                        $tag = Get-BrokerTag -Name 'AutoScaling' -DesktopGroupUid $desktopGroup.Uid -AdminAddress $adminAddress
                        $tagMetaData = ($DesktopGroup.Name + $maintenanceModeMetaData)                       
                        If ($tag.MetaDataMap[$tagMetaData] -ne 'Set')
                        {
                                $xBrokerMachines = $brokerMachines | Where-Object {$_.DesktopGroupName -eq $desktopGroup.Name} | Sort-Object LoadIndex
                                If ($xBrokerMachines.count -ge 2)
                                {
                                        $upperBound = $xBrokerMachines.GetUpperBound(0)
                                        $x = [System.Math]::Round(($upperBound + 1) * ($keepOnline/100))
                                        If ($x -lt 1)
                                        {
                                                $x = 1
                                        }
                                        Do 
                                        {                                                
                                                $highLevelLogOp = Start-LogHighLevelOperation  -AdminAddress $adminAddress -Source "AutoScaling" -StartTime (Get-Date -Format G) -Text "EUC Consulting Auto Scaling: $($xBrokerMachines[$x].MachineName) was put into Maintenance Mode"
                                                Set-BrokerMachine -MachineName $xBrokerMachines[$x].MachineName -InMaintenanceMode $true -AdminAddress $adminAddress
                                                Stop-LogHighLevelOperation  -AdminAddress $adminAddress -EndTime (Get-Date -Format G) -HighLevelOperationId $highLevelLogOp.Id -IsSuccessful $True
                                                $x++
                                        } Until ($x -gt $upperBound)
                                }
                                $tag |  Set-BrokerTagMetadata -Name ($desktopGroup.Name + $maintenanceModeMetaData) -Value 'Set' -AdminAddress $adminAddress
                        }
                }

                # Turn off servers when there is no one left logged in

                $brokerMachines = Get-BrokerMachine -Filter {Tag -ne 'NoAutoScaling' -and Tag -eq 'AutoScaling'} -AdminAddress $adminAddress

                foreach ($desktopGroup in $desktopGroups)
                {
                        foreach ($machine in $brokerMachines)
                        {
                                If ($machine.DesktopGroupName -eq $desktopGroup.Name -and $machine.PowerState -eq "On" -and $machine.InMaintenanceMode -eq $true -and $machine.SessionCount -eq 0)
                                {
                                        $highLevelLogOp = Start-LogHighLevelOperation  -AdminAddress $adminAddress -Source "AutoScaling" -StartTime (Get-Date -Format G) -Text "EUC Consulting Auto Scaling: $($machine.MachineName) had 0 sessions and was turned off"
                                        New-BrokerHostingPowerAction -Action TurnOff -MachineName $machine.MachineName -AdminAddress $adminAddress | Out-Null
                                        Set-BrokerMachine -MachineName $machine.MachineName -InMaintenanceMode $false -AdminAddress $adminAddress
                                        Stop-LogHighLevelOperation  -AdminAddress $adminAddress -EndTime (Get-Date -Format G) -HighLevelOperationId $highLevelLogOp.Id -IsSuccessful $True
                                }
                        }
                }
                
                # Check average load of servers which are left powered on, enable logons if load exceeds reverseScaleLoad. If there are no servers to re-enable logons, servers are booted up.

                foreach ($desktopGroup in $desktopGroups)
                {
                        $brokerMachines = Get-BrokerMachine -Filter {Tag -ne 'NoAutoScaling' -and Tag -eq 'AutoScaling'} -AdminAddress $adminAddress | Sort-Object LoadIndex
                        $averageLoad = (($brokerMachines | Where-Object {$_.DesktopGroupName -eq $desktopGroup.Name -and $_.PowerState -eq 'On' -and $_.InMaintenanceMode -eq $false}) | Measure-Object LoadIndex -Average).Average 
                        $averageLoad = [math]::Round($averageLoad)
                        If ($averageLoad -ge $reverseScaleLoad)
                        {
                                $highLevelLogOp = Start-LogHighLevelOperation  -AdminAddress $adminAddress -Source "AutoScaling" -StartTime (Get-Date -Format G) -Text "EUC Consulting Auto Scaling: Average load accross the Desktop Group $($desktopGroup.Name) has exceeded $($reverseScaleLoad)"
                                Stop-LogHighLevelOperation  -AdminAddress $adminAddress -EndTime (Get-Date -Format G) -HighLevelOperationId $highLevelLogOp.Id -IsSuccessful $True
                                $runningServers = ($brokerMachines | Where-Object {$_.DesktopGroupName -eq $desktopGroup.Name -and $_.PowerState -eq "On"}).Count
                                $enabledServers = ($brokerMachines | Where-Object {$_.DesktopGroupName -eq $desktopGroup.Name -and $_.InMaintenanceMode -eq $false}).Count
                                If ($runningServers -gt $enabledServers)
                                {
                                                ForEach ($machine in $brokerMachines)
                                                {
                                                        If ($averageLoad -ge $reverseScaleLoad)
                                                        {
                                                                If($machine.DesktopGroupName -eq $desktopGroup.Name -and $machine.InMaintenanceMode -eq $true)
                                                                {
                                                                        $highLevelLogOp = Start-LogHighLevelOperation  -AdminAddress $adminAddress -Source "AutoScaling" -StartTime (Get-Date -Format G) -Text "EUC Consulting Auto Scaling: $($Machine.MachineName) was taken out of Maintenance Mode during Scale Down as the other servers were overloaded."
                                                                        Set-BrokerMachine -MachineName $machine.MachineName -InMaintenanceMode $false -AdminAddress $adminAddress
                                                                        Stop-LogHighLevelOperation  -AdminAddress $adminAddress -EndTime (Get-Date -Format G) -HighLevelOperationId $highLevelLogOp.Id -IsSuccessful $True
                                                                        Start-Sleep -Seconds 60
                                                                        $xbrokerMachines = Get-BrokerMachine -Filter {Tag -ne 'NoAutoScaling' -and Tag -eq 'AutoScaling' -and InMaintenanceMode -eq $false -and PowerState -eq 'On'} -AdminAddress $adminAddress
                                                                        $averageLoad = ($xBrokerMachines | Measure-Object LoadIndex -Average).Average
                                                                        $averageLoad = [math]::Round($averageLoad)
                                                                }
                                                        }
                                                 }
                                        
                                }
                                Else
                                {                
                                        $x=0
                                        Do 
                                        {
                                                $xBrokerMachines = $brokerMachines | Where-Object {$_.DesktopGroupName -eq $desktopGroup.Name -and $_.PowerState -eq "Off"}
                                                $highLevelLogOp = Start-LogHighLevelOperation  -AdminAddress $adminAddress -Source "AutoScaling" -StartTime (Get-Date -Format G) -Text "EUC Consulting Auto Scaling: $($xBrokerMachines[0].MachineName) was turned on during Scale Down as the other servers were overloaded."
                                                New-BrokerHostingPowerAction -Action TurnOn -MachineName $xBrokerMachines[0].MachineName -AdminAddress $adminAddress | Out-Null
                                                Set-BrokerMachine -MachineName $xBrokerMachines[0].MachineName -InMaintenanceMode $false -AdminAddress $adminAddress
                                                Stop-LogHighLevelOperation  -AdminAddress $adminAddress -EndTime (Get-Date -Format G) -HighLevelOperationId $highLevelLogOp.Id -IsSuccessful $True
                                                $x ++
                                        } Until ($x -eq $powerOnQuantity)
                                }
                        }
                }
                Start-Sleep -Seconds 180
                KillSwitch
        }
        
        # If ScaleUp loop gets to last 200 seconds, clear tag metadata

        If ((Get-Date -UFormat "%s") -gt ($scaleUpTimeEpoch -200) -and (Get-Date -UFormat "%s") -lt $scaleUpTimeEpoch)
        {
                foreach ($desktopGroup in $desktopGroups)
                {        
                        $tag = Get-BrokerTag -Name 'AutoScaling' -DesktopGroupUid $desktopGroup.Uid -AdminAddress $adminAddress
                        $tag | Set-BrokerTagMetadata -Name $scalingModeMetaData -Value 'null' -AdminAddress $adminAddress
                        $tag | Set-BrokerTagMetadata -Name $maintenanceModeMetaData -Value 'null' -AdminAddress $adminAddress
                        $tag | Set-BrokerTagMetadata -Name $scaleUpEpochMetaData -Value 'null' -AdminAddress $adminAddress
                        $tag | Set-BrokerTagMetadata -Name $scaleDownEpochMetaData -Value 'null' -AdminAddress $adminAddress
                }
                foreach ($machine in $brokerMachines)
                {
                        Set-BrokerMachine -MachineName $machine.MachineName -InMaintenanceMode $false
                        Remove-BrokerTag -Name 'AutoScaling' -Desktop $machine.Uid -AdminAddress $adminAddress
                }
                $highLevelLogOp = Start-LogHighLevelOperation  -AdminAddress $adminAddress -Source "AutoScaling" -StartTime (Get-Date -Format G) -Text "EUC Consulting Auto Scaling: Scaling down has finished, tags have been cleared,machines taken out of Maintenance Mode"
                Stop-LogHighLevelOperation  -AdminAddress $adminAddress -EndTime (Get-Date -Format G) -HighLevelOperationId $highLevelLogOp.Id -IsSuccessful $True
        }       
}

There may be times where you need to clear the tags and tag metadata during testing or if you manually stop the script one day and restart it the next day. For this reason I’ve written this other small script which will clear the tags for you.

Add-PSSnapin Citrix.* -ErrorAction SilentlyContinue
$adminAddress   = 'localhost'

$scalingModeMetaData     = 'ScalingMode'
$scaleUpEpochMetaData    = 'ScaleUpTimeEpoch'
$scaleDownEpochMetaData  = 'ScaleDownTimeEpoch'
$startUpTimeMetaData     = 'StartUpTime'
$shutdownTimeMetaData    = 'ShutdownTime'
$noAutoScalingMetaData   = 'NoAutoScaling'
$maintenanceModeMetaData = 'MaintenanceMode'

$desktopGroups = Get-BrokerDesktopGroup
$brokermachines = Get-BrokerMachine

foreach ($desktopGroup in $desktopGroups)
{
        $tag = Get-BrokerTag -Name 'AutoScaling' -DesktopGroupUid $desktopGroup.Uid -AdminAddress $adminAddress
        $tag | Set-BrokerTagMetadata -Name $scalingModeMetaData -Value 'null' -AdminAddress $adminAddress
        $tag | Set-BrokerTagMetadata -Name ($desktopGroup.Name + $maintenanceModeMetaData) -Value 'null' -AdminAddress $adminAddress
        $tag | Set-BrokerTagMetadata -Name $scaleUpEpochMetaData -Value 'null' -AdminAddress $adminAddress
        $tag | Set-BrokerTagMetadata -Name $scaleDownEpochMetaData -Value 'null' -AdminAddress $adminAddress
}

foreach ($machine in $brokermachines)
{
        Remove-BrokerTag -Name 'AutoScaling' -Desktop $machine.Uid
}

Comments

Krunal Patel

Hi, I have tried your script but seems its not working. May be I have not correctly used the parameters. Can you please guide me

I'd love to reach out to the author of this script. I've written something every similar already and it would be great to touch base and discuss a possible collaboration.

Shaun Ritchie

Thanks but I am not maintaining this script any more.

Can you publish the script on Github so people can contribute with enhancements, open issues,... ?

Join the discussion