Automated backup for Cisco UCS instances

Prerequisites

1.    For each UCS cluster instance logon and create dedicated user with admin rights for backup purposes. 

2.    Enable HTTPS port on UCS cluster instance

 3.    Password for user “backup” out of Cisco UCS should be kept in secure string. It is already created and saved at file D:\UCS_SCRIPT\CISCO_UCS_BCKP_PWD.txt as secure string.

Note:     If there is a need for new password for new Cisco UCS instance different from currently used you have to create new text file for storing new secure string, otherwise you will overwrite D:\UCS_SCRIPT\CISCO_UCS_BCKP_PWD.txt and logon will fail for all previously setup instances.

4.    Create new secure string from plain password.
   i.    Logon with local admin user to Windows box. Keep in mind this user should start scheduled job later so you have to setup it with same Windows credentials.
   ii.    Open Cisco UCS PowerTool or MS PowerShell.

  iii.    Copy following command

Read-Host -AsSecureString | ConvertFrom-SecureString | Out-File D:\UCS_SCRIPT\CISCO_UCS_BCKP_PWD.txt

  paste at OpenTool command line, change name of text file CISCO_UCS_BCKP_PWD_NN.txt, where NN is next available number and click Enter.
  iv.    Command will wait for input of plain password for backup user. Click Enter when finished.

5.    Prepare folders for backup files.
  i.    Go to L:\UCS_BACKUPS (change to your backup destination) and navigate to appropriate datacentre or create new one if it does not exist.
  ii.    Go to appropriate UCS instance within chosen datacenter or create new one if it does not exist.

6.    Copy existing script to new one
  i.   Example: Cisco_UCS_[datacenter]_[UCSinstance]_Backup_Script.ps1”, where [datacenter] is your site and [UCSinstance] is name of UCS instance.


UCS instance backup script 

1.    Backup script should have independence from others and batch file ran from scheduler. This involves each particular script to Load Module: Cisco UCS PowerTool on top of MS PowerShell.

2.    Static parameters

Set-Variable -Name ScriptDir   -Value "D:\UCS_SCRIPT" -Scope Local
Set-Variable -Name BackupPath  -Value "L:\UCS_BACKUPS\[datacenter]\[UCSinstance]" -Scope Local
Set-Variable -Name BackupTypes -Value ('config-system','config-logical','config-all','full-state')
Write-Host "`nStatic parameters`n-----------------`nScriptDir = $($ScriptDir)`nBackupPath = $($BackupPath)`nBackupTypes = $($BackupTypes)`n"
$UCS = "[FQDN of your UCS instance]"
$errmailTo = "you@yourcompany.com"
$errmailFrom = "UCS_[name of currnet UCS instance]_backup@yourcompany.com"
$errmailSMTP = "smtp.yourcompany.com"


  i.    BackupPath should correspond to your newly created backup folders – previous chapter, article 5.
  ii.    BackupTypes describe all possible backups for UCS.
  iii.    $UCS should correspond to name of your new UCS instance.
  iv.    $errmailTo should correspond to your addressee of error messages.
   v.    $errmailFrom should be descriptive which UCS instance sent messages and why.

3.    Start PowerShell transcript and point output to $BackupPath\backuplog.txt. This file is unique for each Cisco UCS instance and keeps logs for last backup job until is overwritten from next start of backup script. Its purpose is only for troubleshooting.

4.    Read credentials, logon to Cisco UCS instance and define error on connecting
 
# Get the credentials setup
$password = Get-Content D:\UCS_SCRIPT\CISCO_UCS_BCKP_PWD.txt | ConvertTo-SecureString
$cred = New-Object System.Management.Automation.PSCredential “backup”,$password
# Login
$login = Connect-Ucs [IP aaa.bbb.ccc.ddd or $UCS] -Credential $cred -ErrorVariable errConnectingUcs | select Ucs,UserName,Version

  i.    Pay attention you get secure string from correct text file.
  ii.    Modify IP address aaa.bbb.ccc.ddd to correspond to your new UCS instance.
  iii.    If connection fails an error message will be send to predefined addressee.

5.    Get one of each type of UCS backup and send an e-mail if backup job is failed.

6.    Disconnect Cisco UCS

7.    Retention policy for backup files - delete all files within backup folder older than 30 days from now, but keep at least last 3 sets with all types of backups (3 daily backups) even they are older than 30 days. This will ensure you will have all type of backups for last 30 days in normal operation, but in case of backup failure you could rely on latest 3 backups.


P.S. If there is a need to take exclusive backup for UCS instance start appropriate script at D:\UCS_SCRIPT\ with extension .ps1 within MS PowerShell or Cisco OpenTool console and keep in mind to evacuate backup files from backup folder if you have intention to keep them more than 30 days. Otherwise create one .bat file containing as many rows you need as following one:

 %WINDIR%\System32\windowspowershell\v1.0\powershell.exe "D:\UCS_SCRIPT\Cisco_UCS_[datacenter]_[UCSinstance]_Backup_Script.ps1”, where [datacenter] is your site and [UCSinstance] is name of UCS instance.

And backup script:

#------------------#
# Preparation work #
#------------------#

#--- Import Cisco PowerTool Modules ---#
if ((Get-Module |where {$_.Name -ilike "CiscoUcsPS"}).Name -ine "CiscoUcsPS")
    {
    Write-Host "Loading Module: Cisco UCS PowerTool Module" -ForegroundColor "Green"
    Import-Module CiscoUcsPs
    }

#--- Static parameters ---#
Set-Variable -Name ScriptDir   -Value "D:\UCS_SCRIPT" -Scope Local
Set-Variable -Name BackupPath  -Value "L:\UCS_BACKUPS\[datacenter]\[UCSinstance]" -Scope Local
Set-Variable -Name BackupTypes -Value ('config-system','config-logical','config-all','full-state')
Write-Host "`nStatic parameters`n-----------------`nScriptDir = $($ScriptDir)`nBackupPath = $($BackupPath)`nBackupTypes = $($BackupTypes)`n"
$Ucs = "[FQDN of your UCS instance]"
$errmailTo = "you@yourcompany.com"
$errmailFrom = "UCS_[name of currnet UCS instance]_backup@yourcompany.com"
$errmailSMTP = "smtp.yourcompany.com"

#----------------------------------#
# Start doing the real backup work #
#----------------------------------#

#--- Start collecting logs for last backup job ---#
start-transcript $BackupPath\backuplog.txt

#--- Get the credentials setup ---#
$password = Get-Content D:\UCS_SCRIPT\CISCO_UCS_BCKP_PWD.txt | ConvertTo-SecureString
$cred = New-Object System.Management.Automation.PSCredential “backup”,$password

#--- Login ---#
$login = Connect-Ucs [IP aaa.bbb.ccc.ddd or $UCS] -Credential $cred -ErrorVariable errConnectingUcs | select Ucs,UserName,Version

#--- Send an e-mail if error occured while connecting to UCS ---#
if ($errConnectingUcs -ne $null) {
        $errmailBody = @()
    $errmailBody += "$(Get-Date) - Error connecting to UCS`n`n"
    $errmailBody += $errConnectingUcs
    Send-MailMessage  -From $errmailFrom -To $errmailTo -Subject "Backup-mUcs Error-- Failed connecting to $($Ucs)" -Body ($errmailBody | Out-String) -SmtpServer $errmailSMTP
        break
}

#--- Get one of each type of UCS backup and send an e-mail if backup job is failed ---#
ForEach ($type in $backuptypes) {
        Backup-Ucs -Type $type  -PreservePooledValues -PathPattern ($BackupPath + '\${ucs}-${yyyy}${MM}${dd}-${HH}${mm}-' + $type + '.xml') -ErrorVariable errBackupUcs
        if ($errBackupUcs -ne $null) {
                $errmailBody = @()
                $errmailBody += "$(Get-Date) - Error running backup of type $($type)`n`n"
                $errmailBody += $error[0]
                Send-MailMessage  -From $errmailFrom -To $errmailTo -Subject "Backup-mUcs Error-- On $($Ucs) backup of type $($type) failed" -Body ($errmailBody | Out-String) -

SmtpServer $errmailSMTP
        }
    else
    {
    Write-Host "`nCisco UCS instance $Ucs created successfully $type type of backup and saved it at $BackupPath `n" -ForegroundColor "Green"
    }
}

#--- Don`t leave a stale session on the UCS ---#
Disconnect-Ucs

# ----------------------------------------------#
# Define parameters for Backup Retention policy #
# ----------------------------------------------#

#--- Define today ---#
$Now = Get-Date

#--- Define retention amount of days ---#
$Days = "30"

#--- Last 3 backups (4 per day) should be kept even they are older than $Days + backuplog.txt file ---#
#--- Example: Keep last 3 backups i.e. 12 files / 4 files per day = 3 days ---#
$OldBackupsFilestobeKept = "13"
$OldBackupsDays = ($OldBackupsFilestobeKept - 1)/4

#--- Define extension of backup files to be deleted ---#
$Extension = "*.xml"

#--- Define LastWriteTime parameter based on $Days ---#
$LastWrite = $Now.AddDays(-$Days)

#--- Get files based on lastwrite filter and specified folder ---#
$Files = Get-Childitem $BackupPath $Extension | Where {$_.LastWriteTime -le "$LastWrite"}

#--- Determine how many files are currently in folder ---# 
#--- Please keep in mind UCS has 4 separate xml backup files per day ---#
$NumberOfFiles = Get-ChildItem $BackupPath $Extension

Write-Host "`nExecuting retention policy for backup files - delete all files older than $Days days, but keep last 3 daily backups `n" -BackgroundColor "Gray" -ForegroundColor 

"Black"

if ($NumberOfFiles.Count -le $OldBackupsFilestobeKept)
    {
    write-host "`nCan not delete last $OldBackupsDays backups even they are older than $Days days, as they are only 3 last full sets of backups`n" -ForegroundColor "DarkRed"
    }
    else
        {
        foreach ($File in $Files)
            {
            if ($File -ne $Null)
                {
                write-host "'nDeleting File $File 'n" -ForegroundColor "DarkRed"
                Remove-Item $File.FullName | out-null
                }
            else
                {
                Write-Host "`nNo more files to delete! `n" -foregroundcolor "Green"
                }
            }
        }

stop-transcript