tl;dr – click here to jump to the bottom and get the script.

Quick History on Deduplication

The current version of Deduplication has been around since at least Server 2012/2012R2. Starting in Server 2016, Storage Spaces Direct (S2D) has had the ability to use dedup on CSVFS_NTFS (and CSVFS_ReFS in Server 2019). This same version of Deduplication is shipped with Azure Stack HCI (thru 22H2) as well.

So when you turn on Dedup, the following jobs are created:

However, the built-in job schedule is… how do I say it… “less than optimal” for most workloads. Digging into those default options (link to Docs here):

So let’s take a look at the BackgroundOptimization schedule. It will run every hour with the following settings:

And while it is configured for throttling and stopping when the system is busy, it can still pound your IO subsystem when you may need those IOPS for more important stuff.

Now everyone’s usage may vary – I tend to use Deduplication for AVD / VDI workloads because it is one of the most obvious choices for awesome efficiency. However, even Microsoft recommends that the default options should be tweaked for Hyperconverged (S2D) scenarios (documentation here):

Since my workloads are most active Monday – Friday, 9AM-5PM (EST), I will tweak my jobs to run a single time during the workday and another time off-hours. I will also configure my GC and Scrubbing jobs to go hard on weekends to really clean things up. Let’s dig into it…

Custom Deduplication Jobs

By running the script below, your Deduplication schedule should now look like this. Note that the default jobs are Enabled = False and the custom jobs have Days and StartTime specified:

Here’s the script to do that. Make sure to tweak or remove regions that you do not feel fit your requirements!

$clusterName   = "CLUSTERNAME.contoso.com"

#########################

# This only needs to be run on a single node of the cluster, so just grab the first server
$clusterNode = (Get-Cluster -Name $clusterName | Get-ClusterNode)[0]

Write-Host "Running script on node $($clusterNode.Name)" -ForegroundColor Cyan

Invoke-Command -ComputerName $clusterNode.Name -ScriptBlock { 

    # Disable the default Weekly GC and Scrub schedules
    Write-Host "Disabling default deduplication schedules" -ForegroundColor Green
    Get-DedupSchedule -Name WeeklyGarbageCollection | Set-DedupSchedule -Enabled $false
    Get-DedupSchedule -Name WeeklyScrubbing | Set-DedupSchedule -Enabled $false
    Get-DedupSchedule -Name BackgroundOptimization | Set-DedupSchedule -Enabled $false
    Get-DedupSchedule -Name PriorityOptimization | Set-DedupSchedule -Enabled $false
    
    ##############################
    # Create replacement schedules
    ##############################

    #region OPT-DAILY-MIDDAY

    # Optimization - Daily MidDay
    #######################################
    # Starts:        Daily 1100 EDT
    # Duration:      4 hours
    # Memory:        75%
    # Priority:      High

    $scheduleName = "CUSTOM-Daily-MidDay-Optimization"

    If (Get-DedupSchedule -Name $scheduleName -ErrorAction Ignore) {
        Write-Warning "$scheduleName - Job already exists"
    }
    Else {
        # Create the job
        New-DedupSchedule -Name $scheduleName -Type Optimization -Days Monday,Tuesday,Wednesday,Thursday,Friday -Start 11:00 -DurationHours 4 -Memory 75 -Priority High
    }

    #endregion OPT-DAILY-MIDDAY

    #region OPT-DAILY-NIGHT

    # Optimization - Daily NIGHT
    #######################################
    # Starts:        Daily 1700 EDT
    # Duration:      4 hours
    # Memory:        75%
    # Priority:      High

    $scheduleName = "CUSTOM-Daily-Night-Optimization"

    If (Get-DedupSchedule -Name $scheduleName -ErrorAction Ignore) {
        Write-Warning "$scheduleName - Job already exists"
    }
    Else {
        # Create the job
        New-DedupSchedule -Name $scheduleName -Type Optimization -Days Monday,Tuesday,Wednesday,Thursday,Friday -Start 17:00 -DurationHours 4 -Memory 75 -Priority High
    }

    #endregion OPT-DAILY-NIGHT

    #region GC-MIDWEEKSHORT

    # Garbage Collection - Midweek Short
    #######################################
    # Starts:        Wednesdays 2200 EDT
    # Duration:      4 hours
    # Full Job:      False
    # Memory:        50%
    # Cores:         75%
    # IO Throttle:   Medium
    # Priority:      Normal
    # Stop Sys Busy: True

    $scheduleName = "CUSTOM-Midweek-GarbageCollection"

    If (Get-DedupSchedule -Name $scheduleName -ErrorAction Ignore) {
        Write-Warning "$scheduleName - Job already exists"
    }
    Else {
        # Create the job
        New-DedupSchedule -Name $scheduleName -Type GarbageCollection -Days Wednesday -Start 22:00 -DurationHours 4 -Memory 50 -Cores 75 -InputOutputThrottleLevel Medium -Priority Normal
        Set-DedupSchedule -Name $scheduleName -Full $false -StopWhenSystemBusy $true
    }

    #endregion GC-MIDWEEKSHORT

    #region GC-ENDWEEKFULL

    # Garbage Collection - End of Week Full
    #######################################
    # Starts:        Friday 2200 EDT
    # Duration:      32 hours
    # Full Job:      True
    # Memory:        50%
    # Cores:         100%
    # IO Throttle:   None
    # Priority:      Normal
    # Stop Sys Busy: True

    $scheduleName = "CUSTOM-EndWeek-GarbageCollection"

    If (Get-DedupSchedule -Name $scheduleName -ErrorAction Ignore) {
        Write-Warning "$scheduleName - Job already exists"
    }
    Else {
        # Create the job
        New-DedupSchedule -Name $scheduleName -Type GarbageCollection -Days Friday -Start 22:00 -DurationHours 32 -Memory 50 -Cores 100 -InputOutputThrottleLevel None -Priority Normal
        Set-DedupSchedule -Name $scheduleName -Full $true -StopWhenSystemBusy $true
    }

    #endregion GC-ENDWEEKFULL

    #region SCRUB-ENDWEEKSHORT

    # Scrubbing - End of Week Short
    #######################################
    # Starts:        Sunday 0100 EDT
    # Duration:      18 hours
    # Full Job:      False
    # Memory:        50%
    # Cores:         100%
    # IO Throttle:   None
    # Priority:      Normal
    # Stop Sys Busy: True

    $scheduleName = "CUSTOM-EndWeek-Scrubbing"

    If (Get-DedupSchedule -Name $scheduleName -ErrorAction Ignore) {
        Write-Warning "$scheduleName - Job already exists"
    }
    Else {
        # Create the job
        New-DedupSchedule -Name $scheduleName -Type Scrubbing -Days Sunday -Start 01:00 -DurationHours 18 -Memory 50 -Cores 100 -InputOutputThrottleLevel None -Priority Normal
        Set-DedupSchedule -Name $scheduleName -Full $false -StopWhenSystemBusy $true
    }

    #endregion SCRUB-ENDWEEKSHORT
}

Leave a Reply

Your email address will not be published. Required fields are marked *