Remote Desktop Gateway Optimizations:
RDG requirement is to use IIS, it also uses for Application Pools, and are inter-related.
As a result of inter-related application pools, the IIS App Pool Recycling is turned on. To keep RDG in memory, disable App Pool Recycling.
Optional Network Optimizations:
If using VMWare, use VMXNET3 (Do not use E1000 and E1000E)
Disable IPv6, QoS, and anything that has to do with packet shaping.
If VMQ is enabled on the VMW Host/Guest, disable it.
Increase the Maximum Input and Output threads to allow for multiple TCP threads inbound.
HKLM\Software\Microsoft\Terminal Server Gateway\Maxiothreads (REG_DWORD) = 5
Increase those IIS Pool Threads:
HKLM\System\CurrentControlSet\Services\InetInfo\Parameters\MaxPoolThreads (REG_DWORD) = 4
HKLM\Software\Microsoft\Rpc\ServerReceiveWindow (REG_DWORD) = 64kb
Adjust as need with a maximum of 1GB.
Disable TCP Offloading on the NIC: https://support.rackspace.com/how-to/disabling-tcp-offloading-in-windows-server-2012/