The following steps could speed up the RDP connection:
The policy that helps to speedup the remote connections (this settings is on client):
Computer Configuration\Windows Settings\Security Settings\Public Key Policies\Certificate Path Validation Settings
on Network Retrieval check the "Define these policy settings" and uncheck both settings below.
Now the RDP Connections are speeded up!!