Remote desktop connection to a server takes very long time to connect

 The following steps could speed up the RDP connection:

1. Open Network Connections in the configuration panel.
2. Right-click the icon of the network interface which is responsible for the connection to your server and select Properties.
3. In the Networking tab, click on the Configure… button.
4. In the next window, switch to the Advanced tab.
5. Click the Large Send Offload Version 2 (IPv4) and change the value to Disabled.
6. Click Start > click Run > type regedit and then click OK.
7. Locate and then click the following registry subkey:
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters
8. In the right pane, make sure that the DisableTaskOffload registry entry exists. If this entry does not exist, add the entry:
DWORD Value: DisableTaskOffload
Value Data: 1
9. Exit Registry Editor.
The following commands need to be entered through the command shell.
Which you can enter by doing: start > run > enter cmd.
netsh interface tcp set global autotuninglevel=disabled
and
netsh interface tcp set global autotuninglevel=highlyrestricted

==================================================================

The policy that helps to speedup the remote connections (this settings is on client):

Computer Configuration\Windows Settings\Security Settings\Public Key Policies\Certificate Path Validation Settings

on Network Retrieval check the "Define these policy settings" and uncheck both  settings below.

Now the RDP Connections are speeded up!!