After installing Windows Update Services and trying to approve the updates you probably get the above error saying an error occurred trying to connect the WSUS server. Click Reset Server Node to try to connect to the server again.
You can try one of the following options to solve the problem.
Option 1: Disable the firewall on the WSUS Server.
Option 2: Increase the amount of memory the WSUS pool uses, which was 1.75 GB by default. Increasing this value to 15 GB proved that WSUS works with no problem and the client devices start to report back and get updates. Make sure that you should have more than 15 GB RAM to perform this step if not, try to increase the memory size depending on the availability of your server memory.
Steps to increase the amount of RAM
Open the IIS on the WSUS server and browse to Application Pools, Wsus Pool and right click & select Advanced Settings.
On the new displayed window select Private Memory Limit (KB) and enter your new Memory Value, please note that the value to enter is in KB.