While performing some maintenance and updates on an ancient Windows 2008 Server VM, I upgraded it to Service Pack 2, in an attempt to resolve various ongoing issues.

Shortly afterwards, I discovered Windows Update was no longer working; it gave an obscure failure code and all the usual efforts to revive it, such as flushing the software distribution cache, were unsuccessful. No matter — Windows 2008 Server is out of support anyway, so there are unlikely to be any important updates, right?

Some time later, I discovered that the recently upgraded VPOP3 mail server, running on the same machine, was complaining that it couldn’t access its activation server to verify its license. This wasn’t a critical failure, but it resulted in reduced functionality.

Checking the mail server logs revealed this error message:

“Server certificate verification failed. Connection aborted" Windows code 0x80090302

Hmm. I used Wireshark to monitor the network traffic when VPOP3 started up, and could see it issuing a HTTPS request to the remote activation server. However, the session ended immediately after the initial handshake — clearly something was amiss. I verified that Chrome running on the same machine could access the Activation website successfully; the issue appeared to be with the Windows 2008 SSL libraries.

A check of the Windows Application event log revealed lots of repeated CAPI2 error events every 10 minutes:

Failed extract of third-party root list from auto update cab at: http://www.download.windowsupdate.com/msdownload/update/v3/static/trustedr/en/authrootstl.cab with error: A certificate chain processed, but terminated in a root certificate which is not trusted by the trust provider.

Next step was to visit the Applications and Services Logs in Event Viewer, then navigate to Microsoft / Windows / CAPI2 and right-click on the Operational Log to Enable CAPI2 logging. After doing so, a lot more detail about CAPI2 activity started to appear. (It’s a good idea to also increase the CAPI2 log size to 4096 Kb in Properties, as the default 1024 Kb can fill up very quickly.)

Repeating the activation test with this additional debugging enabled revealed a new error:

Very odd — apparently the Microsoft Root Certificate Authority 2010 was now considered untrusted. Just in case it had become corrupted, I used certmgr.msc to export a copy of this from another, newer machine and import it to the Windows 2008 Server instance, but this made no difference.

After some more research, I discovered John Thaller’s useful Root Certificate Updates For Legacy Windows github repository. Surely this would sort things out? Unfortunately, while it installed without issue, the problem remained.

Eventually, after a lot of further searching, I came across a Microsoft article from 2019 which described a plan to move Windows Update Services from SHA-1 signatures (now considered insecure) to SHA-2. A key part of this plan was that after August 2020, the Windows root certificates would no longer validate SHA-1 signatures, and all Windows Update Services would require SHA-2.

However, Windows 2008 Server SP2 doesn’t actually support SHA-2 verification. No problem – Microsoft were pushing an update that would add the needed SHA-2 support, so once that update was installed ahead of the transition deadline, everything would be good.

Christmas 2023 turned out to be NOT ahead of the transition deadline (missed it by more than three years, in fact) which explains why the server couldn’t access Windows Updates any more. As VPOP3’s Activation server appears to rely on a Microsoft certificate chain for authentication, it too fell victim to the lack of SHA-2 support.

Fortunately, there is a straightforward solution, as Microsoft describe in a follow-up article on How to Update Windows Devices to SHA-2. Microsoft made available a standalone SHA-2 update download page that has the correct installation file for a variety of different scenarios. Pick the one that matches your system, download and install, reboot, and you’re all set.

And indeed, after doing just that, everything started working normally once again!

All’s well that ends well, then. However, I do feel there was a missed opportunity here for the CAP2 logging to be a bit more explicit about the fact that it was the lack of SHA-2 signing support in the OS, or at least the lack of a verifiable signature in the provided certificate, that was the root cause of the failure – that would have saved a lot of time!

My previous attempt to stop the Windows 10 default route vanishing turned out to be unreliable; after a few Windows Updates, the default route was gone again.

However, I’ve finally found a way to make a statically configured default route properly persistent across reboots (something that other operating systems, and older versions of Windows, have no trouble doing at all).

To recap: for various reasons, I assign static IP addresses to most of my PCs instead of using DHCP to allocate them automatically. As part of the configuration I specify a default gateway to the Internet, like this:

Windows 10 appears unable to remember this default gateway across reboots, especially when I have additional IP subnets configured on the same network interface.

This isn’t a big deal if I’m sitting in front of the PC, since I can fix it quickly. However, it’s a show-stopper if I need to access the PC remotely since it is no longer connected to the Internet. When it happens with our main family PC, it’s even more annoying since the steps to resolve it are not intuitive for my wife and sons.

Today, after yet another such unplanned outage (thank you, Windows Updates), a lightbulb finally went off in my head — why not use the existing Windows Persistent Route capability to add a persistent default route? I’ve now tried this approach on several PCs and it seems to work reliably!

So without further ado, if you too are suffering from this problem, here’s how to fix it.

1. Open a command prompt with Administrator privileges by right-clicking on the Command Prompt option and selecting Run As Administrator, or pressing <WindowsKey>-<R>, typing CMD in the Run box, and pressing <CTRL>-<SHIFT>-<Enter>
2. Type ROUTE DELETE 0.0.0.0 to delete any existing default route; if you don’t have one at the moment, this will give an error which is fine.
3. Type ROUTE ADD -p 0.0.0.0 MASK 0.0.0.0 10.1.1.254 to add a new default route using the IP router gateway 10.1.1.254; replace this with your own gateway IP.

That’s it! Your default route should now be added back automatically whenever you reboot. If you need to change to a different default route, just repeat the steps above.