Time Travel: Running Python 3.7 on XP

To restart my career as a technical writer, I chose a light topic. Namely, running applications compiled with new versions of Visual Studio on Windows XP. I didn’t find any prior research on the topic, but I also didn’t search much. There’s no real purpose behind this article, beyond the fact that I wanted to know what could prevent a new application to run on XP. Our target application will be the embedded version of Python 3.7 for x86.

If we try to start any new application on XP, we’ll get an error message informing us that it is not a valid Win32 application. This happens because of some fields in the Optional Header of the Portable Executable.

Most of you probably already know that you need to adjust these fields as follows:

MajorOperatingSystemVersion: 5
MinorOperatingSystemVersion: 0
MajorSubsystemVersion: 5
MinorSubsystemVersion: 0

Fortunately, it’s enough to adjust the fields in the executable we want to start (python.exe), there’s no need to adjust the DLLs as well.

If we try run the application now, we’ll get an error message due to a missing API in kernel32. So let’s turn our attention to the imports.

We have a missing vcruntime140.dll, then a bunch of “api-ms-win-*” DLLs, then only python37.dll and kernel32.dll.

The first thing which comes to mind is that in new applications we often find these “api-ms-win-*” DLLs. If we search for the prefix in the Windows directory, we’ll find a directory both in System32 and SysWOW64 called “downlevel”, which contains a huge list of these DLLs.

As we’ll see later, these DLLs aren’t actually used, but if we open one with a PE viewer, we’ll see that it contains exclusively forwarders to APIs contained in the usual suspects such as kernel32, kernelbase, user32 etc.

There’s a MSDN page documenting these DLLs.

Interestingly, in the downlevel directory we can’t find any of the files imported by python.exe. These DLLs actually expose C runtime APIs like strlen, fopen, exit and so on.

If we don’t have any prior knowledge on the topic and just do a string search inside the Windows directory for such a DLL name, we’ll find a match in C:\Windows\System32\apisetschema.dll. This DLL is special as it contains a .apiset section, whose data can easily be identified as some sort of format for mapping “api-ms-win-*” names to others.

Searching on the web, the first resource I found on this topic were two articles on the blog of Quarkslab (Part 1 and Part 2). However, I quickly figured that, while useful, they were too dated to provide me with up-to-date structures to parse the data. In fact, the second article shows a version number of 2 and at the time of my writing the version number is 6.

Just for completeness, after the publication of the current article, I was made aware of an article by deroko about the topic predating those of Quarkslab.

Anyway, I searched some more and found a code snippet by Alex Ionescu and Pavel Yosifovich in the repository of Windows Internals. I took the following structures from there.

The data starts with a API_SET_NAMESPACE structure.

Count specifies the number of API_SET_NAMESPACE_ENTRY and API_SET_HASH_ENTRY structures. EntryOffset points to the start of the array of API_SET_NAMESPACE_ENTRY structures, which in our case comes exactly after API_SET_NAMESPACE.

Every API_SET_NAMESPACE_ENTRY points to the name of the “api-ms-win-*” DLL via the NameOffset field, while ValueOffset and ValueCount specify the position and count of API_SET_VALUE_ENTRY structures. The API_SET_VALUE_ENTRY structure yields the resolution values (e.g. kernel32.dll, kernelbase.dll) for the given “api-ms-win-*” DLL.

With this information we can already write a small script to map the new names to the actual DLLs.

This code can be executed with Cerbero Profiler from command line as “cerpro.exe -r apisetschema.py”. These are the first lines of the produced output:

Going back to API_SET_NAMESPACE, its field HashOffset points to an array of API_SET_HASH_ENTRY structures. These structures, as we’ll see in a moment, are used by the Windows loader to quickly index a “api-ms-win-*” DLL name. The Hash field is effectively the hash of the name, calculated by taking into consideration both HashFactor and HashedLength, while Index points to the associated API_SET_NAMESPACE_ENTRY entry.

The code which does the hashing is inside the function LdrpPreprocessDllName in ntdll:

Or more simply in C code:

As a practical example, let’s take the DLL name “api-ms-win-core-processthreads-l1-1-2.dll”. Its hash would be 0x445B4DF3. If we find its matching API_SET_HASH_ENTRY entry, we’ll have the Index to the associated API_SET_NAMESPACE_ENTRY structure.

So, 0x5b (or 91) is the index. By going back to the output of mappings, we can see that it matches.

By inspecting the same output, we can also notice that all C runtime DLLs are resolved to ucrtbase.dll.

I was already resigned at having to figure out how to support the C runtime on XP, when I noticed that Microsoft actually supports the deployment of the runtime on it. The following excerpt from MSDN says as much:

If you currently use the VCRedist (our redistributable package files), then things will just work for you as they did before. The Visual Studio 2015 VCRedist package includes the above mentioned Windows Update packages, so simply installing the VCRedist will install both the Visual C++ libraries and the Universal CRT. This is our recommended deployment mechanism. On Windows XP, for which there is no Universal CRT Windows Update MSU, the VCRedist will deploy the Universal CRT itself.

Which means that on Windows editions coming after XP the support is provided via Windows Update, but on XP we have to deploy the files ourselves. We can find the files to deploy inside C:\Program Files (x86)\Windows Kits\10\Redist\ucrt\DLLs. This path contains three sub-directories: x86, x64 and arm. We’re obviously interested in the x86 one. The files contained in it are many (42), apparently the most common “api-ms-win-*” DLLs and ucrtbase.dll. We can deploy those files onto XP to make our application work. We are still missing the vcruntime140.dll, but we can take that DLL from the Visual C++ installation. In fact, that DLL is intended to be deployed, while the Universal CRT (ucrtbase.dll) is intended to be part of the Windows system.

This satisfies our dependencies in terms of DLLs. However, Windows introduced many new APIs over the years which aren’t present on XP. So I wrote a script to test the compatibility of an application by checking the imported APIs against the API exported by the DLLs on XP. The command line for it is “cerpro.exe -r xpcompat.py application_path”. It will check all the PE files in the specified directory.

I had to omit the contents of the apisetschema global variable for the sake of brevity. You can download the full script from here. The system32 directory referenced in the code is the one of Windows XP, which I copied to my desktop.

And here are the relevant excerpts from the output:

We’re missing 5 APIs from kernel32.dll and 2 from ws2_32.dll, but the Winsock APIs are imported just by _socket.pyd, a module which is loaded only when a network operation is performed by Python. So, in theory, we can focus our efforts on the missing kernel32 APIs for now.

My plan was to create a fake kernel32.dll, called xernel32.dll, containing forwarders for most APIs and real implementations only for the missing ones. Here’s a script to create C++ files containing forwarders for all APIs of common DLLs on Windows 10:

It creates files like the following kernel32.cpp:

The comment on the right (“// XP”) indicates whether the forwarded API is present on XP or not. We can provide real implementations exclusively for the APIs we want. The Windows loader doesn’t care whether we forward functions which don’t exist as long as they aren’t imported.

The APIs we need to support are the following:

GetTickCount64: I just called GetTickCount, not really important
GetFinalPathNameByHandleW: took the implementation from Wine, but had to adapt it slightly
InitializeProcThreadAttributeList: took the implementation from Wine
UpdateProcThreadAttribute: same
DeleteProcThreadAttributeList: same

I have to be grateful to the Wine project here, as it provided useful implementations, which saved me the effort.

I called the attempt at a support runtime for older Windows versions “XP Time Machine Runtime” and you can find the repository here. I compiled it with Visual Studio 2013 and cmake.

So that we have now our xernel32.dll, the only thing we have to do is to rename the imported DLL inside python37.dll.

Let’s try to start python.exe.

Awesome.

Of course, we’re still not completely done, as we didn’t implement the missing Winsock APIs, but perhaps this and some more could be the content of a second part to this article.

12 thoughts on “Time Travel: Running Python 3.7 on XP”

  1. Great analysis!

    I’ll have to re-read it a couple of times to understand the details: it’s really an awesome work (you had to spend quite some time to put the pieces together, do researches, debug windows imports loading, write and debug Cerbero scripts, ecc.).

    Thank you for sharing your knowledge!
    And, of course, welcome back ! You rock!

    Best Regards,
    Tony

  2. This is awesome !
    Since I stuck with XP to access an old ERP via odbc, (Navision, codbc driver),
    this gives me some hope to be able to lift my XP Gateway to 3.7 ….
    Great work !
    yours sincerely
    Robert

      1. Without any intention to advertise, let me show you this:
        http://deneskellner.com/sw/rapidexe

        This is why I was looking for a solution to run Python on XP. It seems too much pain in the basement now; but if you find a solution to extend this tool with something-XP-ish, I’d be glad to add it as a new engine. Aaand, let me know what you think about it anyway!

        (It’s a very simple tool but I really enjoy using it)

        1. Hello Denes,
          I didn’t try it, but as a concept it’s clearly useful. Especially to deploy scripts to non-technical users. Maybe sooner or later I’ll wrap up my writing on XP and also make it simpler to integrate. πŸ™‚

  3. Not a coder and only understood about a third of it (wandered in looking for a way to run Vista/Win7 apps on XP, or at least XP64), but I’m quite impressed by your detective skills. Bravo!

  4. This article is really a pleasure to read, got some questions though, I see you use CFF explorer for many of the shots, which software is used to create this [one](https://ntcore.com/wp-content/uploads/2018/xptm/forwarders.png)?

    Also, you say ‘we’ll find a directory both in System32 and SysWOW64 called β€œdownlevel”, which contains a huge list of these DLLs.’. I’ve installed a fresh windows xp as a virtual-machine, installed vcredist_x86.exe from VisualStudio2015 and I still don’t see neither that downlevel directory nor apisetschema.dll, could you clarify here?

    I see apisetschema.dll is living on my devbox (windows7) though… i assume this is not coming from any visual studio redistributable package?

    All in all, this is really high quality article and I miss more articles like this one across the whole internet πŸ˜‰

    ps. i was trying to build windows xp apps by using nuitka so this article definitely will help me in the task hehe πŸ˜€

    1. Hello BPL,

      thank you, I’m glad you enjoyed the article. I always try to write articles which are a bit different than those usually found across the internet. So hearing that this is appreciated is rewarding. πŸ™‚

      The software used for the screen-shot is the following: https://store.cerbero.io/profiler/download/
      It’s a commercial application I wrote, but it has a free trial.

      It’s also the same application used to execute the Python code. (all those Pro headers are inside the application). The syntax to execute the scripts is explained in the article: search for cerpro.exe -r.

      The downlevel directory is present on Windows 10. I don’t know if it’s present on other versions of Windows. But those DLLs aren’t used. When you install the redist the DLLs can be found under: C:\Program Files (x86)\Windows Kits\10\Redist\ucrt\DLLs. Same for apisetschema.dll. It’s a DLL in Windows 10, but it’s present also on other version of Windows (not XP). You don’t need that DLL. It just contains the mappings to establish where the APIs are actually located.

      I hope this helps!

Leave a Reply

Your email address will not be published. Required fields are marked *