Overclocked

This post comes after a very long hiatus on my side in relation to this personal blog. During the past years I have been very busy with work and other activities, but in the last months I took a break and started to re-think my life.

One of the consequences of this process, has been the revamping of NTCore and the decision to provide it with new content in the shape of articles and programs. In fact, I wanted to start with a technical article, but then some considerations crept into my mind and I wanted to share them.

One of the reasons I stopped writing about interesting things and to dedicate spare time to my IT hobby, was that too much of my time was being spent on work related IT activities not connected to the development of Cerbero Profiler. Anyone who has ever worked for a company with incompetent managers, can understand this perfectly. There are companies, large or small, which kill the passion for whatever you enjoyed doing before working for them.

One classic example is a company which had luck with its first product, because it was the right product at the right time and then tries to replicate its first success with an endless amount of new projects all doomed to fail. The reason they do it is because they don’t want their company to rely only on one product. The reason they fail is because they were lucky, not clever, with their first product.

Unfortunately, the boost of arrogance caused by the first hit is enough to eclipse all the following failures, which may or not, depending on the success of the first product, bring the company to collapse.

The technical workforce in such a company is divided into two groups. The first group works on the first product, aka the cash cow. This group endures enormous pressure, because the entire faith of the company depends on them. Not only that, but the pressure increases whenever money is wasted on the other useless side-projects. The frustration of this group stems from the fact that they are the only ones being put under pressure and that their work has to finance the, from their side perceived, non-work of the others.

The second groups works on the side-projects which are doomed to fail. The clever technical people in this group already know that these projects will fail, but that doesn’t change anything in the decisions taken by the company. The frustration of this group stems from continuously doing useless things, which nobody cares about and not being appreciated like the people in the first group.

In such an environment, it doesn’t matter to which group you belong to, if you understand the big picture or if you just consider it your day job. You’re screwed regardless. The difference is that the people of the first group tend to last longer, but the toxic environment of the company will consume them as well in the long run. The people of the second group are the ones being consumed faster and there’s a reason for that.

I heard that some large companies take into account the psychological effects on a software developer who worked on a major project, which then got canceled. These companies make sure that the employee is then assigned to the development team of an already established product. This is to avoid the re-occurrence of the same situation for the developer and the psychological strain it would generate for him.

If you currently work for a company of the earlier category, I can give you only one advice: resign and do something else. Cultivate crops, hunt, forge steel or build roads. Anything is better than enduring the bullshit of such a place. You can do it for a time if you need to, but you have to know when to stop.

For years I wasn’t able to live from the profits of my commercial product and needed a day job, then in the last years the situation changed, but I still didn’t stop my other activity for a number of reasons. In the beginning profits were still uncertain and I also figured that more money was even better.

The ironic thing is that even though you may earn more money, you are also more inclined to spend it easily. This is because of the work-caused mental fatigue which forces your brain to look for continuous gratification to alleviate the pain. So you end up in a fancy apartment, with a big TV, a nice car, etc. It requires some effort to break the routine and part from that situation. Effort which isn’t caused by the difficulty to give up a materialistic life-style, but to one’s mental fatigue which makes it hard to start any new endeavor.

That isn’t to say that I dislike money. In fact, one of the reasons I changed my life is that the money wasn’t nearly good enough for the amount of stress I had to face. I am neither a materialistic person nor a hippie. I can live with little money or with tons of it. It doesn’t change who I am.

It’s been only 10 months since I changed things and started to re-organize my life. The initial months were spent mostly on personal matters, logistics and recovering my physical health. Even though I always kept in shape and did a lot of sport, the stress still had effects on my overall well-being.

I spent the following months on relaxing my mind, making projects for the future and even starting a new hobby, knife making.

Of course, I still worked on my commercial product from time to time, but even that required a thinking pause as the new 3.0 version approaches and it’s a good point in time for some interesting and major improvements. I also made new important business deals unrelated to my product, which wouldn’t have happened if I hadn’t changed things.

That brings us to now and to my wish to rekindle my passion for IT and to the actual topic of the post.

It’s impossible for someone who grew up playing with SoftICE, like myself, not to notice the differences in approaching the field of IT back then and doing it now. In the past, we spent our time on IRC, which was a lot more fun than Twitter. We had less technologies to focus on. The result was that we were more focused and less distracted.

Not only that. We were small communities in which you could gain appreciation for some days of work writing a small utility or writing an article. Today nobody gives a fuck. Your article or code is just a drop in the ocean or a tweet in the movie “The Birds”.

Nowadays the IT field exploded with many new fields and disciplines, many of which 20 years ago were relegated to academic research, were insignificantly small or weren’t there at all. Distributed computing, machine learning, mobile development, virtualization etc.

At the same time, the amount of people and money in the IT industry also caused the explosion of bullshit. From IT security up until the retarded bullshit of agile development.

Although this may just seem another “things were better before” comment, it’s not really the point of it. There’s a natural process of commercialization from something which is niche to something which becomes common and consumed by the masses, which makes the field for those belonging to the initial niche less appealing. This is normal.

What is interesting is that we lose interest in things today, because we are overclocked. By this technical reference I mean that we are overstimulated. We developed a numbness in regard to technology because we were exposed to too many (mostly useless) innovations in an excessive amount which our brain couldn’t absorb and so it gave up and lost interest.

While, of course, no one can centrally control the amount of innovations which globally come out every day, individual companies can limit the amount of innovations within their own products for our brains to be able to appreciate them.

There’s a reason why nobody cares today when the new Windows is released. Many stopped caring after Windows Vista and most after Windows 7. Remember when the release of a new Windows was a big event? Remember how respected the work of Matt Pietrek and Sven B. Schreiber was? It’s not just because they were pioneers. The reason is that we cared beyond having a resource to help us implement our daily piece of code.

We had the illusion that technology was a progression towards improvement. And now we are disillusioned.

In my old rants against Microsoft, wherein I predict the failure of products like Windows Phone and Silverlight, it is possible to notice the increasing disillusionment. Let me quote an old post from 2011:

Moreover, Windows could be improved to an endless extent without re-inventing the wheel every 2 years. If the decisions were up to me I would work hard on micro-improvements. Introduce new sets of native APIs along Win32. And I’d do it gradually, with care and try to give them a strong coherency. I would try to introduce benefits which could be enjoyed even by applications written 15 years ago. The beauty should lie in the elegance in finding ingenious solutions for extending what is already there, not by doing tabula rasa every time. I would make developers feel at home and that their time and code is highly valued, instead of making them feel like their creations are always obsolete compared to my brand new technology which, by the way, nobody uses.

To be clear, it isn’t just Microsoft. All the big players make the same mistake. During Jobs’ era at Apple we had a controlled amount of improvements which we could appreciate. When Jobs died, Apple became the same as any other company and today nobody cares about Apple products as well.

The gist of my theory is what follows. The majority of people use Windows or the iPhone to do a number of things. While a minority of people may think it’s cool to have yet a slimmer phone without headphone jack or charging it without a wire, these are actually regressions (having to buy new adapters or headphones from Apple, more easily breaking your phone because the back is made out of glass) and they annoy the majority, while also numbing their capacity to absorb improvements.

If you add to your product 50 new things and only 5 of those are actual improvements, even those 5 improvements will become an indistinguishable blur among the other 45 and won’t even be perceived.

And just to hammer my point home, let’s take a Victorinox Swiss Army Knife (yes, I grew up watching MacGyver). It has more than a hundred years of history and it is perfect as it is. Of course, a minority of people may think that adding pizza cutter to it may be essential, but Victorinox doesn’t work for a minority. Yes, every now and then a new model of knife comes out intended for a particular group of people like sailing enthusiasts or IT workers, but the classic models have more or less remained unchanged throughout the decades. What happened is that they went over countless micro-improvements which brought them to the state-of-the-art tools they are today.

An OS, just like any important piece of technology, should give the user the same satisfaction a Victorinox SAK gives to its holder.

These are some of the considerations which crossed my mind while trying to make again my entrance in the IT world. They will reflect on my work and over the next months I will put my money where my mouth is.

NTCore revamped

After over a decade, I finally took two afternoons to revamp this personal web-page and to merge the content of the old NTCore page with the content of its blog (rcecafe.net). All the URLs of the old web-page and blog have been preserved in the process.

The people who voted for this on Twitter are the guilty ones.

You know who you are.

Ctor conflicts

Perhaps the content of this post is trivial and widely known(?), but I just spent some time fixing a bug related to the following C++ behavior.

Let’s take a look at this code snippet:

The output of the code above is:

Whether we compile it with VC++ or g++, the result is the same.

The problem is that although the struct or class is declared locally the name of the constructor is considered a global symbol. So while the allocation size of the struct or class is correct, the constructor being invoked is always the first one encountered by the compiler, which in this case is the one which prints ‘apple’.

The problem here is that the compiler doesn’t warn the user in any way that the wrong constructor is being called and in a large project with hundreds of files it may very well be that two constructors collide.

Since namespaces are part of the name of the symbol, the code above can be fixed by adding a namespace:

Now the correct constructor will be called.

I wrote a small (dumb) Python script to detect possible ctor conflicts. It just looks for struct or class declarations and reports duplicate symbol names. It’s far from perfect.

In my opinion this could be handled better on the compiler side, at least by giving a warning.

ADDENDUM: Myria ‏(@Myriachan) explained the compiler internals on this one on twitter:

I’m just surprised that it doesn’t cause a “duplicate symbol” linker error. Symbol flagged “weak” from being inline, maybe? […] Member functions defined inside classes like that are automatically “inline” by C++ standard. […] The “inline” keyword has two meanings: hint to compiler that inlining machine code may be wise, and making symbol weak. […] Regardless of whether the compiler chooses to inline machine code within calling functions, the weak symbol part still applies. […] It is as if all inline functions (including functions defined inside classes) have __declspec(selectany) on them, in MSVC terms. […] Without this behavior, if you ever had a class in a header with functions defined, the compiler would either have to always inline the machine code, or you’d have to use #ifdef nonsense to avoid more than one .cpp defining the function.

The explanation is the correct one. And yes, if we define the ctor outside of the class the compiler does generate an error.

The logic mismatch here is that local structures in C do exist, local ctors in C++ don’t. So, the correct struct is allocated but the wrong ctor is being called. Also, while the symbol is weak for the reasons explained by Myria, the compiler could still give an error if the ctor code doesn’t match across files.

So the rule here could be: if you have local classes, avoid defining the ctor inside the class. If you already have a conflict as I did and don’t want to change the code, you can fix it with a namespace as shown above.

Creating undetected malware for OS X

This article was originally published on cerbero-blog.com on October the 7th, 2013.

While this PoC is about static analysis, it’s very different than applying a packer to a malware. OS X uses an internal mechanism to load encrypted Apple executables and we’re going to exploit the same mechanism to defeat current anti-malware solutions.

OS X implements two encryption systems for its executables (Mach-O). The first one is implemented through the LC_ENCRYPTION_INFO loader command. Here’s the code which handles this command:

This code calls the set_code_unprotect function which sets up the decryption through text_crypter_create:

The text_crypter_create function is actually a function pointer registered through the text_crypter_create_hook_set kernel API. While this system can allow for external components to register themselves and handle decryption requests, we couldn’t see it in use on current versions of OS X.

The second encryption mechanism which is actually being used internally by Apple doesn’t require a loader command. Instead, it signals encrypted segments through a flag.

Protected flag

The ‘PROTECTED‘ flag is checked while loading a segment in the load_segment function:

The unprotect_segment function sets up the range to be decrypted, the decryption function and method. It then calls vm_map_apple_protected.

Two things about the code above. The first 3 pages (0x3000) of a Mach-O can’t be encrypted/decrypted. And, as can be noticed, the decryption function is dsmos_page_transform.

Just like text_crypter_create even dsmos_page_transform is a function pointer which is set through the dsmos_page_transform_hook kernel API. This API is called by the kernel extension “Dont Steal Mac OS X.kext“, allowing for the decryption logic to be contained outside of the kernel in a private kernel extension by Apple.

Apple uses this technology to encrypt some of its own core components like “Finder.app” or “Dock.app”. On current OS X systems this mechanism doesn’t provide much of a protection against reverse engineering in the sense that attaching a debugger and dumping the memory is sufficient to retrieve the decrypted executable.

However, this mechanism can be abused by encrypting malware which will no longer be detected by the static analysis technologies of current security solutions.

To demonstrate this claim we took a known OS X malware:

Scan before encryption

Since this is our public disclosure, we will say that the detection rate stood at about 20-25.

And encrypted it:

Scan after encryption

After encryption has been applied, the malware is no longer detected by scanners at VirusTotal. The problem is that OS X has no problem in loading and executing the encrypted malware.

The difference compared to a packer is that the decryption code is not present in the executable itself and so the static analysis engine can’t recognize a stub or base itself on other data present in the executable, since all segments can be encrypted. Thus, the scan engine also isn’t able to execute the encrypted code in its own virtual machine for a more dynamic analysis.

Two other important things about the encryption system is that the private key is the same and is shared across different versions of OS X. And it’s not a chained encryption either: but per-page. Which means that changing data in the first encrypted page doesn’t affect the second encrypted page and so on.

Our flagship product, Cerbero Profiler, which is an interactive file analysis infrastructure, is able to decrypt protected executables. To dump an unprotected copy of the Mach-O just perform a “Select all” (Ctrl+A) in the main hex view and then click on “Copy into new file” like in the screen-shot below.

Mach-O decryption

The saved file can be executed on OS X or inspected with other tools.

Decrypted Mach-O

Of course, the decryption can be achieved programmatically through our Python SDK as well. Just load the Mach-O file, initialize it (ProcessLoadCommands) and save to disk the stream returned by the GetStream.

A solution to mitigate this problem could be one of the following:

  • Implement the decryption mechanism like we did.
  • Check the presence of encrypted segments. If they are present, trust only executables with a valid code signature issued by Apple.
  • 3. Check the presence of encrypted segments. If they are present, trust only executables whose cryptographic hash matches a trusted one.

This kind of internal protection system should be avoided in an operating system, because it can be abused.

After we shared our internal report, VirusBarrier Team at Intego sent us the following previous research about Apple Binary Protection:

http://osxbook.com/book/bonus/chapter7/binaryprotection/
http://osxbook.com/book/bonus/chapter7/tpmdrmmyth/
https://github.com/AlanQuatermain/appencryptor

The research talks about the old implementation of the binary protection. The current page transform hook looks like this:

VirusBarrier Team also reported the following code by Steve Nygard in his class-dump utility:

https://bitbucket.org/nygard/class-dump/commits/5908ac605b5dfe9bfe2a50edbc0fbd7ab16fd09c

This is the correct decryption code. In fact, the kernel extension by Apple, just as in the code above provided by Steve Nygard, uses the OpenSSL implementation of Blowfish.

We didn’t know about Nygard’s code, so we did our own research about the topic and applied it to malware. We would like to thank VirusBarrier Team at Intego for its cooperation and quick addressing of the issue. At the time of writing we’re not aware of any security solution for OS X, apart VirusBarrier, which isn’t tricked by this technique. We even tested some of the most important security solutions individually on a local machine.

The current 0.9.9 version of Cerbero Profiler already implements the decryption of Mach-Os, even though it’s not explicitly written in the changelist.

We didn’t implement the old decryption method, because it didn’t make much sense in our case and we’re not aware of a clean way to automatically establish whether the file is old and therefore uses said encryption.

These two claims need a clarification. If we take a look at Nygard’s code, we can see a check to establish the encryption method used:

It checks the first dword in the encrypted segment (after the initial three non-encrypted pages) to decide which decryption algorithm should be used. This logic has a problem, because it assumes that the first encrypted block is full of 0s, so that when encrypted with AES it produces a certain magic and when encrypted with Blowfish another one. This logic fails in the case the first block contains values other than 0. In fact, some samples we encrypted didn’t produce a magic for this exact reason.

Also, current versions of OS X don’t rely on a magic check and don’t support AES encryption. As we can see from the code displayed at the beginning of the article, the kernel doesn’t read the magic dword and just sets the Blowfish magic value as a constant:

So while checking the magic is useful for normal cases, security solutions can’t rely on it or else they can be easily tricked into using the wrong decryption algorithm.

MUI files under the hood

Have you ever copied after Vista a system file like notepad.exe onto the desktop and tried to execute it? Have you ever tried after Vista to modify the resources of a system file like regedit.exe? It’s most likely that neither of the two was a successful operation.

This will be very brief because the topic is very limited and because of my lack of time: bear with me. 🙂

If you try to copy, for instance, notepad.exe onto the desktop and run it in a debugger you will notice that it fails in its initialization routine when trying to load its accelerators. You take a look at the HINSTANCE passed to LoadAccelerators and notice that it’s NULL. You open notepad.exe in a resource viewer and notice that it doesn’t contain accelerator resources. Thus, you realize that the global instance is associated to some external resource as well. Go back to the system folder where you took the system executable and you’ll notice language directories such as “en-US”. Just copy the one which identifies the language of your system to the same directory of notepad.exe. You’ll notice that now notepad.exe runs correctly.

Vista introduced the separation between binary and language dependent resources to allow a single Windows image to contain more than just one language. You can obtain more information about the development aspects on MSDN.

The language directory contains files with names such as “notepad.exe.mui”, one for every file they provide resources for (including dlls). These are very basic PE files which contain only a resource directory and are loaded into the address space of the process as they are.

These files are associated to the main file in two ways:

1) By name: just rename the notepad to test.exe and the MUI file accordingly and it still works.
2) Via resource, as we’ll see.

If you open both notepad.exe and its MUI file with a resource viewer, you’ll see they both contain a “MUI” resource. What this data contains can be roughly understood from the MSDN or SDK:

You’ll find this structure in WinNls.h. However, this structure is for GetFileMUIInfo, it doesn’t match the physical data.

The first DWORD is clearly a signature. If you change it, the MUI is invalidated and notepad won’t run. It is followed by another DWORD describing the size of the structure (including the signature).

These are the two checksums:

These two checksums are probably in the same order of the structure. They both match the ones contained in the MUI file and if you change the second one, the application won’t run.

There are no other association criteria: I changed both the main file and the MUI file (by using a real DLL and just replacing the resource directory with the one of the MUI file) and it still worked.

About the second matter mentioned in the beginning: modification of resources. If you try to add/replace an icon to/in notepad.exe you will most likely not succeed. This is because as mentioned in the MSDN:

There are some restrictions on resource updates in files that contain Resource Configuration(RC Config) data: LN files and the associated .mui files. Details on which types of resources are allowed to be updated in these files are in the Remarks section for the UpdateResource function.

Basically, UpdateResource doesn’t work if the PE file contains a MUI resource. Now, prepare for an incredibly complicated and technically challenging hack to overcome this limitation… Ready? Rename the “MUI” resource to “CUI” or whatever, now try again and it works. Restore the MUI resource name and all is fine.

The new build of the CFF Explorer handles this automatically for your comfort.

This limitation probably broke most of the resource editors for Win32. Smart.