Ctor conflicts

Perhaps the content of this post is trivial and widely known(?), but I just spent some time fixing a bug related to the following C++ behavior.

Let’s take a look at this code snippet:

The output of the code above is:

Whether we compile it with VC++ or g++, the result is the same.

The problem is that although the struct or class is declared locally the name of the constructor is considered a global symbol. So while the allocation size of the struct or class is correct, the constructor being invoked is always the first one encountered by the compiler, which in this case is the one which prints ‘apple’.

The problem here is that the compiler doesn’t warn the user in any way that the wrong constructor is being called and in a large project with hundreds of files it may very well be that two constructors collide.

Since namespaces are part of the name of the symbol, the code above can be fixed by adding a namespace:

Now the correct constructor will be called.

I wrote a small (dumb) Python script to detect possible ctor conflicts. It just looks for struct or class declarations and reports duplicate symbol names. It’s far from perfect.

In my opinion this could be handled better on the compiler side, at least by giving a warning.

ADDENDUM: Myria ‏(@Myriachan) explained the compiler internals on this one on twitter:

I’m just surprised that it doesn’t cause a “duplicate symbol” linker error. Symbol flagged “weak” from being inline, maybe? […] Member functions defined inside classes like that are automatically “inline” by C++ standard. […] The “inline” keyword has two meanings: hint to compiler that inlining machine code may be wise, and making symbol weak. […] Regardless of whether the compiler chooses to inline machine code within calling functions, the weak symbol part still applies. […] It is as if all inline functions (including functions defined inside classes) have __declspec(selectany) on them, in MSVC terms. […] Without this behavior, if you ever had a class in a header with functions defined, the compiler would either have to always inline the machine code, or you’d have to use #ifdef nonsense to avoid more than one .cpp defining the function.

The explanation is the correct one. And yes, if we define the ctor outside of the class the compiler does generate an error.

The logic mismatch here is that local structures in C do exist, local ctors in C++ don’t. So, the correct struct is allocated but the wrong ctor is being called. Also, while the symbol is weak for the reasons explained by Myria, the compiler could still give an error if the ctor code doesn’t match across files.

So the rule here could be: if you have local classes, avoid defining the ctor inside the class. If you already have a conflict as I did and don’t want to change the code, you can fix it with a namespace as shown above.

Creating undetected malware for OS X

This article was originally published on cerbero-blog.com on October the 7th, 2013.

While this PoC is about static analysis, it’s very different than applying a packer to a malware. OS X uses an internal mechanism to load encrypted Apple executables and we’re going to exploit the same mechanism to defeat current anti-malware solutions.

OS X implements two encryption systems for its executables (Mach-O). The first one is implemented through the LC_ENCRYPTION_INFO loader command. Here’s the code which handles this command:

This code calls the set_code_unprotect function which sets up the decryption through text_crypter_create:

The text_crypter_create function is actually a function pointer registered through the text_crypter_create_hook_set kernel API. While this system can allow for external components to register themselves and handle decryption requests, we couldn’t see it in use on current versions of OS X.

The second encryption mechanism which is actually being used internally by Apple doesn’t require a loader command. Instead, it signals encrypted segments through a flag.

Protected flag

The ‘PROTECTED‘ flag is checked while loading a segment in the load_segment function:

The unprotect_segment function sets up the range to be decrypted, the decryption function and method. It then calls vm_map_apple_protected.

Two things about the code above. The first 3 pages (0x3000) of a Mach-O can’t be encrypted/decrypted. And, as can be noticed, the decryption function is dsmos_page_transform.

Just like text_crypter_create even dsmos_page_transform is a function pointer which is set through the dsmos_page_transform_hook kernel API. This API is called by the kernel extension “Dont Steal Mac OS X.kext“, allowing for the decryption logic to be contained outside of the kernel in a private kernel extension by Apple.

Apple uses this technology to encrypt some of its own core components like “Finder.app” or “Dock.app”. On current OS X systems this mechanism doesn’t provide much of a protection against reverse engineering in the sense that attaching a debugger and dumping the memory is sufficient to retrieve the decrypted executable.

However, this mechanism can be abused by encrypting malware which will no longer be detected by the static analysis technologies of current security solutions.

To demonstrate this claim we took a known OS X malware:

Scan before encryption

Since this is our public disclosure, we will say that the detection rate stood at about 20-25.

And encrypted it:

Scan after encryption

After encryption has been applied, the malware is no longer detected by scanners at VirusTotal. The problem is that OS X has no problem in loading and executing the encrypted malware.

The difference compared to a packer is that the decryption code is not present in the executable itself and so the static analysis engine can’t recognize a stub or base itself on other data present in the executable, since all segments can be encrypted. Thus, the scan engine also isn’t able to execute the encrypted code in its own virtual machine for a more dynamic analysis.

Two other important things about the encryption system is that the private key is the same and is shared across different versions of OS X. And it’s not a chained encryption either: but per-page. Which means that changing data in the first encrypted page doesn’t affect the second encrypted page and so on.

Our flagship product, Cerbero Profiler, which is an interactive file analysis infrastructure, is able to decrypt protected executables. To dump an unprotected copy of the Mach-O just perform a “Select all” (Ctrl+A) in the main hex view and then click on “Copy into new file” like in the screen-shot below.

Mach-O decryption

The saved file can be executed on OS X or inspected with other tools.

Decrypted Mach-O

Of course, the decryption can be achieved programmatically through our Python SDK as well. Just load the Mach-O file, initialize it (ProcessLoadCommands) and save to disk the stream returned by the GetStream.

A solution to mitigate this problem could be one of the following:

  • Implement the decryption mechanism like we did.
  • Check the presence of encrypted segments. If they are present, trust only executables with a valid code signature issued by Apple.
  • 3. Check the presence of encrypted segments. If they are present, trust only executables whose cryptographic hash matches a trusted one.

This kind of internal protection system should be avoided in an operating system, because it can be abused.

After we shared our internal report, VirusBarrier Team at Intego sent us the following previous research about Apple Binary Protection:

http://osxbook.com/book/bonus/chapter7/binaryprotection/
http://osxbook.com/book/bonus/chapter7/tpmdrmmyth/
https://github.com/AlanQuatermain/appencryptor

The research talks about the old implementation of the binary protection. The current page transform hook looks like this:

VirusBarrier Team also reported the following code by Steve Nygard in his class-dump utility:

https://bitbucket.org/nygard/class-dump/commits/5908ac605b5dfe9bfe2a50edbc0fbd7ab16fd09c

This is the correct decryption code. In fact, the kernel extension by Apple, just as in the code above provided by Steve Nygard, uses the OpenSSL implementation of Blowfish.

We didn’t know about Nygard’s code, so we did our own research about the topic and applied it to malware. We would like to thank VirusBarrier Team at Intego for its cooperation and quick addressing of the issue. At the time of writing we’re not aware of any security solution for OS X, apart VirusBarrier, which isn’t tricked by this technique. We even tested some of the most important security solutions individually on a local machine.

The current 0.9.9 version of Cerbero Profiler already implements the decryption of Mach-Os, even though it’s not explicitly written in the changelist.

We didn’t implement the old decryption method, because it didn’t make much sense in our case and we’re not aware of a clean way to automatically establish whether the file is old and therefore uses said encryption.

These two claims need a clarification. If we take a look at Nygard’s code, we can see a check to establish the encryption method used:

It checks the first dword in the encrypted segment (after the initial three non-encrypted pages) to decide which decryption algorithm should be used. This logic has a problem, because it assumes that the first encrypted block is full of 0s, so that when encrypted with AES it produces a certain magic and when encrypted with Blowfish another one. This logic fails in the case the first block contains values other than 0. In fact, some samples we encrypted didn’t produce a magic for this exact reason.

Also, current versions of OS X don’t rely on a magic check and don’t support AES encryption. As we can see from the code displayed at the beginning of the article, the kernel doesn’t read the magic dword and just sets the Blowfish magic value as a constant:

So while checking the magic is useful for normal cases, security solutions can’t rely on it or else they can be easily tricked into using the wrong decryption algorithm.

MUI files under the hood

Have you ever copied after Vista a system file like notepad.exe onto the desktop and tried to execute it? Have you ever tried after Vista to modify the resources of a system file like regedit.exe? It’s most likely that neither of the two was a successful operation.

This will be very brief because the topic is very limited and because of my lack of time: bear with me. 🙂

If you try to copy, for instance, notepad.exe onto the desktop and run it in a debugger you will notice that it fails in its initialization routine when trying to load its accelerators. You take a look at the HINSTANCE passed to LoadAccelerators and notice that it’s NULL. You open notepad.exe in a resource viewer and notice that it doesn’t contain accelerator resources. Thus, you realize that the global instance is associated to some external resource as well. Go back to the system folder where you took the system executable and you’ll notice language directories such as “en-US”. Just copy the one which identifies the language of your system to the same directory of notepad.exe. You’ll notice that now notepad.exe runs correctly.

Vista introduced the separation between binary and language dependent resources to allow a single Windows image to contain more than just one language. You can obtain more information about the development aspects on MSDN.

The language directory contains files with names such as “notepad.exe.mui”, one for every file they provide resources for (including dlls). These are very basic PE files which contain only a resource directory and are loaded into the address space of the process as they are.

These files are associated to the main file in two ways:

1) By name: just rename the notepad to test.exe and the MUI file accordingly and it still works.
2) Via resource, as we’ll see.

If you open both notepad.exe and its MUI file with a resource viewer, you’ll see they both contain a “MUI” resource. What this data contains can be roughly understood from the MSDN or SDK:

You’ll find this structure in WinNls.h. However, this structure is for GetFileMUIInfo, it doesn’t match the physical data.

The first DWORD is clearly a signature. If you change it, the MUI is invalidated and notepad won’t run. It is followed by another DWORD describing the size of the structure (including the signature).

These are the two checksums:

These two checksums are probably in the same order of the structure. They both match the ones contained in the MUI file and if you change the second one, the application won’t run.

There are no other association criteria: I changed both the main file and the MUI file (by using a real DLL and just replacing the resource directory with the one of the MUI file) and it still worked.

About the second matter mentioned in the beginning: modification of resources. If you try to add/replace an icon to/in notepad.exe you will most likely not succeed. This is because as mentioned in the MSDN:

There are some restrictions on resource updates in files that contain Resource Configuration(RC Config) data: LN files and the associated .mui files. Details on which types of resources are allowed to be updated in these files are in the Remarks section for the UpdateResource function.

Basically, UpdateResource doesn’t work if the PE file contains a MUI resource. Now, prepare for an incredibly complicated and technically challenging hack to overcome this limitation… Ready? Rename the “MUI” resource to “CUI” or whatever, now try again and it works. Restore the MUI resource name and all is fine.

The new build of the CFF Explorer handles this automatically for your comfort.

This limitation probably broke most of the resource editors for Win32. Smart.

Preparing a bugfix version of CFF Explorer

It has been many years since the last update of what had started as a hobby side-project when I was 19. I’m sorry that I haven’t updated the CFF for such a long time, given that thousands of people use it every day. A few months ago I stopped working for Hex-Rays to fully dedicate myself to my own company and thus I have decided that I have now the time and the energy (barely) to finally update the CFF.

Over the years I’ve received several bugfix requests, but couldn’t oblige because of the lack of time. If you’re interested that a particular fix goes into the upcoming release, please leave a comment under this blog post or drop me an email to ntcore@gmail.com (feel free to repeat the request, as it might have been lost during the years).

Please don’t include radical changes or improvements, we’ll leave that for later maybe. If your company needs professional PE inspection (not editing), I’d advice you to check out my current commercial product at cerbero.io/profiler, which doesn’t cover ‘just’ the Portable Executable format.

UPDATE: Uploaded new version with the following improvements:

– Dropped Itanium version
– Added ENCLog and ENCMap .NET tables
– Modify resources of system files (MUI limitation)
– Fixed resource loop bug
– Fixed MDTables string overflow bug
– Fixed command line scripting bug
– Fixed ‘Select All’ bug in hex editor
– Fixed missing offset check in .NET tables
– Fixed missing reloc size check
– Fixed scripting handles bug
– Use FTs when OFTs are invalid
– Updated UPX

You can continue to leave comments or send me emails. As soon as there are enough new bug reports, I’ll upload a new version. In time, maybe, some small improvements could be included apart from bug fixes.

Companies on the Verge of a Nervous Breakdown

This is basically a continuation of the previous post about the biggest software delusions of the last decade. In hindsight I would have set rather a different tone for what I wrote, less rant and more technical, but the problem is that I keep things on my mind for a long time and never care enough to write them down leaving them rotting until they come out as technological rants. Anyway, rants are always more fun to read, so let’s keep the style.

In this post I’m going to write about some things left out in the previous one and also comment some things which happened in the meanwhile. You might ask what I have to show for my big claims about complex issues? Very little indeed, but does this make them less true? You’ll be the judge. What I try to offer here is a different perspective on issues which are always analyzed from the marketing or business point of view. Trying to explain these things giving technical reasons, offer in my opinion much better explanations than those fished from the flavor-of-the-day marketing magic hat.

After the last post I was sent per email a “graphic that illustrates the 30 years of innovation at Microsoft and their failures along the way” to link on my blog. I don’t care really about the reasons to ask for a link-to. What I want to say is that this graphic made fun of Microsoft’s failures of the decade just by listing some of them. And this is more or less the usual approach I see taken on the subject even by technical blogs. Which means focusing on the facts, rather than trying to understand them.

Windows Phone

Can we say that Lumia/Windows 7 phones flopped or is it still too soon? I think that after some of the articles I’ve read here and there, we might say that. Lumia phones were pushed out by a big carrier in the US (AT&T) and have been subject of a massive marketing campaign, but still they sold less than the dropped and not advertized N9/MeeGo project.

Nokia is laughable for dropping MeeGo! It can’t be stressed enough, because that would’ve been their only chance to regain market share and they completely blew it.

But why? Surely many reasons stand in the background, but in the end of the day one has to consider what is better on the technical level. If your definition of a better phone is how shiny it looks, then important decisions in the mobile industry shouldn’t be left to you. Many think that Apple is leading the smartphone/table industry because of their marketing strategy. While Apple products are often appealing and polished, this can’t be farther from the truth. Take the desktop market. Is Apple leading there? No. Why? Aren’t the products as polished as their counterparts in the mobile market? Or does Apple strangely suck at marketing their desktop products? Sure, Apple computers are expensive, but so are iPhones!

The first rule here is that great products sell themselves. Clearly marketing helps, but no matter how much marketing money you spend on a product which people don’t want, it will not sell, especially in the long term.

Take MeeGo for instance. I don’t mean that this project would’ve rescued Nokia instantly. Probably they would’ve still to endure 1-2 years of losses along the road, but eventually it would’ve flourished. Of this I’m sure. And considering how many people still buy overpriced N9 phones on ebay, I have a point. The trick is that if you know you have a great project at hands, you invest in it, endure some losses in the strong belief that it will eventually succeed.

One might say that this is exactly what is happening to Nokia and Windows Phone, only that they are betting on the wrong horse. It would be an acceptable point of view if we don’t get hands down on the technology itself. MeeGo was a great project, in my opinion it would’ve been the most advanced OS on the mobile market. Compare this with a repackaged Windows Mobile (not based on NT technology) running Silverlight. Alone the fact that a developer is forced to write his apps in Silverlight or XNA, that alone, would be enough to say “case fuckin’ closed!”. Rumors say Windows Mobile 8 will feature a NT kernel and also that developers will be able to compile C++ code. Seems like after enormous pressure, Microsoft had to give in about C++ (wow, that was totally unexpected… except that I wrote it even a year ago and would’ve been clear to anyone which has even a yota of experience as a developer). Even if it’s true, this is totally messed up. Those developers who lost time to port their C++ code to C# for Windows Phone 7 because C++ would never be a part of the toolchain of that OS lost their time probably for nothing. Also, users which are running Windows Mobile 7 won’t get a free update to the next version, which is incredible since both iOS and Android update their OS even for older phones. It should be pretty clear that when you want to take away market share from the biggest in the game, you must offer at least in part something which is better. Now can someone tell me in what regard a Windows Mobile 7 is better than iOS or Android. Leaving out the hardware of Nokia (and I still think that a smartphone without front-camera is pretty silly nowadays) and just focus on the operating system itself. Is there any advantage? Both iOS and Android have many more apps and of higher quality than WP7. iOS is closed just like Windows Mobile 7, while Android is more easy to hack and play with. Both iOS and Android allow C++ to be compiled, while WP7 doesn’t.

Metro and Windows 8

I’m still calling it Metro, but what is it called now? Microsoft lost the brand to a very famous European wholesale chain store. As a friend of mine said, “I would fire the whole marketing team, if they even can’t come up with a brand name which is not already used”. And not only is it used, but it’s used by a very big chain. It’s like calling your new technology “Walmart”, at least google the name first! (maybe it’s because they were forced to use Bing…)

And enough with these flashy marketing names for development technologies! There’s no reason to pretentiously call something “Silverlight”, it makes it only much more ridiculous when it ends up in the shithouse (or silvershithouse). Use dumb prosaic names like Win32, MFC, Qt! It doesn’t fuckin’ matter! What matters is the code and only the code, and after a year or more of hearing about Metro I haven’t yet seen the code! Granted I don’t look for it, I don’t dig it up from some msdn showcase, I don’t go to conferences, but this isn’t a good enough reason. Just google “metro code snippet” or anything similar and it will be hard to come up with results (I’ve found a preview on msdn which is just a collection of small samples which I was too lazy to view all). The code in this case is like a big mistery waiting to be unveiled…

Except that nobody cares! Apart making fun of Metro, I have yet to see anybody waiting impatiently for Metro or even talking about it (apart making fun of the name etc.).

Microsoft got me personally annoyed to a point in which I don’t follow anything they do anymore. I will have to try sooner or later Windows 8 just to guarantee the stability of my own product, but that’s it. I won’t use it nor play with it. I will skip it completely. And all this is ok, because I think that everything Microsoft is doing is not here to stay. Bing, Silverlight, Windows Phone, WPF, Zune (R.I.P.) etc. And time is confirming my claims. Of course, I can’t predict the future, something might change and change the faith of one of these products as well. But with the current management this is very unlikely.

As for what I read about it, the whole new UI is just jaw-dropping stupid. It’s incredible how this trend of “simplifying UIs” got hold of so many projects. Seen what happened to Gnome 3? Seen what happened to Ubuntu when it came out with Unity? Why is Mint now so popular?

Sure people don’t want to learn again things they already can do, but the problem here is that there’s no damn reason to change something which is working perfectly well and put instead something which is just worse. While humans strive for harmony and unity, these concepts can’t be applied to everything. A desktop is a productivity device. It’s efficient, fast and advanced. While a tablet is a device for consumption, it is ideal to read, play games, browse the web. Having one application at a time visible in a desktop is not only a bad idea, it is idiotic beyond imagination. The key point of a desktop is that it allows complex applications to be used, which would be impossible to use on a tablet: Photoshop, Maya, LibreOffice, Premiere etc. And the whole concept of tiles, which to Microsoft is so brilliant is equally moronic. If Microsoft doesn’t drop the whole concept soon enough after the Windows 8 debacle, I will just drop Windows completely.

The complexity of window managers could be solved much more elegantly by providing a basic mode for users which are not technologically capable.

The betrayal

Developers have been “betrayed” by Microsoft numerous times. Like I mentioned in the previous post, Microsoft deprecated and dished out new technologies at a pace that no one could follow, deprecating in a matter of few years what they just claimed to be their newest direction. Hence confusing and frustrating developers who tried to keep up-to-date, while refusing to significantly update existing and widely used technologies.

Or in the case of Windows Phone 7 the few developers who ported their code to C# now read that Windows Phone 8 will allow C++ code to be compiled. Will they be satisfied by this? Same for the users who bought Windows Phone 7 devices: they will not be able to run applications compiled for Windows Phone 8. Well, at least they got the tiles…

Losing the ground

The one thing which differentiates one OS, apart their own intrinsic quality, from the other is the number of applications which run on it. But the quality of the OS increases once there’s enough interest in it, and that interest is again a result of the applications which run on it. So simple right? While Microsoft knows this rule, it did everything it could to annoy developers. Microsoft tried to bind developers to Windows not by pleasing them, but by dishing out ugly technologies which run only on Windows and using their market share to force developers to use them.

Developers, like anyone else, guard their own interests. Many lost faith in Microsoft completely and started looking for safer havens. This surely is true even for other experts, although I can speak only for my own kind.

For instance, how did Microsoft lose its IE market share? I can’t even start judging IE as a product, apart its history of lack of security, its history of ignoring standards making life hell for web developers, its appalling plugin technology. We’re talking about a product which in 2012 considers clicking on a URL such an important event to signal it emitting a click sound. IE lost its market share by being an inferior product. But do you think that users with no technical ability would’ve downloaded and installed Firefox on their own? No, it’s because more technical people advised them to do so. I did it many times. And this is true for many products which make a name for themselves among technical people and from there they get to the masses. By the way, I consider this the best path for a product, because it means it stands on solid ground.

And finally Valve is starting to sell games on Linux. It can’t be stressed enough how important this is, because if this works out and I can’t see why it shouldn’t, it will change everything. If Microsoft loses the game battle to Linux, then they will lose the OS battle. I think this could be the battle of Stalingrad for Microsoft, because once there are enough games on Linux, there’s no end to the ground which Microsoft can lose. At that point Valve could even come out with its own console and compete against XBox. And since the gaming industry is so powerful, it would mean an overwhelming cash and interest injection into Linux, which everybody involved in that OS could benefit from. Of course, I’m speculating here, but does Microsoft understand the potential here?

I don’t think management does. They are hopping from one technology to another: WinForms, no WPF, no Silverlight, no Metro (replace with the new still unknown name), C#, no HTML5+JS. The problem, in the end, is that if as a CEO you don’t know what you are dealing with, you can’t take informed decisions and you will surround yourself with people you can’t evaluate technically. Your decisions will then only be based upon the appearance, the flashy name, how pretentious the concept sounds or how many millions are spent on marketing. A technically capable CEO is not a guarantee for success, but an incapable one is a recipe for failure. Remember what the former CEO of Pepsi did to Apple? Look at what Elop is doing to Nokia or Ballmer to Microsoft.