Agner`s CPU blog

Will Intel be forced to remove the "cripple AMD" function from their compiler?

Many software programmers consider Intel's compiler the best optimizing compiler on the market, and it is often the preferred compiler for the most critical applications. Likewise, Intel is supplying a lot of highly optimized function libraries for many different technical and scientific applications. In many cases, there are no good alternatives to Intel's function libraries.

Unfortunately, software compiled with the Intel compiler or the Intel function libraries has inferior performance on AMD and VIA processors. The reason is that the compiler or library can make multiple versions of a piece of code, each optimized for a certain processor and instruction set, for example SSE2, SSE3, etc. The system includes a function that detects which type of CPU it is running on and chooses the optimal code path for that CPU. This is called a CPU dispatcher. However, the Intel CPU dispatcher does not only check which instruction set is supported by the CPU, it also checks the vendor ID string. If the vendor string says "GenuineIntel" then it uses the optimal code path. If the CPU is not from Intel then, in most cases, it will run the slowest possible version of the code, even if the CPU is fully compatible with a better version.

I have complained about this behavior for years, and so have many others, but Intel have refused to change their CPU dispatcher. If Intel had advertised their compiler as compatible with Intel processors only, then there would probably be no complaints. The problem is that they are trying to hide what they are doing. Many software developers think that the compiler is compatible with AMD processors, and in fact it is, but unbeknownst to the programmer it puts in a biased CPU dispatcher that chooses an inferior code path whenever it is running on a non-Intel processor. If programmers knew this fact they would probably use another compiler. Who wants to sell a piece of software that doesn't work well on AMD processors?

Because of their size, Intel can afford to put more money into their compiler than other CPU vendors can. The Intel compiler is relatively cheap, it has superior performance, and the support is excellent. Selling such a compiler is certainly not a profitable business in itself, but it is obviously intended as a way of supporting Intel's microprocessors. There would be no point in adding new advanced instructions to the microprocessors if there were no tools to use these instructions. AMD is also making a compiler, but the current version supports only Linux, not Windows.

Various people have raised suspicion that the biased CPU dispatching has made its way into common benchmark programs (link link). This is a serious issue indeed. We know that many customers base their buying decision on published benchmark results, and a biased benchmark means an unfair market advantage worth billions of dollars.

The legal battle

AMD have sued Intel for unfair competition at least since 2005, and the case has been settled in November 2009. This settlement deals with many issues of unfair competition, apparently including the Intel compiler. The settlement says:

2.3 TECHNICAL PRACTICES

Intel shall not include any Artificial Performance Impairment in any Intel product or require any Third Party to include an Artificial Performance Impairment in the Third Party’s product. As used in this Section 2.3, “Artificial Performance Impairment” means an affirmative engineering or design action by Intel (but not a failure to act) that (i) degrades the performance or operation of a Specified AMD product, (ii) is not a consequence of an Intel Product Benefit and (iii) is made intentionally to degrade the performance or operation of a Specified AMD Product. For purposes of this Section 2.3, “Product Benefit” shall mean any benefit, advantage, or improvement in terms of performance, operation, price, cost, manufacturability, reliability, compatibility, or ability to operate or enhance the operation of another product.

In no circumstances shall this Section 2.3 impose or be construed to impose any obligation on Intel to (i) take any act that would provide a Product Benefit to any AMD or other non-Intel product, either when such AMD or non-Intel product is used alone or in combination with any other product, (ii) optimize any products for Specified AMD Products, or (iii) provide any technical information, documents, or know how to AMD.

This looks like a victory for AMD. If we read "any Intel product" as Intel's compilers and function libraries, "any Third Party" as programmers using these compilers and libraries, and "Artificial Performance Impairment" as the CPU dispatcher checking the vendor ID string; then the settlement puts an obligation on Intel to change their CPU dispatcher. I will certainly check the next version of Intel's compiler and libraries to see if they have done so or they have found a loophole in the settlement.

Interestingly, this is not the end of the story. Only about one month after the AMD/Intel settlement, the US Federal Trade Commission (FTC) filed an antitrust complaint against Intel. The accusations in the FTC complaint are unusually strong:

Intel sought to undercut the performance advantage of non-Intel x86 CPUs relative to Intel x86 CPUs when it redesigned and distributed software products, such as compilers and libraries.
[...]
To the public, OEMs, ISVs, and benchmarking organizations, the slower performance of non-Intel CPUs on Intel-compiled software applications appeared to be caused by the non-Intel CPUs rather than the Intel software. Intel failed to disclose the effects of the changes it made to its software in or about 2003 and later to its customers or the public. Intel also disseminated false or misleading documentation about its compiler and libraries. Intel represented to ISVs, OEMs, benchmarking organizations, and the public that programs inherently performed better on Intel CPUs than on competing CPUs. In truth and in fact, many differences were due largely or entirely to the Intel software. Intel’s misleading or false statements and omissions about the performance of its software were material to ISVs, OEMs, benchmarking organizations, and the public in their purchase or use of CPUs. Therefore, Intel’s representations that programs inherently performed better on Intel CPUs than on competing CPUs were, and are, false or misleading. Intel’s failure to disclose that the differences were due largely to the Intel software, in light of the representations made, was, and is, a deceptive practice. Moreover, those misrepresentations and omissions were likely to harm the reputation of other x86 CPUs companies, and harmed competition.
[...]
Some ISVs requested information from Intel concerning the apparent variation in performance of identical software run on Intel and non-Intel CPUs. In response to such requests, on numerous occasions, Intel misrepresented, expressly or by implication, the source of the problem and whether it could be solved.
[...]
Intel’s software design changes slowed the performance of non-Intel x86 CPUs and had no sufficiently justifiable technological benefit. Intel’s deceptive conduct deprived consumers of an informed choice between Intel chips and rival chips, and between Intel software and rival software, and raised rivals’ costs of competing in the relevant CPU markets. The loss of performance caused by the Intel compiler and libraries also directly harmed consumers that used non-Intel x86 CPUs.

The remedy that the FTC asks for is also quite farreaching:

Requiring that, with respect to those Intel customers that purchased from Intel a software compiler that had or has the design or effect of impairing the actual or apparent performance of microprocessors not manufactured by Intel ("Defective Compiler"), as described in the Complaint:

1. Intel provide them, at no additional charge, a substitute compiler that is not a Defective Compiler;
2. Intel compensate them for the cost of recompiling the software they had compiled on the Defective Compiler and of substituting, and distributing to their own customers, the recompiled software for software compiled on a Defective Compiler; and
3. Intel give public notice and warning, in a manner likely to be communicated to persons that have purchased software compiled on Defective Compilers purchased from Intel, of the possible need to replace that software.

Maybe the FTC has decided that the AMD/Intel settlement was not a fair and sufficient remedy against Intel's monopoly behavior? The settlement compensates AMD, but not VIA and other microprocessor vendors, and not the customers who have been harmed by insufficient competition and by the "defective" software produced with the Intel compiler.

My own findings

When I started testing Intel's compiler several years ago, I soon found out that it had a biased CPU dispatcher. Back in January 2007 I complained to Intel about the unfair CPU dispatcher. I had a long correspondence with Intel engineers about the issue, where they kept denying the problem and I kept providing more evidence. They said that:

The CPU dispatch, coupled with optimizations, is designed to optimize performance across Intel and AMD processors to give the best results. This is clearly our goal and with one exception we believe we are there now. The one exception is that our 9.x compilers do not support SSE3 on AMD processors because of the timing of the release of AMD processors vs. our compiler (our compiler was developed before AMD supported SSE3). The future 10.x compilers, which enter beta this quarter and release around the middle of the year, will address this now that we’ve had time to tune and adjust to the new AMD processors.

Sounds nice, but the truth is that the CPU dispatcher didn't support SSE or SSE2 or any higher SSE in AMD processors and still doesn't today (Intel compiler version 11.1.054). I have later found out that others have made similar complaints to Intel and got similarly useless answers (link link).

The Intel CPU dispatcher does not only check the vendor ID string and the instruction sets supported. It also checks for specific processor models. In fact, it will fail to recognize future Intel processors with a family number different from 6. When I mentioned this to the Intel engineers they replied:

You mentioned we will not support future Intel processors with non-'6' family designations without a compiler update. Yes, that is correct and intentional. Our compiler produces code which we have high confidence will continue to run in the future. This has the effect of not assuming anything about future Intel or AMD or other processors. You have noted we could be more aggressive. We believe that would not be wise for our customers, who want a level of security that their code (built with our compiler) will continue to run far into the future. Your suggested methods, while they may sound reasonable, are not conservative enough for our highly optimizing compiler. Our experience steers us to issue code conservatively, and update the compiler when we have had a chance to verify functionality with new Intel and new AMD processors. That means there is a lag sometime in our production release support for new processors.

In other words, they claim that they are optimizing for specific processor models rather than for specific instruction sets. If true, this gives Intel an argument for not supporting AMD processors properly. But it also means that all software developers who use an Intel compiler have to recompile their code and distribute new versions to their customers every time a new Intel processor appears on the market. Now, this was three years ago. What happens if I try to run a program compiled with an old version of Intel's compiler on the newest Intel processors? You guessed it: It still runs the optimal code path. But the reason is more difficult to guess: Intel have manipulated the CPUID family numbers on new processors in such a way that they appear as known models to older Intel software. I have described the technical details elsewhere.

Perhaps the initial design of Intel's CPU dispatcher was indeed intended to optimize for known processor models only, without regard for future models. If any of my students had made such a solution that was not future-oriented, I would consider it a serious flaw. Perhaps the Intel engineers discovered the missing support for future processors too late so that they had to design the next generation of their processors in such a way that they appeared as known models to existing Intel software.

After Intel had flatly denied to change their CPU dispatcher, I decided that the most efficient way to make them change their minds was to create publicity about the problem. I contacted several IT magazines, but nobody wanted to write about it. Sad, but not very surprising, considering that they all depend on advertising money from Intel. The only publicity was my own optimization manual where I have described the problem in detail and given instructions on how to replace the unfair CPU dispatcher. I wonder why AMD didn't create public awareness about the problem. Were they obliged to keep quiet about an ungoing lawsuit? And what about VIA/Centaur?

Workarounds

At present, we don't know if or when Intel will make a new compiler and new software libraries that do not check the vendor ID string. In the meantime, here is what we can do about the problem.

• Use another compiler. In my tests, the Gnu compiler for Linux has an optimizing performance similar to the Intel compiler, but the Gnu function library (glibc) is inferior. All other compilers gave lower performance in my tests. There is no other Windows compiler with a similar performance, not even the Gnu compiler for Windows.
    
• Use the Intel software and patch the CPU dispatcher. In my C++ manual, I have provided the code for alternative CPU dispatchers for Intel's compiler and function libraries and descriptions on how to patch them into your software. This, of course, relies on undocumented details of the Intel software. This dispatcher-patch can improve performance on non-Intel processors considerably in many cases.
    
• Never trust any benchmark unless it is open source and compiled with a neutral compiler, such as Gnu or Microsoft.
     
• It is possible to change the CPUID of AMD processors by using the AMD virtualization instructions. I hope that somebody will volunteer to make a program for this purpose. This will make it easy for anybody to check if their benchmark is fair and to improve the performance of software compiled with the Intel compiler on AMD processors.

Links

My Discussion in Aceshardware forum 2007.

Discussion in AMD Developer Forums 2008.

My Discussion in AMDzone 2009.

Discussion in comp.arch 2004.

Complaint to Intel 2004, discussion in slashdot.org.

Mark Mackey, complaint to Intel 2005.

PCMark 2005 benchmark proven unfair. Arstechnica.

Testimony by John Oram regarding BAPCo benchmark organization.

Comment on AMD Developer Central 2005.

AMD files lawsuit 2005.

AMD untitrust complaints 2005.

Settlement agreement between AMD and Intel, 2009.

FTC complaint 2009.

Technical details in my C++ optimization manual.

[Added later:]

Discussion on XtremeSystems Forum.

Felid

«It is possible to change the CPUID of AMD processors by using the AMD virtualization instructions. I hope that somebody will volunteer to make a program for this purpose.»
How exactly we can do that?

As of now the simpliest option is to change vendor strings embedded in executables to compare with CPU's. We're at iXBT.com are doing this right now and will publish test results soon (Intel & AMD CPU's, „Intel“—>„AMD“ & „AMD“—>„Intel“ string change, 4-6 runs per app, 50-100 app's). If it is OK to use your findings (with all the copyrights), we will :)

Yes, it may be true that the same code paths they used for Intel CPUs would have optimized well for AMD CPUs, but Intel probably isn't required to know that. Also, their code path optimizations are apparently targeted for specific vendors (namely, theirs, I guess) and models, not generic strategies, so it would have been a hack for them to apply an Intel-derived code path to AMD processors, because, while being better than no optimizer, the code might also include superfluous and deleterious algorithms as applied to AMD CPUs. Yes, it's pragmatic to do so and would have been better for consumers, but can we ethically require them to engineer a dirty hack into their programming structure? Another reason it's a gray area is that it's possible that the code path optimizations they took were obvious and would likely apply to any modern x86 CPU (though the fact that AMD and Intel are the only two players in the game sort of makes it beg the question), and if only some of them were obvious those optimizations may or may not be modularized already for easy extraction and universal use.

From a business standpoint, this was obviously a strategic decision on Intel's part to undermine AMD, but if it's an ideological grey area whether they undermined AMD or simply took advantage of the fact that they happen to be the producers of one of the CPUs they compile for in order to improve their compiler for their *own* product, then they can't necessarily be faulted, because the law requires evidence and analysis. (Taking in-house advantage isn't necessarily anti-competitive in itself; for example, Microsoft makes plenty of applications that run on their own OS but not on Linux or Mac OS.)

It is true that the settlement they issued flat-out admits wrongdoing; i.e., it implies that they did, in fact, effect an "affirmative or design action" meant to undermine AMD. As far as I can see there are three possible reasons for this.
1. We don't know all the details regarding how their code paths engine was implemented and other contingent issues. Being that it's a gray area, depending on these details we don't know, they could have been clearly in the wrong and thus admitted it.
2. It's a settlement; settlements are always political. Therefore the fact that they made a statement implying wrongdoing doesn't necessarily mean a whole lot. In fact, the excerpt shown doesn't really say *anything* except that Intel will not in the future do things that are already clearly classified as anti-competitive. In other words, it's a formal statement that they won't break the law... and not being able to break the law is already implied.
3. As per Reason 2, given the small excerpt of the settlement shown, it seems possible to me that what they *actually* did is make something up that will sate AMD's lawyers while at the same time leaving the door open for them to either continue the same practice, or cease the practice (if it's too obviously anti-competitive or if they explicitly said they'd cease it elsewhere) but instate similar and/or related practices in the future, on account of the fact that those practices can easily be classified as "failures to act."

Of course it complicates the matter that there is actually a complete instruction set (SSE) that they don't support for the AMD. They have an excuse for that. Whether or not it's valid who knows. Since they do support that instruction set, it would seem deliberate "affirmative action" to disable it on AMD since AMD supports it in exactly the same way that Intel does on a purely functional level (independently of speed), and SSE is generally more efficient when applicable. However, if the decisions for how and when to use SSE instructions are intricately tied in with the rest of their code path algorithm (and possibly rely on internal structure of the CPU design), then the caveats I brought up earlier still apply.

In any case, whether not supporting optimizations on AMD's CPUs was an affirmative design decision to undermine AMD machines or merely a failure to act (to benefit AMD machines), either way, it's clearly wrong for them to publish benchmarks to OEMs, etc. comparing AMD CPUs to Intel CPUs using their own compiler that specifically optimizes for Intel CPUs (based on Vendor ID no less!, but either way) and not for AMD CPUs. It's misleading, and according to the UTC, even when specifically confronted with the issue they would habitually either mislead or directly lie about the cause for the speed difference and whether it could be solved. So *that's* the part that's really devious, and I can see why the FTC sued them. I *hate* companies like that. Incidentally, though, all companies are companies like that.

I think it's arguable whether or not Intel crippled AMD via an "affirmative engineering or design action" as opposed to a "failure to act" (as distinguished in the settlement).

I doubt Intel can be required to optimize specifically for a CPU that's not theirs

Another reason it's a gray area is that it's possible that the code path optimizations they took were obvious and would likely apply to any modern x86 CPU (though the fact that AMD and Intel are the only two players in the game sort of makes it beg the question),

given the small excerpt of the settlement shown, it seems possible to me that what they *actually* did is make something up that will sate AMD's lawyers while at the same time leaving the door open for them to either continue the same practice, or cease the practice (if it's too obviously anti-competitive or if they explicitly said they'd cease it elsewhere) but instate similar and/or related practices in the future, on account of the fact that those practices can easily be classified as "failures to act." [...] However, if the decisions for how and when to use SSE instructions are intricately tied in with the rest of their code path algorithm (and possibly rely on internal structure of the CPU design), then the caveats I brought up earlier still apply.

But of course Intel compiler engineers are not obliged to listen to my advice if doing otherwise enables them to harm their competitors.

In any case, whether not supporting optimizations on AMD's CPUs was an affirmative design decision to undermine AMD machines or merely a failure to act (to benefit AMD machines), either way, it's clearly wrong for them to publish benchmarks to OEMs, etc. comparing AMD CPUs to Intel CPUs using their own compiler that specifically optimizes for Intel CPUs (based on Vendor ID no less!, but either way) and not for AMD CPUs. It's misleading, and according to the UTC, even when specifically confronted with the issue they would habitually either mislead or directly lie about the cause for the speed difference and whether it could be solved. So *that's* the part that's really devious, and I can see why the FTC sued them. I *hate* companies like that. Incidentally, though, all companies are companies like that.

Which compilers did you test?

Did you try e.g. the Portland Group (PGI) compiler? That means, does Intel's compiler (icc) produce faster executables than e.g. the PGI compiler for Intel CPU's? Does icc (with a patched CPU dispatcher) even produce faster executables than e.g. PGI for AMD CPU's?

Not that I'd try to challenge your claims (I haven't benchmarked any of the compilers), I'm just interested. Thanks! :-)

Which compilers did you test?

The article indicates a fair handling of non-Intel processors in the IPP library. This is in accordance with my test results.

What the article doesn't mention is the unfair CPU dispatching in several other Intel function libraries :-)

What I found most interesting, is how this seems to parallel a fight we've been having in the web development community a lot over the last five years or so, on the benefits of feature detection versus browser detection. I'll defer to Nicholas Zakas' explanation here: www.nczonline.net/blog/2009/12/29/feature-detection-is-not-browser-detection/

Now, on the web it's a bit trickier, web browsers' behaviour tends to be considerably more variable than x86 behaviour across different vendors, but ultimately, the desire is the same. Check what is supported, and select code-paths based on what's supported, not based on whether you're on IE or Safari or Firefox, etc. And the bad design is generally made by the developer, not some third-party vendor (though not always, as Nicholas talks about in his post).

I'm going to be publicizing this issue, to the best of my ability, since I think it's important that people understand the implications of choosing to use Intel's compiler, which is otherwise quite high quality. And, while I didn't understand why the antitrust suit was filed by the US government when it was announced, I'm certainly far more interested in it's outcome than I was before.

Unfortunately, this practice is more widespread than I thought. Here are some examples:

1. According to CNET news, Skype has made a deal with Intel to limit the functionality on non-Intel computers, alleging that contemporary AMD processors were not powerful enough. See CNET News and Slashdot News. This limitation has later been removed.
    
2. 64-bit Windows will only run on processors with known vendor names. This is a serious barrier for new companies to enter the x86 processor market and it is an obstackle to emulation. Same problem in FreeBSD. In fact, VIA had to make a feature to change the vendor string in their processors in order to run 64-bit Windows. See analysis by Geoff Chappell. Incidentally, this feature made it possible for me to make my CPUID manipulation program.
    
3. An Intel employee is making a separate branch for the Intel Atom processor in a certain function in the Gnu Libc library (see libc-alpha archive). Intel also have a separate branch for the Atom processor in their IPP library. If this practice spreads then the libraries will soon be bloated with many separate branches for different brands and models of microprocessors. In my opinion, the branching should be based on certain performance characteristics, not on specific processor models. If necessary, add CPUID bits for specific performance parameters, or let the software test which branch is fastest.
    

Bug has been filed at:
defect.opensolaris.org/bz/show_bug.cgi?id=14706

It is possible to change the CPUID of AMD processors by using the AMD virtualization instructions. I hope that somebody will volunteer to make a program for this purpose. This will make it easy for anybody to check if their benchmark is fair and to improve the performance of software compiled with the Intel compiler on AMD processors.

And VIA processors can do this too, which would be useful not only for this, but also because some 64-bit OSes include a hardcoded list of CPU vendor IDs that the OS will run on. Windows is a common 64-bit OS that include such a list. Often older versions of such an OS will not include the CentaurHauls vendor ID. Windows would react by bluescreening the computer at boot. MS was able to release a hotfix that added CentaurHauls to the list, and it is included in Vista SP2 and 7.

PS. If you know how to change the vendor string on a VIA then please tell me. I can't find it and I have tried everything.

You can download the program from www.agner.org/optimize/cpuidfake.zip. It is open source (GPL). Please read the included instructions.

The program requires a VIA Nano processor, and Windows (32 or 64 bit). This program makes it possible to test if a benchmark result depends on the CPUID vendor string. You can also test if the performance of other CPU-intensive programs perform differently depending on the CPUID.

If you find any benchmark or other generally used CPU-intensive program that appears to have an unfair CPU dispatching then please let me know. You may also contact the producer of the program and ask if it has been built with an Intel compiler or Intel function libraries. It is important to know how widespread this problem is.

I have tested the new libraries and found that the CPU dispatching works basically the same way as before. The standard math library, vector math library, short vector math library and the 64-bit version of other math kernel library functions still use an inferior code path for non-Intel processors.

I have found the following differences from previous versions:

• Many functions now have a branch for the forthcoming AVX instruction set, but still only for Intel processors. This will increase the difference in performance between Intel and AMD processors on these functions. Both Intel and AMD are planning to support AVX in 2011.
     
• The CPU dispatcher for the vector math library has a new branch for non-Intel processors with SSE2. Unlike the generic branch, the new non-Intel SSE2 branch is used only on non-Intel processors, and it is inferior in many cases to the branch used by Intel processors with the same instruction set. The non-Intel SSE2 branch is implemented in the 32-bit Windows version and the 32-bit Linux version, but not in the 64-bit versions of the library.
   
• A new Summary Statistics library uses the same CPU dispatcher as the vector math library.

Obviously, I haven't tested all functions in the library. There may be more differences that I haven't discovered. But it is clear that many functions in the new version of the library still cripples performance on non-Intel processors. I don't understand how they can do this without violating the legal settlement with AMD.