FMA instruction set

The FMA instruction set is an extension to the x86 microprocessor instruction set to perform fused multiply–add (FMA) operations on 128-bit and 256-bit operands.: 14-22–14-23 : 613  These operations were extended to 512-bit operands by the AVX-512 extension introduced by Intel and later supported by AMD.

There are two incompatible variants of the FMA instruction set. They have the same operations but differ in the number of operands:: 6, 613–615 

Operations

An FMA operation has the form d=round⁡(±a⋅b±c), where the operands are floating-point scalars or SIMD vectors. The exact value of ±a⋅b±c is rounded only once to the precision of the destination format (i.e. single or double precision).: 14–23 : 613 : 52 : 4  In the IEEE 754-2008 standard for floating-point arithmetic, the operation is called fusedMultiplyAdd and conforming implementations must provide it.: 4, 21  The result is usually more accurate than that of separate multiply and add instructions.: 52 

FMA3 and FMA4 have the same six operations. The operations differ only in the signs of the product and of c.

FMA operations (each result is rounded once): 613–672 : 5-207–5-331 
MnemonicAMD nameIntel nameOperation
VFMADD Multiply and AddFused Multiply-Addd=a⋅b+c
VFMSUB Multiply and SubtractFused Multiply-Subtractd=a⋅b−c
VFNMADD Negative Multiply and AddFused Negative Multiply-Addd=−(a⋅b)+c
VFNMSUB Negative Multiply and SubtractFused Negative Multiply-Subtractd=−(a⋅b)−c
VFMADDSUB Multiply with Alternating Add/SubtractFused Multiply-Alternating Add/Subtract
  • di=ai⋅bi−ci for even i
  • di=ai⋅bi+ci for odd i
VFMSUBADD Multiply with Alternating Subtract/AddFused Multiply-Alternating Subtract/Add
  • di=ai⋅bi+ci for even i
  • di=ai⋅bi−ci for odd i

The two alternating operations exist only for vectors. Vector elements are numbered from 0.

Compatibility

Although FMA3 and FMA4 have the same operations, their instructions differ in operands and encoding. Their instructions have different opcodes in different opcode maps. An instruction of one variant is an invalid opcode on a processor that supports only the other. Each variant has its own CPUID feature flag: FMA for FMA3 and FMA4 for FMA4.: 613–617 

Their operands differ as follows:

  • FMA4 instructions have a destination and three sources. Their operands a, b, c, and d can be four different registers. Either of the last two sources can be in memory.: 6, 613  The fourth register is encoded in an extra byte. Each FMA4 instruction is one byte longer than the matching FMA3 instruction.: 6, 617 
  • FMA3 instructions have three operands. The result is written over the first operand. The destination d is therefore the same register as a, b, or c. Only the third operand can be in memory. Each FMA3 operation has three operand orders numbered 132, 213, and 231. The orders differ in whether the overwritten register and the memory operand are factors or the summand.: 613  The input to be overwritten must first be copied to another register if all three inputs are still needed afterwards. The examples below contain such copies.: 144 

The split between FMA3 and FMA4 followed changes of plan by Intel and AMD. AMD Fellow Dave Christie wrote in May 2009 that AMD could not adopt Intel's December 2008 change of FMA from four to three operands without unacceptable risk to its product schedules. AMD therefore kept the four-operand form from Intel's earlier specification. Josh Walrath of PC Perspective wrote in 2009 that it was "unfortunate that AMD and Intel cannot work together a bit better on these things". Agner Fog wrote in 2009 that Intel, AMD, and VIA "are assigning different codes to identical or equivalent instructions because each keep their innovations secret for as long as possible". In 2011, he listed FMA3 and FMA4 among several pairs of AMD and Intel instructions with incompatible encodings. He wrote that Intel had given its own VEX encodings to instructions that AMD had first defined in the XOP encoding.

Naming

Intel and AMD define the same FMA3 mnemonics in their instruction-set manuals.: 5-207–5-331 : 613–672  The FMA4 mnemonics are defined only in AMD's manual.: 613–672  Each FMA mnemonic consists of a stem from the table above and a postfix for the data format:: 14-21–14-22 : 613–672 

  • PS: packed single precision
  • PD: packed double precision
  • SS: scalar single precision
  • SD: scalar double precision

In a packed instruction, the operation is applied to every element of a vector. In a scalar instruction, it is applied only to the lowest element.: 14–22  An FMA4 mnemonic is the stem followed directly by the postfix, as in VFMADDPS. In an FMA3 mnemonic, the operand order 132, 213, or 231 comes between the stem and the postfix, as in VFMADD213PS.: 613–672 

Rounding

The exact intermediate value of an FMA instruction exists only inside the processor and cannot be examined by software.: 14–23  FMA results are rounded in the rounding mode set in the processor's MXCSR control and status register.: 613  For AVX-512 instructions, a rounding mode can instead be encoded in the 512-bit and scalar forms whose operands are all registers.: 5–222 : 15–12  Under IEEE 754-2008, overflow, underflow, and inexact exceptions in a fused multiply–add can arise only from the addition and not from the multiplication.: 21 

With separate multiply and add instructions, the product is rounded before the addition. The result is then rounded twice: round⁡(round⁡(a⋅b)+c). On x86 processors without FMA3 or FMA4, a result rounded only once is still available from the C fma function.: 239  Intel describes its own implementation as having some cost in speed.: 16 

Contraction by C compilers

In the C programming language, since the 1999 revision, an expression such as a * b + c may be contracted into a single fused operation. Whether contraction is allowed is controlled by the STDC FP_CONTRACT pragma, and its default status is implementation-defined.: 68, 215  When compiling for a processor with FMA instructions:

  • In GCC, which does not support the pragma, contraction is on except in a standard-conforming mode such as -std=c11.
  • In Clang, contraction within a single expression has been on by default since version 14.
  • In Visual C++, contraction has been off by default since Visual Studio 2022.
  • In Intel's compilers, contraction is on by default at optimization level -O1 and above.: 16 

In the examples below, a * b + c is compiled into one FMA instruction.

Results of the same program can therefore differ between builds for processors with and without FMA.: 16  Without FMA, the expression p⋅q+r⋅s with r=p and s=−q is zero. With FMA, either product may be the unrounded one. The two possible results can differ from zero and from each other.: 17 

Intel recommends disabling FMA instructions for the best reproducibility between processors with and without FMA.: 22  Even on systems that conform to IEEE 754-2008, a program may give different results if optimizations that change results (e.g. contraction) are not avoided.: 50–52 

Examples

The following C function computes a⋅b+c on four single-precision values at once, using the vector extension of GCC and Clang:

typedef float float4 __attribute__((vector_size(16)));

float4 madd(float4 a, float4 b, float4 c)
{
    return a * b + c;
}

The listings below are Clang 21's output with optimization (-O2) for FMA3 (-mfma) and for FMA4 (-mfma4), with assembler directives and Clang's comments removed. They are in Intel syntax, where the destination is the first operand.: 616  Under the System V AMD64 calling convention, the vector arguments a, b, and c are passed in registers XMM0, XMM1, and XMM2, with a vector result returned in XMM0.

Compiler output for madd
FMA3 (-mfma)FMA4 (-mfma4)
madd:
    vfmadd213ps xmm0, xmm1, xmm2
    ret
madd:
    vfmaddps xmm0, xmm0, xmm1, xmm2
    ret

In the FMA3 listing, VFMADD213PS writes the result over its first operand, a. In the FMA4 listing, VFMADDPS has a separate destination operand. It writes to XMM0 only because XMM0 is the return register, and could write to any other register without changing a.: 616 

The next function uses all six FMA operations. The array r is passed as a pointer in register RDI. In the listings, each FMA instruction is commented with the element of r that it computes.

typedef float float4 __attribute__((vector_size(16)));

void fma_all_ops(float4 a, float4 b, float4 c, float4 r[6])
{
    r[0] = a * b + c;
    r[1] = a * b - c;
    r[2] = c - a * b;
    r[3] = -(a * b) - c;
    /* subtract in even elements, add in odd */
    r[4] = __builtin_shufflevector(a * c - b, a * c + b,
                                   0, 5, 2, 7);
    /* add in even elements, subtract in odd */
    r[5] = __builtin_shufflevector(b * c + a, b * c - a,
                                   0, 5, 2, 7);
}

Every element of r depends on all three inputs, so a, b, and c must stay intact until the last operation.

Compiler output for fma_all_ops
FMA3 (-mfma)FMA4 (-mfma4)
fma_all_ops:
    vmovaps xmm3, xmm1
    vfmadd213ps xmm3, xmm0, xmm2       ; r[0]
    vmovaps xmmword ptr [rdi], xmm3
    vmovaps xmm3, xmm1
    vfmsub213ps xmm3, xmm0, xmm2       ; r[1]
    vmovaps xmmword ptr [rdi + 16], xmm3
    vmovaps xmm3, xmm1
    vfnmadd213ps xmm3, xmm0, xmm2      ; r[2]
    vmovaps xmmword ptr [rdi + 32], xmm3
    vmovaps xmm3, xmm1
    vfnmsub213ps xmm3, xmm0, xmm2      ; r[3]
    vmovaps xmmword ptr [rdi + 48], xmm3
    vmovaps xmm3, xmm2
    vfmaddsub213ps xmm3, xmm0, xmm1    ; r[4]
    vmovaps xmmword ptr [rdi + 64], xmm3
    vfmsubadd213ps xmm2, xmm1, xmm0    ; r[5]
    vmovaps xmmword ptr [rdi + 80], xmm2
    ret
fma_all_ops:
    vfmaddps xmm3, xmm0, xmm1, xmm2    ; r[0]
    vmovaps xmmword ptr [rdi], xmm3
    vfmsubps xmm3, xmm0, xmm1, xmm2    ; r[1]
    vmovaps xmmword ptr [rdi + 16], xmm3
    vfnmaddps xmm3, xmm0, xmm1, xmm2   ; r[2]
    vmovaps xmmword ptr [rdi + 32], xmm3
    vfnmsubps xmm3, xmm0, xmm1, xmm2   ; r[3]
    vmovaps xmmword ptr [rdi + 48], xmm3
    vfmaddsubps xmm3, xmm0, xmm2, xmm1 ; r[4]
    vmovaps xmmword ptr [rdi + 64], xmm3
    vfmsubaddps xmm0, xmm1, xmm2, xmm0 ; r[5]
    vmovaps xmmword ptr [rdi + 80], xmm0
    ret

Each FMA3 instruction except the last is therefore preceded by a VMOVAPS that copies an input into XMM3, and the FMA instruction then overwrites the copy.: 144  The last FMA3 instruction overwrites c directly, since c is not needed afterwards. In the FMA4 listing, each FMA4 instruction except the last writes its result to XMM3, which is not one of the input registers, and the last overwrites a, which is no longer needed. Nothing is copied, so the only VMOVAPS instructions are the stores to r.

FMA3 instruction set

CPUs with FMA3

  • AMD
    • Piledriver (2012) and newer microarchitectures
      • 2nd gen APUs, "Trinity" (32nm), May 15, 2012
      • 2nd gen "Bulldozer" (bdver2) with Piledriver cores, October 23, 2012
  • Intel

Excerpt from FMA3

Supported commands include

MnemonicOperationMnemonicOperation
VFMADDresult = + a · b + cVFMADDSUBresult = a · b + c for i = 1, 3, ...
result = a · b − c for i = 0, 2, ...
VFNMADDresult = − a · b + c
VFMSUBresult = + a · b − cVFMSUBADDresult = a · b − c for i = 1, 3, ...
result = a · b + c for i = 0, 2, ...
VFNMSUBresult = − a · b − c
Note
  • VFNMADD is result = − a · b + c, not result = − (a · b + c).
  • VFNMSUB generates a −0 when all inputs are zero.

Explicit order of operands is included in the mnemonic using numbers "132", "213", and "231":

Postfix
1
Operationpossible
memory operand
overwrites
132a = a · c + bc (factor)a (other factor)
213a = b · a + cc (summand)a (factor)
231a = b · c + ac (factor)a (summand)

as well as operand format (packed or scalar) and size (single or double).

Postfix
2
precisionsizePostfix
2
precisionsize
SSSingle32 bitSDDouble64 bit
PSx4× 32 bitPDx2× 64 bit
PSy8× 32 bitPDy4× 64 bit

This results in

Encoding Mnemonic Operands Operation
VEX.256.66.0F38.W1 98 /r VFMADD132PDy ymm, ymm, ymm/m256 a = a · c + b
VEX.256.66.0F38.W0 98 /r VFMADD132PSy
VEX.128.66.0F38.W1 98 /r VFMADD132PDx xmm, xmm, xmm/m128
VEX.128.66.0F38.W0 98 /r VFMADD132PSx
VEX.LIG.66.0F38.W1 99 /r VFMADD132SD xmm, xmm, xmm/m64
VEX.LIG.66.0F38.W0 99 /r VFMADD132SS xmm, xmm, xmm/m32
VEX.256.66.0F38.W1 A8 /r VFMADD213PDy ymm, ymm, ymm/m256 a = b · a + c
VEX.256.66.0F38.W0 A8 /r VFMADD213PSy
VEX.128.66.0F38.W1 A8 /r VFMADD213PDx xmm, xmm, xmm/m128
VEX.128.66.0F38.W0 A8 /r VFMADD213PSx
VEX.LIG.66.0F38.W1 A9 /r VFMADD213SD xmm, xmm, xmm/m64
VEX.LIG.66.0F38.W0 A9 /r VFMADD213SS xmm, xmm, xmm/m32
VEX.256.66.0F38.W1 B8 /r VFMADD231PDy ymm, ymm, ymm/m256 a = b · c + a
VEX.256.66.0F38.W0 B8 /r VFMADD231PSy
VEX.128.66.0F38.W1 B8 /r VFMADD231PDx xmm, xmm, xmm/m128
VEX.128.66.0F38.W0 B8 /r VFMADD231PSx
VEX.LIG.66.0F38.W1 B9 /r VFMADD231SD xmm, xmm, xmm/m64
VEX.LIG.66.0F38.W0 B9 /r VFMADD231SS xmm, xmm, xmm/m32

AVX-512

AVX-512 adds 512-bit forms of the FMA3 instructions using a new instruction encoding, the EVEX prefix.: 5–220  The EVEX forms keep the FMA3 mnemonics, operand orders (132, 213, 231), and opcode bytes.: 5–207, 5–220  The VEX-encoded FMA3 forms operate on vectors of up to 256 bits and require the processor to report the FMA feature through the CPUID instruction. The EVEX forms also operate on 512-bit vectors. They require either the AVX10.1 feature or the AVX512F feature, plus AVX512VL for 128-bit and 256-bit vectors.: 5–220, 5–232 

Processors supporting AVX-512 include:

  • Intel
    • Xeon Phi Knights Landing (2016), the first processors to support AVX-512: 193 
  • AMD
    • Zen 4 (2022), AMD's first AVX-512 processor,: 248  which executes 512-bit floating-point instructions, including fused multiply–add, by "double pumping" 256-bit datapaths: 2 
    • Zen 5 (2024), AMD's first core with full 512-bit floating-point datapaths and two FMA units, which executes 512-bit fused multiply–add at full width: 2, 9 

The EVEX forms add:

  • masking of the destination by an opmask register, with unselected elements either left unchanged or set to zero: 5-221–5-222 : 15-1–15-2 
  • broadcast of one 32-bit or 64-bit value from memory to every element of a source vector: 5–221 : 15–11 
  • a rounding mode set by the instruction itself, for 512-bit and scalar forms with register operands: 5–222, 5-232–5-233 : 15–2, 15–12 

The 512-bit forms add a third packed size to postfix 2:

512-bit postfixes: 5–207, 5–220 
Postfix
2
PrecisionSizePostfix
2
PrecisionSize
PSzSingle16× 32 bitPDzDouble8× 64 bit
FMA3 encodings and their requirements: 5–220, 5-232–5-233 
EncodingOperand sizesMaskingBroadcastRounding modeFeature flag
VEX128-bit and 256-bit packed, 32-bit and 64-bit scalarNoNoNoFMA
EVEX.128, EVEX.256128-bit and 256-bit packedYesYesNoAVX512F and AVX512VL, or AVX10.1
EVEX.512512-bit packedYesYesYesAVX512F, or AVX10.1
EVEX scalar32-bit and 64-bit scalarYesNoYesAVX512F, or AVX10.1

In the rounding mode column, "Yes" means the instruction can set its own rounding mode. Otherwise the rounding mode set in the processor's MXCSR register applies.: 613 : 15–12 

For example:: 5–220 

EncodingMnemonicOperandsOperation
EVEX.512.66.0F38.W0 98 /r VFMADD132PSz zmm {k}{z}, zmm, zmm/m512/m32bcst{er} a = a · c + b

In the operands, {k} is the opmask register, {z} means unselected elements are set to zero, m32bcst is a 32-bit value broadcast from memory, and {er} means the instruction can set its own rounding mode.: 5-220–5-222 

FMA4 instruction set

CPUs with FMA4

  • AMD
    • "Heavy Equipment" processors
    • Zen: WikiChip's testing shows FMA4 still appears to work (under the conditions of the tests) despite not being officially supported and not even reported by CPUID. This has also been confirmed by Agner Fog. AMD Official Web Site FMA4 Support Note ZEN CPUs = AMD ThreadRipper 1900x, R7 Pro 1800, 1700, R5 Pro 1600, 1500, R3 Pro 1300, 1200, R3 2200G, R5 2400G.
  • Intel
    • Intel has not released CPUs with support for FMA4.

Excerpt from FMA4

Mnemonic Operands (Intel syntax) Operation
VFMADDPD xmm, xmm, xmm/m128, xmm/m128 a = b·c + d
VFMADDPD ymm, ymm, ymm/m256, ymm/m256
VFMADDPS xmm, xmm, xmm/m128, xmm/m128
VFMADDPS ymm, ymm, ymm/m256, ymm/m256
VFMADDSD xmm, xmm, xmm/m64, xmm/m64
VFMADDSS xmm, xmm, xmm/m32, xmm/m32

History

The incompatibility between Intel's FMA3 and AMD's FMA4 is due to both companies changing plans without coordinating coding details with each other. AMD changed their plans from FMA3 to FMA4 while Intel changed their plans from FMA4 to FMA3 almost at the same time. The history can be summarized as follows:

Compiler and assembler support

Different compilers provide different levels of support for FMA: