Agner`s CPU blog

Software optimization resources | E-mail subscription to this blog | www.agner.org

Matrix multiplication
Author:  Date: 2016-07-29 10:03
There are two problems with matrix multiplication in recent architectures:
1. The biggest problem with matrix multiplication comes from the need to keep many independent accumulators to tolerate pipeline latency. For example, on Intel's Haswell (Xeon E5 v3) processor with two FMA units and a 5-cycle FMA latency, a minimum of 10 accumulators are required. Each of these accumulators must be a full-width SIMD vector in order to use all the FMA hardware. It turns out that 10 accumulators is not a useful number for register blocking of matrix multiplication (5x2 and 2x5 don't provide good re-use), so the optimum algorithms must use 12 accumulators -- leaving only 4 register names for everything else. The optimal blocking uses 3 registers for values that are re-used in non-adjacent code, leaving a single register for handling all of the load-with-broadcast values that are used for the input array that requires transposition.
2. A secondary problem is the absence of transposition support. When going from AVX2 (described above) to AVX-512, the number of accumulators required remains the same, but the wider SIMD registers means that the code must implement 8 load-with-broadcast operations (for the transposed input array) in the same number of cycles that were required for 4 load-with-broadcast operations on Haswell. The only way that this can happen is to use a larger blocking factor for register blocking so that more data re-use occurs and more cycles are available with a free load port. Since AVX2 already required all 16 registers, an AVX-512 implementation clearly requires more -- so it is not surprising that the AVX-512 instruction set provides 32 named registers.

The first problem is probably the easiest to solve. A floating-point multiply-add unit with a single-cycle multiply-accumulate mode is described by S. Jain et al., in “A 90mW/GFlop 3.4GHz reconfigurable fused/continuous multiply-accumulator for floating-point and integer operands in 65nm,” VLSID ’10., 2010. This would reduce the number of accumulators to 2 (one per FMA unit) and (in the AVX2 example) would free 10 registers for holding data for re-use.

The second problem is definitely more difficult to solve. Although it would be relatively easy to build an FMA unit with a built-in broadcast of a scalar element, this would not help with getting the scalar element(s) into the register(s) in the first place. In the standard form of matrix multiplication, the values that need to be broadcast are collected from strided memory addresses (with a stride equal to the dimension of the matrix). Given that most (all?) recent processors have "wide" (multiple 64-bit word) interfaces at all levels of the cache/memory hierarchy, so there is physically no way to get full bandwidth while extracting only one element per block.

A possible route forward would be a SIMD "transpose with broadcast" instruction, taking a single vector of P contiguous elements and broadcasting each element across one of P output SIMD registers. The overall matrix multiplication algorithm would have to be organized so that all of these P elements would be used before being discarded, which restricts the space of possible orderings considerably. I have not worked through the details to see if this leads to a workable solution. A non-SIMD approach based on independent single-lane vector units operating from a scratchpad memory (rather than SIMD registers) might be preferred, but that opens up too many options to think about. Some encouraging developments in high-radix on-chip crossbar technology that would help in any of these approaches to parallelism is reported by Sudhir Satpathy, et al., in a number of publications, including
citeseerx.ist.psu.edu/viewdoc/download;jsessionid=8E92B85798A678099DC7A0C836575697?doi=10.1.1.459.8628&rep=rep1&type=pdf

 
thread Proposal for instruction set - now on Github new - Agner - 2016-06-26
replythread Proposal for instruction set - now on Github new - Joe Duarte - 2016-07-04
last replythread Proposal for instruction set - now on Github new - Agner - 2016-07-04
replythread Proposal for instruction set - now on Github new - Hubert Lamontagne - 2016-07-06
last replythread Proposal for instruction set - now on Github new - Agner - 2016-07-06
last replythread Proposal for instruction set - now on Github new - Hubert Lamontagne - 2016-07-07
last reply Proposal for instruction set - now on Github new - Agner - 2016-07-07
replythread Whole-function vectorization and conditionals new - Sylvain Collange - 2016-08-15
last replythread Whole-function vectorization and conditionals new - Agner - 2016-08-15
last replythread Whole-function vectorization and conditionals new - Sylvain Collange - 2016-08-15
last replythread Whole-function vectorization and conditionals new - Agner - 2016-08-15
last replythread Whole-function vectorization and conditionals new - Sylvain Collange - 2016-08-15
last replythread Whole-function vectorization and conditionals new - Agner - 2016-08-15
reply Number of input dependencies new - Agner - 2016-08-16
last replythread Whole-function vectorization and conditionals new - Sylvain Collange - 2016-08-16
last replythread Whole-function vectorization and conditionals new - Agner - 2016-08-17
last replythread Merging with first operand new - Sylvain Collange - 2016-08-18
last replythread Merging with first operand new - Agner - 2016-08-19
replythread SIMD exceptions are fine with masking new - Sylvain Collange - 2016-08-19
last replythread SIMD exceptions are fine with masking new - Agner - 2016-08-20
reply SIMD exceptions are fine with masking new - Hubert Lamontagne - 2016-08-20
last reply SIMD exceptions are fine with masking new - Sylvain Collange - 2016-08-25
last reply Merging with first operand new - Hubert Lamontagne - 2016-08-19
last replythread Proposal for instruction set - now on Github new - Joe Duarte - 2016-08-17
last replythread Proposal for instruction set - now on Github new - Agner - 2016-08-18
last replythread Proposal for instruction set - now on Github new - Joe Duarte - 2016-08-31
reply Proposal for instruction set - now on Github new - Agner - 2016-08-31
last reply Proposal for instruction set - now on Github new - Jorcy Neto - 2016-09-01
replythread Proposal for instruction set - now on Github new - Yuhong Bao - 2016-07-12
last reply Proposal for instruction set - now on Github new - Hubert Lamontagne - 2016-07-12
replythread Things from MIPS (and novel things) new - Anonymous - 2016-07-28
replythread Things from MIPS (and novel things) new - Agner - 2016-07-28
last reply Things from MIPS (and novel things) new - Hubert Lamontagne - 2016-07-28
last replythread Matrix multiplication new - Agner - 2016-07-29
reply Matrix multiplication new - Hubert Lamontagne - 2016-07-29
last replythread Matrix multiplication - John D. McCalpin - 2016-07-29
last reply Matrix multiplication new - Agner - 2016-07-29
replythread Introduction website new - Agner - 2016-08-01
last replythread Introduction website new - EricTL - 2017-07-17
last replythread Introduction website new - Agner - 2017-07-18
last replythread Introduction website new - EricTL - 2017-07-20
last reply Introduction website new - Agner - 2017-07-20
replythread Proposal for instruction set - now on Github new - Joe Duarte - 2016-08-04
last replythread Proposal for instruction set - now on Github new - Agner - 2016-08-04
last replythread Proposal for instruction set - now on Github new - Hubert Lamontagne - 2016-08-05
replythread Proposal for instruction set - now on Github new - Agner - 2016-08-06
last replythread Proposal for instruction set - now on Github new - fanoI - 2016-08-08
last replythread Proposal for instruction set - now on Github new - Agner - 2016-08-08
last reply Proposal for instruction set - now on Github new - fanoI - 2016-08-09
last replythread Proposal for instruction set - now on Github new - Joe Duarte - 2016-08-08
last replythread Proposal for instruction set - now on Github new - Hubert Lamontagne - 2016-08-09
last replythread Proposal for instruction set - now on Github new - Joe Duarte - 2016-08-11
last replythread Proposal for instruction set - now on Github new - Agner - 2016-08-12
last reply Proposal for instruction set - now on Github new - Hubert Lamontagne - 2016-08-12
replythread Proposal for instruction set - now on Github new - grant galitz - 2016-08-22
reply Proposal for instruction set - now on Github new - Agner - 2016-08-22
last reply Proposal for instruction set - now on Github new - Hubert Lamontagne - 2016-08-24
replythread ARM with scalable vector extensions new - Agner - 2016-08-23
replythread ARM with scalable vector extensions new - Jorcy Neto - 2016-08-23
last reply ARM with scalable vector extensions new - Hubert Lamontagne - 2016-08-26
last reply ARM with scalable vector extensions new - Jorcy Neto - 2016-12-20
replythread Proposal for instruction set - now on Github new - Hubert Lamontagne - 2016-09-05
replythread Proposal for instruction set - now on Github new - Agner - 2016-09-05
replythread Proposal for instruction set - now on Github new - Hubert Lamontagne - 2016-09-05
last replythread Proposal for instruction set - now on Github new - Agner - 2016-09-06
reply Proposal for instruction set - now on Github new - Bigos - 2016-09-06
last replythread Proposal for instruction set - now on Github new - Hubert Lamontagne - 2016-09-06
last replythread Proposal for instruction set - now on Github new - Agner - 2016-09-07
last replythread Proposal for instruction set - now on Github new - Hubert Lamontagne - 2016-09-07
last replythread Proposal for instruction set - now on Github new - Agner - 2016-09-08
last reply Proposal for instruction set - now on Github new - Hubert Lamontagne - 2016-09-08
last replythread Proposal for instruction set - now on Github new - Commenter - 2016-09-07
last reply Proposal for instruction set - now on Github new - Bigos - 2016-09-08
last replythread Paging new - Kurt Baumgardner - 2016-09-09
replythread Paging new - Agner - 2016-09-10
reply Paging new - Hubert Lamontagne - 2016-09-11
last replythread Paging new - Kurt Baumgardner - 2016-09-13
replythread Paging new - Agner - 2016-09-13
last reply Paging new - Kurt Baumgardner - 2016-09-13
last replythread Paging new - Hubert Lamontagne - 2016-09-13
last reply Paging new - Kurt Baumgardner - 2016-09-14
replythread Paging new - Hubert Lamontagne - 2016-09-11
last reply Paging new - Kurt Baumgardner - 2016-09-13
last replythread Paging new - Agner - 2016-09-14
last reply Paging new - Jorcy Neto - 2016-09-18
replythread A null register? new - csdt - 2016-09-23
last replythread A null register? new - Agner - 2016-09-24
last replythread A null register? new - Hubert Lamontagne - 2016-09-24
replythread A null register? new - csdt - 2016-09-26
last reply A null register? new - Agner - 2016-09-27
last replythread Indexed registers new - Kurt Baumgardner - 2016-09-26
last replythread Indexed registers new - Agner - 2016-09-27
replythread Indexed registers new - Kurt Baumgardner - 2016-09-27
last reply Indexed registers new - Agner - 2016-09-28
last replythread Indexed registers new - Hubert Lamontagne - 2016-09-28
last replythread Indexed registers new - Kurt Baumgardner - 2016-10-03
reply Indexed registers new - Agner - 2016-10-03
last replythread Indexed registers new - Hubert Lamontagne - 2016-10-04
last replythread Bilinear Interpolation new - Hubert Lamontagne - 2016-10-28
last replythread Bilinear Interpolation new - Agner - 2016-10-29
last replythread Bilinear Interpolation new - Hubert Lamontagne - 2016-10-29
last replythread Bilinear Interpolation new - Agner - 2016-10-30
last reply Bilinear Interpolation new - Hubert Lamontagne - 2016-10-30
replythread ForwardCom version 1.04 new - Agner - 2016-12-08
replythread ForwardCom version 1.04 new - Matthias Bentrup - 2016-12-12
last replythread ForwardCom version 1.04 new - Agner - 2016-12-12
last reply ForwardCom version 1.04 new - Matthias Bentrup - 2016-12-14
last replythread Async system calls; horizontal packing instruction new - Joe Duarte - 2016-12-14
reply Async system calls; horizontal packing instruction new - Agner - 2016-12-15
last replythread Comparison of instruction sets new - Agner - 2016-12-17
replythread Comparison of instruction sets new - Joe Duarte - 2016-12-28
reply Comparison of instruction sets new - Agner - 2016-12-29
last reply Comparison of instruction sets new - Hubert Lamontagne - 2016-12-30
last reply Comparison of instruction sets new - Hubert Lamontagne - 2017-01-05
replythread ForwardCom version 1.05 new - Agner - 2017-01-22
replythread Syscall/ISR acceleration new - Jonathan Brandmeyer - 2017-01-22
last replythread Syscall/ISR acceleration new - Agner - 2017-01-23
last replythread Syscall/ISR acceleration new - Jonathan Brandmeyer - 2017-01-25
last reply Syscall/ISR acceleration new - Agner - 2017-01-25
replythread ForwardCom version 1.05 new - Jiří Moravec - 2017-01-23
last reply ForwardCom version 1.05 new - Agner - 2017-01-24
last replythread Jump prefetch? new - csdt - 2017-01-27
last replythread Jump prefetch? new - Agner - 2017-01-27
last replythread Jump prefetch? new - csdt - 2017-01-30
last replythread Jump prefetch? new - Agner - 2017-01-30
last replythread Jump prefetch? new - csdt - 2017-01-30
replythread Jump prefetch? new - Agner - 2017-01-31
reply Jump prefetch? new - csdt - 2017-01-31
last replythread Jump prefetch? new - Hubert Lamontagne - 2017-02-01
last replythread Jump prefetch? new - Agner - 2017-02-01
last replythread Jump prefetch? new - Hubert Lamontagne - 2017-02-01
last replythread Jump prefetch? new - Agner - 2017-02-02
last reply Jump prefetch? new - Agner - 2017-02-14
last replythread Jump prefetch? new - Hubert Lamontagne - 2017-01-31
last replythread High precision arithmetic new - fanoI - 2017-03-21
last reply High precision arithmetic new - Agner - 2017-03-21
replythread Intel's Control-flow Enforcement Technology new - Joe Duarte - 2017-04-13
last reply Intel's Control-flow Enforcement Technology new - Agner - 2017-04-14
reply Proposal for instruction set - now on Github new - Agner - 2017-04-27
replythread Assembler with metaprogramming features new - Agner - 2017-07-27
last replythread Assembler with metaprogramming features new - Kai Rese - 2017-08-11
last replythread Assembler with metaprogramming features new - Agner - 2017-08-11
last replythread Assembler with metaprogramming features new - Kai Rese - 2017-08-14
last replythread Assembler with metaprogramming features new - Agner - 2017-08-14
last reply Assembler with metaprogramming features new - Kai Rese - 2017-08-15
replythread Number of register file ports in implementations new - Hubert Lamontagne - 2017-08-22
last replythread Number of register file ports in implementations new - Agner - 2017-08-23
last replythread Number of register file ports in implementations new - Hubert Lamontagne - 2017-08-27
last replythread Number of register file ports in implementations new - Agner - 2017-08-28
reply Number of register file ports in implementations new - Bigos - 2017-08-28
last reply Number of register file ports in implementations new - Hubert Lamontagne - 2017-08-28
last replythread Proposal for instruction set - now on Github new - yeengief - 2017-09-20
replythread Proposal for instruction set - now on Github new - Agner - 2017-09-20
last replythread Proposal for instruction set - now on Github new - yeengief - 2017-09-20
last replythread Proposal for instruction set - now on Github new - Agner - 2017-09-20
last replythread Proposal for instruction set - now on Github new - yeengief - 2017-09-21
last replythread Proposal for instruction set - now on Github new - Agner - 2017-09-21
last replythread Proposal for instruction set - now on Github new - yeengief - 2017-09-21
last reply Proposal for instruction set - now on Github new - Agner - 2017-09-23
replythread Proposal for instruction set - now on Github new - - - 2017-09-22
last reply Proposal for instruction set - now on Github new - Agner - 2017-09-23
last replythread Proposal for instruction set - now on Github new - Hubert Lamontagne - 2017-09-25
last replythread Proposal for instruction set - now on Github new - Agner - 2017-09-26
last reply Proposal for instruction set - now on Github new - Hubert Lamontagne - 2017-09-26