performance in sched_yield (multisocket)

[phirus2]

Hi!!!
I'm using mumps with openblas 0.2.18 compiled from sources with this options:

• gcc 5.3.0
• USE_THREAD = 1
• NUM_THREADS = 128
• NO_WARMUP = 1
• #NO_AFFINITY = 1
• #BIGNUMA = 1
• MAX_STACK_ALLOC = 8128

I'm running my code in some machines multisocket (2 or 4) with 24 until 128 cores

The problem is that most of the time is executed by the system. To be more acurate is doing not I/O is doing Context switch. You can see it in the below pictures

This doesn't happend in 1 socket machine, but if I force to use just one CPU (taskset -c 0-31 MyAPP) the performance is also poor.

What can I do to give you more information and try to help¿¿¿

[brada4]

How big is data sample? What CPU? Maybe all the time is spent ripping tiny data sample between CPU caches or NUMA nodes?

[brada4]

sched_yield is that descheduling your flawed automated debugger is recommending.
man sched_yield ?

[jeromerobert]

@phirus2 What is are the size of the GEMM ? Could you get a sample of them with adding a trace in interface/gemm.c or with a debugger ?

MUMPS is a sparse solver so he is probably working on small blocks on which you can't get any benefits from the OpenBLAS multi-threading. Anyway if blocks are small we should not even enter in blas_thread_server so that's odd.

[brada4]

There is no 32-core intel CPU being manufactured
Results are fake at best. There is no issue.

[phirus2]

hwloc-info
depth 0:        1 Machine (type #1)
 depth 1:       4 NUMANode (type #2)
  depth 2:      4 Socket (type #3)
   depth 3:     4 L3Cache (type #4)
    depth 4:    64 L2Cache (type #4)
     depth 5:   64 L1dCache (type #4)
      depth 6:  64 L1iCache (type #4)
       depth 7: 64 Core (type #5)
        depth 8:        128 PU (type #6)

cat /proc/cpuinfo | grep -i "model name" -B4 | tail -n5
processor       : 127
vendor_id       : GenuineIntel
cpu family      : 6
model           : 63
model name      : Intel(R) Xeon(R) CPU E7-8860 v3 @ 2.20GHz

model CPU

@brada4 As you can see it is not a fake CPU.

@jeromerobert Ok i'm execute a bigger problem and give you the sizes of my sistem to analize what is happening

[phirus2]

I'm resolving an sparse matriz equation with this sizes:

• A ==> [(50_50_50) x (50_50_50) ] == [125k x 125k]. For each row there is 7 nz value. So the matriz has aprox 7MB
• B /X ==> [50_50_50] == [125k] ~= 1MB

But internally is doing LU decomposition and I guess it' is using dens matriz. because the total ram used by the program is ~1GB. Changing the size to 100x100x100 the total size of the problem went up to ~15GB. I think is enough big to share avoid that context switch

The profiler info of openBlas is

        ====== BLAS Profiling Result =======

  Function      No. of Calls   Time Consumption   Efficiency  Bytes/cycle  Wall Time(Cycles)
idamax        :    5306040            0.19%           0.000%      2.75   875257291
dgemm         :    9533360           91.66%           2.334%      0.85   353009864
dtrsm         :    6395160            8.15%           3.402%      0.75   -425599732
 --------------------------------------------------------------------
Total         :   21234560                            2.416%      0.85

The extra info of mumps is ....

 =================================================
 MUMPS compiled with option -Dmetis
 =================================================
L U Solver for unsymmetric matrices
Type of parallelism: Working host

 ****** ANALYSIS STEP ********

 Resetting candidate strategy to 0 because NSLAVES=1

 ... Structural symmetry (in percent)=   98
 ... No column permutation
 Ordering based on METIS
 ELAPSED TIME SPENT IN METIS reordering  =      0.9891

Leaving analysis phase with  ...
INFOG(1)                                       =               1
INFOG(2)                                       =          208155
 -- (20) Number of entries in factors (estim.) =        85425349
 --  (3) Storage of factors  (REAL, estimated) =        85425349
 --  (4) Storage of factors  (INT , estimated) =         2406061
 --  (5) Maximum frontal size      (estimated) =            3644
 --  (6) Number of nodes in the tree           =           52849
 -- (32) Type of analysis effectively used     =               1
 --  (7) Ordering option effectively used      =               5
ICNTL(6) Maximum transversal option            =               0
ICNTL(7) Pivot order option                    =               7
Percentage of memory relaxation (effective)    =              20
Number of level 2 nodes                        =               0
Number of split nodes                          =               0
RINFOG(1) Operations during elimination (estim)=   1.449D+11
 ** Rank of proc needing largest memory in IC facto        :         0
 ** Estimated corresponding MBYTES for IC facto            :       964
 ** Estimated avg. MBYTES per work. proc at facto (IC)     :       964
 ** TOTAL     space in MBYTES for IC factorization         :       964
 ** Rank of proc needing largest memory for OOC facto      :         0
 ** Estimated corresponding MBYTES for OOC facto           :       278
 ** Estimated avg. MBYTES per work. proc at facto (OOC)    :       278
 ** TOTAL     space in MBYTES for OOC factorization        :       278
 ELAPSED TIME IN ANALYSIS DRIVER=       1.2585
Entering DMUMPS 5.0.1 driver with JOB, N, NZ =   5      132651        1061208

 ****** FACTORIZATION STEP ********


 GLOBAL STATISTICS PRIOR NUMERICAL FACTORIZATION ...
 NUMBER OF WORKING PROCESSES              =             1
 OUT-OF-CORE OPTION (ICNTL(22))           =             0
 REAL SPACE FOR FACTORS                   =      85425349
 INTEGER SPACE FOR FACTORS                =       2406061
 MAXIMUM FRONTAL SIZE (ESTIMATED)         =          3644
 NUMBER OF NODES IN THE TREE              =         52849
 MEMORY ALLOWED (MB -- 0: N/A )           =             0
 Convergence error after scaling for ONE-NORM (option 7/8)   = 0.85D+00
 Maximum effective relaxed size of S              =     116410103
 Average effective relaxed size of S              =     116410103
 GLOBAL TIME FOR MATRIX DISTRIBUTION       =      0.0203
 ** Memory relaxation parameter ( ICNTL(14)  )            :        20
 ** Rank of processor needing largest memory in facto     :         0
 ** Space in MBYTES used by this processor for facto      :       964
 ** Avg. Space in MBYTES per working proc during facto    :       964

 ELAPSED TIME FOR FACTORIZATION           =      7.0425
 Maximum effective space used in S   (KEEP8(67))  =      97008403
 Average effective space used in S   (KEEP8(67))  =      97008403
 ** EFF Min: Rank of processor needing largest memory :         0
 ** EFF Min: Space in MBYTES used by this processor   :       805
 ** EFF Min: Avg. Space in MBYTES per working proc    :       805

 GLOBAL STATISTICS
 RINFOG(2)  OPERATIONS IN NODE ASSEMBLY   = 1.642D+08
 ------(3)  OPERATIONS IN NODE ELIMINATION= 1.449D+11
 INFOG (9)  REAL SPACE FOR FACTORS        =      85425349
 INFOG(10)  INTEGER SPACE FOR FACTORS     =       2406061
 INFOG(11)  MAXIMUM FRONT SIZE            =          3644
 INFOG(29)  NUMBER OF ENTRIES IN FACTORS  =      85425349
 INFOG(12)  NUMBER OF OFF DIAGONAL PIVOTS =            15
 INFOG(13)  NUMBER OF DELAYED PIVOTS      =             0
 INFOG(14)  NUMBER OF MEMORY COMPRESS     =             0
 ELAPSED TIME IN FACTORIZATION DRIVER=       7.0875


 ****** SOLVE & CHECK STEP ********


 STATISTICS PRIOR SOLVE PHASE     ...........
 NUMBER OF RIGHT-HAND-SIDES                    =           1
 BLOCKING FACTOR FOR MULTIPLE RHS              =           1
 ICNTL (9)                                     =           1
  --- (10)                                     =           0
  --- (11)                                     =           0
  --- (20)                                     =           0
  --- (21)                                     =           0
  --- (30)                                     =           0
 ** Rank of processor needing largest memory in solve     :         0
 ** Space in MBYTES used by this processor for solve      :       961
 ** Avg. Space in MBYTES per working proc during solve    :       961

 TIME to scatter right-hand-sides =       0.002222
 TIME in solution step (fwd/bwd)  =       0.383480
  .. TIME in forward (fwd) step   =          0.237609
  .. TIME in backward (bwd) step  =          0.145850
 TIME to gather solution          =       0.001802
 ELAPSED TIME IN SOLVE DRIVER=       0.4491
Entering DMUMPS 5.0.1 driver with JOB, N, NZ =   5      132651        1061208

.

[jeromerobert]

@phirus2 I'm not sure how to deduce the size (m x n) of the gemm blas calls from this information. I (we ?) don't the MUMPS internal and don't know how it use GEMM. You have to get this information from OpenBLAS it self with adding trace or using gdb.

[martin-frbg]

Do you see an improvement when you limit the number of threads to the count of full-featured cores ? Hyperthreading is not going to help for compute-intensive tasks, what you see could be each two threads competing for access to the floating-point hardware.

[phirus2]

I thought that but decreasing the number of threads (64, 32, 16) the performance efficiency is the same. Even fixing to same cores using taskset to avoid the context switch. But I still have the poor performance efficiency.

@jeromerobert have modified the src to print the headers of dgemm. First of all... there are soooo many calls in just one iteration of mumps solver (110k) and seems that many of them are matrix of ¿1x1?

TRANSA:N  TRANSB:N  M:1  N:1  K:1 alpha:-1.000000 a:-0.140344  ldA:2 b:1.572134 ldb:2  beta:1.000000  c:0.000000  ldc:2
TRANSA:N  TRANSB:N  M:1  N:1  K:1 alpha:-1.000000 a:-0.141675  ldA:2 b:1.583998 ldb:2  beta:1.000000  c:0.000000  ldc:2
TRANSA:N  TRANSB:N  M:1  N:1  K:1 alpha:-1.000000 a:-0.142773  ldA:2 b:1.593849 ldb:2  beta:1.000000  c:0.000000  ldc:2
TRANSA:N  TRANSB:N  M:1  N:1  K:1 alpha:-1.000000 a:-0.142272  ldA:2 b:1.589344 ldb:2  beta:1.000000  c:0.000000  ldc:2
TRANSA:N  TRANSB:N  M:1  N:1  K:1 alpha:-1.000000 a:-0.140708  ldA:2 b:1.575375 ldb:2  beta:1.000000  c:0.000000  ldc:2
TRANSA:N  TRANSB:N  M:1  N:1  K:1 alpha:-1.000000 a:-0.140055  ldA:2 b:1.569573 ldb:2  beta:1.000000  c:0.000000  ldc:2
TRANSA:N  TRANSB:N  M:1  N:1  K:1 alpha:-1.000000 a:-0.142308  ldA:2 b:1.589665 ldb:2  beta:1.000000  c:0.000000  ldc:2

Actually there are:

• ~45k calls with N * M == 1
• ~71k calls with N * M < 36
• ~100k calls with N * M < 1000
• ~10k calls with 1000<N*M<10000
• ~2k calls with 10000<N*M<244000

So the problem is of mumps an his partitioning????

[jeromerobert]

So the problem is of mumps an his partitioning????

Yes and that was expected. I don't think any sparse solver could rely on BLAS for parallelization. OpenBLAS dgemm will only enable threading when M* N * K > 8192 * GEMM_MULTITHREAD_THRESHOLD (= 32768 most of the time). So in your case very few of them will be run in parallel.

The problem is that in OpenBLAS does not select the number of threads very smartly. If you have a matrix with M_N_K = 32769 it will run on all the core. In your case it will be very inefficient because 32769 is much to small for 64 cores. I guess that even 244000 is much too small for 64 cores. That's why sched_yield takes so much time in your profiling. Too be more efficient we should select the right number of threads between 1 and 64 depending on the size of the matrix (ex: 4 or 6 for MNK=24400). But this is not implemented.

To get good performance with MUMPS you should use MPI not multi-threading. If you absolutely want a shared memory solution, try PaStiX.

[brada4]

You have 4 10-core cpus
7mb is microscopic, best results are in 10 cores and less

[brada4]

45k calls to m.n=1 should not call gemm but gemv instead

[phirus2]

Absolutely agree. Thousands of gemm with 1x1 'matrix' is a mistake.

Thanks for the advise and the explanation of the limits of parallelization. I will take a look to that library pastix and also report the results to mumps

[brada4]

You can see NUMA stride in sysctl. You need at least a stride per multicore CPU in node, never less