Low XMR-STAK hash rate on two cards in a 8 card Linux NVIDIA rig

Question

Recently I switched from windows to ubuntu server 16 in order to support more than 8 nvidia cards. However before even getting to install more than 8 cards, I see some issues with the hash rate. (running XMR-STAK, latest. CUDA 9.0, latest. Latest NVIDIA drivers, Ubuntu server 16 latest build, clean install) Check out the 60 seconds and 15 minutes averages. Ran it overnight and the problem persists.

As you can see, cards 0 and 2 (1060 and 1070) are woefully underperforming, even compared to identical cards on the same rig that were ordered at the same time and are the same age. Strangely, in windows, this problem does not occur.

My config.txt:

"cpu_threads_conf" : [
],

"use_slow_memory" : "no_mlck",

"nicehash_nonce" : false,

"aes_override" : null,

"use_tls" : false,
"tls_secure_algo" : false,
"tls_fingerprint" : "",

"pool_list": [
    {
        "pool_address" : "nyc01.supportxmr.com:7777",
        "wallet_address" : "omitted",
        "pool_password" : "omitted",
        "use_nicehash" : false,
        "use_tls" : false,
        "tls_fingerprint": "",
        "pool_weight": 640
    }
],

"call_timeout" : 2,
"retry_time" : 2,
"giveup_limit" : 0,

"verbose_level" : 4,
"print_motd" : false,

"h_print_time" : 30,

"daemon_mode" : true,

"output_file" : "",
"flush_stdout" : false,

"httpd_port" : 0,
"http_login" : "",
"http_pass" : "",

"prefer_ipv4" : true,
"currency" : "monero",

My nvidia.txt: (I tried with default settings as well, these are modified settings but it does not help at all)

    "gpu_threads_conf" :
[
  // gpu: GeForce GTX 1070 architecture: 61
  //      memory: 8013/8114 MiB
  //      smx: 15
  { "index" : 0,
    "threads" : 10, "blocks" : 120,
    "bfactor" : 0, "bsleep" :  0,
    "affine_to_cpu" : false,
  },
  // gpu: GeForce GTX 1060 6GB architecture: 61
  //      memory: 5993/6072 MiB
  //      smx: 10
  { "index" : 1,
    "threads" : 8, "blocks" : 120,
    "bfactor" : 0, "bsleep" :  0,
    "affine_to_cpu" : false,
  },
  // gpu: GeForce GTX 1060 6GB architecture: 61
  //      memory: 5993/6072 MiB
  //      smx: 10
  { "index" : 2,
    "threads" : 8, "blocks" : 120,
    "bfactor" : 0, "bsleep" :  0,
    "affine_to_cpu" : false,
  },
  // gpu: GeForce GTX 1070 architecture: 61
  //      memory: 8013/8114 MiB
  //      smx: 15
  { "index" : 3,
    "threads" : 10, "blocks" : 120,
    "bfactor" : 0, "bsleep" :  0,
    "affine_to_cpu" : false,
  },
  // gpu: GeForce GTX 1070 architecture: 61
  //      memory: 8013/8114 MiB
  //      smx: 15
  { "index" : 4,
    "threads" : 10, "blocks" : 120,
    "bfactor" : 0, "bsleep" :  0,
    "affine_to_cpu" : false,
  },
  // gpu: GeForce GTX 1060 6GB architecture: 61
  //      memory: 5993/6072 MiB
  //      smx: 10
  { "index" : 5,
    "threads" : 8, "blocks" : 120,
    "bfactor" : 0, "bsleep" :  0,
    "affine_to_cpu" : false,
  },
  // gpu: GeForce GTX 1060 6GB architecture: 61
  //      memory: 5993/6072 MiB
  //      smx: 10
  { "index" : 6,
    "threads" : 8, "blocks" : 120,
    "bfactor" : 0, "bsleep" :  0,
    "affine_to_cpu" : false,
  },
  // gpu: GeForce GTX 1070 architecture: 61
  //      memory: 8013/8114 MiB
  //      smx: 15
  { "index" : 7,
    "threads" : 10, "blocks" : 120,
    "bfactor" : 0, "bsleep" :  0,
    "affine_to_cpu" : false,
  },

],

Does anyone have any tips to what could be causing this or potential things I should try?

Thank you for any help!

Answer 1

It's been some time since last time i mined with Nvidia GPUs, but let's try with some tips.

First of all, there are 2 versions of the last xmr-stak (2.0 and 2.1). I tried both (amd) and had slighlty better results from 2.0. I encourage you to try both.

Let's recommend you some settings:

-About blocks: take note of the number of SMX of your cards (10 for 1060 and 15 for 1070). The default first config run by xmr-stak its 3 times smx (30 for 1060 and 45 for 1070). If you are fine tuning, you want to increase this value smx times value each test (30-40-50 and so on for 1060; 45-60-75 and so on for 1070). blocks value must be multiple of smx.

-About threads: all i have to say about threads its that they have to be multiple of 8 (8,16,24,32 and so on).

Ok, being explained that, i can recommend you, if you are using, lets say GPU0 for display, to put some bfactor (8) and bsleep (100), just in the card its displaying image. You can do this for testing purposes, if you dont notice improvement and/or stability, just leave it both 0.

Before proceeding on testing, please leave all cards on stock settings (clocks and voltage), and test fine tuning EACH CARD AND EACH SETTING (B or T) per test, because if you fine tune more than 1, it will be hard to know which one is crashing if the system crashes.

Some good combinations? you have to try by yourself. I have seen 8x160 and/or 80x30 gtx 1060s doing over 500H. Those values complies with the rules above explained. For the 1070s you can start with 8x45 and going up with blocks in increments of 15 (smx value) each test.

Last, but not least important, remember VRAM consumption and 2Mb rule. If you want to know how much RAM will it cost the card with each settings, multiply B*T*2=VRAM used. So, a 80x30 value in a gtx 1060 will use about 80*30*2= 4.8GB of VRAM. You cannot run a setting with more than available GPU VRAM, it wont work.

Remember to document each result, to ensure you can max out your perfomance. Fine tuning is the most beautiful and time consuming activity when mining. Im pretty sure you will have so much fun and will spend a good chunk of time fine tuning each card.

Last recomendation: start with "threads 8", fine tune blocks number accordly until you find the limit (unstable mining and/or crash or GPU VRAM limit). Take note of the last good value, both for settings and hashrate. After you have managed to do so try with "threads 16", rinse and repeat. It can be the case that your card likes a middle value instead of so much blocks or threads.

Its a lil long post, but i think i have include most basics here. I dont get tired of this: TEST EACH CARD, dont think that because you found a good setting in one card, lets say, a GTX 1070, it will work in another GTX 1070 just fine. Nothing more wrong about that. There is something called "silicon lottery".

Once you have found nice limits in all of your cards, proceed with overclock/undervolt or w/e you want to do in smi. Again, for EACH CARD separately. never assume a setting will work in another GPU, even if they are twins. This guide works with every NVIDIA card. I hope this helps. Kind regards...