Skip to content

sudden crash on RTX 3070 for long task #2212

Open

Description

@HiroshiMatsumoto

Thank you Niko1221-san for creating such a wonderful tool !!!

I've been using this for a week now.
Untill yesterday, I was running small errands such as small coding and OCR and translation tasks.
Yesterday, i made it make batch for translation task for about 2,000 documents, not one whole document at once per iteration but small chunk of text, as small as surrounding context is retained.
Sample tests went smoothly all perfection, but when it came to running batch execution for all documents, it crashes for unknown reason.
By unknown reason, I mean I looked up log files under /var/log and looked at nvidia-smi query for any clues, only thing I found was graphic board gets lower voltage on nvidia-smi before crash.
And by crash, what actually happens is i access this machine through ssh and run the task under tmux session and when this problem occurs it kicks me out of ssh and tmux sesion is gets killed.
It seems as if it went on reboot, but i can ssh back in asap as i get kicked out.
I tried several times but it occurs at uncertain points.

it ouccred in the variant time-window; 10 mins, some longer minuts, over 1hr.
It does not occur on certain translation-target document since it can pick up where it left and continues.

There was no OOM error or kernel panic or anything in the log for around the time this incident occured.

my setup is the following.
CPU: AMD Ryzen 9 3950X 16-Core Processor
MEM: DDR4 128GB
Graphic: NVIDIA GeForce RTX 3070 8GB
Disk Space:

$ df -Th | grep -iE "^/dev"
/dev/nvme0n1p2 ext4      916G  455G  414G  53% /
/dev/nvme1n1p1 ext4      234G  772M  221G   1% /media/XXXXXX
/dev/sda       ext4      916G  577G  293G  67% /media/YYYYYY
$ cat /etc/os-release
PRETTY_NAME="Ubuntu 24.04.5 LTS"
NAME="Ubuntu"
VERSION_ID="24.04"
VERSION="24.04.5 LTS (Noble Numbat)"

Strata is on /dev/sda .
Tested on both /dev/nvme1n1p1 and /dev/nvme0n1p2 diskspaces but same crash occurs.

my setup.sh command is the following

./setup.sh  --setup --host 0.0.0.0 --api-key "XXXXXXXX" --model IQ3_XXS --context 65536 --vision cpu --yes

Is there anything anyone who can think of what might be the cause of this issue ?

here is a part of the nvidia-smi log around the time crash occured.

timestamp, temperature.gpu, power.draw [W], utilization.gpu [%], memory.used [MiB], clocks.current.sm [MHz], clocks_event_reasons.active
2026/10/13 01:31:31.783, 69, 144.96 W, 100 %, 7559 MiB, 2010 MHz, 0x0000000000000000
2026/10/13 01:31:36.784, 69, 157.08 W, 100 %, 7559 MiB, 2010 MHz, 0x0000000000000000
2026/10/13 01:31:41.785, 70, 154.16 W, 100 %, 7559 MiB, 2010 MHz, 0x0000000000000000
2026/10/13 01:31:46.785, 70, 155.40 W, 100 %, 7559 MiB, 2010 MHz, 0x0000000000000000
2026/10/13 01:31:51.786, 70, 160.50 W, 100 %, 7559 MiB, 2010 MHz, 0x0000000000000000
2026/10/13 01:31:56.787, 69, 145.12 W, 93 %, 7559 MiB, 2010 MHz, 0x0000000000000000
2026/10/13 01:32:01.789, 69, 145.24 W, 100 %, 7559 MiB, 2010 MHz, 0x0000000000000000
2026/10/13 01:32:06.790, 64, 74.88 W, 100 %, 7559 MiB, 2025 MHz, 0x0000000000000000
2026/10/13 01:32:11.792, 62, 73.98 W, 100 %, 7559 MiB, 2025 MHz, 0x0000000000000000
2026/10/13 01:32:16.793, 61, 73.22 W, 100 %, 7559 MiB, 2025 MHz, 0x0000000000000000
2026/10/13 01:32:21.794, 61, 72.68 W, 100 %, 7559 MiB, 2040 MHz, 0x0000000000000024
2026/10/13 01:32:26.796, 60, 72.37 W, 100 %, 7559 MiB, 2040 MHz, 0x0000000000000000
2026/10/13 01:32:31.797, 59, 71.56 W, 100 %, 7559 MiB, 2040 MHz, 0x0000000000000000
2026/10/13 01:32:36.798, 59, 70.84 W, 100 %, 7559 MiB, 2040 MHz, 0x0000000000000000

Thank you in advance !!

Activity

  1. Niko1221 commented on Oct 13, 2026

    @Niko1221
    Owner

    Thanks @HiroshiMatsumoto. A program in user space cannot reboot a Linux machine, and an ssh session dropped together with tmux, with no OOM and no panic in the logs, is what a hard reset or power loss looks like. Together with the voltage drop you saw on the card, power (the PSU or the GPU's power delivery under a long sustained load) is the most likely cause. Some checks:

    1. Did it reboot? last -x | head and journalctl --list-boots. Then look at the boot before the crash: journalctl -b -1 -k | tail -100. An NVIDIA Xid line (journalctl -b -1 -k | grep -i xid) or a sudden end with no shutdown messages tells the two cases apart.
    2. Cap the card's power. sudo nvidia-smi -pl 180 (the 3070 defaults to 220 W; the setting resets at reboot), then run the same batch. If it survives with a lower limit, it is power. nvidia-smi -q -d POWER shows the limits.
    3. The PSU. A 3950X under all-core load plus the 3070 at its transient peaks needs a good 650-750 W unit. Spikes well above the average power can trip an older or smaller unit. Try another PSU if you can, or check the cables (separate 8-pin cables, not one daisy-chained).
    4. Lighter load from Strata's side: a lower --pcie-frac (for example 0.2) or --pool-workers 8 reduces the combined CPU + GPU draw. If that stops the reboots, it points the same way.

    If journalctl -b -1 shows something before the reset (Xid, MCE, thermal), please post it here and we will look again.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions