Repository navigation
sudden crash on RTX 3070 for long task #2212
Description
Thank you Niko1221-san for creating such a wonderful tool !!!
I've been using this for a week now.
Untill yesterday, I was running small errands such as small coding and OCR and translation tasks.
Yesterday, i made it make batch for translation task for about 2,000 documents, not one whole document at once per iteration but small chunk of text, as small as surrounding context is retained.
Sample tests went smoothly all perfection, but when it came to running batch execution for all documents, it crashes for unknown reason.
By unknown reason, I mean I looked up log files under /var/log and looked at nvidia-smi query for any clues, only thing I found was graphic board gets lower voltage on nvidia-smi before crash.
And by crash, what actually happens is i access this machine through ssh and run the task under tmux session and when this problem occurs it kicks me out of ssh and tmux sesion is gets killed.
It seems as if it went on reboot, but i can ssh back in asap as i get kicked out.
I tried several times but it occurs at uncertain points.
it ouccred in the variant time-window; 10 mins, some longer minuts, over 1hr.
It does not occur on certain translation-target document since it can pick up where it left and continues.
There was no OOM error or kernel panic or anything in the log for around the time this incident occured.
my setup is the following.
CPU: AMD Ryzen 9 3950X 16-Core Processor
MEM: DDR4 128GB
Graphic: NVIDIA GeForce RTX 3070 8GB
Disk Space:
$ df -Th | grep -iE "^/dev"
/dev/nvme0n1p2 ext4 916G 455G 414G 53% /
/dev/nvme1n1p1 ext4 234G 772M 221G 1% /media/XXXXXX
/dev/sda ext4 916G 577G 293G 67% /media/YYYYYY
$ cat /etc/os-release
PRETTY_NAME="Ubuntu 24.04.5 LTS"
NAME="Ubuntu"
VERSION_ID="24.04"
VERSION="24.04.5 LTS (Noble Numbat)"Strata is on /dev/sda .
Tested on both /dev/nvme1n1p1 and /dev/nvme0n1p2 diskspaces but same crash occurs.
my setup.sh command is the following
./setup.sh --setup --host 0.0.0.0 --api-key "XXXXXXXX" --model IQ3_XXS --context 65536 --vision cpu --yesIs there anything anyone who can think of what might be the cause of this issue ?
here is a part of the nvidia-smi log around the time crash occured.
timestamp, temperature.gpu, power.draw [W], utilization.gpu [%], memory.used [MiB], clocks.current.sm [MHz], clocks_event_reasons.active
2026/10/13 01:31:31.783, 69, 144.96 W, 100 %, 7559 MiB, 2010 MHz, 0x0000000000000000
2026/10/13 01:31:36.784, 69, 157.08 W, 100 %, 7559 MiB, 2010 MHz, 0x0000000000000000
2026/10/13 01:31:41.785, 70, 154.16 W, 100 %, 7559 MiB, 2010 MHz, 0x0000000000000000
2026/10/13 01:31:46.785, 70, 155.40 W, 100 %, 7559 MiB, 2010 MHz, 0x0000000000000000
2026/10/13 01:31:51.786, 70, 160.50 W, 100 %, 7559 MiB, 2010 MHz, 0x0000000000000000
2026/10/13 01:31:56.787, 69, 145.12 W, 93 %, 7559 MiB, 2010 MHz, 0x0000000000000000
2026/10/13 01:32:01.789, 69, 145.24 W, 100 %, 7559 MiB, 2010 MHz, 0x0000000000000000
2026/10/13 01:32:06.790, 64, 74.88 W, 100 %, 7559 MiB, 2025 MHz, 0x0000000000000000
2026/10/13 01:32:11.792, 62, 73.98 W, 100 %, 7559 MiB, 2025 MHz, 0x0000000000000000
2026/10/13 01:32:16.793, 61, 73.22 W, 100 %, 7559 MiB, 2025 MHz, 0x0000000000000000
2026/10/13 01:32:21.794, 61, 72.68 W, 100 %, 7559 MiB, 2040 MHz, 0x0000000000000024
2026/10/13 01:32:26.796, 60, 72.37 W, 100 %, 7559 MiB, 2040 MHz, 0x0000000000000000
2026/10/13 01:32:31.797, 59, 71.56 W, 100 %, 7559 MiB, 2040 MHz, 0x0000000000000000
2026/10/13 01:32:36.798, 59, 70.84 W, 100 %, 7559 MiB, 2040 MHz, 0x0000000000000000
Thank you in advance !!
Activity
Niko1221 commentedon Oct 13, 2026
Thanks @HiroshiMatsumoto. A program in user space cannot reboot a Linux machine, and an ssh session dropped together with tmux, with no OOM and no panic in the logs, is what a hard reset or power loss looks like. Together with the voltage drop you saw on the card, power (the PSU or the GPU's power delivery under a long sustained load) is the most likely cause. Some checks:
last -x | headandjournalctl --list-boots. Then look at the boot before the crash:journalctl -b -1 -k | tail -100. An NVIDIAXidline (journalctl -b -1 -k | grep -i xid) or a sudden end with no shutdown messages tells the two cases apart.sudo nvidia-smi -pl 180(the 3070 defaults to 220 W; the setting resets at reboot), then run the same batch. If it survives with a lower limit, it is power.nvidia-smi -q -d POWERshows the limits.--pcie-frac(for example 0.2) or--pool-workers 8reduces the combined CPU + GPU draw. If that stops the reboots, it points the same way.If
journalctl -b -1shows something before the reset (Xid, MCE, thermal), please post it here and we will look again.