[georgeguimaraes]

A compiled function with a deep graph stops the VM with SIGBUS on macOS:

chain = fn x ->
  Enum.reduce(1..10_000, x, fn _, acc -> acc |> Nx.sin() |> Nx.add(0.001) end)
end

Nx.Defn.jit(chain, compiler: EMLX).(Nx.tensor(0.5, backend: EMLX.Backend))

MLX compiles a graph with compile_dfs, which recurses once for each level of the graph. The program compiles on the worker thread, and std::thread gets the default stack of a secondary thread. On macOS that stack is 512 KB, and a chain of 10,000 ops overflows it. The crash report shows the fault in mlx::core::detail::compile_dfs on the worker thread. I found this when an ONNX model with an unrolled LSTM over audio frames crashed for long inputs.

With this change, the worker starts with pthread_create and a 64 MB stack. Stack use is about 200 bytes per level, so a chain of 200,000 ops compiles, and 400,000 still overflows. 64 MB is huge for a stack, but it is address space, not memory. The OS maps a stack page only when the thread first uses it. A worker that never compiles a deep graph uses only the few pages it needs, and a deep compile uses only the pages for the depth that it reaches. On Linux, glibc threads usually get 8 MB, so the limit goes up there too.

The test compiles the chain above, 20,000 ops. It stops with SIGBUS when I set the stack back to 512 KB.

A compile_dfs without recursion in MLX would remove the limit. This change makes the graphs that people build today safe.