Ramp of lanes <= 1 exception when cross-compile Halide for RISC-V

[dkurt]

Tried tutorial https://halide-lang.org/tutorials/tutorial_lesson_11_cross_compilation.html on x86 host machine (Ubuntu 20.04) but for RISC-V target architecture. Unfortunately, there is an exception from IR.cpp. The exception appears only when .vectorize(x, 16) is applied, so problem is somewhere in RISC-V Vectorization support.

Internal Error at /home/dkurt/Halide/src/IR.cpp:272 triggered by user code at : Condition failed: lanes > 1: Ramp of lanes <= 1

Everything is OK locally for x86 target.

• Halide @ 16243da
• LLVM 15.0.2

Code:

#include "Halide.h"

using namespace Halide;

int main(int argc, char** argv) {
    Func brighter;
    Var x, y;

    Param<uint8_t> offset;
    ImageParam input(type_of<uint8_t>(), 2);
    std::vector<Argument> args(2);
    args[0] = input;
    args[1] = offset;

    brighter(x, y) = input(x, y) + offset;

    brighter.bound(x, 0, 64).bound(y, 0, 64);

    brighter.vectorize(x, 16).parallel(y);

    Target target;
    target.os = Target::Linux;
    target.arch = Target::RISCV;
    target.bits = 64;

    std::vector<Target::Feature> features;
    features.push_back(Target::RVV);
    target.set_features(features);

    try {
        brighter.compile_to_static_library("compiled", args, "brighter", target);
    } catch(Halide::InternalError& ex) {
        std::cout << ex.what() << std::endl;
    }
}

Stacktrace:

__GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
50      ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) bt
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1  0x00007ffff14d1859 in __GI_abort () at abort.c:79
#2  0x00007ffff175a911 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
#3  0x00007ffff176638c in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
#4  0x00007ffff17663f7 in std::terminate() () from /lib/x86_64-linux-gnu/libstdc++.so.6
#5  0x00007ffff176637f in std::rethrow_exception(std::__exception_ptr::exception_ptr) () from /lib/x86_64-linux-gnu/libstdc++.so.6
#6  0x00007ffff2f00062 in Halide::Internal::run_with_large_stack(std::function<void ()> const&) (action=...) at /home/dkurt/Halide/src/Util.cpp:793
#7  0x00007ffff20a2b31 in Halide::Internal::CodeGen_LLVM::compile (this=0x5555556acaf0, input=...) at /home/dkurt/Halide/src/CodeGen_LLVM.cpp:566
#8  0x00007ffff209dc94 in Halide::codegen_llvm (module=..., context=...) at /home/dkurt/Halide/src/CodeGen_LLVM.cpp:41
#9  0x00007ffff24c4cd3 in Halide::compile_module_to_llvm_module (module=..., context=...) at /home/dkurt/Halide/src/LLVM_Output.cpp:410
#10 0x00007ffff252a2cb in Halide::Module::compile (this=0x7fffffffdca8, Python Exception <class 'AttributeError'> 'NoneType' object has no attribute 'pointer': 
output_files=std::map with 2 elements) at /home/dkurt/Halide/src/Module.cpp:597
#11 0x00007ffff257049f in Halide::Pipeline::compile_to_static_library (this=0x7fffffffdd40, filename_prefix="compiled", args=std::vector of length 2, capacity 2 = {...}, 
    fn_name="brighter", target=...) at /home/dkurt/Halide/src/Pipeline.cpp:390
#12 0x00007ffff22ba02e in Halide::Func::compile_to_static_library (this=0x7fffffffddd0, filename_prefix="compiled", args=std::vector of length 2, capacity 2 = {...}, 
    fn_name="brighter", target=...) at /home/dkurt/Halide/src/Func.cpp:3325
#13 0x000055555555ab6b in main (argc=1, argv=0x7fffffffe0c8) at /home/dkurt/halide_demo/main.cpp:40

[zvookin]

Mainline RISC-V support is pretty stale. I'll post a branch with the current good state of RISC-V support today and reply back here.

[zvookin]

Branch pushed here: https://github.com/halide/Halide/tree/riscv_update .

[dkurt]

@zvookin, thanks, that solved the problem!

brighter.vectorize(x, 16).parallel(y);

Target target;
target.os = Target::Linux;
target.arch = Target::RISCV;
target.bits = 64;
target.vector_bits = 16 * 8;

[zvookin]

Please let me know if there are any issues. This should allow access to LMUL via specifying vectorizations that are larger or smaller than the number of lanes implied by the vector_bits_* target flag and the data type. Also should support vector reductions and some set of special operations like averaging add. Hoping to get this merged soonish, but the vector predication changes will need a bit of code cleanup first.

[dkurt]

Hi, @zvookin! Tried run on riscv64 host and got "Unsupported CPU type" at RuntimeDyldELF::resolveRelocation method. Am I right that expected Halide usage is only AOT for RISC-V rather than JIT?

https://github.com/llvm/llvm-project/blob/de9d80c1c579e39cc658a508f1d4ba1cd792e4d5/llvm/lib/ExecutionEngine/RuntimeDyld/RuntimeDyldELF.cpp#L1076

Code snippet

#include <iostream>
#include "Halide.h"

using namespace Halide;

int main(int argc, char** argv) {
    Func brighter;
    Var x, y;

    // Declare the arguments.
    Param<uint8_t> offset;
    ImageParam input(type_of<uint8_t>(), 2);
    std::vector<Argument> args(2);
    args[0] = input;
    args[1] = offset;

    // Define the Func.
    brighter(x, y) = input(x, y) + offset;

    // brighter.vectorize(x, 16);

    Target target;
    target.os = Target::Linux;
    target.arch = Target::RISCV;
    target.bits = 64;

    std::cout << target.supported() << std::endl;

    std::vector<Target::Feature> features;
    features.push_back(Target::RVV);
    target.set_features(features);

    Buffer<uint8_t> output(std::vector<int>{2, 64});
    brighter.realize(output, target);
    return 0;
}

Unsupported CPU type!
UNREACHABLE executed at /home/dkurt/llvm-project/llvm/lib/ExecutionEngine/RuntimeDyld/RuntimeDyldELF.cpp:1076!
Aborted

[dkurt]

Also, if it's possible, may I ask which configuration do you use for RVV testing?

I've tried build AOT pipeline with Halide and standalone application which is linked against a generated library by https://github.com/riscv-collab/riscv-gnu-toolchain. Without vectorization, everything is fine, however vectorized scheduling produces Illegal instruction:

Halide code

#include "Halide.h"

using namespace Halide;

int main(int argc, char** argv) {
    Func brighter;
    Var x, y;

    Param<uint8_t> myoffset("myoffset");
    ImageParam input(type_of<uint8_t>(), 2);
    std::vector<Argument> args(2);
    args[0] = input;
    args[1] = myoffset;

    brighter(x, y) = input(x, y) * myoffset;

    brighter.bound(x, 0, 64).bound(y, 0, 64);

    brighter.vectorize(x, 16).parallel(y);

    Target target;
    target.os = Target::Linux;
    target.arch = Target::RISCV;
    target.bits = 64;
    target.vector_bits = 16 * 8;

    std::cout << target << std::endl;

    std::vector<Target::Feature> features;
    features.push_back(Target::RVV);
    target.set_features(features);

    brighter.print_loop_nest();

    try {
        brighter.compile_to_static_library("compiled", args, "brighter", target);
    } catch(Halide::InternalError& ex) {
        std::cout << ex.what() << std::endl;
    }
}

Application

#include <vector>
#include <iostream>

#include "compiled.h"
#include "HalideBuffer.h"

int main(int argc, char** argv) {
    Halide::Runtime::Buffer<uint8_t> input(64, 64), output(64, 64);
    uint8_t offset = 6;

    brighter(input, offset, output);
}

Compilation flags:

/opt/riscv/bin/riscv64-unknown-linux-gnu-g++ runner.cpp compiled.a -lpthread -ldl -I ./ -I ~/halide-install/include/ -o runner
/opt/riscv-rvv/bin/qemu-riscv64 ./runner

[steven-johnson]

Am I right that expected Halide usage is only AOT for RISC-V rather than JIT?

No, we expect JIT to work for all our CPU targets -- I'm gonna guess that the RISC-V code just hasn't been tested (enough) and you've found a bug. :-/

(@zvookin is out this week -- unfortunately I don't know what configuration he's using for testing.)

[dkurt]

Oh, thank you @steven-johnson. Would it be correct if I open a new issue? As this starting one was solved by moving to https://github.com/halide/Halide/tree/riscv_update branch.

Or maybe I could solve it locally, must be something wrong with my QEMU usage.

[steven-johnson]

Would it be correct if I open a new issue?

Yes, please.

[dkurt]

🎉 Solved Illegal instruction problem by using QEMU with -cpu rv64,v=true flags. See #7082