`-mattr` should be encoded in the IR for LTO's sake #50505

twd2 · 2021-07-22T08:14:18Z


Bugzilla Link	51161
Version	unspecified
OS	Linux
Blocks	#4440
Attachments	The diagnostic information.
CC	@asb,@efriedma-quic,@fhahn,@MaskRay,@nathanchance,@nickdesaulniers,@smithp35,@twd2

Extended Description

When compiling the following code with LTO:

$ cat test.c
void _start()
{
    asm volatile (
        ".balign 4\n\t"
        "c.nop\n\t"
        ".balign 4\n\t"
        "c.nop"
    );
}

$ clang -march=rv64imac -O2 -flto -mno-relax -c test.c
$ ld.lld test.o

lld fails with unable to write nop sequence of 2 bytes.
The diagnostic information is attached.

It looks like to be due to the HasStdExtC check failure in RISCVAsmBackend::writeNopData.
And, if I use ld.lld -mllvm -mattr=+c test.o, lld can generate proper machine code.
According to Nick Desaulniers [1], -mattr should be encoded in the IR for LTO's sake.

My LLVM version: commit 1b61d83 ("[Inline] Add test for #49933 (NFC)")
[1] https://lore.kernel.org/linux-riscv/CAKwvOdmNji0AbYUiOSfb5cLD+g7YCpXk4oDupa8gTfgzYmxvBg@mail.gmail.com/

The text was updated successfully, but these errors were encountered:

fhahn · 2021-07-26T09:11:50Z

Created attachment 25046 [details]

It looks like to be due to the HasStdExtC check failure in
RISCVAsmBackend::writeNopData.
And, if I use ld.lld -mllvm -mattr=+c test.o, lld can generate proper
machine code.
According to Nick Desaulniers [1], -mattr should be encoded in the IR for
LTO's sake.

Yes it should, but you are not passing -mattr=+c to the clang invocation. Does it work if you add -mattr=+c to your clang line? I assume the +c feature is not part of -march=rv64imac

twd2 · 2021-07-26T10:40:15Z

Created attachment 25046 [details]

It looks like to be due to the HasStdExtC check failure in
RISCVAsmBackend::writeNopData.
And, if I use ld.lld -mllvm -mattr=+c test.o, lld can generate proper
machine code.
According to Nick Desaulniers [1], -mattr should be encoded in the IR for
LTO's sake.

Yes it should, but you are not passing -mattr=+c to the clang
invocation. Does it work if you add -mattr=+c to your clang line? I assume
the +c feature is not part of -march=rv64imac

Hi, I don't know how to pass -mattr=+c to clang:

$ clang -march=rv64imac -mattr=+c -O2 -flto -mno-relax -c test.c
clang-13: error: unknown argument: '-mattr=+c'

But, rv64imac actually implies the +c feature:

llvm-project/clang/lib/Driver/ToolChains/Arch/RISCV.cpp

Line 417 in f1cbea3

case 'c':

fhahn · 2021-07-26T10:46:24Z

But, rv64imac actually implies the +c feature:
https://github.com/llvm/llvm-project/blob/
f1cbea3/clang/lib/Driver/ToolChains/Arch/
RISCV.cpp#L417

Strange, the target feature gets added as expected when looking at Godbolt: https://godbolt.org/z/4avz69d3v

Can you check the IR in test.o and see if it has "target-features"="...,+c,..." in the attributes of the function?

twd2 · 2021-07-26T10:57:09Z

But, rv64imac actually implies the +c feature:
https://github.com/llvm/llvm-project/blob/
f1cbea3/clang/lib/Driver/ToolChains/Arch/
RISCV.cpp#L417

Strange, the target feature gets added as expected when looking at Godbolt:
https://godbolt.org/z/4avz69d3v

Can you check the IR in test.o and see if it has
"target-features"="...,+c,..." in the attributes of the function?

My output is similar to yours:

$ clang -march=rv64imac -O2 -flto -mno-relax -c test.c
$ test llvm-dis test.o -o -       
; ModuleID = 'test.o'
source_filename = "test.c"
target datalayout = "e-m:e-p:64:64-i64:64-i128:128-n64-S128"
target triple = "riscv64-unknown-linux-gnu"

; Function Attrs: nounwind
define dso_local void @&#8203;_start() local_unnamed_addr #&#8203;0 {
  tail call void asm sideeffect ".balign 4\0A\09c.nop\0A\09.balign 4\0A\09c.nop", ""() #&#8203;1, !srcloc !&#8203;6
  ret void
}

attributes #&#8203;0 = { nounwind "frame-pointer"="none" "min-legal-vector-width"="0" "no-trapping-math"="true" "stack-protector-buffer-size"="8" "target-features"="+64bit,+a,+c,+m,-relax,-save-restore" }
attributes #&#8203;1 = { nounwind }

!llvm.module.flags = !{#0, !&#8203;1, !&#8203;2, !&#8203;3, !&#8203;4}
!llvm.ident = !{#5}

!&#8203;0 = !{i32 1, !"wchar_size", i32 4}
!&#8203;1 = !{i32 1, !"target-abi", !"lp64"}
!&#8203;2 = !{i32 1, !"SmallDataLimit", i32 8}
!&#8203;3 = !{i32 1, !"ThinLTO", i32 0}
!&#8203;4 = !{i32 1, !"EnableSplitLTOUnit", i32 1}
!&#8203;5 = !{!"clang version 13.0.0"}
!&#8203;6 = !{i32 49, i32 61, i32 81, i32 105}

^0 = module: (path: "test.o", hash: (0, 0, 0, 0, 0))
^1 = gv: (name: "_start", summaries: (function: (module: ^0, flags: (linkage: external, visibility: default, notEligibleToImport: 1, live: 0, dsoLocal: 1, canAutoHide: 0), insts: 2))) ; guid = 9315384113491396937
^2 = flags: 8
^3 = blockcount: 1
$ test ld.lld test.o 
LLVM ERROR: unable to write nop sequence of 2 bytes

The target-features attributes seem to be ignored by lld (?)

efriedma-quic · 2021-07-27T20:38:16Z

I think this is what's happening:

You pass the "c" feature to clang.
clang encodes "c" feature into the target-features of each function.
You pass the bitcode into lld.
The RISCV asm printer knows it's in a "c" function, and adjusts... but never actually passes the information on to the MC layer.
MC layer doesn't realize compressed ops are supposed to be legal, and prints an error.

Probably the answer here is that the RISCV asmprinter should be calling emitDirectiveOptionRVC() somewhere, so the output is unambiguous.

twd2 · 2021-07-29T21:24:34Z

This code doesn't work as well:

void _start(void)
{
    asm volatile (
        ".option push\n\t"
        ".option rvc\n\t"
        ".balign 4\n\t"
        "c.nop\n\t"
        ".balign 4\n\t"
        "c.nop\n\t"
        ".option pop"
    );
}

twd2 · 2021-07-29T21:34:24Z

I find the following patch is related to this issue:
https://reviews.llvm.org/D45962

twd2 · 2022-04-26T04:29:48Z

I find the above PR is merged, and after some tests, I think this issue is solved.

nickdesaulniers · 2022-04-26T04:44:33Z

Thanks for the follow up; closing.

Allow LTO to be selected for RISC-V, only when LLD >= 14, since there is an issue [1] in prior LLD versions that prevents LLD to generate proper machine code for RISC-V when writing `nop`s. I have tested enabling LTO for `defconfig`. The LLD took ~2m21s and ~3GiB on our Intel Xeon Gold 6140 server and produced an 18MiB Image. The image can boot to shell using an archriscv rootfs on QEMU. I have also tested it for `allyesconfig` without COMPILE_TEST, FTRACE, KASAN, and GCOV. The LLD took ~7h03m and ~335GiB on the server, successfully producing a 1.7GiB Image. Unfortunately, we cannot boot this image because the `create_kernel_page_table()` -> `alloc_pmd_early()` -> `BUG_ON()` logic limits the image to be < 1GiB. Maybe we can fix it in a separate patch further. Disable LTO for arch/riscv/kernel/pi, as llvm-objcopy expects an ELF object file when manipulating the files in that subfolder, rather than LLVM bitcode. [1] llvm/llvm-project#50505, resolved by LLVM commit e63455d5e0e5 ("[MC] Use local MCSubtargetInfo in writeNops") Tested-by: Wende Tan <twd2.me@gmail.com> Signed-off-by: Wende Tan <twd2.me@gmail.com> Co-developed-by: Nathan Chancellor <nathan@kernel.org> Signed-off-by: Nathan Chancellor <nathan@kernel.org>

Allow LTO to be selected for RISC-V, only when LLD >= 14, since there is an issue [1] in prior LLD versions that prevents LLD to generate proper machine code for RISC-V when writing `nop`s. I have tested enabling LTO for `defconfig`. The LLD took ~2m21s and ~3GiB on our Intel Xeon Gold 6140 server and produced an 18MiB Image. The image can boot to shell using an archriscv rootfs on QEMU. I have also tested it for `allyesconfig` without COMPILE_TEST, FTRACE, KASAN, and GCOV. The LLD took ~7h03m and ~335GiB on the server, successfully producing a 1.7GiB Image. Unfortunately, we cannot boot this image because the `create_kernel_page_table()` -> `alloc_pmd_early()` -> `BUG_ON()` logic limits the image to be < 1GiB. Maybe we can fix it in a separate patch further. Disable LTO for arch/riscv/kernel/pi, as llvm-objcopy expects an ELF object file when manipulating the files in that subfolder, rather than LLVM bitcode. [1] llvm/llvm-project#50505, resolved by LLVM commit e63455d5e0e5 ("[MC] Use local MCSubtargetInfo in writeNops") Tested-by: Wende Tan <twd2.me@gmail.com> Signed-off-by: Wende Tan <twd2.me@gmail.com> Co-developed-by: Nathan Chancellor <nathan@kernel.org> Signed-off-by: Nathan Chancellor <nathan@kernel.org> Signed-off-by: Björn Töpel <bjorn@rivosinc.com>

nickdesaulniers · 2023-10-16T15:30:58Z

This came up again in ClangBuiltLinux/linux#1942, and is causing issues reported in #65090. Reopening.

Allow LTO to be selected for RISC-V, only when LLD >= 14, since there is an issue [1] in prior LLD versions that prevents LLD to generate proper machine code for RISC-V when writing `nop`s. To avoid boot failures in QEMU [2], '-mattr=+c' and '-mattr=+relax' need to be passed via '-mllvm' to ld.lld, as there appears to be an issue with LLVM's target-features and LTO [3], which can result in incorrect relocations to branch targets [4]. Once this is fixed in LLVM, it can be made conditional on affected ld.lld versions. Disable LTO for arch/riscv/kernel/pi, as llvm-objcopy expects an ELF object file when manipulating the files in that subfolder, rather than LLVM bitcode. [1] llvm/llvm-project#50505, resolved by LLVM commit e63455d5e0e5 ("[MC] Use local MCSubtargetInfo in writeNops") [2] ClangBuiltLinux#1942 [3] llvm/llvm-project#59350 [4] llvm/llvm-project#65090 Tested-by: Wende Tan <twd2.me@gmail.com> Signed-off-by: Wende Tan <twd2.me@gmail.com> Co-developed-by: Nathan Chancellor <nathan@kernel.org> Signed-off-by: Nathan Chancellor <nathan@kernel.org>

Allow LTO to be selected for RISC-V, only when LLD >= 14, since there is an issue [1] in prior LLD versions that prevents LLD to generate proper machine code for RISC-V when writing `nop`s. To avoid boot failures in QEMU [2], '-mattr=+c' and '-mattr=+relax' need to be passed via '-mllvm' to ld.lld, as there appears to be an issue with LLVM's target-features and LTO [3], which can result in incorrect relocations to branch targets [4]. Once this is fixed in LLVM, it can be made conditional on affected ld.lld versions. Disable LTO for arch/riscv/kernel/pi, as llvm-objcopy expects an ELF object file when manipulating the files in that subfolder, rather than LLVM bitcode. [1] llvm/llvm-project#50505, resolved by LLVM commit e63455d5e0e5 ("[MC] Use local MCSubtargetInfo in writeNops") [2] ClangBuiltLinux#1942 [3] llvm/llvm-project#59350 [4] llvm/llvm-project#65090 Tested-by: Wende Tan <twd2.me@gmail.com> Signed-off-by: Wende Tan <twd2.me@gmail.com> Co-developed-by: Nathan Chancellor <nathan@kernel.org> Signed-off-by: Nathan Chancellor <nathan@kernel.org> Signed-off-by: Björn Töpel <bjorn@rivosinc.com>

Allow LTO to be selected for RISC-V, only when LLD >= 14, since there is an issue [1] in prior LLD versions that prevents LLD to generate proper machine code for RISC-V when writing `nop`s. To avoid boot failures in QEMU [2], '-mattr=+c' and '-mattr=+relax' need to be passed via '-mllvm' to ld.lld, as there appears to be an issue with LLVM's target-features and LTO [3], which can result in incorrect relocations to branch targets [4]. Once this is fixed in LLVM, it can be made conditional on affected ld.lld versions. Disable LTO for arch/riscv/kernel/pi, as llvm-objcopy expects an ELF object file when manipulating the files in that subfolder, rather than LLVM bitcode. [1] llvm/llvm-project#50505, resolved by LLVM commit e63455d5e0e5 ("[MC] Use local MCSubtargetInfo in writeNops") [2] ClangBuiltLinux#1942 [3] llvm/llvm-project#59350 [4] llvm/llvm-project#65090 Tested-by: Wende Tan <twd2.me@gmail.com> Signed-off-by: Wende Tan <twd2.me@gmail.com> Co-developed-by: Nathan Chancellor <nathan@kernel.org> Signed-off-by: Nathan Chancellor <nathan@kernel.org> Reviewed-by: Conor Dooley <conor.dooley@microchip.com>

Allow LTO to be selected for RISC-V, only when LLD >= 14, since there is an issue [1] in prior LLD versions that prevents LLD to generate proper machine code for RISC-V when writing `nop`s. To avoid boot failures in QEMU [2], '-mattr=+c' and '-mattr=+relax' need to be passed via '-mllvm' to ld.lld, as there appears to be an issue with LLVM's target-features and LTO [3], which can result in incorrect relocations to branch targets [4]. Once this is fixed in LLVM, it can be made conditional on affected ld.lld versions. Disable LTO for arch/riscv/kernel/pi, as llvm-objcopy expects an ELF object file when manipulating the files in that subfolder, rather than LLVM bitcode. [1] llvm/llvm-project#50505, resolved by LLVM commit e63455d5e0e5 ("[MC] Use local MCSubtargetInfo in writeNops") [2] ClangBuiltLinux#1942 [3] llvm/llvm-project#59350 [4] llvm/llvm-project#65090 Tested-by: Wende Tan <twd2.me@gmail.com> Signed-off-by: Wende Tan <twd2.me@gmail.com> Co-developed-by: Nathan Chancellor <nathan@kernel.org> Signed-off-by: Nathan Chancellor <nathan@kernel.org> Signed-off-by: Björn Töpel <bjorn@rivosinc.com>

Allow LTO to be selected for RISC-V, only when LLD >= 14, since there is an issue [1] in prior LLD versions that prevents LLD to generate proper machine code for RISC-V when writing `nop`s. To avoid boot failures in QEMU [2], '-mattr=+c' and '-mattr=+relax' need to be passed via '-mllvm' to ld.lld, as there appears to be an issue with LLVM's target-features and LTO [3], which can result in incorrect relocations to branch targets [4]. Once this is fixed in LLVM, it can be made conditional on affected ld.lld versions. Disable LTO for arch/riscv/kernel/pi, as llvm-objcopy expects an ELF object file when manipulating the files in that subfolder, rather than LLVM bitcode. [1] llvm/llvm-project#50505, resolved by LLVM commit e63455d5e0e5 ("[MC] Use local MCSubtargetInfo in writeNops") [2] ClangBuiltLinux#1942 [3] llvm/llvm-project#59350 [4] llvm/llvm-project#65090 Tested-by: Wende Tan <twd2.me@gmail.com> Signed-off-by: Wende Tan <twd2.me@gmail.com> Co-developed-by: Nathan Chancellor <nathan@kernel.org> Signed-off-by: Nathan Chancellor <nathan@kernel.org> Reviewed-by: Conor Dooley <conor.dooley@microchip.com> Link: https://lore.kernel.org/r/20231017-riscv-lto-v4-1-e7810b24e805@kernel.org Signed-off-by: Palmer Dabbelt <palmer@rivosinc.com>

llvmbot transferred this issue from llvm/llvm-bugzilla-archive Dec 11, 2021

nickdesaulniers closed this as completed Apr 26, 2022

EugeneZelenko added mc Machine (object) code and removed lld labels Apr 26, 2022

vit9696 mentioned this issue Jul 14, 2022

[MIPS] -mno-abicalls is ignored for LTO #56192

Open

twd2 mentioned this issue Oct 14, 2023

RISC-V linker relaxation breaks booting with LTO ClangBuiltLinux/linux#1942

Open

nickdesaulniers reopened this Oct 16, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

`-mattr` should be encoded in the IR for LTO's sake #50505

`-mattr` should be encoded in the IR for LTO's sake #50505

twd2 mannequin commented Jul 22, 2021

fhahn commented Jul 26, 2021

twd2 mannequin commented Jul 26, 2021

fhahn commented Jul 26, 2021

twd2 mannequin commented Jul 26, 2021

efriedma-quic commented Jul 27, 2021

twd2 mannequin commented Jul 29, 2021

twd2 mannequin commented Jul 29, 2021

twd2 commented Apr 26, 2022

nickdesaulniers commented Apr 26, 2022

nickdesaulniers commented Oct 16, 2023

-mattr should be encoded in the IR for LTO's sake #50505

-mattr should be encoded in the IR for LTO's sake #50505

Comments

twd2 mannequin commented Jul 22, 2021

Extended Description

fhahn commented Jul 26, 2021

twd2 mannequin commented Jul 26, 2021

fhahn commented Jul 26, 2021

twd2 mannequin commented Jul 26, 2021

efriedma-quic commented Jul 27, 2021

twd2 mannequin commented Jul 29, 2021

twd2 mannequin commented Jul 29, 2021

twd2 commented Apr 26, 2022

nickdesaulniers commented Apr 26, 2022

nickdesaulniers commented Oct 16, 2023

`-mattr` should be encoded in the IR for LTO's sake #50505

`-mattr` should be encoded in the IR for LTO's sake #50505