Bug#1076564: pahole BTF processing seems flaky on powerpc

[Domenico Andreoli]

CCing debian-kernel and debian-powerpc

Adding Alan and Jiri to the CC list.

Hi Arnaldo,

Package: pahole
Version: 1.27-1
Severity: normal
6.8.12-1: https://buildd.debian.org/status/fetch.php?pkg=linux&arch=powerpc&ver=6.8.12-1&stamp=1717234422&raw=1
6.9.9-1: https://buildd.debian.org/status/fetch.php?pkg=linux&arch=powerpc&ver=6.9.9-1&stamp=1720906547&raw=1
6.10-1~exp1: https://buildd.debian.org/status/fetch.php?pkg=linux&arch=powerpc&ver=6.10-1%7Eexp1&stamp=1721287862&raw=1
Note, these log files are up to 270 MB in size and should be
downloaded; at least Firefox becomes unresponsive when trying to
display them.
For each of these, the failure seems to start with an error from
[102044] ARRAY (anon) type_id=99491 index_type_id=14 nr_elems=12 Error emitting BTF type
Encountered error while encoding BTF.

Does the above error ring any bell?

Nope

Is there anything I can do to help?

https://buildd.debian.org/status/fetch.php?pkg=linux&arch=powerpc&ver=6.10-1%7Eexp1&stamp=1721287862&raw=1
+ LLVM_OBJCOPY=powerpc-linux-gnu-objcopy pahole -J -j --btf_features=encode_force,var,float,enum64,decl_tag,type_tag,optimized_func,consistent_func --lang_exclude=rust .tmp_vmlinux.btf
Can I have access to that .tmp_vmlinux.btf file so that I can try to
reproduce it here?

I don't have access to the build host (blaauw2) and I've some doubts
it would still have that file.

Maybe our kernel team and powerpc porters have suggestions?

Dom

[John Paul Adrian Glaubitz]

Hi Domenico,

Post by Domenico Andreoli

Is there anything I can do to help?

https://buildd.debian.org/status/fetch.php?pkg=linux&arch=powerpc&ver=6.10-1%7Eexp1&stamp=1721287862&raw=1
+ LLVM_OBJCOPY=powerpc-linux-gnu-objcopy pahole -J -j --btf_features=encode_force,var,float,enum64,decl_tag,type_tag,optimized_func,consistent_func --lang_exclude=rust .tmp_vmlinux.btf
Can I have access to that .tmp_vmlinux.btf file so that I can try to
reproduce it here?

I don't have access to the build host (blaauw2) and I've some doubts
it would still have that file.
Maybe our kernel team and powerpc porters have suggestions?

I have root access to all powerpc/ppc64 machines (buildds and porterbox).

I'm cleaning up the porterbox now, disk is quite full, then you can try
to build the kernel package on perotto.debian.net or I can try it myself.

I have seen the bug myself and I wanted to debug it, but the attempt was
foiled by the fact that the disk on perotto is full (again).

Will take care of it and let you know when it's (some hours).

Adrian

[Domenico Andreoli]

Post by John Paul Adrian Glaubitz
Hi Domenico,

Hi!

Post by John Paul Adrian Glaubitz

Post by Domenico Andreoli

Is there anything I can do to help?

https://buildd.debian.org/status/fetch.php?pkg=linux&arch=powerpc&ver=6.10-1%7Eexp1&stamp=1721287862&raw=1
+ LLVM_OBJCOPY=powerpc-linux-gnu-objcopy pahole -J -j --btf_features=encode_force,var,float,enum64,decl_tag,type_tag,optimized_func,consistent_func --lang_exclude=rust .tmp_vmlinux.btf
Can I have access to that .tmp_vmlinux.btf file so that I can try to
reproduce it here?

I don't have access to the build host (blaauw2) and I've some doubts
it would still have that file.
Maybe our kernel team and powerpc porters have suggestions?

I have root access to all powerpc/ppc64 machines (buildds and porterbox).
I'm cleaning up the porterbox now, disk is quite full, then you can try
to build the kernel package on perotto.debian.net or I can try it myself.
I have seen the bug myself and I wanted to debug it, but the attempt was
foiled by the fact that the disk on perotto is full (again).
Will take care of it and let you know when it's (some hours).

That's great, thank you.

Post by John Paul Adrian Glaubitz
Adrian
--
.''`. John Paul Adrian Glaubitz
: :' : Debian Developer
`. `' Physicist
`- GPG: 62FF 8A75 84E0 2956 9546 0006 7426 3B37 F5B5 F913

[Ben Hutchings]

Post by John Paul Adrian Glaubitz
Hi Domenico,

Post by Domenico Andreoli

Is there anything I can do to help?

https://buildd.debian.org/status/fetch.php?pkg=linux&arch=powerpc&ver=6.10-1%7Eexp1&stamp=1721287862&raw=1
+ LLVM_OBJCOPY=powerpc-linux-gnu-objcopy pahole -J -j --btf_features=encode_force,var,float,enum64,decl_tag,type_tag,optimized_func,consistent_func --lang_exclude=rust .tmp_vmlinux.btf
Can I have access to that .tmp_vmlinux.btf file so that I can try to
reproduce it here?

I don't have access to the build host (blaauw2) and I've some doubts
it would still have that file.
Maybe our kernel team and powerpc porters have suggestions?

I have root access to all powerpc/ppc64 machines (buildds and porterbox).
I'm cleaning up the porterbox now, disk is quite full, then you can try
to build the kernel package on perotto.debian.net or I can try it myself.
I have seen the bug myself and I wanted to debug it, but the attempt was
foiled by the fact that the disk on perotto is full (again).

I had a go yesterday and ran into the same problem. I couldn't
reproduce with a small kernel config (allnoconfig + BPF + DEBUG_INFO +
DEBUG_INFO_BTF) and there wasn't enough disk space to build even one of
the Debian kernel flavours.

Post by John Paul Adrian Glaubitz
Will take care of it and let you know when it's (some hours).

Thank you!

Ben.

[John Paul Adrian Glaubitz]

Post by Ben Hutchings
I had a go yesterday and ran into the same problem. I couldn't
reproduce with a small kernel config (allnoconfig + BPF + DEBUG_INFO +
DEBUG_INFO_BTF) and there wasn't enough disk space to build even one of
the Debian kernel flavours.

Post by John Paul Adrian Glaubitz
Will take care of it and let you know when it's (some hours).

Thank you!

There are now 120 GB of free disk space. Let me know if that's sufficient
or whether I need to clean up more, probably asking others to clean up
their home directories.

Adrian

[Ben Hutchings]

Post by John Paul Adrian Glaubitz

Post by Ben Hutchings
I had a go yesterday and ran into the same problem. I couldn't
reproduce with a small kernel config (allnoconfig + BPF + DEBUG_INFO +
DEBUG_INFO_BTF) and there wasn't enough disk space to build even one of
the Debian kernel flavours.

Post by John Paul Adrian Glaubitz
Will take care of it and let you know when it's (some hours).

Thank you!

There are now 120 GB of free disk space. Let me know if that's sufficient
or whether I need to clean up more, probably asking others to clean up
their home directories.

I've now done 10 kernel builds on perotto (4 builds of just the
"powerpc" flavour and then 2 builds of all 3 flavours) and not
reproduced this. I'm thinking this may be machine-dependent in some
way.

Looking again at all the build logs since DEBUG_INFO_BTF was enabled
for powerpc, we have:

Successes:

Version Builder
----------------------------------
6.9.10-1 debian-project-be-2
6.9.7-1 debian-project-be-1
6.8.12-1 debian-project-be-2
6.8.11-1 blaauw
6.8.9-1 debian-project-be-2
6.7.12-1 blaauw
6.7.9-2 debian-project-be-2
6.7.4-1~exp1 blaauw
6.7.1-1~exp1 debian-project-be-2
6.6.15-1 blaauw
6.6.13-1 blaauw
6.6.11-1 debian-project-be-1
6.6.9-1+b1 debian-project-be-1
6.6.8-1 debian-project-be-2
6.6.4-1~exp1 blaauw
6.5.13-1 debian-project-be-1
6.5.10-1 debian-project-be-2
6.5.8-1 blaauw
6.5.6-1 debian-project-be-2
6.5.3-1 blaauw
6.5~rc7-1~exp1 debian-project-be-1
6.5~rc6-1~exp1 blaauw
6.4.13-1 blaauw
6.4.11-1 blaauw
6.4.4-3 blaauw

Failures:

Version Builder Failure mode
----------------------------------------------------------------------------
6.10-1~exp1 blaauw this bug
6.9.9-1 blaauw this bug
6.9.8-1 kapitsa this bug
6.9.2-1~exp1 blaauw this bug
6.8.12-1 kapitsa this bug
6.7-1~exp1 debian-project-be-2 compiler OOM; not powerpc-specific
6.6.3-1~exp1 blaauw kernel-wedge failed; not powerpc-specific
6.4.4-2 blaauw out of disk space

Ignoring the unrelated failures, kapitsa has a 0% success rate (but
with only 2 attempts), blaauw an 80% success rate, and debian-project-
be-{1,2} have 100% success rates.

I don't know what differences there are between these builders that
might be relevant.

Ben.

[John Paul Adrian Glaubitz]

Hi Ben,

Post by Ben Hutchings
I don't know what differences there are between these builders that
might be relevant.

For kapitsa, the installed host system is powerpc while all the others
run the ppc64 port.

As for the hardware:

kapitsa runs bare-metal (inside an LPAR) on a POWER8 machine:

***@kapitsa:~# grep model /proc/cpuinfo
model : IBM,8284-22A
***@kapitsa:~#

Both blaauw and perotto are KVMs running on watson which runs
Debian's ppc64el port (little-endian):

***@watson:~# grep model /proc/cpuinfo
model : 8247-42L
***@watson:~#

***@blaauw:~# grep model /proc/cpuinfo
model : IBM pSeries (emulated by qemu)
***@blaauw:~#

***@perotto:~# grep model /proc/cpuinfo
model : IBM pSeries (emulated by qemu)
***@perotto:~#

Both debian-project-be-01 and debian-project-be-02 are KVMs running
on OpenStack at OSUOSL's OpenPOWER platform:

***@debian-project-be-1:~# grep model /proc/cpuinfo
model : IBM pSeries (emulated by qemu)
***@debian-project-be-1:~#

***@debian-project-be-2:~# grep model /proc/cpuinfo
model : IBM pSeries (emulated by qemu)
***@debian-project-be-2:~#

Adrian