Boot failure with ppc64 port on iMacs G5

[tuxayo]

Hi again :)

I tried snapshots/2024-01-31/debian-12.0.0-ppc64-NETINST-1.iso

And was able to start booting from usb with:
boot usb0/***@1:,\boot\grub\powerpc.elf
(typed in Open Firmware shell)
(usb0 is the top port)

Grub worked, and then I tried default install (the 1st option) and it
started loading during like 2 minutes.
And then it got stuck with some superposition of the messages
smp_core99_probe
and
the stuff before
DO-QUIESCE finisedBooting Linux via __start() @ 0x0000000002090000 ...

Full message of the two iMacs G5 ↓↓↓
https://transfert.facil.services/r/Ksfq_2VM9_#tDATBcLXzB0zkAEaiqm9gfLfsaXliVJ13rQxKUHgUmA=
https://transfert.facil.services/r/Zs1h1jEtb2#jufjxv6+1DfHnO3TSfhmYD+teOvY46sGClHyz7SiXd4=

Same when starting expert install.

Models: M9249 and M9845, see:
https://en.wikipedia.org/wiki/IMac_G5

Any hint of what might be the issue?

Cheers,

[tuxayo]

There seems to be a regression in the kernel which affects PowerPC 970 machines,
i.e. PowerMac G5 CPUs. The issue needs to be bisected and reported upstream.
If you have the time, I would really appreciate if you could test the various
snapshots and let me know which kernel is the first to not work. I expect that
the breakage occurred somewhere around kernel 6.3 or so.

No problem, I'm going there every week and can do as much testing as
necessary even if it takes months. We got a donation of a standard PC so
we have now one publicly usable PC at the library.
So I can take my time to try to make something out of the iMacs.

Results:
- 2023-06-18: ok
- running strings on iso's vmlinuz yields 6.3.7-1 (2023-06-12)
- 2023-11-16: fail
- 6.5.10-1 (2023-11-03)

As expected, the regression happened during the big gap between
snapshots >_<

Anything more I can test?
(cross compiling old kernels is unfortunately too much over my head)

It's weird that it seems the regression is about booting the iso.
Otherwise, net installs from old iso like what Claudia did wouldn't work
after rebooting after install.

Cheers,

[tuxayo]

I'm not sure you'll be able to install debian with that cd image.
I tried about a week ago on my iMac G5 iSight (ati x600 graphics) model,
had to use 2023-05-16 image with kernel options
"radeon.agpmode=1 radeon.modeset=1 video=offb:off nomodeset"
in grub.

Ok so you had the same issue when booting a recent iso, right?
smp_core99_probe
DO-QUIESCE finisedBooting Linux via __start() @ 0x0000000002090000 ...

Thanks for the warnings about the next steps. One of my iMacs has an ATI
Radeon 9600. I was hopping it would work better than the other one with
the nVidia GeForce FX 5200 Ultra.

Looking at this:
https://wiki.debian.org/PowerPC/FAQ#How_do_I_get_graphics_working.3F
you might have hope to get away without nomodeset and thus still having
2D acceleration (I'm not sure if your kernel params leave you with any
kind of acceleration)

Cheers,

[Michael Ellerman]

Post by tuxayo
I tried snapshots/2024-01-31/debian-12.0.0-ppc64-NETINST-1.iso
(typed in Open Firmware shell)
(usb0 is the top port)
Grub worked, and then I tried default install (the 1st option) and it
started loading during like 2 minutes.
And then it got stuck with some superposition of the messages
smp_core99_probe
and
the stuff before

There seems to be a regression in the kernel which affects PowerPC 970 machines,
i.e. PowerMac G5 CPUs. The issue needs to be bisected and reported upstream.

I have a quad G5 that is booting mainline happily.

I used to have an iMac G5 but it died.

If you have the time, I would really appreciate if you could test the various
snapshots and let me know which kernel is the first to not work. I expect that
the breakage occurred somewhere around kernel 6.3 or so.

Can someone send the .config for the kernel in question? I could try
that on my machine here.

cheers

[Michael Ellerman]

Hi Michael,

Post by Michael Ellerman

There seems to be a regression in the kernel which affects PowerPC 970 machines,
i.e. PowerMac G5 CPUs. The issue needs to be bisected and reported upstream.

I have a quad G5 that is booting mainline happily.

it's a really tricky problem because it seems to depend on how the kernel image
is booted.

Post by Michael Ellerman
https://cdimage.debian.org/cdimage/ports/snapshots/2024-02-25/debian-12.0.0-ppc64-NETINST-1.iso

but the kernel will boot fine when installing in an existing system which was installed
with an installation CD which uses an older kernel.

Post by Michael Ellerman
https://cdimage.debian.org/cdimage/ports/snapshots/2023-06-18/debian-12.0.0-ppc64-NETINST-1.iso

I have not really figured out yet what the problem is.

OK.

That second iso boots OK for me in qemu. It boots grub and then the
kernel loads and shows:

Loading ...
OF stdout device is: /***@f0000000/mac-***@c/***@13000/ch-***@13020
Preparing to boot Linux version 6.3.0-1-powerpc64 (debian-***@lists.debian.org) (gcc-12 (Debian 12.3.0-2) 12.3.0, GNU ld (GNU Binutils for Debian) 2.40) #1 SMP Debian 6.3.7-1 (2023-06-12)
Detected machine type: 0000000000000400
command line: BOOT_IMAGE=/install/vmlinux --- quiet
memory layout at init:
memory_limit : 0000000000000000 (16 MB aligned)
alloc_bottom : 0000000005e70000
alloc_top : 0000000030000000
alloc_top_hi : 0000000080000000
rmo_top : 0000000030000000
ram_top : 0000000080000000
copying OF device tree...
Building dt strings...
Building dt structure...
Device tree strings 0x0000000005e80000 -> 0x0000000005e80560
Device tree struct 0x0000000005e90000 -> 0x0000000005ea0000
Quiescing Open Firmware ...
Booting Linux via __start() @ 0x0000000002000000 ...
Hello World !
smp_core99_probe
smp_core99_bringup_done
Starting system log daemon: syslogd, klogd.

And eventually starts the installer.

That's using no VGA, so possibly there's something wrong with the video
setup on real hardware:

$ qemu-system-ppc64 -nographic -vga none -M mac99,via=pmu -smp 1 -m 2G -nic user -drive file=$HOME/debian-12.0.0-ppc64-NETINST-1.2023-06-18.iso,format=raw,media=cdrom -boot d

I'll try and find time to test it on my actual G5 next week when I'm in
the office.

cheers

[Brad Boyer]

The problem is that the newer image doesn't boot and currently I don't know
why because installing the exact same kernel later from the package manager
into an installed system works yields a bootable system with the latest
kernel.
The installer images are built from the same kernel package which makes the
whole thing even more confusing.

Perhaps it's a memory allocation problem before the kernel has fully
configured the MMU? I know in the past the kernel could be touchy
about how memory was mapped in the very early stages of boot. If we
happened to be near some limit, even a small increase in the size of
the kernel might have pushed us over the edge.

I'm not sure of the exact details, but perhaps the memory setup is
slightly different when booted from the CD image. I haven't looked
at recent images, but I know in the past the boot process was
different on CD compared to after the install.

Brad Boyer
***@allandria.com

[Michael Ellerman]

Post by Michael Ellerman
That second iso boots OK for me in qemu. It boots grub and then the
Loading ...
Detected machine type: 0000000000000400
command line: BOOT_IMAGE=/install/vmlinux --- quiet
memory_limit : 0000000000000000 (16 MB aligned)
alloc_bottom : 0000000005e70000
alloc_top : 0000000030000000
alloc_top_hi : 0000000080000000
rmo_top : 0000000030000000
ram_top : 0000000080000000
copying OF device tree...
Building dt strings...
Building dt structure...
Device tree strings 0x0000000005e80000 -> 0x0000000005e80560
Device tree struct 0x0000000005e90000 -> 0x0000000005ea0000
Quiescing Open Firmware ...
Hello World !
smp_core99_probe
smp_core99_bringup_done
Starting system log daemon: syslogd, klogd.
And eventually starts the installer.

Yep, the second, older image works as expected. However, the recent one does not
and I have absolutely no clue why.

I actually tested both, and both work, but then I cited the wrong one in
my email >_<

So at least on qemu that newer kernel is OK:

Preparing to boot Linux version 6.6.15-powerpc64 (debian-***@lists.debian.org) (gcc-13 (Debian 13.2.0-13) 13.2.0, GNU ld (GNU Binutils for Debian) 2.42) #1 SMP Debian 6.6.15-2 (2024-02-04)
...
Booting Linux via __start() @ 0x0000000004800000 ...
Hello World !
smp_core99_probe
smp_core99_bringup_done
Starting system log daemon: syslogd, klogd.


cheers

[John Paul Adrian Glaubitz]

Hi Michael,

Post by Michael Ellerman

Yep, the second, older image works as expected. However, the recent one does not
and I have absolutely no clue why.

I actually tested both, and both work, but then I cited the wrong one in
my email >_<
...
Hello World !
smp_core99_probe
smp_core99_bringup_done
Starting system log daemon: syslogd, klogd.

Did you get around testing the images on real hardware?

And can tell me what command line you used for booting with QEMU?
Maybe that gives us a clue where the problem is.

Users are still reporting boot lockups with kernel 6.6.x.

Adrian

[John Paul Adrian Glaubitz]

Hi,

Post by Michael Ellerman

p, the second, older image works as expected. However, the recent one does not
and I have absolutely no clue why.

I actually tested both, and both work, but then I cited the wrong one in
my email >_<
...
Hello World !
smp_core99_probe
smp_core99_bringup_done
Starting system log daemon: syslogd, klogd.
https://cdimage.debian.org/cdimage/ports/tests/ppc64-test-20240511/debian-12.0.0-ppc64-NETINST-1.iso

Still hangs for me with QEMU emulating a PPC970.

Will retest once kernel 6.8.x enters Debian unstable.

Adrian

[John Paul Adrian Glaubitz]

Hi Michael,

Post by Michael Ellerman

Yep, the second, older image works as expected. However, the recent one does not
and I have absolutely no clue why.

I actually tested both, and both work, but then I cited the wrong one in
my email >_<
...
Hello World !
smp_core99_probe
smp_core99_bringup_done
Starting system log daemon: syslogd, klogd.
https://salsa.debian.org/kernel-team/linux/-/blob/master/debian/patches/bugfix/powerpc/fbdev-offb-Update-expected-device-name.patch?ref_type=heads

Removing this patch unbreaks the console on PowerMac G5.

Post by Michael Ellerman
https://cdimage.debian.org/cdimage/daily-builds/daily/current/ppc64el/iso-cd/debian-testing-ppc64el-netinst.iso
https://people.debian.org/~glaubitz/debian-bug-1033058.mp4
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1085949

This issue can be easily reproduced with QEMU.

Aditi Mishra (CC'ed) is trying to reproduce the issue as well, but so far has been unsuccessful.

Adrian