Title was "AMD ROCm hipBLAS and LLM use (ExLlamaV2 and any other parallel / model splitting associated)" but it's over 80 characters so I had to re-write it.
Has anyone installed AMD ROCm hipBLAS, successfully, to their PS4 Linux? From what I quickly checked Linux kernel seems to be somewhere between 5.4-5.15 depending on hardware (and older are available aswell). And then there is this news topic here about 6.6 kernel.
I've lately become quite fascinated in the capabilities of LLMs, especially if many models were used as parts of a tool / platform. 'Hivemind' keeps popping to my mind.
I have RX 9070 XT here in my desktop PC. It's LLM use would seem to be mostly tied to ROCm. Of course Vulkan can be used, but Vulkan is lesser (in performance). I haven't really wondered far in understanding the possibilities of splitting a single LLM model into two or more pieces that could then be used in parallel style with 2x+ GPUs. From what little I managed to scavenge, it seems that even if a person would manage to have several motherboards with GPU in each -> ethernet speed would be a very noticeable bottleneck.
Nonetheless: has anyone tried to use PS4 Linux for local LLM purpose? I'm fascinated with the idea of using RX 9070 XT + PS4 Pro GPU / hardware together, although I suspect it may be a stupid idea (for example, that LAN speed might be a severe bottleneck).
If anyone is intrested in trying PS4 Linux ROCm, I have no idea what the ideal route would be; but I think https://github.com/spack/spack and/or https://github.com/lamikr/rocm_sdk_builder might be options. I have very little "great knowledge" on hardware, coding, etc. so I am myself very limited in all of this. But nonetheless I would assume that I'd know if PS4 Linux can be teamed with desktop PC by the end of summer months, I guess. We'll see.
(If it actually is so that ethernet / LAN speed is a great bottleneck, well I wonder if it really is. Or how large issue this is.)
PS4 GPU(s) would seem to be either this https://www.techpowerup.com/gpu-specs/playstation-4-gpu.c2085 and https://www.techpowerup.com/gpu-specs/playstation-4-pro-gpu.c2876 (???)
