Differences

This shows you the differences between two versions of the page.

--- linux64 [2026/07/18 11:25] – [ollama in podman for ai] bengt
+++ linux64 [2026/07/18 21:37] (current) – [ollama in podman for ai] bengt
@@ Line 866: / Line 866: @@
   cd $HOME
   loginctl enable-linger ollama
-  podman run -d --rm -v /home/ollama/ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama
+  podman run -d --rm -v /home/ollama/ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama # See below for flags when using gpu
   $ chmod u+x start.sh
 Then setup a start.sh-script containing the startup command line and maybe start this using "lxterminal -t ollama -e sudo -u ollama /home/ollama/start.sh". Replace "-d" with "-it" in the podman command line to see the output if starting this way.
@@ Line 883: / Line 883: @@
 You can add the ollama integration to home assistant. On issues check your firewall config. Be sure to have "conversation:" "assist_pipeline:" in configuration.yaml if you do not use the default config before trying to setup assist pipelines.
-After buying a 5060 16GB ti for AI to have next to my 1660 ti super used for the console I needed to switch to nvidia drivers and configure podman:
+After buying a 5060 16GB ti for AI to have next to my 1660 ti super used for the console I needed to switch to nvidia drivers and configure podman. I already have rpmfusion-nonfree repo enabled so I just did this:
-  # dnf config-manager --set-enabled rpmfusion-nonfree-nvidia-driver
+  # dnf update # To avoid issues below. Reboot if new kernel.
   # dnf install -y akmod-nvidia xorg-x11-drv-nvidia-cuda
-  Wait for akmod to complete before rebooting. Use "sudo akmods --force" perhaps.
+  # akmods --force
+  Wait for akmod to complete building drivers before rebooting. Check with top?
+Note that I lost audio after this for the new kernel that got installed at the same time. Turns out that those modules were not installed as they were not needed to compile the nvidia driver. Update and reboot before or install lacking kernel-modules and kernel-modules-extra. This should also work "dnf install kernel-$(uname -r)".
+Configure podman:
+  # dnf install nvidia-container-toolkit
+  # nvidia-ctk cdi generate --output=/etc/cdi/nvidia.yaml
+  # nvidia-ctk cdi list
+  # podman run --rm --security-opt label=disable --device nvidia.com/gpu=1 ubuntu nvidia-smi # Check gpu index to the one you want to use
+  # vi start.sh
+  Add "--security-opt label=disable --device nvidia.com/gpu=1" with the index of the wanted gpu or all to use the 1660 ti for some layers in my case. Doubled the performance compared to spilling over to the cpu. But models splits early.
+  # nvidia-smi # Check ram usage for gpus. Watch out for too large context windows spilling over as it will spread not prioritizing the 5060 ti.
+If you run into context size problems (it got capped at 4096) you can add "-e OLLAMA_CONTEXT_LENGTH=32768" to the podman line. But then you will probably want to cap this client side for the larger models using both gpus. Using "ollama ps" will show the active max context size. Check "nvidia-smi" for when the context makes the model split.
 ===== Software =====