User Tools

Site Tools


linux64

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
Next revision
Previous revision
linux64 [2026/10/31 09:07] – [vllm in podman for fast qwen3.8:27b ai] bengtlinux64 [2026/11/07 19:26] (current) – [Using pi.dev as ai harness] bengt
Line 905: Line 905:
 ==== vllm in podman for fast qwen3.8:27b ai ==== ==== vllm in podman for fast qwen3.8:27b ai ====
  
-I found https://www.reddit.com/r/LocalLLM/comments/1wnapgs/qwen_38_27b_on_dual_rtx_5060_ti_130_toks_full/ and https://github.com/skrodahl/qwen38-27B-dual-rtx5060 which claims much higher token/s than I get with ollama due to using mtp prediction.+I found https://www.reddit.com/r/LocalLLM/comments/1wnapgs/qwen_38_27b_on_dual_rtx_5060_ti_130_toks_full/ and https://github.com/skrodahl/qwen38-27B-dual-rtx5060 which claims much higher token/s than I get with ollama due to using mtp prediction. With this setup using nvfp4 for model and kv cache and 94% of VRAM I get: GPU KV cache size: 463,166 tokens, Maximum concurrency for 262,144 tokens per request: 1.77x
  
 So I translated that recipe to a podman line to use with my startup script serving on the ollama port so I do not have to reconfigure clients using the openai interface: So I translated that recipe to a podman line to use with my startup script serving on the ollama port so I do not have to reconfigure clients using the openai interface:
 <code> <code>
-tag=v0.30.0+tag=v0.32.0
 port=11434 port=11434
  
 cd $HOME cd $HOME
-loginctl enable-linger ollama+loginctl enable-linger $USER
  
 # vllm according to https://github.com/skrodahl/qwen38-27B-dual-rtx5060 # vllm according to https://github.com/skrodahl/qwen38-27B-dual-rtx5060
Line 934: Line 934:
   --api-key none \   --api-key none \
   --trust-remote-code \   --trust-remote-code \
-  --kv-cache-dtype fp8 \ +  --kv-cache-dtype nvfp4 \ 
-  --max-model-len 240000 \+  --max-model-len 262144 \
   --max-num-seqs 2 \   --max-num-seqs 2 \
   --compilation-config '{"cudagraph_mode": "PIECEWISE"}' \   --compilation-config '{"cudagraph_mode": "PIECEWISE"}' \
Line 953: Line 953:
 </code> </code>
  
-It failed to let vllm download the model trying to divide by 0 but when I dowloaded the model using the wget method from the github page it worked! Also reduced  "--max-model-len 262144" and "--gpu_memory_utilization 0.95" to avoid running out of VRAM. It allocates RAM outside of those 95% and it could run out during compaction.+It failed to let vllm download the model trying to divide by 0 but when I dowloaded the model using the wget method from the github page it worked! Also reduced "--max-model-len 262144" and "--gpu_memory_utilization 0.95" to avoid running out of VRAM. It allocates RAM outside of those 95% and it could run out during compaction. 
 + 
 +<code> 
 +REPO=gittensor-model-hub/Qwen3.8-27B-NVFP4-RTX5090 
 +REV=0cc27958cefbbe231782ec8511de8c4eb5233348 
 +mkdir -p ~/ai/models/gittensor-NVFP4 && cd ~/ai/models/gittensor-NVFP4 
 +for f in config.json generation_config.json hf_quant_config.json \ 
 +         model.safetensors.index.json model-00001-of-00003.safetensors \ 
 +         model-00002-of-00003.safetensors model-00003-of-00003.safetensors \ 
 +         tokenizer.json tokenizer_config.json vocab.json merges.txt \ 
 +         chat_template.jinja preprocessor_config.json processor_config.json \ 
 +         video_preprocessor_config.json; do 
 +  wget -c "https://huggingface.co/$REPO/resolve/$REV/$f" 
 +done 
 +</code>
  
 This is right on the edge for what my anker c300x ups can handle. I may need to get a new ups. You can tell the cards to use less max power with "nvidia-smi -pl 150" as root. I added "ollama ALL=(ALL) NOPASSWD: /usr/bin/nvidia-smi" with visudo to be able to set this with sudo in the LLM start script. This is right on the edge for what my anker c300x ups can handle. I may need to get a new ups. You can tell the cards to use less max power with "nvidia-smi -pl 150" as root. I added "ollama ALL=(ALL) NOPASSWD: /usr/bin/nvidia-smi" with visudo to be able to set this with sudo in the LLM start script.
  
 +==== Using pi.dev as ai harness ====
  
 Switched to pi.dev to be able to configure qwen3.8:27b to use medium instead of xhigh thinking: Switched to pi.dev to be able to configure qwen3.8:27b to use medium instead of xhigh thinking:
Line 986: Line 1001:
 In this https://github.com/earendil-works/pi/issues/2020 thinkingFormat changes to qwen or maybe qwen-chat-template. In this https://github.com/earendil-works/pi/issues/2020 thinkingFormat changes to qwen or maybe qwen-chat-template.
  
-I decided to install a cpu compactor "pi install npm:pi-vcc" to avoid running out of context window while compacting. Means that I can use the full server side context window client side. Needs config to replace the auto/default compactor, check ~/.pi/agent/pi-vcc-config.json. It falls back to the default compactor when summary is less than <4096 tokens which again use the LLM that can run out of tokens. So I asked pi to fix the plugin setting. (I first tried npm:pi-compact but then it sometimes answered in Chinese and had a remember tool that I do not want.)+A also did the following to make pi behave: 
 +  - Install a cpu compactor "pi install npm:@sting8k/pi-vcc" (plain npm:pi-vcc is abandoned and buggy). This means that I can use the full server side context window client side. 
 +  - Use "/settings" in pi and configure "HTTP idle timeout" to disabled. For local models with long output this can aovid "Error: Terminated". 
 +  - You can also configure theme in "/settings" to match your terminal colors. 
 +  - It is possible to control if all pre-entered steering prompts should be sent at once with "/settings " and "Steering mode" all. 
  
-But this was suggested by the LLM to control output better:+Editing the system promt was also suggested to control output better:
 <code> <code>
 $ cat ~/.pi/agent/APPEND_SYSTEM.md $ cat ~/.pi/agent/APPEND_SYSTEM.md
-## Reply language +## House rules
- +
-Always respond and output exclusively in the language of the user's latest message (normally English or Swedish and never Chinese). +
- +
-## Docs & comments+
  
-Keep documentation and code comments brief and concise: state intent and the non-obvious "why" in one or two lines. Do not restate what the code already says.+- **Keep documentation and code comments brief and concise:** State intent and the non-obvious "why" in one or two lines. Do not restate what the code already says. 
 +- **Short commit messages:**  
 +  - Subject: A normal Git subject line is 50 characters or fewer, hard limit 72. Create a commit-msg hook to enforce subject length <72 if not already in place. 
 +  - Body: A few lines of intent only (what + why). 
 +- Always **respond and output** exclusively in the language of the user's latest message (normally English or Swedish and never Chinese).
 </code> </code>
  
linux64.1793437642.txt.gz · Last modified: by bengt

Donate Powered by PHP Valid HTML5 Valid CSS Driven by DokuWiki