linux64
Differences
This shows you the differences between two versions of the page.
| Both sides previous revisionPrevious revisionNext revision | Previous revision | ||
| linux64 [2026/10/30 22:27] – [vllm in podman for fast qwen3.8:27b ai] bengt | linux64 [2026/11/07 19:26] (current) – [Using pi.dev as ai harness] bengt | ||
|---|---|---|---|
| Line 905: | Line 905: | ||
| ==== vllm in podman for fast qwen3.8:27b ai ==== | ==== vllm in podman for fast qwen3.8:27b ai ==== | ||
| - | I found https:// | + | I found https:// |
| So I translated that recipe to a podman line to use with my startup script serving on the ollama port so I do not have to reconfigure clients using the openai interface: | So I translated that recipe to a podman line to use with my startup script serving on the ollama port so I do not have to reconfigure clients using the openai interface: | ||
| < | < | ||
| - | tag=v0.30.0 | + | tag=v0.32.0 |
| port=11434 | port=11434 | ||
| cd $HOME | cd $HOME | ||
| - | loginctl enable-linger | + | loginctl enable-linger |
| # vllm according to https:// | # vllm according to https:// | ||
| Line 934: | Line 934: | ||
| --api-key none \ | --api-key none \ | ||
| --trust-remote-code \ | --trust-remote-code \ | ||
| - | --kv-cache-dtype | + | --kv-cache-dtype |
| - | --max-model-len | + | --max-model-len |
| --max-num-seqs 2 \ | --max-num-seqs 2 \ | ||
| --compilation-config ' | --compilation-config ' | ||
| Line 953: | Line 953: | ||
| </ | </ | ||
| - | It failed to let vllm download the model trying to divide by 0 but when I dowloaded the model using the wget method from the github page it worked! Also reduced | + | It failed to let vllm download the model trying to divide by 0 but when I dowloaded the model using the wget method from the github page it worked! Also reduced " |
| + | |||
| + | < | ||
| + | REPO=gittensor-model-hub/ | ||
| + | REV=0cc27958cefbbe231782ec8511de8c4eb5233348 | ||
| + | mkdir -p ~/ | ||
| + | for f in config.json generation_config.json hf_quant_config.json \ | ||
| + | | ||
| + | | ||
| + | | ||
| + | | ||
| + | | ||
| + | wget -c " | ||
| + | done | ||
| + | </ | ||
| This is right on the edge for what my anker c300x ups can handle. I may need to get a new ups. You can tell the cards to use less max power with " | This is right on the edge for what my anker c300x ups can handle. I may need to get a new ups. You can tell the cards to use less max power with " | ||
| + | ==== Using pi.dev as ai harness ==== | ||
| Switched to pi.dev to be able to configure qwen3.8:27b to use medium instead of xhigh thinking: | Switched to pi.dev to be able to configure qwen3.8:27b to use medium instead of xhigh thinking: | ||
| Line 973: | Line 988: | ||
| " | " | ||
| " | " | ||
| - | " | + | " |
| " | " | ||
| " | " | ||
| Line 986: | Line 1001: | ||
| In this https:// | In this https:// | ||
| - | I decided | + | A also did the following |
| + | - Install | ||
| + | - Use "/settings" | ||
| + | | ||
| + | - It is possible to control if all pre-entered steering prompts should be sent at once with "/ | ||
| - | But this was suggested | + | Editing the system promt was also suggested to control output better: |
| < | < | ||
| $ cat ~/ | $ cat ~/ | ||
| - | ## Reply language | + | ## House rules |
| - | + | ||
| - | Always respond and output exclusively in the language of the user's latest message (normally English or Swedish and never Chinese). | + | |
| - | + | ||
| - | ## Docs & comments | + | |
| - | Keep documentation and code comments brief and concise: | + | - **Keep documentation and code comments brief and concise:** State intent and the non-obvious " |
| + | - **Short commit messages:** | ||
| + | - Subject: A normal Git subject line is 50 characters or fewer, hard limit 72. Create a commit-msg hook to enforce subject length <72 if not already in place. | ||
| + | - Body: A few lines of intent only (what + why). | ||
| + | - Always **respond and output** exclusively in the language of the user's latest message (normally English or Swedish and never Chinese). | ||
| </ | </ | ||
linux64.1793399267.txt.gz · Last modified: by bengt