...
| Code Block |
|---|
source llm_env/bin/activate #pip install open-webui==0.2.5 pip install open-webui # 0.6.10 open-webui serve |
| Model | sec to load the model | layers to GPU | |
|---|---|---|---|
DeepSeek R1 Distill Llama 70B | 54.25 | 81/81 | |
llama3.3:70b | 53.34 | 81/81 | |
Qwen3 32B | 28.04 | 65/65 | |
phi3:14b | 19.09 | 41/41 | |
openchat7b | 6.53 | 33/33 | |
llama4:scout | |||
Llama 3.1 70B Instruct 2024 12 | |||
gemma3:27b | |||
mistral-small3.1:24b |
llama.cpp
https://github.com/ggml-org/llama.cpp
...