...
| Model | prompt eval rate | second prompt | eval rate | |||||
|---|---|---|---|---|---|---|---|---|
| Name | Params, B | SIZE, GB | CPU | GPU | CPU | GPU | CPU | GPU |
| gemma3:12b | 12 | 8.1 | 15.73 | - | 109.35 | - | 6.55 | - |
| gemma3:4b | 4 | 3.3 | 48.55 | - | 284.47 | - | 17.88 | - |
| gemma3:1b | 1 | 0.8 | 140.40 | - | 177.24 | - | 45.26 | - |
orca-mini:3b | 3 | 2.0 | 68.21 | 1049.24 | 21.91 | |||
orca-mini:7b | 7 | 3.8 | 33.20 | 576.83 | 11.76 | |||
orca-mini:13b | 13 | 7.4 | 17.27 | 307.94 | 6.23 | |||
orca-mini:70b | 70 | 38 | 3.28 | 62.05 | 1.24 | |||
| phi4:14b-q4_K_M | 14 | 9.1 | 12.98 | 9.87 | 100.02 | 113.45 | 5.78 | 6.77 |
| phi4-mini:3.8b-q4_K_M | 3.8 | 2.5 | 49.26 | - | 186.63 | - | 18.83 | - |
| phi4:14b-fp16 | 14 | 29 | 12.45 | 11.53 | 35.53 | 40.16 | 2.08 | 2.09 |
| openthinker:32b-v2-fp16 | 32 | 65 | 4.45 | 7.31 | 20.24 | 22.31 | 0.81 | 0.90 |
| openthinker:32b | 32 | 19 | 5.93 | 4.59 | 66.31 | 69.74 | 2.60 | 2.78 |
| dolphin-phi:2.7b | 2.7 | 1.6 | 85.67 | 86.81 | 744.07 | 649.43 | 25.42 | 21.73 |
| dolphin3:8b | 3.8 | 4.9 | 26.04 | 30.97 | 325.85 | 373.30 | 10.76 | 12.58 |
| tinyllama:1.1b | 1.1 | 0.6 | 198.18 | 112.98 | 2595.12 | 2211.21 | 62.99 | 57.53 |
| deepseek-v2:16b | 16 | 8.9 | 59.47 | 15.83 | 361.51 | 175.02 | 24.39 | 12.00 |
| phi3:14b | 14 | 7.9 | 15.60 | 10.51 | 101.53 | 128.59 | 6.07 | 7.67 |
| llama3.3:70b | 70 | 42 | 2.60 | 1.54 | 21.35 | 23.37 | 1.25 | 1.37 |
| mistral-small3.1:24b | 24 | 15 | 7.71 | - | 1321.32 | - | 3.64 | - |
| llama4:scout | 17 | 67 | 11.14 | - | 1683.33 | - | 4.81 | - |
| openchat:7b | 7 | 4.1 | 30.47 | 27.15 | 273.39 | 361.04 | 11.10 | 14.81 |
| qwen3:32b | 32 | 20 | 5.67 | 2.84 | 38.88 | 41.60 | 2.53 | 2.73 |
| gemma3:27b | 27 | 17 | 6.60 | - | 49.38 | - | 3.04 | - |
| deepseek-r1:70b | 70 | 42 | 2.63 | 0.89 | 12.39 | 14.13 | 1.24 | 1.38 |
...