Same model, same Q4_K_M label: 5.02, 5.07 and 5.27 bits per weight

Same model, same Q4_K_M label: 5.02, 5.07 and 5.27 bits per weight(github.com)

9 points by logickkk1 3 hours ago | 2 comments

p_stuart82 2 hours ago |

Does this catch intermittent fallback between requests?

logickkk1 2 hours ago | |

Yes, picchio monitors a running llama-server on a timer and flags when the prefill/decode ratio goes CPU-shaped. I hot-swapped one mid-run. probe 4 caught ENGAGED -> NOT ENGAGED.

p_stuart82 2 hours ago |

Does this catch intermittent fallback between requests?

logickkk1 2 hours ago | |

Yes, picchio monitors a running llama-server on a timer and flags when the prefill/decode ratio goes CPU-shaped. I hot-swapped one mid-run. probe 4 caught ENGAGED -> NOT ENGAGED.