A new inference engine to run Kimi K3 2.78T parameter with 29GB of RAM

A new inference engine to run Kimi K3 2.78T parameter with 29GB of RAM(marcobambini.substack.com)

4 points by marcobambini 31 days ago | 2 comments

Current speed is “approximately one third of a token per second”

marcobambini 31 days ago | |

Right, we trade speed for the ability to run a 2.7T-parameter model while preserving accuracy. It is a first version, and we plan to improve the inference performance.

armchairhacker 31 days ago |

Current speed is “approximately one third of a token per second”

marcobambini 31 days ago | |

Right, we trade speed for the ability to run a 2.7T-parameter model while preserving accuracy. It is a first version, and we plan to improve the inference performance.