Hetzner is working on LLM Inference

Hetzner is working on LLM Inference(sliplane.io)

39 points by jonas_scholz 2 hours ago | 13 comments

This seems like a smart move, given their ability to host efficiently. I approve of efforts to make the cost of inference for smaller useful models slowly approach 'close to zero' and there are many good paths for getting there. It is useful for companies to get fast hosting for the class of smaller models they may end up hosting in house.

jonas_scholz 36 minutes ago | |

I really hope they dont stop at the small models though! The bigger ones that dont fit on a single GPU are more interesting I think

swiftcoder 46 minutes ago |

It would certainly be interesting to have a highly respected EU-native inference provider, if only to make the regulatory gods happy

jonas_scholz 41 minutes ago | |

agree. I have a usecase for a bigger open-weight model hosted by an EU company and the selection isnt really great, hard to make the regulatory gods (and the devs) happy at the same time right now

sofixa 32 minutes ago | |

Scaleway have two separate (one fully managed one a bit less) services for that:

https://www.scaleway.com/en/generative-apis/

https://www.scaleway.com/en/inference/

jonas_scholz 30 minutes ago | | |

yea, but no prompt caching right? This makes it unusable for my usecase at least, the cost would be insane

ano-ther 1 hour ago |

Good to see more developments in this space. I quite like this service, which is a little further than Hetzner and has several models to choose from: https://www.infomaniak.com/en/hosting/ai-services

jonas_scholz 1 hour ago | |

Interesting, didn’t know about them! Weird model selection though, no glm or deepseek?

archerx 1 hour ago | |

Infomaniak is such a shitty company, I had to use them on a previous job I worked at and dealing with them was awful.

MASNeo 36 minutes ago | | |

That makes it sound like dealing with Hetzner is easy. Is the case?

embedding-shape 1 hour ago |

> The enable_thinking option is worth mentioning. Without it, the model can spend a surprising amount of the completion budget reasoning before it returns a visible answer.

Straight up the opposite, which the name makes abundantly clear, with the option it does reasoning, without it it doesn't...

jonas_scholz 1 hour ago | |

My writing wasnt clear here I think, without the option it defaults to reasoning enabled. With "without it" I meant without enable_thinking=False!

cyanydeez 44 minutes ago | | |

llamacpp has a reasoning-budget and reasoning-message setting that can both be a global or per header setting. Using it, you can stop it's reasoning token count and insert a message at the point you stopped it.

This allows both the client and server to customize it. I typically use a message that tells it to compact the conversation and use subagents. I find the reasoning gets bloated when it's failed to do whatever task it's doing and often times it either has too little context (subagents) or its context is bloated (compact).

This works fairly well to get it to extend workable life up to ~1M on a local model.