Kimi K3 Architecture Overview and Notes(sebastianraschka.com) |
Kimi K3 Architecture Overview and Notes(sebastianraschka.com) |
Because it's RNN-like, it has an inherent idea that X comes before Y which comes before Z in the sequence XYZ. Transformers, by default, don't have that. They operate on sets, unordered collections of unique items. They have no idea where those items are in relation to each-other so you have to clue them in.
Because There are 3 KDA layers per attention layer, and 3 KDA layers before the first attention layer, every single token position is going to be able to learn information about where it is in the sequence before the first actual attention layer.
RoPE is actually a bit destructive, so being able to omit it like this is very convenient. Models like Gemma-4 have a similar structure with 5:1 Sliding Window Attention (SWA) layers for every global attention layer. These are cheap, shitty attention layers which handle local information and which go in-between the big powerful ones, KDA serves the same role in this model. In Gemma only the SWA layers have RoPE while the global attention layers omit it. SWA is actual attention, even if it only operates on a small sliding window, so it needs the positional embedding. KDA isn't, so it doesn't.
>Curious to see if it holds up at frontier scale.
I don't know how much more frontier scale you can get than this, but yes, there's no reason why that wouldn't work at larger scales. Honestly, more parameters just makes it easier for the KDA layers to communicate that positional information better.
> Seems to benchmark
yes, but in human usage the differences show up
SWA has a similar issue. Unless you keep the entire KV prefix lying around (which is not unreasonable: you retain flop + bandwidth benefits but lose capacity benefits), you need to start 1 window back from the rollback point, in order to refill the sliding window before going into normal prefill.
1. https://news.ycombinator.com/item?id=49019012 (original chinese)
2. https://news.ycombinator.com/item?id=49052912 (translated english [pdf])