Try SambaNova chat: 1T param LLM, 500 tokens/SEC(coe-1.cloud.snova.ai) |
Try SambaNova chat: 1T param LLM, 500 tokens/SEC(coe-1.cloud.snova.ai) |
Each of our nodes has a huge amount of DDR attached, in addition to copious amounts of on-chip HBM and SRAM. This allows the system to switch between a variety of different models of different sizes and architectures at lightning speed. A highlight is one based on Llama2 7b, similar to the Groq demo, but executing with bf16/fp32 instead of int8. (And using only 8 chips instead of 568!)