I’m interested in playing with vector databases to detect interesting anomalies in a large volume of logs, like 1TB / day.

Is it reasonable to attempt to generate embeddings for every log event that hits the system? At 1TB/day, it’s like 1B log events per day, over 10k per second.

Would I just have to sample some tiny percentage of log events to generate embeddings for?

The volume feels too high, but I’m curious if others do this successfully. I want this to be reasonably cheap, like less than 1 cent per million log events.

Twitter seems to be doing something like this for all tweets at much higher volume. But I don’t want to spend too much money :)