Ask HN: How to build full text search at scale? Data Store - Elasticsearch
Scale - 10 million writes/day (500 GB/day), about 100K search queries per day Trying to figure out 1 - How to control access to data (multi-tenancy where there are ~100K tenants) 2 - Database design - Indexes and Shards and best practices around mixing different types of documents in a single index. |
Elasticsearch is built on-top-of Lucene which is a Java API that you can use in pretty much any application. If you already have a system that can search the MySQL clusters then I would recommend hooking Lucene into that system instead of standing up another one.
So basically, you need to run a web server that serves a search page in which your users can create and submit queries. The web server receives the query and then routes it to the appropriate search handler. The web server should handle access control and there are several standard different approaches for this.
Solr and ElasticSearch can both be used in this manner.
>> Database design - Indexes and Shards and best practices around mixing different types of documents in a single index.
Depends a lot on what your users want to get in their search results. A first step would be to identify the primary text fields they want to search in each document type, then create a standard text field in the schema into which each document's primary content gets indexed. You can get fancy by running different document types through different analyzer/tokenizer chains (for example if they were in different languages) and you can do a lot of 'cheating/preprocessing' here so that the primary search text field has good information in it.
Choose one system and learn it well.
Search Engine - Elasticsearch