Elasticsearch

Elasticsearch is a distributed, RESTful search and analytics engine built on Apache Lucene, designed for horizontal scalability, speed, and flexibility. Unlike traditional databases optimized for transactional workloads, Elasticsearch excels at full-text search (think Google-like search for your application), log and event data analysis, real-time aggregations, and exploring massive datasets. Data is stored as JSON documents and indexed for near-instantaneous search across millions of records. Elasticsearch is the "E" in the ELK Stack (Elasticsearch, Logstash, Kibana)—the industry-standard platform for log aggregation, monitoring, and observability. Its power comes from sophisticated text analysis (stemming, synonyms, fuzzy matching), distributed architecture scaling to petabytes, and rich aggregation framework for analytics dashboards.

Best For:
Elasticsearch excels for applications requiring sophisticated search capabilities and log analysis. It's ideal for adding Google-quality search to your application (e-commerce product search, content discovery, enterprise document search), centralized log aggregation and analysis (collecting logs from hundreds of servers, searching/analyzing for debugging), application performance monitoring and observability (tracking metrics, traces, errors), security information and event management (SIEM), and real-time analytics dashboards. Elasticsearch is particularly valuable when you need full-text search with relevance ranking, faceted navigation (filters/refinements), auto-complete suggestions, or log analysis across distributed systems. It's the default choice for search-heavy applications.

Common Questions

Elasticsearch vs. PostgreSQL full-text search: which to use?
Elasticsearch offers significantly more powerful search features: better relevance scoring, sophisticated text analysis (35+ language analyzers, custom tokenizers), fuzzy matching, geospatial search, aggregations, and highlighting. PostgreSQL's full-text search handles basic needs (simple search within a single database). Use PostgreSQL when search is minor feature alongside transactional data. Use Elasticsearch when search is core feature, you need advanced capabilities (auto-complete, "did you mean?", facets), or you're searching across multiple data sources. Common pattern: store data in PostgreSQL, sync to Elasticsearch for search.

How much does Elasticsearch cost?
Elasticsearch is free and open source (Apache 2.0 license for core, SSPL for Elasticsearch itself). Self-host Elasticsearch cluster free on your infrastructure. Elastic Cloud (managed service) costs from $95/month (4 GB RAM cluster) to thousands for production deployments. AWS OpenSearch Service (Amazon's fork of Elasticsearch) starts ~$15/month (t3.small.search) to thousands for large clusters. Most costs come from infrastructure (RAM, storage, CPU)—Elasticsearch is resource-intensive. Many companies self-host free Elasticsearch but pay for Kibana visualizations, premium features (security, alerting), or commercial support.

Is OpenSearch the same as Elasticsearch?
OpenSearch is an AWS fork of Elasticsearch 7.10, created when Elastic changed Elasticsearch licensing from Apache 2.0 to SSPL (Server Side Public License) in 2021. OpenSearch is 100% open source (Apache 2.0) and compatible with Elasticsearch 7.x APIs. It's actively developed by AWS and community. Use Elasticsearch for latest features, commercial support from Elastic, and integrated Elastic stack (Kibana, Beats). Use OpenSearch for true open source, AWS integration, and avoiding SSPL licensing. Both are excellent; choice often depends on cloud provider (AWS → OpenSearch, others → Elasticsearch) and licensing preferences.

Can Elasticsearch be used as a primary database?
No, Elasticsearch is designed for search and analytics, not transactional data storage. It lacks ACID transactions, foreign key constraints, and durability guarantees of traditional databases. Data in Elasticsearch should be derived from authoritative source (PostgreSQL, MongoDB). Use Elasticsearch as search layer on top of primary database. Elasticsearch can lose data during cluster failures or splits—always have source data elsewhere. Pattern: write to PostgreSQL (source of truth), sync to Elasticsearch (search layer), read from both depending on operation.

Pricing & Plans

Elasticsearch Open Source (Free): Core search and analytics engine with distributed architecture, full-text search, aggregations, geospatial queries, and basic security. Free forever under Elastic License and SSPL. Self-host on any infrastructure. No restrictions on data size or nodes.

Elastic Cloud (Managed Elasticsearch):

• Standard Tier: $95/month (4 GB RAM, 96 GB storage, single AZ). Includes Elasticsearch, Kibana, basic security. $225/month (8 GB, 192 GB). Production tiers: $450-2,000+/month with high availability, multiple AZs, dedicated instances.
• Enterprise Tier: Adds machine learning, advanced security (RBAC, SAML), alerting, and SLA. Pricing: ~$500-5,000+/month based on scale.
• Serverless (Preview): Pay-per-use model. Still in early access.

AWS OpenSearch Service (Amazon's Fork):

• Development: t3.small.search (1.5 GB RAM) ~$15/month. t3.medium.search (3 GB) ~$30/month. Good for testing.
• Production: m6g.large.search (8 GB) ~$100/month. r6g.xlarge.search (32 GB) ~$350/month. Production clusters typically $200-2,000+/month with Multi-AZ, automated snapshots. UltraWarm storage (cheaper cold data) reduces costs.

Self-Hosting Costs: Infrastructure only (servers/VMs with sufficient RAM and fast storage). Elasticsearch recommends 64 GB RAM per node for production, SSD storage, multiple nodes for resilience. Expect $100-1,000+/month depending on scale. More complex operations but maximum cost control.

Limitations

Elasticsearch is resource-intensive, requiring substantial RAM (entire index must fit in memory for performance), fast SSD storage, and multiple nodes for production—expect higher infrastructure costs than traditional databases. It's not suitable as a primary data store due to lack of ACID transactions, potential data loss during failures, and no foreign key constraints. Complex queries can be slow despite indexing—poorly designed aggregations across billions of documents may timeout. Elasticsearch has a learning curve: understanding mappings (schema), analyzers, index settings, and cluster health requires expertise. Updates and deletes are expensive operations (Lucene immutability requires marking deleted, reindexing). Version upgrades across major versions can require reindexing all data. Elasticsearch licensing changed from Apache 2.0 to SSPL in 2021, causing AWS and others to fork (OpenSearch), creating ecosystem fragmentation.

Getting Started

Install Elasticsearch: Docker (docker run -d -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" -e "xpack.security.enabled=false" docker.elastic.co/elasticsearch/elasticsearch:8.11.0), or download from elastic.co. Start server (runs on port 9200). Check status: curl localhost:9200. Create index and add document:

# Create document
curl -X POST "localhost:9200/products/_doc" -H 'Content-Type: application/json' -d'
{
  "name": "Laptop",
  "brand": "Dell",
  "price": 899,
  "category": "Electronics"
}'

# Search documents
curl -X GET "localhost:9200/products/_search" -H 'Content-Type: application/json' -d'
{
  "query": {
    "match": { "name": "laptop" }
  }
}'

# Aggregation (facets)
curl -X GET "localhost:9200/products/_search" -H 'Content-Type: application/json' -d'
{
  "size": 0,
  "aggs": {
    "brands": { "terms": { "field": "brand.keyword" } }
  }
}'

Use Elasticsearch from your application (Python example: pip install elasticsearch):

from elasticsearch import Elasticsearch
es = Elasticsearch(["http://localhost:9200"])

# Index document
es.index(index="products", document={"name": "Laptop", "price": 899})

# Search
result = es.search(index="products", query={"match": {"name": "laptop"}})
print(result['hits']['hits'])

Install Kibana for visualizations and exploration (docker run -d -p 5601:5601 docker.elastic.co/kibana/kibana:8.11.0). Configure index mappings before indexing production data (define analyzers, field types). Use bulk API for importing large datasets. Monitor cluster health with _cluster/health endpoint. Learn query DSL (match, term, bool, range), aggregations (terms, date_histogram, stats), and relevance tuning. Follow Elastic's documentation and free training courses.