Optimizing Elasticsearch performance requires trade-offs between business priorities and actual cluster configuration. This includes ensuring that thread pools are properly sized to support the expected query and indexing workload.
Another critical factor is sizing the disk capacity for each node in your cluster. It’s important to monitor disk utilization and consider the size of index, search, and bulk queues.
Use A Ha Cluster
HA clusters distribute business applications across multiple servers to minimize the impact of system failure and downtime. You can often restore operations within 30 seconds to a few minutes. However, many are still wondering how to deploy Elasticsearch on Kubernetes.
If you have a HA cluster, search, and indexing requests will be automatically routed to another node in case your primary node fails. This ensures that you will continue accessing your data and allows you to diagnose and fix the issue quickly.
Adding new hardware to your HA cluster increases computing, memory, and storage – the three main resources that affect Elasticsearch performance. But other, less obvious factors can cause performance bottlenecks. A common one is a limited number of primary shards. Adding more nodes will increase capacity but may still limit indexing throughput on your most active indices if a fixed number of primary shards restricts them.
Reduce The Number Of Shards.
Creating many indices and shards can cause significant management load and degrade indexing and search performance. The optimal number of shards and indices will depend on your data, SLAs, and growth patterns. You should run a performance test with a small subset of your data to determine the correct number of shards and indices to use.
The number of shards per node affects read, write, and fault recovery performance. In some cases, the primary shards of an index may be spread too thinly across nodes. This can cause hotspots that require excessive CPU and disk bandwidth, resulting in slow searches.
The index disk space is another important metric to consider when determining the number of shards. The shards in an index must be initialized, started, and relocated to perform ingest operations. Additionally, unused mapped fields carry an overhead for disk and memory usage.
Freeze Old Indices
The way data is organized can have a big impact on query performance. It’s important to organize indices to match the expected query patterns. For example, if queries search for documents based on time ranges, it might be best to organize by date rather than a more document-oriented approach.
Mapping updates can be resource-intensive and negatively impact query performance. It’s important to minimize mapping changes by ensuring that the mapping accurately reflects the data schema and only making changes when necessary.
Unused indices eat memory and slow down search speed. Using the freeze API allows you to stop unused indices from consuming memory. Frozen indices can still be queried but cannot receive new writes. This optimization is beneficial for read-only indices that are rarely used and will benefit from being force-merged into a single segment, which can use simpler and more efficient data structures to perform searches.
Scale Horizontally
Horizontal scaling is a great way to handle unexpected spikes in demand and ensure that your product’s servers remain available at all times. It also reduces latency for users in different geographic regions and improves fault tolerance.
One of the most important factors in Elasticsearch’s performance is how well the system utilizes its hardware resources. When deploying in a Kubernetes environment, adding nodes as virtual machines (VMs) or containers results in shared hardware resources such as CPU cores, RAM, and storage. This can lead to poor cluster performance if the total container CPU limits exceed the maximum allowable limit.
Another important factor is how well the cluster performs when handling index-heavy workloads. Indexing requires a lot of memory and can be an important source of bottlenecks for Elasticsearch at scale. To improve performance, consider adding more workers to the stateful set. This will increase the number of parallel tasks that can be handled, allowing for more efficient processing of search and indexing requests.
Optimize Storage
The disks that store Elasticsearch data can become a bottleneck for indexing performance. To address this issue, consider using fast storage like SSDs for your Elasticsearch cluster.
Another way to improve indexing speed is to reduce the number of segments per shard. This will help reduce the time a search request spends checking each shard segment. This can be done by triggering the Force Merge API on one or more of your indices. However, balancing the benefits of reducing index size against the cost of performing frequent merges is important.
You can also improve indexing performance by setting the number of fields to cache for a given shard in ES. ES must load a field into the cache whenever a field is used, which takes time. Once it is in the cache, subsequent field uses will be faster. However, this technique can be risky if you have many indices and use a large number of fields in your queries.
