01 - Introduction

Background

Elasticsearch is an open source analytics and full-text search engine. It is often used for enabling search functionality for various applications. It provides various search related functionalities such as Full Text Search, Synonmyms, Boosting etc.

Elastic Search can also be used for analytical purposes and other Machine Learning. Elasticsearch can also be used for Anomality Detection and other various scenarios.

How does it work ?

Data is stored in Elasticsearch as documents. A document’s data is separated into fields (Similar to colunns in relational databases). A document is just a json object, which contains fields and values.

Why Elastic Search ?

Elastic Search is easy to use and highly scalable. It’s based on Apache Lucene. It is distributed by nature, it is able to scale very well on large data volumes and throughputs. Elastic Search has a very vibrant community which makes it a very helpful if you need any help with Elastic Search.

Elastic Stack

Elasticstack consists of Elastic Search, Kibana, Logstash and XPack

Lucene based Search Engine

Kibana

An analytics and visualization platform which sits on top of elastic search to provide you dashboards and analytical tool to query elastic search data

Logstash

An event processing pipeline

  • Inputs -
  • Filters - Data Enrichment
  • Outputs -

Logstash is horizontally scalable. Logstash pipeline is defined in a proprietary markup format that is similar to JSON.

X-Pack

Set of additional plugins added to the Elasticsearch & Kibana, which enables features such as Security, Monitoring, Alerting, Reporting etc.

Security

Adds authentication and authorization for Elastic Search and Kibana. In regards to authentication, Kibana can integrate with LDAP, Active Directory and other technologies to provide authentication. You can also add users and roles, and configure exactly what a given user or role is allowed to access.

Monitoring

Gain insight into how the Elastic Stack in running

  • CPU
  • Memory Usage
  • Disk Space
  • Setup Alerting for Elastic Stack

Reporting

  • Reports can be generated on-demand or scheduled
  • Generate reports when certain conditions are fulfilled
  • Export data

Machine Learning

  • Enables machine learning for Elasticsearch & KIbana
  • Anomaly Detection
  • Forecasting
  • Graph - Analyze the relationship in your data
  • Elasticsearch SQL - Query Elasticsearch with SQL

Beats

A collection of data shippers that sends data to Elasticsearch or Logstash