Elastic Stack is a combination of open-source products that is Elasticsearch, Kibana, Logstash, and Beats known as (ELK stack) that are designed to reliably and securely take data from any source and format, then search, analyze, and visualize it in real-time.

Elastic Stack Components

Elasticsearch is a distributed RESTful search engine with the ability to address a large number of use-cases. It is based on Java and searches through data in a fast lightning motion with powerful analytics that scale at ease. It detects hardware failures to keep your data safe and available by having a secondary cluster as a backup.

Logstash is a data collection engine that integrates data from any source, transforms it, and sends it to your favorite ‘stash’. Logstash filters parse events, identify name fields to build structured data with grok and decipher geo coordinates from IP addresses. Logstash has a framework with plugins that allow you to configure your pipeline in your own way.

Kibana is a user interface that builds data visualizations in a simple and intuitive manner that lets you visualize data and navigate the Elastic stack. It has a canvas that lets you be creative with your live data with logos, colors, and designs unique to you. It allows you to build customized dashboards that enable deeper analysis. You can also export Kibana visualizations as a link, a PDF, or an image.

Beats is an added product that was not there before. They are data shippers installed as agents on servers to send operational data to Elasticsearch. The fast includes Audit data, metrics, log files, cloud data, windows events logs, and network traffic. Beats can send data directly to Elasticsearch or via Logstash.

This guide will show you how to install Elastic Stack 8 on KDE Neon|Kubuntu systems.

Installation Requirements

A KDE Neon|Kubuntu Server with 4GB RAM.
A non-root sudo user.

Install Elastic Stack 8 on KDE Neon / Kubuntu

Update your system packages before you perform any other installation.

### Kubuntu ###
sudo apt update && sudo apt upgrade -y

### KDE Neon ###
sudo apt update && sudo pkcon update -y

Install missing dependencies.

sudo apt install vim wget apt-transport-https curl gpgv gpgsm gnupg-l10n gnupg dirmngr -y

Once that is done proceed to the next steps.

Step 1. Install Java on KDE Neon / Kubuntu

Install OpenJDK 17 using the following command

sudo apt install default-jdk

Confirm the installation by running

$ java --version
openjdk 11.0.15 2022-04-19
OpenJDK Runtime Environment (build 11.0.15+10-Ubuntu-0ubuntu0.20.04.1)
OpenJDK 64-Bit Server VM (build 11.0.15+10-Ubuntu-0ubuntu0.20.04.1, mixed mode, sharing)

Configure the JAVA_HOME path using the following command.

$ sudo vim /etc/profile
export JAVA_HOME=/usr/lib/jvm/default-java

Confirm the settings:

source /etc/profile
echo $JAVA_HOME

Step 2. Install Elasticsearch on KDE Neon / Kubuntu

Add the official Elasticsearch GPG repository key.

curl -fsSL https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo gpg --dearmor -o /etc/apt/trusted.gpg.d/elastic.gpg

Install the Elastic Stack 8 repository using the following command.

echo "deb https://artifacts.elastic.co/packages/8.x/apt stable main" | sudo tee /etc/apt/sources.list.d/elastic-8.x.list

Run a system update.

sudo apt update

Install Elasticsearch 8 on KDE Neon|Kubuntu.

sudo apt install elasticsearch -y

Note down the password for the superuser which you will use to log in to configure Elasticseach.

Step 3. Configure Elasticsearch 8 on KDE Neon / Kubuntu

Most of the settings are already preconfigured. We will change the network host to localhost. You can also add your actual IP address to disable external access.

Access the configuration file using a text editor.

sudo nano /etc/elasticsearch/elasticsearch.yml

Then edit the following lines by uncommenting.

cluster.name: elk8
http.port: 9200

Setting ‘network.host:localhost‘ ensures you access it from that interface alone. You can also set it to 0.0.0.0 to allow access from all interfaces

network.host: 0.0.0.0

Disable SSL security settings:

# Enable security features
xpack.security.enabled: false

xpack.security.enrollment.enabled: false

# Enable encryption for HTTP API client connections, such as Kibana, Logstash, and Agents
xpack.security.http.ssl:
  enabled: false
  keystore.path: certs/http.p12

# Enable encryption and mutual authentication between cluster nodes
xpack.security.transport.ssl:
  enabled: false

Elasticsearch automatically sets the JVM heap size. If you want to manually adjust the heap size, add the ‘Xms‘ and ‘Xms‘  JVM arguments to a custom JVM options file with the extension .options and store it in the jvm.options.d/ directory. 

sudo nano /etc/elasticsearch/jvm.options.d/jvm.options

Set the minimum and maximum heap size as follows. The size might differ from your memory size.

-Xms512m
-Xmx512m

Save and exit the file. Start Elasticsearch service

sudo systemctl restart elasticsearch

You can also enable it to start at boot time.

sudo systemctl enable elasticsearch

Check its status

$ systemctl status elasticsearch
● elasticsearch.service - Elasticsearch
     Loaded: loaded (/lib/systemd/system/elasticsearch.service; enabled; vendor preset: enabled)
     Active: active (running) since Thu 2022-04-14 00:18:10 EAT; 3min 22s ago
       Docs: https://www.elastic.co
   Main PID: 75851 (java)
      Tasks: 61 (limit: 4572)
     Memory: 2.3G
     CGroup: /system.slice/elasticsearch.service
             ├─75851 /usr/share/elasticsearch/jdk/bin/java -Xshare:auto -Des.networkaddress.cache.ttl=60 -Des.network>
             └─76125 /usr/share/elasticsearch/modules/x-pack-ml/platform/linux-x86_64/bin/controller

Use curl command to test connectivity to Elasticsearch:

$ curl -XGET http://127.0.0.1:9200
{
  "name" : "kubuntu",
  "cluster_name" : "elk8",
  "cluster_uuid" : "52V_MNO7QBa46eLOmUzWXQ",
  "version" : {
    "number" : "8.2.0",
    "build_flavor" : "default",
    "build_type" : "deb",
    "build_hash" : "b174af62e8dd9f4ac4d25875e9381ffe2b9282c5",
    "build_date" : "2022-04-20T10:35:10.180408517Z",
    "build_snapshot" : false,
    "lucene_version" : "9.1.0",
    "minimum_wire_compatibility_version" : "7.17.0",
    "minimum_index_compatibility_version" : "7.0.0"
  },
  "tagline" : "You Know, for Search"
}

Step 4. Install Logstash on KDE Neon / Kubuntu

To install Logstash use the following command

sudo apt install logstash -y

Start and enable the service using the following commands.

sudo systemctl start logstash
sudo systemctl enable logstash

To check the status, use the following command.

$ systemctl status logstash
logstash.service - logstash
     Loaded: loaded (/etc/systemd/system/logstash.service; enabled; vendor preset: enabled)
     Active: active (running) since Thu 2022-04-14 00:35:14 EAT; 13s ago
   Main PID: 77417 (java)
      Tasks: 15 (limit: 4572)
     Memory: 395.9M
     CGroup: /system.slice/logstash.service
             └─77417 /usr/share/logstash/jdk/bin/java -Xms1g -Xmx1g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupanc>

Apr 14 00:35:14 kdeneon systemd[1]: Started logstash.
Apr 14 00:35:14 kdeneon logstash[77417]: Using bundled JDK: /usr/share/logstash/jdk
Apr 14 00:35:14 kdeneon logstash[77417]: OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated i>

Step 5. Install Kibana on KDE Neon / Kubuntu

To install Kibana, run the commands below.

sudo apt install kibana -y

Configure Kibana by going to its configuration file.

sudo nano /etc/kibana/kibana.yml

Uncomment the following lines.

server.port: 5601
server.host: "0.0.0.0"
elasticsearch.hosts: ["http://localhost:9200"]

Start and enable the service

sudo systemctl restart kibana
sudo systemctl enable kibana

Allow the Kibana port through the firewall.

sudo ufw allow 5601/tcp

Check Kibana status using the following command

$ systemctl status kibana
● kibana.service - Kibana
     Loaded: loaded (/lib/systemd/system/kibana.service; enabled; vendor preset: enabled)
     Active: active (running) since Thu 2022-04-14 00:54:02 EAT; 36s ago
       Docs: https://www.elastic.co
   Main PID: 80109 (node)
      Tasks: 11 (limit: 4572)
     Memory: 295.7M
     CGroup: /system.slice/kibana.service
             └─80109 /usr/share/kibana/bin/../node/bin/node /usr/share/kibana/bin/../src/cli/dist

Apr 14 00:54:02 kdeneon systemd[1]: Started Kibana.
Apr 14 00:54:20 kdeneon kibana[80109]: [2022-04-14T00:54:20.310+03:00][INFO ][plugins-service] Plugin "metricsEntitie>
Apr 14 00:54:20 kdeneon kibana[80109]: [2022-04-14T00:54:20.401+03:00][INFO ][http.server.Preboot] http server runnin>
Apr 14 00:54:20 kdeneon kibana[80109]: [2022-04-14T00:54:20.454+03:00][INFO ][plugins-system.preboot] Setting up [1] >
Apr 14 00:54:20 kdeneon kibana[80109]: [2022-04-14T00:54:20.457+03:00][INFO ][preboot] "interactiveSetup" plugin is h>
Apr 14 00:54:20 kdeneon kibana[80109]: [2022-04-14T00:54:20.499+03:00][INFO ][root] Holding setup until preboot stage>
Apr 14 00:54:20 kdeneon kibana[80109]: i Kibana has not been configured.
Apr 14 00:54:20 kdeneon kibana[80109]: Go to http://localhost:5601/?code=681763 to get started.

You can also access the Kibana dashboard using the URL http://localhost:5601/?code=681763. The URL might be different on your end.

Step 6. Install Filebeat on KDE Neon / Kubuntu

There are several beats that Elastic search uses to collect data from sources and transport them. We will focus on Filebeat which sends file logs.

To install filebeat, use the following command.

sudo apt install filebeat -y

To configure Filebeat, access its configuration file

sudo nano /etc/filebeat/filebeat.yml

Comment the following lines

#around line 135 
#output.elasticsearch:
  # Array of hosts to connect to .
  # hosts: ["localhost:9200"]

Then uncomment the following lines.

output.logstash:
  hosts: ["localhost:5044"]

This disables filebeat to send outputs directly to Elasticsearch and via Logstash.

Enable the filebeat modules

$ sudo filebeat modules enable system
Enabled system

Next, we load the index template into Elasticsearch.

sudo filebeat setup --index-management -E output.logstash.enabled=false -E 'output.elasticsearch.hosts=["localhost:9200"]'

Start and enable the service

sudo systemctl restart filebeat
sudo systemctl enable filebeat

Check the status of the service

$ systemctl status filebeat
● filebeat.service - Filebeat sends log files to Logstash or directly to Elasticsearch.
     Loaded: loaded (/lib/systemd/system/filebeat.service; enabled; vendor preset: enabled)
     Active: active (running) since Thu 2022-04-14 00:58:49 EAT; 865ms ago
       Docs: https://www.elastic.co/beats/filebeat
   Main PID: 81150 (filebeat)
      Tasks: 7 (limit: 4572)
     Memory: 49.1M
     CGroup: /system.slice/filebeat.service
             └─81150 /usr/share/filebeat/bin/filebeat --environment systemd -c /etc/filebeat/filebeat.yml --path.home>

Apr 14 00:58:49 kdeneon systemd[1]: filebeat.service: Scheduled restart job, restart counter is at 8.
Apr 14 00:58:49 kdeneon systemd[1]: Stopped Filebeat sends log files to Logstash or directly to Elasticsearch..
Apr 14 00:58:49 kdeneon systemd[1]: Started Filebeat sends log files to Logstash or directly to Elasticsearch..
Apr 14 00:58:49 kdeneon filebeat[81150]: {"log.level":"info","@timestamp":"2022-04-14T00:58:49.586+0300","log.origin">
Apr 14 00:58:49 kdeneon filebeat[81150]: {"log.level":"info","@timestamp":"2022-04-14T00:58:49.587+0300","log.origin">

Step 7. Access Kibana Dashboard

You can now access the Kibana dashboard using the URL http://localhost:5601/?code=681763.

Generate the token using and copy from the terminal and paste it to configure Elasticsearch.

sudo /usr/share/elasticsearch/bin/elasticsearch-create-enrollment-token -s kibana

Then log in using the superuser username ‘elastic’ and the password generated on installation.

Then the welcome page opens. Click on ‘explore on my own‘.

Then you will be taken to the next page as shown below.

Create Visualizations with Kibana

Click on the menu then select Dashboard.

For the first time, you can add sample data to use as examples. Once data is added, click on Create visualization.

Then you can drag records to the panel to show the visualized data. You can also change the timeline to have data to show on the visual pane.

The visualized data appears as shown below.

  • You can save and return to the dashboard to view it.
  • You can share the visualization and download it as CSV.
  • You can also choose the way you want the data to be visualized e.g Pie chart.

Then once saved, it appears on the dashboard like below.

You can add more visual data and arrange them on the dashboard.

Conclusion

From this guide, we have installed Elastic stack 8 on KDE Neon|Kubuntu. We have configured each component with the required settings for default use. We have also seen how we can visualize data through Kibana which allows us to have an intuitive web interface that helps us identify different relations among data.

LEAVE A REPLY

Please enter your comment!
Please enter your name here