How to Set Up ELK Stack on Ubuntu 24.04

Introduction

Before we begin talking about how to set up ELK Stack on Ubuntu 24.04, let's briefly understand – What is ELK Stack?

ELK Stack is a powerful data analytics solution used by businesses to manage and analyze large volumes of data. It consists of three core components:

1. Elasticsearch: A search and analytics engine that stores, searches, and analyzes data in real-time.
2. Logstash: A tool that collects, processes, and transports data from multiple sources for indexing.
3. Kibana: A visualization platform that helps users explore, visualize, and navigate data stored in Elasticsearch.

ELK Stack simplifies the process of collecting, analyzing, and visualizing data, enabling businesses to gain valuable insights and make informed decisions. It's widely used for monitoring applications, security analytics, and log management.

In this tutorial, you will set up ELK Stack on Ubuntu 24.04. We will also address a few FAQs on how to set up ELK Stack on Ubuntu 24.04.

Prerequisites

• An Ubuntu 24.04 dedicated server with sudo privileges is required.
• The server should have a minimum of 4GB of RAM (more is recommended for handling larger datasets).
• Additionally, create an A record that points to your server's IP address, such as kibana.example.com.

Step 1: Update Your System

Make sure your system is up-to-date before you begin.

sudo apt update
sudo apt upgrade -y

Step 2: Install Java

Elasticsearch needs Java to function. You can use OpenJDK, a free and open-source version of the Java Platform, for this purpose.

sudo apt install openjdk-17-jdk -y

Check the installation:

java -version

Step 3: Install Elasticsearch

Add the Elasticsearch GPG key:

wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -

Add the Elasticsearch Repository:

sudo sh -c 'echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" > /etc/apt/sources.list.d/elastic-7.x.list'

Next, install Elasticsearch:

sudo apt update
sudo apt install elasticsearch -y

Start and enable Elasticsearch:

sudo systemctl start elasticsearch
sudo systemctl enable elasticsearch

Verify the installation:

curl -X GET "localhost:9200/"

Output

{
  "name" : "elk",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "yMKeohL1SnafmH9eiMXPAA",
  "version" : {
    "number" : "7.17.21",
    "build_flavor" : "default",
    "build_type" : "deb",
    "build_hash" : "d38e4b028f4a9784bb74de339ac1b877e2dbea6f",
    "build_date" : "2024-04-26T04:36:26.745220156Z",
    "build_snapshot" : false,
    "lucene_version" : "8.11.3",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

Step 4: Install Logstash

Run the following command to install Logstash:

sudo apt install logstash -y

Start and enable Logstash:

sudo systemctl start logstash
sudo systemctl enable logstash

Step 5: Configure Elasticsearch

To enable authentication, you need to activate xpack security. Modify the Elasticsearch configuration file:

sudo nano /etc/elasticsearch/elasticsearch.yml

Include the following security settings:

xpack.security.enabled: true
xpack.security.authc.api_key.enabled: true

Restart Elasticsearch:

sudo systemctl restart elasticsearch

Configure passwords for built-in users:

Use the following command to set passwords for built-in users, including the elastic user:

sudo /usr/share/elasticsearch/bin/elasticsearch-setup-passwords interactive

You will be prompted to enter passwords for the following users:

Initiating the setup of passwords for reserved users elastic,apm_system,kibana,kibana_system,logstash_system,beats_system,remote_monitoring_user.
You will be prompted to enter passwords as the process progresses.
Please confirm that you would like to continue [y/N]

Follow the prompts and set the passwords as instructed.

Step 6: Install Kibana

Run the following command to install Kibana:

sudo apt install kibana -y

Start and enable Kibana:

sudo systemctl start kibana
sudo systemctl enable kibana

Next, open the Kibana configuration file:

sudo nano /etc/kibana/kibana.yml

Uncomment and set the following lines, or add them to the end of the file:

Note: Substitute kibana.example.com with your domain name, add the xpack section, and replace elasticsearch.password with your actual password.

server.port: 5601
server.host: "0.0.0.0"
server.name: "kibana.example.com"
elasticsearch.hosts: ["http://localhost:9200"]
elasticsearch.username: "kibana_system"
elasticsearch.password: "password@123"

# Add following lines in the file:
xpack.security.enabled: true

Save the changes and exit.

To apply the updates, restart Kibana:

sudo systemctl restart kibana

Step 7: Configure Logstash to Collect and Parse Logs

Generate a configuration file for Logstash:

sudo nano /etc/logstash/conf.d/logstash.conf

Here is an example configuration for reading syslog messages:

input {
  file {
    path => "/var/log/syslog"
    start_position => "beginning"
  }
}

filter {
  grok {
    match => { "message" => "%{SYSLOGTIMESTAMP:timestamp} %{SYSLOGHOST:hostname} %{DATA:program} (?:\[%{POSINT:pid}\])? %{GREEDYDATA:message}" }
  }
  date {
    match => [ "timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
    timezone => "UTC"
  }
}

output {
  elasticsearch {
    hosts => ["https://your_domain:9200"]
    index => "syslog-%{+YYYY.MM.dd}"
    ssl => true
    cacert => "/etc/letsencrypt/live/your_domain/fullchain.pem"
    user => "logstash_system"
    password => "your_logstash_system_password"
  }
}

Note: Replace your_logstash_system_password with the password you set in step 5.

Save the changes and exit.

Restart Logstash to apply the new configuration:

sudo systemctl restart logstash

Step 8: Install and Configure Nginx

Next, install Nginx to act as a proxy server, allowing access to Elasticsearch via your domain name.

sudo apt install nginx -y

Create a new Nginx configuration file for Kibana:

sudo nano /etc/nginx/sites-available/kibana.example.com

Note: Substitute kibana.example.com with your domain name.

Add the following configuration to the file, using your actual domain name in place of kibana.example.com

server {
    listen 80;
    server_name kibana.example.com;

    location / {
        proxy_pass http://localhost:5601;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}

Activate the configuration:

sudo ln -s /etc/nginx/sites-available/kibana.example.com /etc/nginx/sites-enabled/

Test the Nginx configuration and reload it:

sudo nginx -t
sudo systemctl reload nginx

Step 9: Configure Firewall (If applicable):

If you have UFW firewall enabled, follow this step to add the HTTP and HTTPS ports:

ufw allow 80/tcp
ufw allow 443/tcp
ufw reload

Step 10: Obtain SSL Certificates with Let's Encrypt Certbot

First, install Certbot with the following command:

sudo apt install certbot python3-certbot-nginx -y

Obtain and install the SSL certificate by following the prompts.

Replace kibana.example.com with your actual domain name.

sudo certbot --nginx -d kibana.example.com

Certbot will automatically configure the SSL settings for Nginx. Check the configuration file /etc/nginx/sites-available/kibana.example.com to ensure it includes the SSL settings.

Step 11: Test the Setup

Finally, access Kibana using your domain name:

Open your web browser and go to https://kibana.example.com. You should see the Kibana login screen.

We have successfully demonstrated how to install the ELK Stack on an Ubuntu 24.04 server.

FAQs to Set Up ELK Stack on Ubuntu 24.04

Conclusion

We hope this tutorial helped you understand how to set up ELK Stack on Ubuntu 24.04.

If you have any queries, please leave a comment below, and we’ll be happy to respond to them for sure.