The search and analytics engine  Elas­tic­search is one of the best open source solutions for indexing and struc­tur­ing large databases. However, it’s possible to gain valuable insights in later analysis of raw data if it’s vi­su­al­ized in a clear and easily un­der­stand­able form. The vi­su­al­iza­tion tool, Kibana, has been developed es­pe­cial­ly for pre­sent­ing Elas­tic­search data and will be discussed in this tutorial.

What is Kibana?

Kibana is an ex­ten­si­ble web interface for visually rep­re­sent­ing collected data. Together with Elas­tic­search and the data pro­cess­ing tool Logstash, it forms the so-called ELK stack (also called Elastic Stack). This open-source suite enables users to collect data from different server sources (and in any format), arrange it, and prepare it for an­a­lyt­i­cal purposes. In addition to the ability to visualize the data processed by Logstash and Elas­tic­search, Kibana also offers automatic real-time analysis, a very flexible search algorithm, and different data view types (his­tograms, graphs, pie charts, etc.). In the dashboard, the in­di­vid­ual in­ter­ac­tive vi­su­al­iza­tions can then be combined to form a dynamic overall picture that can be filtered and searched.

Mz8avM1oOKQ.jpg To display this video, third-party cookies are required. You can access and change your cookie settings here.

As a web-based ap­pli­ca­tion written in JavaScript, Kibana can be used across platforms. Costs only arise if you use the hosting service, Elastic Cloud, offered by the developer. This paid service allows you to implement and organize a secure Elas­tic­search Kibana cluster on Amazon or Google without having to provide your own resources.

Kibana tutorial: First steps with the vi­su­al­iza­tion tool

Kibana offers a huge range of functions that can be used to display prepared database stocks. However, before you can filter and visualize the in­for­ma­tion in the dashboard so that the desired key values can easily be viewed, analyzed, and evaluated in the long term, you have a good bit of work ahead of you. With this Kibana tutorial we want to make it easier for you to get started with the powerful web interface. This article explains how to install Kibana correctly, how to create your own dashboard, and how to integrate existing data into Elastic’s vi­su­al­iza­tion tool.

Step 1: How to get Kibana up and running

Since Kibana is designed to display data that has been indexed using Elas­tic­search, you will first need to install the search and analytics engine. The cor­re­spond­ing packages for Windows, macOS, and Linux can be found in the Elas­tic­search download center. The pre­req­ui­site is that you have a current Java runtime en­vi­ron­ment (64-bit) installed.

Kibana itself is also available as cross-platform software for Windows, macOS, and Linux (RPM, DEB). Since the ap­pli­ca­tion is based on the JavaScript runtime en­vi­ron­ment Node.js, the various in­stal­la­tion packages also contain the necessary Node.js binaries used to run the vi­su­al­iza­tion tool – sep­a­rate­ly main­tained versions are not supported. The different packages (ZIP com­pressed) can be found like Elas­tic­search on the Elastic website.

Note

Linux and Mac users can also install Kibana from the Elastic repos­i­to­ry using the apt and yum package managers. Detailed in­struc­tions can be found in the online manuals.

Once you have unpacked the Kibana package, run the bin/kibana (macOS, Linux) or bin\kibana.bat (Windows) file to get the Kibana server up and running.

Image: Windows 10 command line: Kibana server initialization
Ini­tial­iz­ing the Kibana server takes a few seconds.

You then access the Kibana backend via the address "http://localhost:5601" in your browser (Elas­tic­search must already be running for this to work).

Image: Kibana: home screen
The Kibana user interface.

Step 2: Feed Kibana with data

To be able to take a closer look at the Kibana dashboard and its functions in this tutorial, the ap­pli­ca­tion must first be supplied with data. On the Elastic website, there are three free down­load­able database samples, which we use here for testing purposes. The three databases listed above are "shake­speare.json" (database of the complete works of William Shake­speare), "accounts.zip" (set of fic­ti­tious accounts), and "logs.jsonl.gz" (set of randomly generated log files).

Download and unzip the three files (account and log file) and save them to the location of your choice.

Image: Database samples for Kibana on elastic.co
To download the sample databases, you can also simply click on the re­spec­tive links – the download will then start au­to­mat­i­cal­ly.

Before you submit the data, it is necessary to create mappings for the Shake­speare and server log database fields. These mappings divide the documents in the index into logical groups and also specify the prop­er­ties of the fields, such as their search­a­bil­i­ty. The ap­pro­pri­ate tool for con­fig­ur­ing the mappings is the console, which can be found in the Kibana interface under the menu items "Dev Tools" à "Console."

Image: Kibana: Console plugin
The Kibana console plugin provides an interface that allows you to interact directly with Elas­tic­search’s REST API.

Now insert the following mappings in sequence via PUT request:

PUT /shakespeare
{
 "mappings": {
  "doc": {
   "properties": {
    "speaker": {"type": "keyword"},
    "play_name": {"type": "keyword"},
    "line_id": {"type": "integer"},
    "speech_number": {"type": "integer"}
   }
  }
 }
}
PUT /logstash-2015.05.18
{
    "mappings": {
        "log": {
            "properties": {
                "geo": {
                    "properties": {
                        "coordinates": {
                            "type": "geo_point"
                        }
                    }
                }
            }
        }
    }
}
PUT /logstash-2015.05.19
{
    "mappings": {
        "log": {
            "properties": {
                "geo": {
                    "properties": {
                        "coordinates": {
                            "type": "geo_point"
                        }
                    }
                }
            }
        }
    }
}
PUT /logstash-2015.05.20
{
    "mappings": {
        "log": {
            "properties": {
                "geo": {
                    "properties": {
                        "coordinates": {
                            "type": "geo_point"
                        }
                    }
                }
            }
        }
    }
}
Image: Kibana console: inputting database mappings
By clicking on the green triangle symbol, you can display the answer to the request entered in the Kibana console.

Now use the bulk API of Elas­tic­search to load the data records via curl in the terminal. In Windows, use the Pow­er­Shell with Invoke-Rest­Method instead (code example below):

curl -H 'Content-Type: application/x-ndjson' -XPOST 'localhost:9200/bank/account/_bulk?pretty' --data-binary @accounts.json
curl -H 'Content-Type: application/x-ndjson' -XPOST 'localhost:9200/shakespeare/doc/_bulk?pretty' --data-binary @shakespeare_6.0.json
curl -H 'Content-Type: application/x-ndjson' -XPOST 'localhost:9200/_bulk?pretty' --data-binary @logs.jsonl
Invoke-RestMethod "http://localhost:9200/bank/account/_bulk?pretty" -Method Post -ContentType 'application/x-ndjson' -InFile "accounts.json"
Invoke-RestMethod "http://localhost:9200/shakespeare/doc/_bulk?pretty" -Method Post -ContentType 'application/x-ndjson' -InFile "shakespeare_6.0.json"
Invoke-RestMethod "http://localhost:9200/_bulk?pretty" -Method Post -ContentType 'application/x-ndjson' -InFile "logs.jsonl"
Image: Windows Powershell, which displays the loading of the Kibana data set
Make sure that you execute the commands for loading the Kibana data sets in the directory where the three sample databases are located.
Note

Depending on the computing power, inputting the data sets can take several minutes.

Switch back to the Kibana console to verify the success of the loading process with the following GET request:

GET /_cat/indices?v

If the data is in­te­grat­ed as planned, the output looks like this:

Image: Kibana console: output of the index request
The health status "yellow" indicates that Elas­tic­search has assigned all primary shards (parts of a data index), but not all replicas (back-ups of the shards).

Step 3: Defining a first index pattern

In order for Kibana to know which data it should process, you must create cor­re­spond­ing patterns for the indices "Shake­speare," "bank," and "logstash." You can define the former as follows:

  1. Open the menu "Man­age­ment" and click on "Index Patterns." When you create the first index pattern, the "Create index pattern" page will open au­to­mat­i­cal­ly. Al­ter­na­tive­ly, you can access it using the button with this written on it.
  2. Enter "shakes*" into the field "Index pattern" and click on "Next step."
  3. Since no special con­fig­u­ra­tion is required for this pattern, skip the next setup step and complete the pattern creation directly by clicking "Create index pattern."
Image: Kibana interface: menu for creating the index pattern
A newly created index pattern must match at least one index that contains existing data. Otherwise, you cannot continue the setup process.

Repeat the steps for the pattern "ba*" which is au­to­mat­i­cal­ly assigned to the index "bank."

Finally, define an index pattern called "logstash*" for the three server log indexes. With this pattern, however, you do not skip the con­fig­u­ra­tion menu, but select the entry "@timestamp" in the dropdown menu "Time Filter field name," as these records contain time series data. Then click on "Create index pattern" as for the two previous patterns.

Image: Kibana: time filter configuration menu for the logstash* pattern
Time series data, such as server logs, are data records with data that is or has been con­tin­u­ous­ly recorded.

Step 4: Browse inserted datasets

After you feed your Kibana server with records, you can now start an Elas­tic­search search query to search these records and filter the results. To do this, go to the “Discover” menu in Kibana and select the index pattern for your search using the small triangle icon in the left menu bar. As part of this Kibana dashboard tutorial, we have chosen the account record (ba*):

Image: Kibana: list of the entries of a fictitious data set
Kibana lists the in­di­vid­ual entries of the selected record, with the ba* record con­tain­ing typical bank account cat­e­gories such as balance, name, age, etc.

As a test, you can filter the bank account record to see only accounts that meet certain criteria. For example, to specif­i­cal­ly search for accounts that have a balance of over $47,500 and belong to people over 38 years of age, type the following command in the search box:

balance:>47500 AND age:>38

Discover then returns the entries for the four accounts 97, 177, 878, and 916 that cor­re­spond to the selected criteria.

Image: Kibana: results of data set filtering
With the button "Add a filter" you can easily define your own filters for the selected database.

With the button "Add a filter" you can easily define your own filters for the selected database.

Step 5: Visualize data

With the prepa­ra­tions made so far in this Kibana tutorial, you are now able to visualize the im­ple­ment­ed data to bring your dashboard to life. Here is where an example of a pie chart for the bank account database is created. On the one hand, this diagram should show which pro­por­tion of the total of 1,000 accounts falls into certain bank account balance ranges, and on the other hand, how the age-specific dis­tri­b­u­tion within these divisions turns out.

In the first step, open the "Visualize" menu and click on "Create a vi­su­al­iza­tion" to get a list of the available vi­su­al­iza­tion types. Then select the "Pie" option.

Image: Kibana: selection menu for the visualization types
The vi­su­al­iza­tion types available in Kibana include heat maps, bar charts, and area charts.

At the beginning, you will only see a simple circle that sum­ma­rizes all entries of the database, since no cat­e­gories have been defined yet. These are also called "buckets" in Kibana and can be created under the menu item of the same name.

To first define the in­di­vid­ual account balance divisions, click on "Split Slices" and select "Range" from the "Ag­gre­ga­tion" drop-down menu:

Image: Kibana: definition window for diagram categories
You can use the "Range" type to create in­di­vid­ual value ranges for selected fields of a data record.

Under "Field" search for the entry "balance" and click on it, then click on the "Add Range" button four times to define the following six bank account balance cat­e­gories:

0 999
1000 2999
3000 6999
7000 14999
15000 30999
31000 50000

Then click “Apply Changes” (triangle symbol), and the pie chart will show the dis­tri­b­u­tion of accounts according to the defined account balance divisions.

Image: Kibana: pie chart with defined categories
Kibana au­to­mat­i­cal­ly creates a legend to the created diagram, whose position (right, left, top, bottom) can be adjusted in the options.

In the second step, you add another ring to the diagram that vi­su­al­izes the dis­tri­b­u­tion of age classes for the in­di­vid­ual account balance areas. To do this, click on "Add sub-buckets," then on "Split Slices" again and select "Terms" from the drop-down menu. Search under "Field" for the entry "age" and accept the changes via "Apply changes."

Image: Kibana: pie chart with defined categories and subcategories
If you move the mouse over the in­di­vid­ual com­po­nents of a diagram, Kibana will show you further tool tips unless you have de­ac­ti­vat­ed this in the options.

You can now save the vi­su­al­iza­tion very easily using the "Save" button located in the upper menu bar.

Step 6: Or­ga­niz­ing the dashboard

The Kibana Dashboard is covered briefly in this tutorial, so you'll create your first test dashboard using the search and vi­su­al­iza­tion you saved in steps 4 and 5. To do this, select the dashboard in the page nav­i­ga­tion and then click "Create new dashboard" and then "Add." Kibana will now au­to­mat­i­cal­ly list all saved vi­su­al­iza­tions or Elas­tic­search searches:

Image: Kibana dashboard: menu for adding new panels
The search function makes it easy to find the desired dashboard com­po­nents, even with a large number of vi­su­al­iza­tions or Elas­tic­search search results.

Left-click to add the account balance vi­su­al­iza­tion and the sample search result to the dashboard, whereupon you can view both in separate panels in the dashboard:

Image: Kibana dashboard: tutorial test example
The "Share" button allows you to share your Kibana dashboard with other users or integrate it into your website at any time.

You can now modify the panels, for example by resizing them or changing their position. It is also possible to display a vi­su­al­iza­tion or search result on the entire screen, or delete it from the dashboard. With many vi­su­al­iza­tions, you can also use “Inspect” to display ad­di­tion­al in­for­ma­tion about the un­der­ly­ing data and queries.

Note

If you remove a panel from the Kibana dashboard, the saved vi­su­al­iza­tion or search will be preserved.

Go to Main Menu