Install Elasticsearch, Kibana 4 , fluentd (Opensource splunk) with syslog clients

So used splunk some times but it has its limit (money) so now Im testing

1. Java

first install java on your server. Get java from here http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html

 yum localinstall jdk-8u25-linux-x64.rpm

And install it on your server.

2. Elasticsearch

Get it from here http://www.elasticsearch.org/download I installed the rpm and run

https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-1.4.0.Beta1.noarch.rpm
yum localinstall elasticsearch-1.4.0.Beta1.noarch.rpm

I hade to make some settings in this file my vps only hade 512m

vi /etc/sysconfig/elasticsearch
/etc/init.d/elasticsearch start

So moving on

3. Kibana 4

Download kibana from here http://www.elasticsearch.org/overview/kibana/installation/

cd /var/www/html
wget https://download.elasticsearch.org/kibana/kibana/kibana-4.0.0-BETA1.1.tar.gz
tar zxvf kibana-4.0.0-BETA1.1.tar.gz
mv kibana-4.0.0-BETA1.1 kibana
chown apache:apache -R kibana

4. Install fluentd

http://docs.fluentd.org/articles/install-by-rpm

curl -L http://toolbelt.treasuredata.com/sh/install-redhat.sh | sh

Install gems needed

yum install libcurl-devel
/usr/lib64/fluent/ruby/bin/fluent-gem install fluent-plugin-elasticsearch

open this file and have only this in the file

vi /etc/td-agent/td-agent.conf
<match td.*.*>
type tdlog
apikey YOUR_API_KEY
auto_create_table
buffer_type file
buffer_path /var/log/td-agent/buffer/td
</match>
<source>
type syslog
port 42185
tag syslog
</source>
<source>
type forward
</source>
<match syslog.**>
type elasticsearch
logstash_format true
flush_interval 10s # for testing
</match>

 

Restart the agent

/etc/init.d/td-agent restart

Time for sending some logs to the server

5. Client

in rsyslog open the file /etc/rsyslog.conf and add at the buttom

*.* @127.0.0.1:42185

  3 comments for “Install Elasticsearch, Kibana 4 , fluentd (Opensource splunk) with syslog clients

  1. April 5, 2014 at 12:07 am

    Nice Post, I was comparing logstash with fluentd as a replacement to my Splunk install and your post helped. Thanx!

  2. Sarveshwar Singh
    May 13, 2016 at 5:37 am

    In my early days of Linux(10 days), your post helped me a lot. Thanks!

    I tried to install step [4. Install fluentd] at first. Got following error

    Building native extensions. This could take a while…
    ERROR: Error installing fluent-plugin-elasticsearch:
    ERROR: Failed to build gem native extension.

    [Solution]
    gcc was also required,

    yum install libcurl-devel

    yum -y install libcurl-devel gcc

Leave a Reply

%d bloggers like this: