elk(日志监控系统搭建),elastic search,kibana,logstash,filebeat搭建

knight_black_bob

浏览: 823155 次
性别:
来自: 北京

最近访客更多访客>>

bian1024

lli

cqh520llr

hxgdragon

博主相关

博客

微博

相册

留言

关于我

文章分类

社区版块

存档分类

博客分类：

elk
elastic search
kibana
logstash

elk elasticsearch kibana logstash 搭建

架构图

演示效果

日志输入

kibana 查看

35. elk 安装

准备工作

wget -c https://download.elastic.co/elasticsearch/release/org/elasticsearch/distribution/rpm/elasticsearch/2.3.3/elasticsearch-2.3.3.rpm
wget -c https://download.elastic.co/logstash/logstash/packages/centos/logstash-2.3.2-1.noarch.rpm
wget https://download.elastic.co/kibana/kibana/kibana-4.5.1-1.x86_64.rpm
wget -c https://download.elastic.co/beats/filebeat/filebeat-1.2.3-x86_64.rpm

35.0 java 安装

yum install java-1.8.0-openjdk -y

35.1 elasticsearch 安装

yum localinstall elasticsearch-2.3.3.rpm -y
systemctl daemon-reload
systemctl enable elasticsearch
systemctl start elasticsearch
systemctl status elasticsearch
systemctl status elasticsearch -l
检查 es 服务
rpm -qc elasticsearch
	/etc/elasticsearch/elasticsearch.yml
	/etc/elasticsearch/logging.yml
	/etc/init.d/elasticsearch
	/etc/sysconfig/elasticsearch
	/usr/lib/sysctl.d/elasticsearch.conf
	/usr/lib/systemd/system/elasticsearch.service
	/usr/lib/tmpfiles.d/elasticsearch.conf
修改防火墙对外
firewall-cmd --permanent --add-port={9200/tcp,9300/tcp}
firewall-cmd --reload
firewall-cmd  --list-all

35.2 安装 kibana

yum localinstall kibana-4.5.1-1.x86_64.rpm –y
systemctl enable kibana
systemctl start kibana
systemctl status kibana
systemctl status kibana -l

检查kibana服务运行
netstat -nltp

firewall-cmd --permanent --add-port=5601/tcp
firewall-cmd --reload
firewall-cmd  --list-all
访问地址 http://192.168.206.130:5601/

35.3 安装 logstash

yum localinstall logstash-2.3.2-1.noarch.rpm –y
cd /etc/pki/tls/ && ls
创建证书
openssl req -subj '/CN=baoyou.com/' -x509 -days 3650 -batch -nodes -newkey rsa:2048 -keyout private/logstash-forwarder.key -out certs/logstash-forwarder.crt
cat /etc/logstash/conf.d/01-logstash-initial.conf

input {
  beats {
    port => 5000
    type => "logs"
    ssl => true
    ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt"
    ssl_key => "/etc/pki/tls/private/logstash-forwarder.key"
  }
}

filter {
  if [type] == "syslog-beat" {
    grok {
      match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
      add_field => [ "received_at", "%{@timestamp}" ]
      add_field => [ "received_from", "%{host}" ]
    }
    geoip {
      source => "clientip"
    }
    syslog_pri {}
    date {
      match => [ "syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]
    }
  }
}

output {
  elasticsearch { }
  stdout { codec => rubydebug }
}

启动logstash
systemctl start logstash
/sbin/chkconfig logstash on
检查服务
 netstat -ntlp

添加防火墙对外
firewall-cmd --permanent --add-port=5000/tcp
firewall-cmd --reload
firewall-cmd --list-all



配置 es
cd /etc/elasticsearch/
mkdir es-01
mv *.yml es-01
vim elasticsearch.yml

http:
  port: 9200
network:
  host: baoyou.com
node:
  name: baoyou.com
path:
  data: /etc/elasticsearch/data/es-01


systemctl restart elasticsearch
systemctl restart logstash

3.4 filebeat 安装

yum localinstall filebeat-1.2.3-x86_64.rpm -y

cp logstash-forwarder.crt /etc/pki/tls/certs/.

cd /etc/filebeat/ && tree

vim filebeat.yml
filebeat:
  spool_size: 1024
  idle_timeout: 5s
  registry_file: .filebeat
  config_dir: /etc/filebeat/conf.d
output:
  logstash:
    hosts:
    - elk.test.com:5000
    tls:
      certificate_authorities: ["/etc/pki/tls/certs/logstash-forwarder.crt"]
    enabled: true
shipper: {}
logging: {}
runoptions: {}

mkdir conf.d && cd conf.d

vim authlogs.yml
filebeat:
  prospectors:
    - paths:
      - /var/log/secure
      encoding: plain
      fields_under_root: false
      input_type: log
      ignore_older: 24h
      document_type: syslog-beat
      scan_frequency: 10s
      harvester_buffer_size: 16384
      tail_files: false
      force_close_files: false
      backoff: 1s
      max_backoff: 1s
      backoff_factor: 2
      partial_line_waiting: 5s
      max_bytes: 10485760

vim syslogs.yml
filebeat:
  prospectors:
    - paths:
      - /var/log/messages
      encoding: plain
      fields_under_root: false
      input_type: log
      ignore_older: 24h
      document_type: syslog-beat
      scan_frequency: 10s
      harvester_buffer_size: 16384
      tail_files: false
      force_close_files: false
      backoff: 1s
      max_backoff: 1s
      backoff_factor: 2
      partial_line_waiting: 5s
      max_bytes: 10485760

service filebeat start
chkconfig filebeat on
netstat -aulpt

访问地址 http://192.168.206.130:5601/

备注：参看文章 elk 日志监控系统

http://467754239.blog.51cto.com/4878013/1700828/

https://my.oschina.net/itblog/blog/547250

https://www.ibm.com/developerworks/cn/opensource/os-cn-elk/

http://www.cnblogs.com/hanyifeng/p/5509985.html （我用该文章搭建成功了）

http://blog.csdn.net/dabokele/article/details/51765136

https://cloud.tencent.com/community/article/562397