CentOS 安装 ELK

rensanning

浏览: 3514977 次
性别:
来自: 大连

最近访客更多访客>>

blogtester

lliiqiang

film

newer_fisher

博主相关

博客

微博

相册

留言

关于我

博客专栏

: 使用Titanium Mo...
浏览量：37500

: Cordova 3.x入门...
浏览量：604395

: 常用Java开源Libra...
浏览量：678169

: 搭建 CentOS 6 服...
浏览量：87329

: Spring Boot 入...
浏览量：399865

: 基于Spring Secu...
浏览量：69086

: MQTT入门
浏览量：90519

文章分类

社区版块

存档分类

博客分类：

Linux

ELK（Elasticsearch, Logstash, Kibana），三个开源软件搭建的实时日志分析平台。

官网：
https://www.elastic.co/products

版本：

Elasticsearch 2.3.4 ：日志索引、存储、查询
Logstash 2.3.4 ：日志收集、过滤、转发
Kibana 4.5.3 ：可视化查询 Elasticsearch 的数据
Beats

前提：
设置hostname（做SSL证书需要）、安装Java（Elasticsearch和Logstash需要）

(1)导入GPG key

rpm --import https://packages.elastic.co/GPG-KEY-elasticsearch

(2)添加Yum仓库

# vi /etc/yum.repos.d/elasticsearch.repo
  [elasticsearch-2.x]
  name=Elasticsearch repository for 2.x packages
  baseurl=https://packages.elastic.co/elasticsearch/2.x/centos
  gpgcheck=1
  gpgkey=https://packages.elastic.co/GPG-KEY-elasticsearch
  enabled=1
# vi /etc/yum.repos.d/kibana.repo
  [kibana-4.5]
  name=Kibana repository for 4.5.x packages
  baseurl=http://packages.elastic.co/kibana/4.5/centos
  gpgcheck=1
  gpgkey=http://packages.elastic.co/GPG-KEY-elasticsearch
  enabled=1
# vi /etc/yum.repos.d/logstash.repo
  [logstash-2.3]
  name=Logstash repository for 2.3.x packages
  baseurl=https://packages.elastic.co/logstash/2.3/centos
  gpgcheck=1
  gpgkey=https://packages.elastic.co/GPG-KEY-elasticsearch
  enabled=1
# vi /etc/yum.repos.d/beats.repo
  [beats]
  name=Elastic Beats Repository
  baseurl=https://packages.elastic.co/beats/yum/el/$basearch
  enabled=1
  gpgkey=https://packages.elastic.co/GPG-KEY-elasticsearch
  gpgcheck=1

(3)安装Elasticsearch

# yum -y install elasticsearch
# chkconfig --add elasticsearch
# chkconfig elasticsearch on
# rpm -qc elasticsearch
# vi /etc/elasticsearch/elasticsearch.yml
  network.host: localhost
# service elasticsearch start
# service elasticsearch status
# netstat -nltp | grep -E '9200|9300'
# curl -X GET 'http://localhost:9200'

确认Elasticsearch的索引

# curl http://localhost:9200/_cat/indices

(4)安装Kibana

# yum -y install kibana
# chkconfig --add kibana
# chkconfig kibana on
# rpm -qc kibana
# vi /opt/kibana/config/kibana.yml
  server.host: "localhost"
# service kibana start
# service kibana status
# netstat -nltp | grep 5601
# curl -X GET 'http://localhost:5601'

Kibana Dashboards

# cd /usr/local/src/
# curl -L -O https://download.elastic.co/beats/dashboards/beats-dashboards-1.1.0.zip
# unzip beats-dashboards-1.1.0.zip
# cd beats-dashboards-1.1.0
# ./load.sh

(5)安装Logstash

# yum -y install logstash
# chkconfig --add logstash
# chkconfig logstash on

# hostname
# cd /etc/pki/tls
# openssl req -subj '/CN=ELK_server_fqdn/' -x509 -days 3650 -batch -nodes -newkey rsa:2048 -keyout private/logstash-forwarder.key -out certs/logstash-forwarder.crt

# vi /etc/logstash/conf.d/02-beats-input.conf
input {
  beats {
    port => 5043
    ssl => true
    ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt"
    ssl_key => "/etc/pki/tls/private/logstash-forwarder.key"
  }
}
# vi /etc/logstash/conf.d/10-syslog-filter.conf
filter {
  if [type] == "syslog" {
    grok {
      match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
      add_field => [ "received_at", "%{@timestamp}" ]
      add_field => [ "received_from", "%{host}" ]
    }
    syslog_pri { }
    date {
      match => [ "syslog_timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
    }
  }
}
# vi /etc/logstash/conf.d/30-elasticsearch-output.conf
output {
  elasticsearch {
    hosts => ["localhost:9200"]
    sniffing => true
    manage_template => false
    index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
    document_type => "%{[@metadata][type]}"
  }
}
# service logstash configtest
# service logstash start
# service logstash status

(6)安装Filebeat

# cp /tmp/logstash-forwarder.crt /etc/pki/tls/certs/logstash-forwarder.crt
# cd ~
# curl -O https://gist.githubusercontent.com/thisismitch/3429023e8438cc25b86c/raw/d8c479e2a1adcea8b1fe86570e42abab0f10f364/filebeat-index-template.json
# curl -X PUT 'http://localhost:9200/_template/filebeat?pretty' -d@filebeat-index-template.json
# yum -y install filebeat
# chkconfig --add filebeat
# chkconfig filebeat on
# vi /etc/filebeat/filebeat.yml
# egrep -v '^$|^#|^\s+#' /etc/filebeat/filebeat.yml
filebeat:
  prospectors:
    -
      paths:
        - /var/log/*.log
      input_type: log
      document_type: syslog
  registry_file: /var/lib/filebeat/registry
output:
  logstash:
    hosts: ["localhost:5043"]
    tls:
      certificate_authorities: ["/etc/pki/tls/certs/logstash-forwarder.crt"]
shipper:
logging:
  files:
    rotateeverybytes: 10485760 # = 10MB
# service filebeat start
# service filebeat status
# curl -XGET 'http://localhost:9200/filebeat-*/_search?pretty'

(7)安装Topbeat

# cd ~
# curl -O https://raw.githubusercontent.com/elastic/topbeat/master/etc/topbeat.template.json
# curl -XPUT 'http://localhost:9200/_template/topbeat' -d@topbeat.template.json
# yum -y install topbeat
# vi /etc/topbeat/topbeat.yml
# egrep -v '^$|^#|^\s+#' /etc/topbeat/topbeat.yml
output:
  logstash:
    hosts: ["localhost:5043"]
    tls:
      certificate_authorities: ["/etc/pki/tls/certs/logstash-forwarder.crt"]
# service topbeat start  
# service topbeat status  
# curl -XGET 'http://localhost:9200/topbeat-*/_search?pretty'

(8)收集Nginx日志

# mkdir -p /opt/logstash/patterns
# chown logstash: /opt/logstash/patterns
# vi /opt/logstash/patterns/nginx
NGUSERNAME [a-zA-Z\.\@\-\+_%]+
NGUSER %{NGUSERNAME}
NGINXACCESS %{IPORHOST:clientip} %{NGUSER:ident} %{NGUSER:auth} \[%{HTTPDATE:timestamp}\] "%{WORD:verb} %{URIPATHPARAM:request} HTTP/%{NUMBER:httpversion}" %{NUMBER:response} (?:%{NUMBER:bytes}|-) (?:"(?:%{URI:referrer}|-)"|%{QS:referrer}) %{QS:agent}
# chown logstash: /opt/logstash/patterns/nginx
# vi /etc/filebeat/filebeat.yml
    -
      paths:
        - /var/log/nginx/access.log
      document_type: nginx-access
# service filebeat restart
# vi /etc/logstash/conf.d/11-nginx-filter.conf
filter {
  if [type] == "nginx-access" {
    grok {
      match => { "message" => "%{NGINXACCESS}" }
    }
  }
}
# service logstash restart

(9)收集Apache日志

# vi /etc/filebeat/filebeat.yml
    -
      paths:
        - /var/log/apache2/access.log
      document_type: apache-access
# service filebeat restart
# vi /etc/logstash/conf.d/12-apache.conf
filter {
  if [type] == "apache-access" {
    grok {
      match => { "message" => "%{COMBINEDAPACHELOG}" }
    }
  }
}
# service logstash restart

参考：
https://www.elastic.co/guide/index.html
https://www.digitalocean.com/community/tutorial_series/centralized-logging-with-logstash-and-kibana-on-centos-7

ELK环境搭建.rar (2.9 MB)
下载次数: 1

分享到：

抓取开发者头条分享的所有文章 | 各云服务平台的架构图

2016-07-19 10:02
浏览 2565
评论(0)
分类:开源软件
查看更多

发表评论

您还没有登录,请您登录后再发表评论

最近访客更多访客>>

博主相关

博客专栏

文章分类

社区版块

存档分类

最新评论

CentOS 安装 ELK

评论

发表评论

相关推荐

最近访客 更多访客>>

博主相关

博客专栏

文章分类

社区版块

存档分类

最新评论

CentOS 安装 ELK

评论

发表评论

相关推荐

logrotate入门

Bash简要入门

搭建独立图像处理服务（Thumbor）

搭建分布式文件存储服务（FastDFS）

CentOS 安装 OpenAM

CentOS 安装 Nexus

CentOS 安装 KVM

CentOS 安装 Android

CentOS 安装 Jenkins

CentOS 7 vs CentOS 6的不同

搭建 CentOS 6 服务器(15) - Cacti、Nagios

搭建VirtualBox + Vagrant + CoreOS + Docker环境

搭建 CentOS 6 服务器(13) - Keepalived、HAProxy、LVS

搭建 CentOS 6 服务器(4) - TigerVNC

搭建 CentOS 6 服务器(12) - squid、Varnish

搭建 CentOS 6 服务器(11) - rsync、Amanda

搭建 CentOS 6 服务器(10) - Samba

搭建 CentOS 6 服务器(14) - CVS、SVN、Git

搭建 CentOS 6 服务器(9) - vsftpd、ProFTPD

搭建 CentOS 6 服务器(8) - Apache/Nginx/Jetty、Tomcat、WildFly

最近访客更多访客>>