用ELK打造強大的日志分析平臺,具體拓撲結構如下:
創(chuàng)新互聯(lián)自2013年創(chuàng)立以來,先為廬江等服務建站,廬江等地企業(yè),進行企業(yè)商務咨詢服務。為廬江企業(yè)網站制作PC+手機+微官網三網同步一站式服務解決您的所有建站問題。在這里我們將進行kafka+filebeat+ELK5.4的部署
各軟件版本
jdk-8u131-linux-i586.tar.gz filebeat-5.4.0-linux-x86_64.tar.gz elasticsearch-5.4.0.tar.gz kibana-5.4.0-linux-x86_64.tar.gz logstash-5.4.0.tar.gz kafka_2.11-0.10.0.0.tgz1、JDK安裝配置(略過)
2、ELK安裝與配置
創(chuàng)建ELK用戶,并進行文件解壓 1.elasticsearch配置 [elk@localhost elasticsearch-5.4.0]$ vi config/elasticsearch.yml ..... network.host: 192.168.12.109 # # Set a custom port for HTTP: # http.port: 9200 .......... 保存,啟動 [elk@localhost elasticsearch-5.4.0]$ nohup bin/elasticsearch & 驗證 # [elk@localhost elasticsearch-5.4.0]$ curl http://192.168.12.109:9200 { "name" : "aCA2ApK", "cluster_name" : "elasticsearch", "cluster_uuid" : "Ea4_9kXZSaeDL1fYt4lUUQ", "version" : { "number" : "5.4.0", "build_hash" : "780f8c4", "build_date" : "2017-04-28T17:43:27.229Z", "build_snapshot" : false, "lucene_version" : "6.5.0" }, "tagline" : "You Know, for Search" } 2、kibana安裝與配置 [elk@localhost kibana-5.4.0-linux-x86_64]$ vi config/kibana.yml ## Kibana is served by a back end server. This setting specifies the port to use. server.port: 5601 # Specifies the address to which the Kibana server will bind. IP addresses and host names are both valid values. # The default is 'localhost', which usually means remote machines will not be able to connect. # To allow connections from remote users, set this parameter to a non-loopback address. server.host: "192.168.12.109" .......... # The URL of the Elasticsearch instance to use for all your queries. elasticsearch.url: "http://192.168.12.109:9200" .......... [elk@localhost kibana-5.4.0-linux-x86_64]$ nohup bin/kibana & 在瀏覽器訪問 能訪問即可3、kafka安裝與配置
這里我們只做單機192.168.12.105部署單節(jié)點《centos kafka單包單機部署》
4、logstah安裝與配置
[elk@localhost logstash-5.4.0]$ vi nginx.conf 這里新生成一個配置文件 input { kafka { codec => "json" topics_pattern => "logstash-.*" bootstrap_servers => "192.168.12.105:9092" auto_offset_reset => "latest" group_id => "logstash-g1" } } filter { if "nginx-accesslog" in [tags] { grok { match => { "message" => "%{IPORHOST:http_host} %{IPORHOST:clientip} - %{USERNAME:remote_user} \[%{HTTPDATE:timestamp}\] \"(?:%{WORD:http_verb} %{NOTSPACE:http_request}(?: HTTP/%{NUMBER:http_version})?|%{DATA:raw_http_request})\" %{NUMBER:response} (?:%{NUMBER:bytes_read}|-) %{QS:referrer} %{QS:agent} %{QS:xforwardedfor} %{NUMBER:request_time:float} %{GREEDYDATA:traceID}"} } mutate { convert => ["status","integer"] convert => ["body_bytes_sent","integer"] convert => ["request_time","float"] } geoip { source=>"remote_addr" } date { match => [ "timestamp","dd/MMM/YYYY:HH:mm:ss Z"] } useragent { source=>"http_user_agent" } } if "tomcat-accesslog" in [tags] { grok { match => { "message" => "%{IPORHOST:clientip} \[%{HTTPDATE:timestamp}\] \"(?:%{WORD:http_verb} %{NOTSPACE:http_request}(?: HTTP/%{NUMBER:http_version})?|%{DATA:raw_http_request})\" %{NUMBER:response} (?:%{NUMBER:bytes_read}|-) %{QS:referrer} %{NUMBER:request_time:float} %{GREEDYDATA:traceID}"} } date { match => [ "timestamp","dd/MMM/YYYY:HH:mm:ss Z"] } } } output { elasticsearch { hosts => ["192.168.12.109:9200"] index => "logstash-%{type}-%{+YYYY.MM.dd}" document_type => "%{type}" } #stdout { codec => rubydebug } } 保存,并啟動 [elk@localhost logstash-5.4.0]$ nohup bin/logstash -f nginx.conf &5、filebeat安裝與配置
將filebeat分別拷貝到需要采集的服務器,進行解壓,在這里我們分別采集Nginx,tomcat日志
Nginx服務器
[user@localhost filebeat-5.4.0-linux-x86_64]$ vi filebeat.yml filebeat.prospectors: - input_type: log paths: - /data/programs/nginx/logs/access.log tags: ["nginx-accesslog"] document_type: nginxaccess tags: ["nginx-test-194"] output.kafka: enabled: true hosts: ["192.168.12.105:9092"] topic: logstash-%{[type]} [user@localhost filebeat-5.4.0-linux-x86_64]$nohup filebeat -c filebeat.yml &tomcat服務器
[user@localhost filebeat-5.4.0-linux-x86_64]$ vi filebeat.yml filebeat.yml filebeat.prospectors: - input_type: log paths: - /data/tomcat/logs/localhost_access_log* tags: ["tomcat-accesslog"] document_type: tomcataccess tags: ["tomcat103"] output.kafka: enabled: true hosts: ["192.168.12.105:9092"] topic: logstash-%{[type]} [user@localhost filebeat-5.4.0-linux-x86_64]$nohup filebeat -c filebeat.yml &完成以上,我們的平臺就搭建好了,接下來我們創(chuàng)建索引
輸入:logstash-nginxaccess*
輸入logstash-tomcataccess*
數據通過filebeat到kafka、ELK成功展示出來
來張炫圖
另外有需要云服務器可以了解下創(chuàng)新互聯(lián)cdcxhl.cn,海內外云服務器15元起步,三天無理由+7*72小時售后在線,公司持有idc許可證,提供“云服務器、裸金屬服務器、高防服務器、香港服務器、美國服務器、虛擬主機、免備案服務器”等云主機租用服務以及企業(yè)上云的綜合解決方案,具有“安全穩(wěn)定、簡單易用、服務可用性高、性價比高”等特點與優(yōu)勢,專為企業(yè)上云打造定制,能夠滿足用戶豐富、多元化的應用場景需求。
當前題目:filebeat+kafka+ELK5.4安裝與部署-創(chuàng)新互聯(lián)
標題網址:http://bm7419.com/article14/cdgige.html
成都網站建設公司_創(chuàng)新互聯(lián),為您提供網站排名、商城網站、全網營銷推廣、微信公眾號、關鍵詞優(yōu)化、企業(yè)建站
聲明:本網站發(fā)布的內容(圖片、視頻和文字)以用戶投稿、用戶轉載內容為主,如果涉及侵權請盡快告知,我們將會在第一時間刪除。文章觀點不代表本網站立場,如需處理請聯(lián)系客服。電話:028-86922220;郵箱:631063699@qq.com。內容未經允許不得轉載,或轉載時需注明來源: 創(chuàng)新互聯(lián)