ROMAIN ROUGNY rrougny@vub.ac.be at 15:34, 27 November 2015

2015-11-27T15:34:06Z

← Older revision		Revision as of 15:34, 27 November 2015
Line 6:		Line 6:
	** '''/opt/logstash/patterns''': contains the patterns that can be used in the filters		** '''/opt/logstash/patterns''': contains the patterns that can be used in the filters
	<br>		<br>


	* To understand all the eventual patterns to match parts of your log line, have a look at the pattern file '''grok-patterns'''.		* To understand all the eventual patterns to match parts of your log line, have a look at the pattern file '''grok-patterns'''.
	** a pattern looks like:		::''' ''[To help you with the patterns and the grok filter, you can use [http://grokconstructor.appspot.com/do/match this excellent site], where you can input log lines, patterns, the section of grok filter related to the patterns, etc, to construct your match.]''' ''

			:* a pattern looks like:
	PATTERN_1 ${NUMBER:stored_in_my_number}%{SPACE}{IP:ip}		PATTERN_1 ${NUMBER:stored_in_my_number}%{SPACE}{IP:ip}
	PATTERN_2 %{INT:floor} %{GREEDYDATA:address}		PATTERN_2 %{INT:floor} %{GREEDYDATA:address}
Line 28:		Line 32:
	}		}

	To help you with the patterns and the grok filter, you can use [http://grokconstructor.appspot.com/do/match this excellent site], where you can input log lines, patterns, the section of grok filter related to the patterns, etc, to construct your match.

	* You then need to restart logstash:		* You then need to restart logstash:

ROMAIN ROUGNY rrougny@vub.ac.be: Created page with "You need to ssh to our elk machine: log10 contains logstash, elasticsearch as well as kibana. ssh root@log10.iihe.ac.be * There are 2 directories to work with: ** '''/etc/lo..."

2015-11-27T15:14:20Z

Created page with "You need to ssh to our elk machine: log10 contains logstash, elasticsearch as well as kibana. ssh root@log10.iihe.ac.be * There are 2 directories to work with: ** '''/etc/lo..."

New page

You need to ssh to our elk machine: log10 contains logstash, elasticsearch as well as kibana.
ssh root@log10.iihe.ac.be

* There are 2 directories to work with:
** '''/etc/logstash/conf.d/''' : contains the filters to parse you log lines
** '''/opt/logstash/patterns''': contains the patterns that can be used in the filters
<br>
* To understand all the eventual patterns to match parts of your log line, have a look at the pattern file '''grok-patterns'''.
** a pattern looks like:
PATTERN_1 ${NUMBER:stored_in_my_number}%{SPACE}{IP:ip}
PATTERN_2 %{INT:floor} %{GREEDYDATA:address}
PATTERN_TOT %{PATTERN_1}%{SPACE}%{PATTERN2}
:* PATTERN_TOT would match:
0412345678 127.0.0.1 666 road to higgs

my_number => 0412345678
ip => 127.0.0.1
floor => 666
address => road to higgs
<br>

* For logs sent through (r)syslog, there will always be a bunch of information (like timestamp, program used, Facility, host, host_ip) that are prefixed to your log line. It is already extracted in '''/etc/logstash/conf.d/logstash-complex.conf'''
"<%{POSINT:syslog_pri}>%{SYSLOGTIMESTAMP:timestamp} %{IPORHOST:logsource} (?:%{PROG:program}(?:\[%{POSINT:pid}\])?: )?%{GREEDYDATA:msg}",:
Therefore your log line is stored in '''message''' (msg is mutated in message), and that's what you need to match using the grok filter. <br>
To match lines like the example before, expecting you have put the patterns in a file, you just need to make a myservice.conf file in '''/etc/logstash/conf.d/''' with:
filter{
grok { match => [ "message", "%{PATTERN_TOT}"] }
}

To help you with the patterns and the grok filter, you can use [http://grokconstructor.appspot.com/do/match this excellent site], where you can input log lines, patterns, the section of grok filter related to the patterns, etc, to construct your match.

* You then need to restart logstash:
service logstash restart

* Your log lines should now be filtered in the '''Logstash Search''' [http://log10.iihe.ac.be/index.html#/dashboard/elasticsearch/Logstash%20Search dashboard].
:: You should see some new fields appearing for each line of logs if you've constructed your grok matching correctly!

Log parsing with logstash - Revision history

ROMAIN ROUGNY rrougny@vub.ac.be at 15:34, 27 November 2015

ROMAIN ROUGNY rrougny@vub.ac.be: Created page with "You need to ssh to our elk machine: log10 contains logstash, elasticsearch as well as kibana. ssh root@log10.iihe.ac.be * There are 2 directories to work with: ** '''/etc/lo..."