#logstash

thanks!

can i use host parameter in output pulgin to send the logs to more than one server??

nene: which output plugin?

logstash output plugin

untergeek: ^^

nene: there is no logstash output plugin. Did you mean the elasticsearch output plugin?

yes elasticearch output plugin

no, you cannot send to multiple hosts that way

however, you can send to multiple hosts by declaring another elasticsearch output in the output block

ok

untergeek: i am getting below error

java.io.StreamCorruptedException: invalid internal transport message format

paste your configuration into a gist or something, please, and link here

it seems that something that should be connecting isn't, but I can't tell from this one line

I'll be back in about 15 minutes

try to change the name of tour cluster!!

so please wait until I'm here again

of your!!!

indeed, don't use protocol => node, use protocol => http (prevents the need to use cluster name)

that will be the default in Logstash 2.0 anyway

bbl

rubytor: i am using diff cluster names for each host

hello, guys

want to ask, is somebody using latest logstash in production (is it stable?)?

trying to move on it earlier, but have few issues with it

I'm talking about 1.5 branch

i have amessage i filter with grok but want to replace/rename the %{URIHOST} to logsource field , any way to do this ?

rename => [ "URIHOST", "logsource" ] in mutate not seems to do the trick

I'm back

tiv: I use 1.5.4

I have no problems with it

tiv: sometimes issues are with plugins, rather than the core Logstash release

tiv: if you have problems, let us know, we would like to help address them

untergeek, thank you. will try to move on today

violet-rpi, you can try to add field with name you want, then remove old field

violet-rpi, also, you can just modify|write your own grok pattern for that

but i need the contet of the old field in the new field

content*

i'm trying to parse nginx error logs , but it is kind of hard, i'm kinsd of new to this

violet-rpi, try to add separated mutate {...} script

violet-rpi, do you use one config file?

violet-rpi: %{URIHOST} in grok would identify the data, but not store it in a field. %{URIHOST:field_name} would store the URIHOST value in field_name

http://pastebin.com/6AbFaVEu the part of the nginx parsing so far

Title: [JSON] grok { break_on_match => "false" - Pastebin.com (at pastebin.com)

uber: thanks i'll try that

violet-rpi: looking at your config, the only fields you're keeping are pid and message.out

violet-rpi, can you give output of this?

violet-rpi, if you have it

Title: [JSON] { "_index": "logstash-2015.09.29", "_type": "logs", "_id": "K3y_4X3FQoC - Pastebin.com (at pastebin.com)

violet-rpi: see, there is no URIHOST field. You should assign these to fields to use them

violet-rpi, %{URIHOST:logsource}"

well without the rename => [ "%{URIHOST}", "%{logsource}" ] it shows the urihost

%{URIHOST:logsource} seems a reasonable way to put it there

violet-rpi, %{<PATTERN>:<field_name>}

so like this rename => [ "%{URIHOST:logsource}", "%{logsource}" ] ?

violet-rpi: no, in your grok rule

match => [ "message", "%{SYSLOG5424PRI}%{CISCOTIMESTAMP} %{URIHOST:logsource}

you shouldn't need the mutate rename after that

right like i did with pid thanks

violet-rpi, yep: http://pastebin.com/Wf1uJd8U

Title: grok { break_on_match => "false" match => [ "message", "%{SYSLOG5424PRI}%{ - Pastebin.com (at pastebin.com)

violet-rpi, *source :)

tiv: thanks

violet-rpi, urw

hi! can anyone give me a hint how i can replace a “grep” filter for new logstash version?

martinseener, tell us more

what actually do you want?

iam using a grep filter to match messages containing special severities and program names to add fields and tags. those tags are later used to send the output not just to ES but to nagios

but grep is not available anymore with LS 1.5.3 which i am about to upgrade from 1.3.1

martinseener, https://discuss.elastic.co/t/migrating-filter-p...

<http://tinyurl.com/ouqg5me>; (at discuss.elastic.co)

tiv: ahh thanks alot! yeah i was looking for this ;)

so basically replace grep with conditional and mutate ;) thanks!

martinseener, you can try ;)

i got about 30 of those, so ill start with just one and comment out the others…i think ill get it working ;)

just wondering that logstash book is still receiving updates, like logstash itself

it would be interesting to see diff :)

Im using xml filter and XPATH inside.... /xxxx/yyyy/wwww/text() mmmm So of course all my output data is STRING..... Can I cast them to number, date

because when I try to use them on kibana...it say that I dont have any NUMBER field

rubytor, why not use mutate filter to change data type?

tiv: let me read it!!

<http://tinyurl.com/kgbfwmq>; (at www.elastic.co)

anyone know why some of my logs are tagged with _grokparsefailure even though every field has been parsed perfectly?

rubytor, if you need to use your data-time as timestamps, you need https://www.elastic.co/guide/en/logstash/curren...

<http://tinyurl.com/mclv6ms>; (at www.elastic.co)

olivier_2, this is because some of grok patterns include another grok patterns

olivier_2, that give you grokparsefailure

tiv: thank you... I'll check!!

olivier_2, especially syslogbase, if I'm right (can't remember now, have the same issue in the past)

you are probably right tiv

thank you

olivier_2, I'm remember! what input filter do you use?

if something not tcp or udp (syslog, for example)

this filters include grok filters inside, that give you fault

i made a huge mistake.

just use tcp or udp for syslog inputs

i mean, hello o/~

tiv: this is what i have atm

are there any standard patterns for logback, postfix, dovecot logs?

Title: input { # this input is used by logstash-forwarder lumberjack { # The - Pastebin.com (at pastebin.com)

olivier_2, syslog input give you grokparsefailure

olivier_2, because actually this is the mix of few simple filters

olivier_2, i'm using raw tcp or udp for all syslogs, for this reason (your issue) :)

meena, there is not

right. then it will apply the default crappy SYSLOGBASE grok pattern