#logstash

I have a question related to using Grok in logstash. Does Grok not support non-capturing matching groups?

For example. With the string ' productname-1.0-el-6-x86_64.tar.gz ' and the regex ' ^(?:[^-]*\-){1}([0-9/.]*) ' it matches the '1.0'. Within the Grok debugger though, it matches ' productname-1.0 ' .

It appears that it doesn't support (or I am not correctly using) the non-capturing groups. It actually appears (or, again, I am not using correctly) matching groups at all.

Does anyone have any insight into this?

to not support matching groups at all*

faxm0dem: ok so collectd is sending stuff to logstash. i'm using kibana and i have not idea how to set this sutff up to show me for example just cpu logs of host A

faxm0dem: can you advise?

deni: in the query box: host:<hostname>

or expand out the filters, add one there

RichardRaseley: why escape the '-' ?

or click on the 'host' in the table view at the bottom, filter from there

I am... not?

whack: ^^

whack: Oh, derp. While true, still doesn't relate to the core question I posed above.

ahh, no idea; I can't test at the moment due to other work

mp___: host:hostname gives me a parse error....just typing the hostname shows some data but not sure what

deni: host:"hostname"?

mp___: that works...but i think it yields the same effect as just "hostname" (without the quotes)

mp___: what about parsing on specific things? ie cpu from collectd?

deni: it's all just fields / queries

deni: there's a link to help with the query language in kibana itself

Do any of you use logstash for system monitoring? I currently use nagios and nsca to report results back to nagios but I'm wondering if it might be reasonable to log the info I need with logstash and then parse it and send alerts to nagios based on that. That way I don't have to run nsca and logstash/redis.

mp___: i regret not going to all those elastic search talks on all those conferences right about now.

hah.

RichardRaseley: your pattern matches just fine

RichardRaseley: filter { grok { match => { "message" => "^(?:[^-]*\-){1}([0-9/.]*)" } } } matches productname-1.0-el-6-x86_64.tar.gz for me

whack: Exactly - it isn't honoring the non-capturing groups. It shouldn't be capturing anything before the first '-' as indicated by ^(?:[^-]*\-){1}

whack: Check out this ( http://regex101.com/r/fU9bN5 )

Title: Online regex tester and debugger: JavaScript, Python, PHP, and PCRE (at regex101.com)

whack: That is what I am *expecting* (which might not line up with reality).

The first match group is just the ' 1.0 ' in that example.

RichardRaseley: I'm not sure what you're asking. Nothing in your pattern requests being captured by grok

grok doesn't save unnamed captures

>> Regexp.new("^(?:[^-]*\-){1}([0-9/.]*)").match("productname-1.0-el-6-x86_64.tar.gz")

=> #<MatchData "productname-1.0" 1:"1.0">

^^ not really seeing what you're calling out? It matches just the '1.0' in ruby.

Grok doesn't capture anything because you didn't tell it to capture anything

I want you to get this working, but I think I'm confused.

whack: So perhaps I am thinking about this incorrectly. Let me restate and try to ask the right question.

whack: For string ' productname-1.0-el-6-x86_64.tar.gz ' and regex ' ^(?:[^-]*\-){1}([0-9/.]*) ' the first match group consists of just ' 1.0 '. Yet if I do something like ' (?<version>^(?:[^-]*\-){1}([0-9/.]*)) ' in the grok debugger it returns ' productname-1.0 '. How can I ask grok to only capture the first match group?

My expectation for the output of ' (?<version>^(?:[^-]*\-){1}([0-9/.]*)) ' was ' 1.0 '.

(?:[^-]*\-){1}(?<version>[0-9/.]*)

your pattern asks to capture everything as 'version'

not just the nuumber part

whack: Wow

Whack OK

whack: OK *

I screw that stuff up alllll the time

time to revist all those honza's talks that i missed i guess :D

*revisit

whack: That is a tremendous help. I have to embed what I want Grok to capture right in the regex, not rely on the whole regex to spit out the thing that grok will capture

yup!

RichardRaseley: in the common case you probably could just do this

whack: :: clouds part ::

[^-]+-%{NUMBER:version}

rather

^[^-]+-%{NUMBER:version}

match anything not a dash up to the first dash, then capture a number called 'version'

whack: Well, I have some things to think about.

=P

hehe

%{...} is grok's syntax for including a known regexp pattern

anyhoo, back to working ;)

whack: Thanks!

Ga just tried updating ES and Kibana.

Nothing in Kibana seems to work.

doug_f what version of ES?

ES 0.90.7 -> 1.0.1

someone mentioned earlier that the output section on logstash needs to be es_http instead of es

to account for the changes from 0.90 and 1.x

Is there a recommended version of ES for the latest kibana? I am already using the ES_http

I was expecting to shut down update and startup and have my old indexes work.

Kibana now just complains about indexes matching timespan.

what's a good way to grab supervisord logs?

no indexes I mean.

I'm running 0.90.7 with logstash/kibana @ 1.3.3

as for the other questions I really don't know enough to answer

(like how to fix the no indexes, etc)

So I ran into a bug it seemed with kibana. . . Looked at the kibana-3.0.0milestone5.zip which I thought was the one I was running and found it differed. Updated got prompted to update ES to at least 0.90.7 and figured I would go for broke.

It broke.