#logstash

simmel: how familiar are you with logstash and graphite?

Wolfsrudel: Not at all I'm afraid.

key value pairs go in, graphs go out! Magic! ; P

ok, i'll wait for electrical :)

i need some kind of uber-magic :D

"uber" meaning super-power-high-definition

sort of

need coffee

http://explainshell.com/ <<< this is cool

Title: explainshell.com - match command-line arguments to their help text (at explainshell.com)

ehlo :-) I'm looking at outputs for elasticsearch - we have quite a number of messages passing through our system, so I would like the elasticsearch output, but only against ES 0.90, becasue that's what our ES cluster runs

Ca Kibana 3 show the Stream Panel from former version?

Can Kibana 3 show the Stream Panel from former version?

yes

elasticsearch_http

index.html#/dashboard/file/logstash.json

ptrf, elasticsearch_http - that is what I try also

swkide: but there are two issues

first, what if one of our cluster nodes go down? Ideally I need a reverse proxy infront

agreed

and also, I dont really like the overhead of one tcp handshake per messages, when there's a lot of messages comming in

ES is clustering und proxying itself, isn't it?

as far as I understand, they donÄt make a session per line

hum hum, is it streaming?

Wolfsrudel: yes, the elasticsearch output opens up an elasticsearch instance and joins the cluster

<http://goo.gl/N7mYtA>; (at logstash.net)

Wolfsrudel: the host field of the elasticsearch_http plugin accepts a string value, not an array.

as I understand it

Wolfsrudel: so if the target ES node becomes unavailable, then what ?

exactly

yes, but now you can use the "Power" of http loadbalacing and proxying and so

now, do I have to downgrade our ES cluster or should I have a more loose interpretation of the version note

?

basically, if you're using the elasticsearch_http plugin, you probably want to point it at an LB in front of ES.

so the es_http isn't clustering like the es output?

s/like/as

logstash has an internal ES client, which only talks to cluster up to 0.20.5 I think

swkide: 0.20.6

phrawzty, thx

Wolfsrudel: no, the es_http speaks http (hence the name)

Wolfsrudel: the es speaks es.

yeah

I want es, but version 0.90

Wolfsrudel: and http isn't a cluster-aware protocol, so...

hm, didn't know that. i only have one node, so it's not that critical

maybe it's a TIAS

ptrf: then put an LB in front of it.

but whack mentioned that 0.90 should work too... maybe with logstash 1.2

ptrf: haproxy works (i've tested this exact scenario)

i agree, though the http client proves quick enough

ptrf: i've heard of people using nginx to do the same.

There's also this: https://github.com/dotcloud/hipache

Title: dotcloud/hipache · GitHub (at github.com)

Who again said kibana 3 can show the stream panel of former versions - could you please tell me how

there are always options :)

nginx is awesome

swkide: are you talking about a live streaming table of new entries?

rdw200169, yes

rdw200169, how many logrows are entred per seconde in my case

well, all things considered, elasticsearch doesn't really work like that, so there was probably some backend processing going on on the server

since kibana3 is all js and html, there is no backend processor, so you just graph/table/term what you want, and set the relative time to a sensible value and tell it to refresh every few seconds

for example (we use elasticsearch for much more than logstash) I use kibana 3 to show me a pretty real-time chart of indexed items per day/hour/minute (etc..) depending on how far I want to go back

you just have to be somewhat careful b/c the resolution of data will affect the results from the facet query against elasticsearch

hi peeps - this is quick qn, but I couldn't find an answer on the docs page for logstash. I'm not familiar with threads or Jruby, so this may be a silly question, but are outputs nonblocking when they are run on log entries with logstash?

It looks like, at least for charting, that the date_histogram facet will only return at least 1minute resolution (not 1 second): http://www.elasticsearch.org/guide/reference/ap...

<http://goo.gl/3fPLg0>; (at www.elasticsearch.org)

I want to find a way to log some events to an instance of Piwik over it's HTTP API, but I'm not sure if this would be a terrible idea performance wise

ok cool, that is enought

ok cool, that is enough

thx a lot rdw200169

the most anoying thing in processing logs are date formats

swkide: if you want higher resolution, look at the graphite output; graphite can get you realtime metrics on your data ;)

rdw200169, sure - we have a running graphite also, but that is next step ;-)

damn apache uses two different formats. one for access other for error. i'm afraid to think what i will get when will start processing syslog where logs comes from tons of other services.

Tokeiito: you're preaching to the choir on that one, I'm currently in the process of refactoring <everything> to use ISO8601 for my/our sanity's sake

Tokeiito: figuring out how to trick logstash into correctly parsing PST/PDT timestamps from a timestamp was not a fun task, all because someone chose 'z' instead of 'Z' when setting the timestamp formatting *grumble*

uff i feel you

hello

i wonder how logstash filters are evaluated, in which order

is the order in the description the order in which they are executated?

daks_: according to the docs, the filters are applied sequentially: http://logstash.net/docs/1.1.13/life-of-an-event

Title: logstash - open source log management (at logstash.net)

yes in fact, thanks for the link

so i'll use "order" directive in puppet-logstash

daks_: you can use tags, though, to control execution, such that every event that passes thru a logstash filter config only 'hits' the filters they need to; you can add tags while filtering, which means you can have sub-filter statements that only filter events based on criteria set in a previous filter

ok

whats the way to put name on such expression: ${WORD}.${word}.${WORD} ?

i need to name all this group

or is is the only way throw custom patterns?

found it

(?'vhost'%{WORD}.%{WORD}.%{WORD})

Tokeiito: It's called named group captures, see http://www.regular-expressions.info/named.html

Title: Regex Tutorial - Named Capturing Groups - Backreference Names (at www.regular-expressions.info)