#logstash

ES is on there own machines... 6 ES nodes i-xxxx; 1 LS node (w/ 5 workers named globalAggregator); 1 LS node (w/ 10 workers, using random names)

I'm still confused I think

so what problem are you tracking down?

There are 6 i-xxxxx names (as I would expect 1/node) There are 6 globalAggregator names (5 workers... +1); there are 11 random names (10 workers... +1)

I mean what does 5 vs 6 nodes cause you? What problem? ;)

nothing... I just thought it was odd

the way workers are implemented probably causes this.

I asked for 5 workers and got 6 connections... that's all.

gotcha, so no problem, just a weirdness?

not a problem per se... just an oddity

cool, just verifying

ok, quick meeting... bbiab

ronnocol: confirmed.

the way workers are implemented causes this.

ronnocol: no i hear ya. i did not want to manipulate it on storage end. i just did not think my question through.

any simple way from the command line to verify the version of LS I'm running?

java -jar logstash.jar version

thanks!

Hi All, We have a problem where logstash doesn't appear to read from the end of a file. Always starts from the start again. I believe this https://logstash.jira.com/browse/LOGSTASH-429 bug could be the cause. Does anyone know any updates on this bug or workarounds?

Title: [LOGSTASH-429] File Input - .sincedb file is broken on Windows - logstash.jira.com (at logstash.jira.com)

i'm seeing errors like this https://gist.github.com/untergeek/8199707 after upgrading ES to 0.90.10 + marvel plugin. logstash at 1.2.2

Title: Error generated by Logstash node joining ES cluster with marvel beta. Doesnt generate [cluster/stats/n]? (at gist.github.com)

confused

you're using logstash 1.2.2?

marvel requires elasticsearch 0.90.9 or newer, iirc, and logstash 1.2.2 ships with 0.90.3

i am not using the ES that ships with LS. ES installed separately

joemiller: are you using the 'elasticsearch' output?

yeah. should i use _http output instead?

probably in this case yes

the _http output uses http, the other output joins the cluster like a regular node would

ok. i thought perhaps, but it is successfully adding data to ES

so it shows up in the node list and marvel will likely try to query it

ahh. ok

I suspect the error you see is harmless

why :(

<http://tinyurl.com/c68vfat>; (at en.wikipedia.org)

if you upgrade to logstash 1.3.3 you can use protocol => transport and avoid this

jkitchen: lol, syslog

is there much performance difference with the http proto?

syslog is an acronym for "not what you think"

haha yea

maybe i'll just do that. ls 1.3.3 was next on the list

joemiller: 10-15% slower in some cases, but only if you're already at peak throughput

I'm half debating just pointing haproxy's syslog output directly at logstash

but I'm a bit worried abotu then not really knowing when i'm overloading logstash with haproxy traffic

avleen: oh, in case I didn't say so, I'm open to a patch that lets you tune the size of SizedQUeue

at least if it goes to local syslog and then I route it through there into rabbitmq I can watch the queue

avleen: I'm also considering making workers tune themselves somehow automatically. We need all filters to be threadsafe first though

whack: multiline inputs.

do the multilining in the input.

boom.

jkitchen: metrics isn't threadsafe in the latest release either

though there's a patch in master that makes it so

neat

I'm back in the game now, so I'll be able to catch up

:D

and this place actually wants to use logstash. I don't need to sell it to upper management :P

whack: is there an alternative counter to metrics that can be used in the filter section?

and them be like "why is that useful"

volker-: alternative counter?

and me just crying.

jkitchen: woot

whack: a variable I can use as counter

volker-: you can count anything from an event

at least, I think you can

yeah, meter and timer supports sprintf values

meter => "apache.%{response_code}"

assuming you have a response_code field, will count/rate/etc by that value

apache.200, apache.302, apache.404

whack: which is always processed immediately in the output section?

"always processed immediately" ?

I don't understand

you want metrics to emit a count immediately?

whack: no, I want a workaround for metrics{}

whack: I think only statsd would do it

what do you need work around?

metrics is a filter plugin, what do you need to work around? I don't think I have context on your issue.

whack: workaround for LOGSTASH-1845

Jira issue [LOGSTASH-1845] metrics counting what is not matched - logstash.jira.com - https://logstash.jira.com/browse/LOGSTASH-1845

whack: I need another way to get reliable counts

Hi All: Does anyone have a workaround for https://logstash.jira.com/browse/LOGSTASH-429 ?

Title: [LOGSTASH-429] File Input - .sincedb file is broken on Windows - logstash.jira.com (at logstash.jira.com)

volker-: that's a bug in the pipeline, not in metrics

it has to do with the way filters which emit events out-of-band are handled by other filters.

it is effectively counting your 'metrics' events

whack: don't know, I looked at the metrics plugin and didn't see anything. But I need to get reliable stats

I am aware. This is a bug.

whack: bug is the one thing, workaround the other :)

bugs have time, workarounds don't :)

there are no workarounds at this time that I am aware of

whack: how about using statsd output with a conditional around it?

https://logstash.jira.com/browse/LOGSTASH-1695 is the bug.

Title: [LOGSTASH-1695] Metric filter events (possibly other filters that add events to the stream) are not applied against conditionals correctly - logstash.jira.com (at logstash.jira.com)

volker-: that will work, yes

New news from newjiraissues: volker created LOGSTASH-1845 - metrics counting what is not matched <https://logstash.jira.com/browse/LOGSTASH-1845>; || Ryan O'Keeffe created LOGSTASH-1695 - Metric filter does not work when multiline is also used. <https://logstash.jira.com/browse/LOGSTASH-1695>;

whack: ok, thanks

whack: if I check the #1695 example, it uses multiple filter{} statements. Is there a benefit in it?

logstash doesn't make any distinction between 1 or 10000 filter{} statements

they all execute the same (as if you had 1)

I know how to solve 1695 I just haven't done it yet

Hey, ES and Kibana guys!

I was thinking of using OpenTSDB for my metrics crunching, but it's nastier to deploy than I like.

If I blast a billion ( timestamp, metric, tags ) objects into ES, how easy will it be to query that with Kibana? What kind of hardware would you suspect I'd need to get my graphs back in not-frustrating time frames?

whack: for the drop I found a workaorund, so I will try statsd :)

garthk: I never looked in OpenTSDB, but I use here metrics => graphite where nagios can pull for some things