#logstash

near88: ish

ish?

near88: I am here "ish"

oh hi

near88: did you want help with something?

yeah, but it seems to be more of a elasticsearch issue

a complex search, do you think you could help me?

i have nginx log requests, and i want to find when an ip address or an user only goes to websites once

possibly - ask the question and we can see

like if it goes to web1,web2,web3 return him to me, but if he goes web1,web2,web1,web3 dont

oh

yeah you probably want a facet I think - http://www.elasticsearch.org/guide/en/elasticse...

<http://tinyurl.com/n2968lv>; (at www.elasticsearch.org)

mmm but I want to somehow know if there are repeated records..

near88: see also this I dug up - http://stackoverflow.com/questions/19102220/how...

<http://tinyurl.com/ksoqpru>; (at stackoverflow.com)

near88: if that doesn't help then rashidkpc or electrical both work at ElasticSearch and may be able to help

thanks a lot james, I think I want some kind of nested query, where Im sure that for an user all the terms count is 1

hi all, am I correct in assuming that the new conditional filtering stuff isn't covered yet by puppet-logstash?

whack: around?

avleen: mostly we've been watching the joins and leaves go by

lol

poor guy, i be tthe baby is running rings around him

avleen: ah yes, the lack of contiguous sleep. Dont' miss it a bit.

avleen: hey, but I could entertain you with a couple of questions if you want.

hehehe. sure! give me just a few mins, type them up while you wait? ;-) I'm just fixing one teeny tiny problem in my config, then i'm all yours

avleen: cool. Q#1: what do you think the best practice is for maintaining a "drop messages that look like X" on the floor list?

avleen: Q#2: any tips on how to monitor for when a log source _stops_ reporting? e.g. lumberjack gets killed on a host, no reports from host in X minutes, so scream?

avleen: for #2, I can see writing some sort of query to ES that goes in our monitoring system, but curious if there is a better way.

avleen: for #1, I'm not clear where people are putting this info in their configs. Surely we all have known noise, I'm curious where people put that and their process around maintaining it (e.g. do people take their instance down each time they tweak this?)

Q#1: Do you mean other than the "drop" filter? You could do it before you ship the logs to logstash, but I don't know the best way to do that really. I do this, and I use a drop filter for stuff i don't care about. i don't even log that I did it because I *just don't care* :-)

Q#2: I'm about to implement this. I'm going to send stats to graphite each time I see a log line for a particular file.

avleen: yup, I speak of the drop filter. So do you keep everything you drop in the main config?

then have nagios monitor the data in graphite

there is an alternative, where you can submit a passive alert ot nagios every time you see a log file, but that might kill the nagios server.

avleen: cool re: Q#2. I'm pretty sure I have seen nagios/graphite integration thingees (kinda like the ones for rrdtool)

http://github.com/etsy/nagios/ i think there's a check_graphite_data in there

(or in one of the repos, i wrote it ;-)

nice.

I'm going to be implementing the monitoring stuff in the next 1-2 days, would be happy to let you know how it works out :)

i ned exactly this

*need

avleen: yes, please. We had a similar issue where some of our lumberjacks went down due to a cert error, I want to know when that happens again.

avleen: don't see a nagios repo, but I will dig.

Ah, found it: https://github.com/etsy/nagios_tools

Title: etsy/nagios_tools · GitHub (at github.com)

looks great.

avleen: ok, so back to drop filters, how do you specify yours? I can just see it getting pretty big, and as I said, I'm not keep on the restarting of logstash if I can help it.

avleen: oh, wait, maybe we are talking two different things here. Let's say I wanted to keep around a file with regexps that get dropped on the way in (since I can't tell lumberjack to drop on the way out).

that list of regexes would only get read when lgostash starts

:)

so you'd need to restart logstash

And where in the config would you put it?

I keep one large monolithic config, and order my rules based on volume. Eg, the logs which come in most frequently have their rules at the top. My filter config is one big if / else if / else if / else if / else if / ....