#logstash

linuxwebguy: yeah, I've done exactly that in hte past with a ruby block

linuxwebguy: yes. compute the lag and drop the old ones.

PsyanidE_: what exactly are you trying to test parsing?

yes

i want it for debugs

debugs of what?

whatever I am troubleshooting

how you configure LS depends largely on whatever data is coming in

differnet apps have different log formats, causing you to to write different regex patterns for thhem

there's not really a one-size-fits-al

all*

well since it will be TCP/UDP snmp stuff should work outta the box?

there is no such thing

there is no default config, no default listener, no default output or default filter

of course ha

so you'll need to create a tcp/udp/whatever input to ingest your data

you /may/ need zero or more filters to massage the data into a format that is more useful

torrancew/rastro: so I could use a ruby block to calculate the lag, then evaluate, and drop. I like that approach. thanks

and you'll need to create one or more outputs (if you plan on using kibana, you need at least an elasticsearch output)

apparently I have some work to do

linuxwebguy: np

yes kibana

linuxwebguy: i store the lag for everything, so I would compute it in the ruby block as a new field, then do a conditional drop{} back out in the regular LS config.

rasto: that makes sense

I've done the opposite, of doing the calculations and dropping all within the ruby block

yeah, that makes no sense :)

does if you're not using the field for anything else :)

it would be nice to see the lag

i have an ES health dashboard that shows the lag as a way of diagnosing problem.

then be able to graph, etc

rastro: I only analyzed lag between LS and ES at the time

and even that, we did externally, by comparing ES API output and LS metrics output in graphite

torrancew: i've yet to find a good way to compute the lag between @timestamp and _timestamp. Love to have that available, too.

yeah, that's why I did it externally

rastro: could do it with an es input, if you're careful to update the existing docs

read in event, use ruby filter to convert both _timestamp and @timestamp into ruby Time objects, subtract and store the diff in a new field, normalized as desired

torrancew: eek. updating documents is on my 'evil' list. Too much merging makes me nervous.

then in output{}, if [timestamp_lag_field] { elasticsearch { ### stuff to update doc goes here } } else { elasticsearch { ### Normal ES Output Here } }

yep, me too

torrancew: need a scripted field in my mapping or something.

rastro: oooh, is that a thing?

holy crap, there is!

<http://tinyurl.com/nkj6qoj>; (at www.elastic.co)

thought i just made it up.

hi

So i wrote a filter plugin but my logstash choke up although the service is running logs are not going into elastic search can someone help

Title: [Ruby] repdb.db - Pastebin.com (at pastebin.com)

is this the official logstash channel?

MuneMunk3y: as official as it gets :)

Where can I find information about logstash-output-elasticsearch plugin and how it handles document_id. I have it setup to deduplicate if I rerun the same logs but version number isn't changing and it doesn't appear if I change field, the value updates

thanks rastro. anyone wanna take a look at my plugin and tell me why its choking up logstash after a few minutes of running properly

MuneMunk3y: probably impossible for me to spot just from glancing through your code. You might toss some logging in the filter and crank up debug in logstash.

i am not a coder by profession just a sysadmin who likes to poke around.... basically i wrote this python code http://pastebin.com/Nk1PmubH thats called by the filter plugin and updates the log

Title: [Python] RepDB.py - Pastebin.com (at pastebin.com)

can some one help me port the python code directly into the ruby filter http://pastebin.com/K1DBY1YN

Title: [Ruby] repdb.db - Pastebin.com (at pastebin.com)

MuneMunk3y: your python code runs a query. you want to update an existing document with those results, or ??

no, basically i am calling the python in the filter plugin.

MuneMunk3y: to do *what*?

so as syslogs come in, i use grok to parse out ip / dns / hash / url etc and and use filter{ repdb { chksource => val } }

after grokking the value out

similar to how geoip works i am just getting the reputation out

i have another index in elastic search called bro that has reputation data

any thoughts @rastro

MuneMunkey: sorry, i got disconnected. missed anything in the last, oh, 13 minutes... :(

so as syslogs come in, i use grok to parse out ip / dns / hash / url etc and and use filter{ repdb { chksource => val } }

here is the filter http://pastebin.com/K1DBY1YN

Title: [Ruby] repdb.db - Pastebin.com (at pastebin.com)

python code http://pastebin.com/Nk1PmubH

Title: [Python] RepDB.py - Pastebin.com (at pastebin.com)

MuneMunkey: you're at about the 10-foot level. If you bounced up to the 1,000-foot level, it might help. So, after reading more of your code, you're trying to do a lookup of an incoming event against another index?

yeah

MuneMunkey: have you seen the translate{} filter?

i admit i am noob at this and I am not a coder by anymeans

no i have not

i was told not to use translate for large LUTs

MuneMunkey: also, depending on where you need the translation, you should be able to do it in ES: https://www.elastic.co/blog/terms-filter-lookup

my repDB has about 10Million IOCs

Title: Terms Filter Lookup | Elastic (at www.elastic.co)

MuneMunkey: ok, so how about the elasticsearch{} filter?

just reading up the elastic search filter piece

Does anyone know of a good filter for parsing asterisk logs?

duczen: not offhand, however, which logs are you trying to parse?

call records, or just general service logs?

IIRC call records can be written as csv, which can be parsed with the csv filter

As much as possible - probably /var/log/asterisk/full

offhand, don't recall what's in that one

Interesting about the call records - we have all that being stored in a mysql db as well so those might not be as important

*nod*