#logstash

Jay: yeah, derivequeries on the field that contains to page

contains the* page

rdw200169, ah, I see. not sure it will really be too much of an issue either way. I only have ~2GB of logs.

and what is the most lightweight shipper option?

mbelaninja: lumberjack

not logstash

i figured as much with the jvm and all...

thx devOpsEv

yaeh, you can find lumberjack on google, just search 'lumberjack logstash'

beaver is also fairly lightweight, but needs python

whack wrote that one, too

er, not beaver... I dunno about beaver, I was referring to LS

er

lumberjack

gah

heh, yeah

*watching from the shadows*

stalker!

*bamf! disappears in a tiny puff of smoke*

devOpsEv: Thanks, it works.. In this case, i have messages like this in the log file "130338130830-INFO: Search query: apple" "130338130830-INFO: Search query: ((apple AND iphone))" "130338130830-INFO: Search query: ((_search_y:2013) AND (apple AND Iphone))" each on separate lines, if i have to use kibana to find what is the most 5 searched keywords for that day, is it possible through derived queries or something.

yes, but you probably need to analyze the search query field in ES with a mapping

so that queries are tokenized

what I mean is, you don't care that apple AND iphone are done on the same search query from your users, right? you just want a total of most popular keywords?

that is to say, you don't need to know that apple and iphone were in the same query in Kibana, necessarily, right?

Yes thats right, i just want the most popular keywords

yeah

in this case apple 3 and iphone 2

so you probably want to tokenize those fields in ES

hm

yeah, you want to tokenize that field

and kibana will you results for each individual term in that string

so apple AND iphone would actually count apple and iphone separately

you'd just need a "-AND" in the Kibana query so it doesn't also count the 'and' in each event

hope

er

I hope I'm making sense

there are a couple different queries and terms and systems that I'm talking about, I udnerstand if it's a bit ambiguous what I'm saying

ok.. Got it.. I have not tokenized it since this occurs in a log with all other messages which follow different pattern , the token we use is time-severity: message , only 3 . Even if i could get "apple and iphone

" 2 and "apple" 1 thats ok

in ES, you can set a mapping for that specific type, for that specific field, so that only that field is tokenized

then you don't have to worry about all other messages

help

sory

**

wrong window ;)

hehe

oh

but thank anyway

in that case

Password:

anyone use OSSEC with logstash?

Oh ok, how to do that for specific field

also is there a way to apply regex patterns for the derive queries.. I see that * works ..

Jay, you have to define the mapping in a json query and use curl to send it to ES, check the elasticsearch guide at www.elasticsearch.org for more info

match => [ "syslog_timestamp", "_ISO8601" ] how would I make this work??

Xian, is that for a date filter?

yes

Xian, is not working now?

Xian, for starters, _ISO8601 isn't an option there, have to remove the _

devOpsEv: Thanks .. and whether derivequeries support regex ?

typo....

Jay: everything in Kibana uses ,,(lucene) query syntax

Jay: http://lucene.apache.org/core/old_versioned_doc...

cool

Xian, a typo in irc or in your config?

both, its fixed

ah, yay :)

*the typo

Regex queries aren't supported yet

they will be soon

rashidkpc: awesome

however they will require elasticsearch 0.90

I have the same issue chandler1 had earlier with the timezones being off

Xian, I believe it will put them in UTC

it does put everything in UTC, this is normal ?

yeah

good enough for me

yeah, still working on that, its complicated

its one of those things were the easy fix will just be a huge headache later

s/were/where

devOpsEv: Thanks

logstashbot: Thanks for the link

Jay: Error: "Thanks" is not a valid command.

hehe

rashidkpc, regex or timezones?

Is there any way to ship all the logs logStash is getting before any changes are made to another logStash box?

brdude: yeah, just don't put any filters in your config

brdude: probably your best bet is redis output and setup redis on the receiving box, then use the redis input on logstash on that box

devOpsEv: I still want the original box to do it's things on the logs. I just want to have the untouched data to test out logstash 1.2 withougth going to the actual boxes any making them ship them to a second logstash box

so you want to process locally with logstash, but also send the logs off somewhere else before hand?

yes

how is the local logstash inputting the logs?

some tcp and some rsyslog

Couldn't you set up Redis to use a Pub/Sub channel and just have multiple subscribers to it?

all TCP actually

with your current setup, you could just have rsyslog listen for logs, then send them to two destinations, one of them being your local logstash

youd have to change your logstash input ports probably

any good references for a cisco filter

Xian: I actually joined the channel to ask about whether there would be interest in some Cisco Grok patterns I've developed.

@GregMefford yes that would be awesome

Thanks guys. Will look more into it and report back

so you have rsyslog listen on 514, then one of your *.* destinations is localhost:50000 or something, and logstash listens on port 50000

I have patterns to match just about every message that I've seen coming from my ASA firewall and I was wondering whether I should contribute them to https://github.com/logstash/logstash/blob/maste... ?

<http://goo.gl/Wjpb4n>; (at github.com)

GregMefford: I don't see any harm in submitting a pull request ;)

I was hoping someone could give feedback on how to properly package and test a set of Grok patterns, because it's working for me, but I feel like it would be more maintanable for others if there were some test cases or something.

GregMefford: I would love to get a copy of those since I'm planning to pull logs from my ASAs in the very near future (as well as all my network equipment).

brdude: there is a clone filter, which can clone an event and then you can tag that event so no other filters work on it, then set an output to only send events tagged with the tag, if that makes sense

debOpsEv: that actually sounds like the way to go. Thanks.

GregMefford: the packaging is literally how they appear in the github, just a flat text file with PATTERN NAME (so.*me|re|gex[pattern]{0.3})

samdoran, Xian: Ok, I'll work on getting a pull request together (probably this weekend). LogStash has really worked wonders for making sense of ASA logs.

using the json filter, what if I want the values to be put in the @fields array just like 'kv' filter?

devops did you ever find the solution for source_host = ip?

what would I use as target? or leave it blank?

i see there is a beta DNS filter

GregMefford, if your logs are standard for your ASA firewall then they're probably fine - most people pick apart the patterns and just use the bits they need if they have custom logs and such

GregMefford: That's very good to hear.

thansen: they should automatically end up keyed to @fields

thansen, if not you may need to specify target => "@fields"