#logstash

is anyone out there using s3 input? and if so, how are you dealing with the fact that @timestamp is off by ~15 mins (due to delayed log delivery to bucket by aws)? i thought of updating @timestamp with value of actual event. but then i read that jordan sissel recommends not updating @timestmp as it could have unexpected results

maseda: I don't thinK I've ever recommended that

you should always try to use the timestamp in the logs themselves if you can

this jordan sissel guy sounds like he's never used logstash ho ho ho

wonder if he's any relation to the author v.similar names

eper: heh. i misquoted him. he said dont use mutate to update @timestamp

oh yeah always use date {}

has anyone experienced this "Starting Elasticsearch Server:sysctl: permission denied on key 'vm.max_map_count'" with openvz ?

maseda: yeah, if you try to use mutate to set @timestamp now, you will get an error.

whack: i had trouble using log-provided timestamp since field was string versus date type in ES

whack: roger that.

whack: i previously tried range searches on my the timestamps of actual event and ES complained

whack: maybe my query was hosed though. is it possible to range search on ES string field?

sounds unrelated, what was the error?

yes, you can raange search strings

whack: crapola. i must have screwed up my query then. i will try again. blessings to you

whack: does the date need to be in a special format though? i was using dates from tomcat like "2014-01-21 13:41:28,024"

maseda: i'm confused. Are we talking strings or dates, and with elasticsearch, right?

whack: yes this is purely ES question at this point. i wanted to search like this: "url_access_time:[now-1h TO now]". that blew up on me. but then i tried similar search on @timestamp and it was successful. then i looked at ES mappings and low and behold url_access_time was string and @timestamp was date.

right

you can't do time searches on strings

you can only do time searches on dates ;)

whack: quick question... if necessary would it be an issue load balancing the TCP inputs of logstash behind a hardware load balancer? I can't think of any since each connection is an individual transaction

nemish: "each connection is an individual transaction" ? You only send one log per connection?

eper: do you have any idea how to add a field for sender id for your exim filter?

weird nobody have written a complete exim filter yet :P

whack: true... it could be batch of events... but you get the point... there shouldn't be any issue sending to 2 LS servers sitting behind a LB right? the LB locks each connection... i could make them sticky if need be I guess

nope, it should be fine

you don't need to make them sticky

whack: i might have misundertood you before... when you said "always try to use the timestamp in the logs themselves if you can", you meant @timestamp right?

maseda: you are pulling logfiles from s3, yes?

frankS2: not of the top of my head, tried hard to forget the format of exim logs as they make me a sad panda

whack: correct

maseda: there's some weird timestamp format *in* that file, yes?

whack: tomcat as well

whack: correct

maseda: you want to use the timestamp *in* the tomcat log as the @timestamp

whack: thanks... if i had LS running on all hosts i probably would just LB redis... but since I use many different methods of getting into LS as the aggregate... then REDIS... and then LS pops off REDIS to ES

you achieve this with the date filter

whack: ahh... i am having problems with that.

whack: i cant seem to find correct pseudocode. do i just need match and that is it?

whack: so you made a featured appearance in the marvel video huh? ;)

whack: the other problem is that s3 log format has ugly apache-style date and i wanted to keep JODA

nemish: hehe

rashidkpc: any special requirements for load balancing a bunch of kibana web frontends? sticky necessary?

maseda: you use grok to match the date text from your log, then use the date filter to parse it

nemish: no, kibana is stateless

in fact, kibana is literally just static html and javascript

whack: cool... so 6 kibana web heads it is :P

can not see much use in load balancing kibana unless its for HA

nemish: I doubt you need that.

whack: more the merrier ;)

nemish: once kibana is served (say, from apache), it's unlikely to hit apache again.

it hits elasticsearch for search, but nothing else after, iirc

whack: i have done the grok. i have extracted out %{MONTHDAY}/%{MONTH}/%{YEAR}:%{TIME} %{INT:TIMEOFFSET} with it. i guess you are saying that i just put syntax similar to this in match option for date filter and then @timestamp will show as "06/Jan/2014:13:51:25 +0000" for example? if so, will @timestamp still be range-searchable with that ugly apache-style date format?

in logstash 1.2.2 and beyond, you can set a 'target' for the date format if you need multiple date fields

but yes

you can take your apache log timestamp format, parse it and store it in @timestamp using the date filter

which will let you do time range searches

whack: Yesterday you made a change to allow multiple geoip filters. I tested that today and it works, but I'm running into the infamous "UndefinedConversionError: ""\xF3"" from ASCII-8BIT to UTF-8" errors now. That was the only change I introduced so I'm guessing it has something to do with that plugin. I need to do some experimentation to even have enough to write a trouble ticket.

Just wanted to give you a heads-up in case you could see something obvious on the geoip filter side of things.

csd126: can you give me an IP lookup that results in that?

csd126: would love to see where the bad dat comes from; gotta force all the crap from geoip to be UTF-8

whack: Sure, give me a minute to see if I can find one.

question!

hmm

on TCP input %{host} becomes IP:PORT on UDP input %{host} is just IP

logstash applies things in the order it finds them in the config file. does this also apply to inputs and outputs? or are inputs effectively *always* before *all* filters and outputs are always effectively after all filters?

that must be a bug O_o

eper: bug.

*stops the tcp rsyslog rollout*

like if I have file A with input, filter output... and file B with input filter output, will the filter and output from file A also be applied to the input in file B?

:p

jkitchen: logically speaking, in a pipeline, you can't have an input happen *after* a filter

is there away to pump flat file logs from windows into logstash nicely ?

whack: Do I need to have elasticsearch in my pipeline to see these UTF-8 errors?

(right now I'm just displaying them to stdout)

moving multiline from LS to beaver has made life better... now I don't need a second event to kick the first one out of the queue.

when running logstash in agent mode, is there a good limit for memory ? I'm thinking of using it on some web servers and want to pipe php, httpd access/error logs to logstash indexer

grok help please: i have a pattern that "works" in grokdebug, but breaks because of a double quote. (a log line is from a tomcat access log and begins like this: 10.189.254.4 - - [30/Jan/2014:15:06:15 -0500] "POST /foo... I want to parse out the http verb separately, so I don't want to use a QS. Escaping the double quote doesn't seem to work. Seems like this should be really simple. Thoughts?

itsted: try single qutoes around the pattern instead of double

match => { "message" => 'something "%{WORD:verb} ... " ...' }

how do I configure logstash to pull local log files?

linux?

everything in /var/log

or include the " in your grok sequence: CMSEXTREQ "%{WORD:verb} /.*/(?<provider>.*)..."

which is what I do

whack: that didn't work either (even though I tried it earlier). I believe Ruby would need the double quotes to do the string interpolation stuff.