#logstash

whack: thanks that makes sense. So really my question is then: does logstash instantiate one filter object per thread, or one that’s globally shared between the threads?

if I delve into that code again i’ll delete the comment

michaelhart: for filters it's shared

whack: ouch. so it is really not thread safe then. zmq says that context can be shared but not sockets. putting a mutex around it won’t help performance either

michaelhart: you could make it threadsafe with some work

whack: I’m willing to do it, as I really need it. Is there a plugin I could use as an example?

cmdstation, try this on Centos

"version" : "1.7.0_25", "vm_name" : "OpenJDK 64-Bit Server VM",

To me this is the only difference with Ubuntu

hm ill look into it

coolacid: needs more logs in the repo :)

@cmdstation: do not install the headless packages

shoerain, feel free to PR a contribution ;)

yes I know that

just 10 lines of a lot of common log formats would be all I'd want (apache, nginx, uwsgi, varnish, solr, rabbitmq, etc etc)

this is the eaxact same version as at work, seems only difference

coolacid: sure, maybe end of week

Heck, do up some log lines, clean them up, and write some cool configs like I did with weblogs ;)

now, all i get is no results because no indices found, hmm

Quick question, I'm trying to parse the date on the first line of a multiline filter. That works great, but it keeps adding in dates from the other lines, which default to the current time, so I get one correct time and a bunch of wrong times. Any ideas here?

Jarth: I only have java-1.7.0-openjdk.x86_64 1:1.7.0.55-2.4.7.1.el6_5 updates and that gave me the bug

ah, so it's fixed now ?

andrewvc: is it in your gist here: https://gist.github.com/andrewvc/2dc7c74aea22a3... ?

<http://tinyurl.com/mdnqn9e>; (at gist.github.com)

Yep, I can make a more minimal one though

<http://tinyurl.com/lpfb8uk>; (at gist.github.com)

I'm just testing with that subset of it

Once again, thanks for all the help

I've tried various sorts of 'if' blocks around there, no luck

Just refreshed the gist to include stdin / stdout

Installing jdk 8 for logstash is no good right

andrewvc: you'll want to do date and grok first in most cases

andrewvc: mostly because it makes somethings easier

however, what you're doing should work. Is there no 'timestmap' field in your event?

Wait, so the multiline goes last?

andrewvc: that's an option

Here are public apologies for ranting and also, for using Ubuntu, but i have too

but anyway, if you can pastebin the full event itself after filtering i could help that

hmmm whack: making multiline go last broke the whole thing

Here's the output I get with the config as it stands

<http://tinyurl.com/lpfb8uk>; (at gist.github.com)

coolacid: one thing I wanted to ask: have you tried parsing the 'request/querystring/API endpoint' out of %{COMBINEDAPACHELOG}? I haven't figured out a way to do that and it would be kind of useful. I would love to see 'request' broken up into smaller key/value pairs

shoerain, one could either put that into the original GROK, or GROK the resulting field into components.

shoerain, it would be possible if you know your REGEX well enough ;)

I just get multivalued lines like: "@timestamp" => [

[0] "2014-01-29T17:33:10.953Z",

[1] "2014-04-29T20:23:12.413Z"

],

coolacid: I know my regex, don't know my logstash configs :(

Where the first line is the correct value. There's no way to have mutate just grab the first value off the @timestamp array is there?

shoerain, I can help you with that. I gotta run out for a bit, let's hook up later.

cool, if you have an example you can point to, that would be great too

is it a separate `grok { ... }` block or within the original `grok { ... }` block?

shoerain, simplest would be a second grok on the new field..

must be running.. bbiaw

Do i need redis or not for logstash to start indexing files ?

Jarth: you dont

andrewvc: the @timestamp array is a bug in 1.4.0

thanks zquad

oh, thanks for the heads up whack, so there's no workaround eh

do i need thsi ?

pip install elasticsearch-curator

still nothing indexed

I have this

output { elasticsearch { host => "localhost" } } in a separate file

whack: Ah, I see it in the ticket. Thanks for finding that, I was going nuts

i define input in dedicated files

whack: ping, if you're around :)

I have a quick question about raw fields. Does anyone know of a good resource that could explain their use to me? I have a field called loglevel that I am trying to report on in kibana, and the widget keeps targeting loglevel.raw instead and that field is blank for some reason even though loglevel is populated properly.

hmm, i've answered my own question, still no indices, now what

avleen: who are you calling whack?

talking tough over the internet

whack: Q about kv{} - do you know why it has such a complex regex scan, rather than just doing split() on the text?:-)

zquad: hah

actually that's not my real question

because the moment i hit 'enter' i realised why

the real question is a "bug" where I'm trying to split querystrings, but sometimes someone will hit http://example.com/foo?12345

and '12345' ends up as a string rather than {"12345": ""}

and then elastic search cries on trying to insert it

Hey all is here a config or cookbook out there that deals with parsing a php error log? I started writing a config to do it and quickly came to realize it would be much more work then I had originally thought to merge all the multilines and what not.

actually I take back what I said, because I think this works, and I solves my "missing value" problem:

Jarth_: you need either a running instance of elasticsearch on the default port (try 'curl -XGET http://localhost:9200'; to make sure it's running)

yeah, it works