#logstash

crazyphil, you're a hero!

found something?

typing it works. I did it in each file separately

w00t. my ipv6 sage tshirt got here!

you weren't typing the files?

doh, wrong channel

no, i assumed that each config file will be handled separately

so adding type => myTypeA

and then doing an if in the filter/output did the trick

this is however beyond intuitive and I would have never guessed this is how it worked

ah, yes, otherwise they'd all be dumped to the same place

yeah, a bit weird

thank you Sir, you saved my day :D

glad I could help someone

I'm on the verge of reveting to LS 1.5 and ES 1.7

reverting

I can feel your pain :)

what's wrong with the config?

whack: who are you asking?

both of you

I suppose :)

well pandaa__ I managed to get fixed

regarding this comment: 16:44 < pandaa__> this is however beyond intuitive and I would have never guessed this is how it worked

^^ this behavior was requested by some users many years ago

oh really

that makes sense

(the current behavior, I mean, of concatenating the files together)

actually

I expected it to be each file being processed as one file

so that i can write different configs and just run and it will do the right thing

I'm having an issue where logstash isn't pushing the proper index template into ES

pandaa__: it's probably 50/50 who expects one behavior vs the other

yeah, probably. I am perfectly fine with this, I just didn't know about it

and I can't for the life of me figure out why not

crazyphil: is it uploading the wrong one? or not updating your new one?

pandaa__: yeah we should make it more obvious :|

oh, can I wildcard folders in the input? for example: path => /log/**/*log

pandaa__: for file input? Yeah :)

** means recursive

perfect

whack: I'm not sure what's going wrong, all I know is that I had to replace a .json output for elastic in my logstash setup, and the new .json index format isn't ending up in ES

crazyphil: ahh, do you have template_overwrite set to true?

<http://tinyurl.com/kppzy3z>; (at www.elastic.co)

electrical referred me to this earlier today: https://github.com/logstash-plugins/logstash-ou...

<http://tinyurl.com/o2cjorq>; (at github.com)

I think by default it only pushes a template if there is none existing

well, when I ask ES about templates, I only get one for marvel and nothing else

interesting

can you link me to your config?

LS, ES or both?

LS

<http://tinyurl.com/phjxzh9>; (at gist.github.com)

now I just noticed that manage_template is set to false

so if I replaced the contents of elasticsearch-template.json, would I need to set that to true?

yeah :)

really

ok

manage_template has good default behavior

it will only push a template if none already exist

I would have thought since it was a default file I modified it would "just accept" it

the default _is_ manage_template => true

yeah, but if I can't find the template in ES, that means it doesn't exist?

so you telling Logstash not to manage template tells it not to manage the template ;P

correct :)

`manage_template => false` makes Logstash not even attempt to push the template

whack: Error: "manage_template" is not a valid command.

logstashbot: *hug*

whack: Error: "*hug*" is not a valid command.

lol

the bot is back

ok, cleaned all my indexes out, trying again

ok

now I see a logstash template

woo!

so that means the ES and LS default templates both changes (for the worse)

now wtf, kibana is telling me no data is present

crazyphil: kibana I think defaults to "last 15 minutes" - are you sure there's data in that time frame?

or make sure oyur query would actually return results

I did, set it to 60 days

do I need to create a log4j.properties file on logstash server to use input log4j?

and I see a logstash-2015.10.30 index

using sense I go do this: GET /logstash-2015.10.30/nginx-access/_search

and I see my data

hmm, perhaps I'll kick kibana in the pants

kunersdorf: nope you shouldn't need to

does this make sense then:

/opt/logstash/bin/logstash --configtest -f /etc/logstash/conf.d/12-log4j.conf

log4j:WARN No appenders could be found for logger (org.apache.http.impl.conn.PoolingHttpClientConnectionManager).

log4j:WARN Please initialize the log4j system properly.

log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#no... for more info.

<http://tinyurl.com/byco3h>; (at logging.apache.org)

Configuration OK

this is config: xhttps://gist.github.com/anonymous/12048ead7b47...

<http://tinyurl.com/na9a7yd>; (at gist.github.com)

kunersdorf: you can ignore those warnings

@whack k, ty

If I create a field with add_field in grok, is there a way I can specify for that field to be shown in the output section when running from the command line?

"shown in the output section" - what do you mean?

Currently when running from the command line, I use output { stdout {} } as the output section. However, when I input log data, the output I get does not include the new field I created, and I was hoping there was a way to show it.

chris_19: sounds like your field is not being added as you expect.

@rastro hmmm.,

Hi

Anyone here is using logstash input s3 plugin?

@rastro the field is there when I search with kibana. But from the command line, the output I get basically matched the input.

I would like to known a solution to scale the logstash input s3 on more than one machine!

chris_19: try output { stdout { codec => json } } ?

ah, thanks. will do

@rastro perfect! Thanks!

?