#logstash

Anyone know of options for indexing my logs different ways in MongoDB? I'm using the output plugin that ships with logstash, but I don't see any options for building different types of indexes.

_jc: why mongo? go with the flow (elasticsearch) imho

semiosis: I agree ES is awesome, but I need a database, not just a distributed text search engine (ie lucene). I need to build some custom indexes in mongo to support queries for my application.

I'm consdering writing my own python etl scripts to load into mongo and build the indexes I need. Let logstash deliver the logs, and then process/insert them into the db using my scripts. Want to make sure there isn't a better way before I go down this road.

_jc: maybe i just dont see the distinction between "a distributed text search engine" and a database

_jc: wish i could be more help but i gave up on mongo about a day into it :/

too hard to justify adding more moving parts to my infrastructure.

also.. mongodb.

semiosis I would really like to leverage mongos aggregation/map-reduce capabilities to build indexes. I plan to use both mongo and ES. Interesting writeup on SO -- http://stackoverflow.com/questions/12723239/ela...

<http://tinyurl.com/n27kuuc>; (at stackoverflow.com)

_jc: read that SO page

_jc: so are you going to be polling the DB for new events that match a query?

_jc: if you could try explaining your goal maybe someone will have an idea

asking about mongo indexes might be a bit too specific for this channel on a friday night

volter: It was the JDK after all. I uninstalled openjdk, installed the Oracle JDK (the new version 8) and logstash is now running and not exiting. I'll proceed from here trying to get a working configuration. Where's the best place to let the maintainers know there's a possible issue with openjdk and logstash-forwarder/lumberjack?

manos: irc is always a good first stop for reporting bugs. the right answer though is ,,(jira)

manos: https://logstash.jira.com/secure/Dashboard.jspa

semiosis: thanks.

manos: do you have error/output from the "exiting" on openjdk?

i'd like to take a look

seanhagen yeah, would like to use mongo as my "store" for all events, and build indexes (example aggregated reports for entities in logs, like an ip address). Would also use ES for text searches of the logs.

yes, it's earlier in this channel

pastebins

semiosis yeah, sorry .. the mongo channel is dead. :)

_jc: no worries, just sayin you might have better luck here during daylight in CA, US

semiosis: Today's channel starting with my appearance contains pretty much the entire issue as I know it. I provided my config file, commands for generating certs, and the debug output when attempting to run logstash with lumberjack on openjdk

manos: a link to the conversation at https://botbot.me/freenode/logstash/ would help, otherwise i have to search

Title: Logs for #logstash | BotBot.me [o__o] (at botbot.me)

you'd know better when you said it

i don't actually have scrollback on this machine :/

Title: Logs for #logstash | BotBot.me [o__o] (at botbot.me)

thanks!

manos: would've been better if you had used 'agent -v -v' to get debug output. if it's convenient could you do that now?

if not, no big deal

Sorry, I've already uninstalled openjdk and installed Oracle JDK. I only have an hour or so to spend before I need to call it quits and I've got to figure out the Windows side of sending logs.

ok no sweat

manos: do you know/recall: what version of logstash? what version of JDK?

semiosis: logstash 1.4.0, java-1.7.0-openjdk-1.7.0.51-2.4.4.1.el6_5.x86_64

ok

manos: lmk if you make a jira ticket. i'm going to try to reproduce

manos: and you say this works with the latest jdk8? i get the same crash on both oracle jdk 1.7u51 and 1.8u20, haven't even tried openjdk yet!

Title: [LOGSTASH-2069] logstash exits with signed fields invalid using openjdk and lumberjack with tls/ssl certs - logstash.jira.com (at logstash.jira.com)

Hmm. I think I have a false positive.

crud. I commented out the ssl lines.

hah

rather, I commented out all the lumberjack stuff

I was trying to get back to a simple config and build up from there.

that'll fix it

Guess I'll be amending my jira report :(

just make a point about following the simple readme instructions & failing. whack hates it when that happens to newbies.

Ok, I'm getting the same error.

I just edited my jira ticket to indicate I'm getting the same error.

semiosis: are you testing on CentOS?

hell no

i mean, i'm all for centos, as long as other people are using it, not me

debuntu is my preference

ubuntu saucy at the moment

ubrowntu