#logstash

azkaran34, forscience I would take just one input file to see if you can get a working config and then start adding more logfiles to see what causes the error

ok

OverlordAlex: yeah. some issues with Jira. we are moving to use Github issues.

electrical, Ahh I see, thanks.

hello

I have setup a simple app to send messages > logstash via the ZMQ input plugin

I have both the app (PUB) and logstash (SUB) using tcp://127.0.0.1:2120

and while I can see other messages (from the filesystem), and while the logstash config passes config test..

I'm not finding messages in ES/kibana

I'm new to debugging this sort of thing in logstash

my actual error is RuntimeError: Unknown frame type 49. It's essentially the same error as here: https://logstash.jira.com/browse/LOGSTASH-1851

Title: [LOGSTASH-1851] Exception Unknown frame type: 69 - logstash.jira.com (at logstash.jira.com)

§S 1

luminous: hmm. that's strange.. anything in the LS or ES logs?

OverlordAlex: still got error with the latest release of LS? ( 1.4.2 )

If so, could you open up an Github issue and link to the jira issue?

electrical, yes, running version 1.4.2. Will open an issue on github now

electrical: for the times where I have the LS config correct, I see acknowledgment of the file/zmq inputs setup, and ES output, that is all

hmm okay. could you add the --verbose flag to the startup and see what happens?

oh!

silly me..

LoadError: The libzmq library (or DLL) could not be loaded

I have libzmq installed on the host

hrm

is there anythign special you need to do for LS to use the zmq lib?

just ensure it is in the correct path

I will check that to confirm

ah, so LS is looking in /usr/local/lib

but it's in /usr/lib/

symlinking did not help :(

electrical: any suggestions?

LS assumes all libs to be in the own Gem path.

how did you start LS?

LS wants some specific paths

electrical: LS is started with init.d script from the .deb

I have restarted the service

I might have it right now

hm okay. it uses the ffi-rzmq gem that's bundled with LS

yes, and I need to provide the libzmq.so

it's named differently in ubuntu, on a different path

Ah okay

must admit i never worked with it so far

zmq is awesome

hrm, now LS is not stopping, even with kill -HUP

there we go

hrm, my app won't bind to the socket if LS is there first

Hi everyone

I have a question on kibana <--> elasticsearch

Are you guys awake?

nope. sleeping :-)

fire away Saichovsky

Maybe I can wake you up electrical :-)

I have two different indexes

logstash-%{YYYY.MM.dd} and syslog-%{YYYY.MM.dd}

logstash is only collecting from the default

When I click on the index tab on logstash, it doesn't seem to allow multiple indexes

Is there a way to go around this challenge?

</question>

that's correct.. that feature isn't in kibana 3. I think it will be done for Kibana 4 but not 100% sure. i can ask when the kibana devs wake up :-)

hmmm... maybe I should've asked in the kibana room, no?

or rather I should try that

Sorry. it is implemented :-)

Title: Support multiple index patterns by rashidkpc · Pull Request #905 · elasticsearch/kibana · GitHub (at github.com)

was beginning this year

which version is kibana now on ?

awesome.... lemmi check it out. Thanks!

and is this ^^ patch in that version already ?

I'm running version 1.1.1

of kibana?

Kibana is at 3.1 the PR is in since 3.0 M5

ah, i'm still at 3.0M4

i needz to updatez

:)

haha yeah :p

Btw. i've seen some stuff from Kibana 4, its gonna be awesome :D

oh.. sorry... that's my elasticsearch version :-)

what i'm really missing now is way to limit users to different indexes

it's kibana 3.1.0

Saichovsky: okay should be in that version then

great idea sysmonk

i know work is being done on ACL/Auth stuff for ES

so how do I get kibana to access multiple ES indexes?

electrical: woot woot, really waiting for that

Saichovsky: logstash-]YYYY.MM.DD,[syslog-]YYYY.MM.DD for example as the index field

currently just have some nginx reverse-proxying to ES to limit the ability to drop/modify indexes

sysmonk: yeah. ACL stuff will be coming but i have no clear ETA yet.. but i do know allot of work is being done on it

Thanks guys. Actually Kibana has some examples just above the text box showing how one can point it to multiple indexes... I am so blind!

hahaha :-)

electrical, I've submitted the issue at https://github.com/elasticsearch/logstash/issue...

<http://tinyurl.com/nuymjjw>; (at github.com)

for now I've just commented out the "raise" in the ruby code

OverlordAlex: cool thanks! :-) will poke the LSF dev about it

The damn thing won't pick up saved changes

I set the indexes to [logstash-]YYYY.MM.DD,[syslog-]YYYY.MM.DD

but it's still not reading the syslog index

do I have to edit some JSON file or what?

shouldn't need to

when will kibana4 be released?

where does kibana save the changes made?

save into es, named `kibana-int`

I don't know what's up, but it's now reading

What's kibana-int - a file?

chenryn: i think its planned for end of this year.

@chenryn

how do you search in kibana for a type, field, or tag that you've added to a message passing through logstash?

@Saichovsky it's also an index, just like your logstash-2014.07.01

oh, I'm getting somehwere

{:timestamp=>"2014-07-01T10:57:43.775000+0000", :message=>"ZeroMQ error while in recv_string", :error_code=>-1, :level=>:error}

my bazillions of msg are streaming in, but erroring out..

luminous: ohw.. hmm

can't say i ever saw that

electrical: from cratch yes. so thats why iv looked at rabbitmq.