#logstash

why is logstash throwing debug log output by default now? i didnt put any debug flags into scripts and its turned on Oo

usually id do that by filling the LS_OPTS wouldnt i

Hi all! I am trying to ingest existing files (on s3) into elasticsearch using Logstash. However, logstash is just waiting for new files to get into S3. Is it possible to ingest old, existing files using Logstash (without copying)?

Renee_: you want to parse already existing files, is that what u mean?

Yes indeed

then start_position => "beginning" input parameter is probly what you're looking for

Does that also work for "files" input connector?

BTW thanks iNs for the quick response :)

file not files

and yes, it does

its most likely for it than anything else actually imo

remember tho, while parsing existing files logstash saves the 'progress' of read file

so given that, if you will want to re-parse them again, remember about .sincedb files

easiest way for me is to define sincedb_path for log type so you can easily figure out which one to delete

Thanks I will try this

but then again, im rather a newbie so remmeber to double check what im sayin, lol :p

@iNs :)

I get "Unknown setting 'start_position' for s3 {:level=>:error}" using "start_position => "beginning". Also tried setting sincedb_path to "/dev/null". That gives me a "Error: no time information in "" {:level=>:error}"

you sure youre putting that into input file?

https://gist.github.com/penekk/118b4adfd1bf4fcb... like this Renee_

<http://tinyurl.com/qjb29t9>; (at gist.github.com)

thx. you are using the "file" input while I am using "s3". Think that's it

might be

Like this: http://pastebin.com/5Ape1WC1

Title: input { s3 { type => 's3' access_key_id => 'HJKHHLUHJHJKJKJCKQA' - Pastebin.com (at pastebin.com)

u need to investigate the bucket input then

cuz thats the one you're using

is there a version of kibana which will work with elasticsearch 2.0.0 without problems: https://gist.github.com/igalic/fdd4fe60271e0db2... ?

<http://tinyurl.com/pj6pg4a>; (at gist.github.com)

im using latest kibana with elastic 2.0 without this issue meena

iNs: kaythanks

meena: check ur default index, the .kibana one, maybe its malformed somehow or dunno, ootb it worked for me

i on the other hand, am still having problem with excessive debugging logs from logstash

Hey, I got a question. Is it possible to have logstash output logs directly to cloudwatch logs? I noticed the cloudwatch plugin, but I get the feeling that one wants to push to cloudwatch metrics?

I want it to accept logs from monolog and directly push to cloudwatch logs

Is that possible or do i have to write to a log file and have the agent push it?

In what kind of a situation is it even possible for logstash to only be running at about 10-15 events per second?

Having a hard time searching for "slow logstash" issues since 8k eps is considered slow in some cases lol.