#logstash

It's going in to ES

If that's what you're asking.

that sounds like an HTTP response code to me

The log has a lot of double quotes, and some symbols, but I've never run into an issue with that before. I can greedydate the whole log and not have problems.

I have 4 nodes each with 1gb jvm heap; and my log indices are growing to over 7gb per day. when i go to kibana i'm getting "courier fetch: N of 50 shards failed". I'm backing up each day to s3 and deleting it from the cluster and things work ok through the morning, then that problem comes back. Do I have any options besides making my cluster bigger or putting less data in it?

es v1.4.4

can you add heap to the nodes?

7gb per day is not query-able in fullw ith only 4gb of heap in the cluster

at least, not particularly well

most of the time i'm just looking at 15minutes slices

but occasionally want to search for some string for, maybe, the past 30 days

doesn't have to be fast at that scale

hard to get around the fact that querying involves heap

hwo many days do you keep on the cluster?

for now, just a single day

but i'd like to keep 30 days

so, yeah, you're going to need more heap

ES recommendation is to give ES 50% of your RAM as heap

(last I knew anyway)

if you're not doing that, start there

one option that i could do is get picky-er about what i store, throw away logs that aren't errors or metrics i want to track

sure, but first, you've got to solve your 1day problem, before you even think about the 30d one

and it would definitely make sense to start by making sure your cluster follows best practices from upstream

In Kibana, is it possible to auto expand all the rows so there's no need to click on each one?

hey guy's i'm testing ELK 5.0-alpha and getting a problem on elasticsearch output plugin.

Currently i'm running a cluster of two nodes, and trying to push the logs with output { elasticsearch { host=>['myip'] } } but i got the error "Translation missing"

but when the ES is in localhost and i use just: elasticsearch {} works perfectly

have no idea whyis happening

using the elapsed module with logstash 2.3.1, I have timeout set to 60, but I'm seeing events with elapsed_time on the order of 200,000. quite puzzling

also confusing that the docs indicate fields will be named like elapsed.time, elapsed.match, etc., but the code clearly uses underscores, not dots

keith4: probably got missed int he recent refactor

a recent ES version enforced that no dots be in field namesa

guys, found my problem there was a typo ! host => should be hosts

[sqs_message][pseudoReq][auditReqInfo][userSettings][attributes] -> How can I make sure this exists all the way down the path without a bunch of nested ifs?

Hey has anyone here parsed multiline logs before where each line shares a certain ID (e.g. Auditd logs)? I found a ticket on it that was marked as resolved last year...but can't seem to find any info.

Joel: have you tried running multiple instances of logstash?

Someone asked me what happens when my logstash instance shuts down. Mine doesn't. Is not shutting down a new feature?

jakinov_: use the multiline FILTER (not codec), parse ID out into field, use that as part of stream_id for filter

torrancew: I'll try that out thanks

hey guys. having some trouble converting to a float. is there something wrong with this syntax?

mutate { add_field => { "tps_count" => "0.00027777777778" } convert => { "tps_count" => "float" } }

platrelefs: what are you seeing?

the field is present as a string

in elasticsearch?

yes