#logstash

yeukhon: each index consumes heap space, so if they have the same (or even nearly the same) retention, then you should keep them in the same index.

rastro thanks. and i would rather use index_type to group similar items, so to make query slightly more efficient.

sorry i know this is more ES/Kibana question :-)

somewhat related, i'm looking into tips on how to retain data as long as possible

leaving data compression aside, what limitations are regarding number of indexes/shards on a single node?

rpetre ES? Single node you lose HA and scalability.

i notice ES spawns a lot of threads, maybe one per shard?

yeukhon: the number of nodes and the hardware is pretty much given

rpetre oh i read it the wrong way :-)

i'm trying to see how long i can promise to keep data searchable

rpetre: you will eventually run out of heap with more open indexes/shards.

rpetre: the idea is to add more nodes with more ram.

i've read a lot about space optimizations, but i see having lots of indexes is not very fun

rpetre: also, you can reduce the heap footprint with doc_values

paramedic tends to kill my browser, for instance :)

we have a cluster with a lot of shards and we would hit the rec 64k file descriptors and have to set something higher

ulimit is something we dislike to set.

rastro: any resources i could read related to that?

the machine is pretty much dedicated to this, so i don't care about various os tune-ups i might make

rpetre: http://www.elastic.co/guide/en/elasticsearch/gu...

<http://tinyurl.com/m35xekm>; (at www.elastic.co)

rastro: so cpu cores is not a limiting factor, usually?

rpetre: http://www.elastic.co/guide/en/elasticsearch/gu...

<http://tinyurl.com/pdy4acl>; (at www.elastic.co)

rpetre: i find LS to be cpu-limited, and ES to be RAM-limited.

i'm not very experienced in managing jvm apps

rastro: aha, good to know

I'll probably give LS workers until it keeps redis empty

ah found my issue

Title: tcp output plugin does not send newlines · Issue #1650 · elastic/logstash · GitHub (at github.com)

that was a head scratcher

rastro, yeukhon : any idea if it's worth keeping monthly indexes instead of daily?

they'll probably get pretty big, but i'll have to see if storage permits

rpetre: it's harder to change mappings, and you lose granularity with retention. Also, there's some limit on recommended shard size, i think.

rpetre: i just started thinking about this lately, but i think from what i gather above…match with data pattern and rention. if i want to look at 14 days, 21 days, it probably be easier to just do daily, if my data is a monthly type, makes more sense with monthly? but rastro can correct me.

i currently keep a year worth of webserver logs, i'd love to be able to expose those to users instead of me awk-ing and perl-ing through them avery now and then

rpetre: absolutely!

rpetre: another option is to close the index, so it's still around but won't be searched until reopened.

i need to have at least yearly retention of data for audit purposes as well, i was a bit bummed when i realized it's probably not feasible in elk

rpetre: yeah searchbility and retention are two things

we retain the logs in a backup (like S3 and then Glacier)

yeah, us as well

but at least webserver logs are useful

"are clients using this obsolete interface?"

"have we seen this guy before?"

hey…how can I check if logstash is running when deployed as a service? I thought “ps aux | grep logstash” would do the trick

but I don’t see anything…

on linux

FoosMasta: then it's not running :)

rpetre: start building 64GB machines to add to the cluster. lather, rinse, repeat.

rpetre: doc_values seems to be giving us a 10x gain.

rastro: i have _one_ 64GB machine (actually two, but i'll have to use the other one for graphite, which will kill my io)

wait if you want more than 32, and take advantage of it, you must run 64?

rpetre: moving to three will give you more heap in the cluster and also an IO boost.

i've actually added the other machine as an ES node, but i'm keeping logstash off it, it's there just for kibana and grafana redundancy

yeukhon: you're supposed to give 50% of the ram to ES, but below 32GB, so a machine with 64GB is the "largest" you would need.

yeah, well, this one will have to justify its own budget first

rpetre: of course

why is the 32GB limit there?

jvm

i've read it last night as well

the jvm performance starts to deteriorate with heaps over 32gb IIRC

rastro ah you are right :-) well if you want 64 you'd go 128 then… but I really think no one runs that size. Usually larger heap -> can mean longer pause. I think there are jvm options to make that less painful.

oh, i see

probably one can have multiple ES instances on the same box

yeukhon: if all you had was a machine with 128G of ram, then run two nodes on that one box.

yeah, that's a work-around

split up into different instances of jvm

right.

^^^ like rastro just said!

yeukhon: but you're sharing disk and other busses.

i noticed in the routing settings it can tell if there are multiple nodes on the same server

and power supplies ;)

rastro: well, you're assuming they are (and you're likely right)

but you could certainly point each es at different mount points, and they could certainly be attached over different buses

no refuting the power supply point, though!

are there any other good ES maintenance tools besides curator i should look at?

do you have head/kopf/$something already?

yes, but i mean for automated index management

not that I know of

kopf is pretty nice, it helps me write json better than in cli :)

jq can be of help, too

i'm thinking maybe after a while i can do some other tricks to indexes, like reimporting data with fewer fields

i keep telling myself to learn jq's minilanguage better, but i keep postponing it

maybe once we start using ec2 more

the rationale, as I understand it, behind limiting heap size to 32gigs, is that the JVM can compress pointers and only use a 32bit pointer up to 32gigs of heap. Once you pass 32gigs of heap the JVM has to use 64bit pointers and so you effectively lose heap size.

<http://tinyurl.com/kjezw4v>; (at blog.codecentric.de)

it mentions this reason in the memory usage page linked ;)

ah I didn't see a link. I only saw reference to larger heap size resulting in longer GC pauses

which isn't really the primary reason.

<http://tinyurl.com/m35xekm>; (at www.elastic.co)

the fielddata limit circuit breaker is interesting as well

I assume I have to use something like the json codec to pass events over udp and keep the fields intact, right?

there is a field in the log that shows up as : james.myers

how can i combine it into one field. also, sometimes there is only one name :(

Looking for some help using field substitution in a string field of the DNS filter. It seems to not substitute my variable.