#logstash

hey guys, do you know if elasticsearch does allocate mem ressource on each node with regard to memory available ?

i.e I have a "light" node with 8G and two big nodes with 16G and I wonder if ES will smartly allocate more shards on big nodes to avoid "out of heap space" problems on the light one

hi

how to drop a event in logstash ? I want to drop the message which doesn't match any of my grep pattern

vks: http://logstash.net/docs/1.1.13/filters/grep

Title: logstash - open source log management (at logstash.net)

drop => true

multiline - does the log need to be be 'all together' in the file ?

think that makes no sense. Syslog file I have [a] cows \n [a] more cows \n [b] somebody else \n [a] last cow

so i'm presuming multi log would need all the [a]'s together to treat them as 1 event, but as its syslog its got events inebwteen :/

eper: you need some way to group events together

eper: no, http://logstash.net/docs/1.1.13/filters/multiline

Title: logstash - open source log management (at logstash.net)

It does let you specify a pattern by which to group events

but how long would it keep looking for that pattern?

That I don't know

viq: i have multiple grep pattern . if drop => true logstash is dropping all the event

vks: or you could add tag with successful grep, and drop events without the tag

thats what i am doing now

viq: but i want to remove tag i had added while grep was matched

New news from newjiraissues: Smaine Kahlouch created LOGSTASH-1231 - Date filter doesn't match my field <https://logstash.jira.com/browse/LOGSTASH-1231>;

I've got some logs in elasticsearch via logstash, and using kibana as a front end. The logs contain xml that have a field we'd like to use. Can this be done dynamically?

ie after data is in elasticsearhc.. almost need some sort of plugin sitting between kibana and elasticsearch

or in splunk i'd probably do a subsearch process xml and create a new field from the result

I want to drop the events which doesn't match with grep pattern http://pastebin.com/sKCjGED3

Title: filter { grep { match => [ "@message", "TenantId:\s\sCRITICAL|INFO" ] - Pastebin.com (at pastebin.com)

ah.. this doesn't look promising https://github.com/rashidkpc/Kibana/issues/121

Title: Feature Request: Support Client Generated Dynamic field searching/terms 路 Issue #121 路 rashidkpc/Kibana 路 GitHub (at github.com)

New news from newjiraissues: f62 created LOGSTASH-1232 - How to drop all events which doesn't match the pattern ? <https://logstash.jira.com/browse/LOGSTASH-1232>;

if anybody is curious about the csv stuff I was chatting about last night… I went ahead and added a grok filter and pulled out what I needed :)

hmm accidentally joined log stash to a 0.9x cluster. It appears to work but I know it is not supposed to

eper: regular ES output? really?

yeah hmm

eper: what version of ES?

1.13

its not supposed to work it was an accidental puppet run changed outback back hmm

its updating through kibana says so and no errors but i am sure its not supposed to join

maybe it'll explode when it tries to create an index

as its already created todays index so just updating

eper: I've never seen elasticsearch 0.20 join a cluster with 0.90

jamesturnbull: about 500 million events/day

don't disagree I'm sure it will explode come new index time

no I mean

it won't join

oh

Anyone try beaver with multiple upstream redis servers?

it will start, and may appear to work, but the cluster won't show any nodes with another version

and the clustesr will be separate

hostname: search01a version: 0.90.2

and logs are appearing O-o

[2013-07-31 14:36:31,581][INFO ][cluster.service ] [search01a] added {[Obliterator][UeHMRR31S9SoglIxVFAmgw][inet[/212.23.x.x:9300]]{client=true, data=false},}, reason: zen-disco-receive(join from node[[Obliterator][UeHMRR31S9SoglIxVFAmgw][inet[/212.23.x.x:9300]]{client=true, data=false}])

oh wells i did intend to use _http output but puppet ran and put it back without me noticing

In the grep filter, if I have several good-sized patterns that I want to match against in an OR fashion, is it possible to split it into several lines without breaking the matching?

adamjt: not at this time