#logstash

techminer: should be the same.

drdre: I'd expect a PUSH to a url with 'templates' in it's path somewhere

techminer: @message is the field created by logstash automatically containing the message field on arrival - message is different field - possibly created by the action of a filter.

I pushed it to set the template yesterday and it took it.

So today as it use the new template it broke?

So I have a central loghost that adds data to the front of the log entries. An extra timestamp, the origin host, and origin program. I want to split off the extra parts the loghost adds, grab just the origin host, and then process the remaining original log message. Is grok what I want to use?

grok { match => [ "@message", "%{SYSLOGBASE:logger_data} %{GREEDYDATA:original_message}" ] }

tecminer: Yep - that'll create 2 fields - logger_data and original_message pulled from @message

Then I could grok the origin out of the "logger_data" and finish processing the "original_message"… ???

drdre: Ahh, I see - sorry, misunderstood what you were doing...

I need to detect IP address in a field, then try reverse DNS on it. Is there a better way than grokking the field to match an %{IP}

nope

drdre: It looks like you've set a field in your mapping to be cast to the wrong type, and ES is having issues casting it back again

ok.

It'd be nice to be able to say: if [host] =~ /%{IP}/

drdre; If I had to guess, I'd say you had a field called PagedBytes, and you've set it to a string, not a number in the mapping

drdre: sorry, ignore the 'PagedBytes' bit - but some field is mis-cast

could not find pagedbytes

when a template is added will it impact all the indices? Indices create before the template map is added and after?

my ES cluster seems to be indexing data still but Kibana is throwing out an error. I pulled the Last Elasticsearch Query and curled on the command line and got that.

drdre: new templates only affect new indexes.

labrown: Thanks!

welcome

is there an easy way to clear all elasticsearch data?

Mattch or anybody: I am still figuring out curl ... how do I compare indices?

hrdcore0x1: delete the index

looking for this mis-cast.

labrown: using the curl -XDELETE ? what if.. I have a lot of indexes like 'logstash-2013.08.21' 'logstash-2013.08.22' etc

drdre; Just have to grab the template from ES and then do a json diff locally I thnk...

labrown: I'd like to not have to iterate all the indexes, but instead just delete all info

hrdcore0x1: should accept wildcards - e.g. logstash-*

mattch: excellent thanks

curl -XDELETE 'http://localhost:9200/' will blow away ALL indexes.

labrown: A good reason to set 'action.disable_delete_all_indices: true' in elasticsearch.yml :)

Mattch: agree

What's the best way to sample an incoming event stream? Say I want to keep only 10%, or 1% of the logs (for a development server, for example)

never mind. found it.

It states in the docs for the elastic search output plugin (http://logstash.net/docs/1.3.3/outputs/elastics...) that: "Your elasticsearch cluster must be running elasticsearch 0.90.9". Does that really mean >= 0.90.9? Or does it mean what it says, only 0.90.9. Elasticsearch is now at 0.90.10 so I wanted to check.

<http://tinyurl.com/ohwcmbm>; (at logstash.net)

can someone point me to the documentation for the forwarder config file?

Guest79662: https://github.com/elasticsearch/logstash-forwa... there's a bit about it in the readme

<http://tinyurl.com/odobhay>; (at github.com)

(if that's what you'r ereferring to, anyway)

mikeybisbo: ES 0.90.9 is known to work (and is tested). You are welcome to try other releases (like 0.90.10) and they *should* work, if they don't, it will be obvious in that you will get lots of errors.

@akat yes there's a bit but I'm looking for more details. For instance, what is the appropriate value for "fields type"

@akat yes there's a bit but I'm looking for more details. For instance, what is the appropriate value for "fields type"?

erp why you message twice

strings

Guest79662: fields are just string-string mappings

@whack string-string mapping between what? and what?

@whackI guessing "fields" is supposed to describe a log entry, the various fields on the log entry, etc

Guest79662: oh, I suppose I should be more clear

Guest79662: every line read from files by logstash-forwarder becomes an event. Using 'fields' you can attach extra data to every event

This is useful for marking what kind of log it is, perhaps some other data like cluster name, purpose, role, etc

it's a way to annotate logs before they leave the forwarder

Whack got it

Guest79662: So let's say you're forwarding Apache logs, you might forward access.log, ssl_access.log, error.log, etc

but you might want to mark them all as one kind

so you could set it to "apache"

good example might be the host name ?

whack: okay great, I'll have a play with 0.90.10! thanks