#logstash

rastro - "search_analyzer": "default_search"

I'm unable to reach DreamObjects storage, presumably due to their maintenance. Does anyone know a mirror from which I can download the logstash JAR file?

banik, i flipped most of my log fields to "index" : "not_analyzed", which kept it from tokenizing everything.

New news from newjiraissues: Jason Martin created LOGSTASH-1542 - Allow time-macros in file definitions <https://logstash.jira.com/browse/LOGSTASH-1542>;

whack: is that you in the kibana video ?

Hi... where can I see the kind of fields the "geoip" plugin can add?

I've read about %{geoip.longitude} and %{geoip.latitude}

is there a %{geoip.country} or similar?

Title: logstash - open source log management (at logstash.net)

hi. I was wondering if I use glob like "/var/log/*.log" in my file input then it would match multiple file, how would I distinguish between those in Kibana? Is there a way to set type == filename or something like that, or should I tag them somehow (how to extract filename in that case)?

under fields

there's the built in GeoLiteCity fields

ouch

stonith, thanks... I missed that!

gansbrest: the @source_file will be in kibana

ok, so that's going to be added only if I use globs right? What if I set just one file on some other box (like I have now). I don't see that @source_file field, is there a way to include it for even single files?

gansbrest: the best way to know is trying

check the different fields that show up in kibana and you'll see if there are ways to differentiate between events

I mean, between source files

JoeJulian: you here?

gansbrest: oops was @source_path; should be everywhere even without a glob

Hi... again with geoip :) I'm trying to use "bettermap", and it seems it need a field with longitude and latitude

I read from Jordan in this thread http://comments.gmane.org/gmane.comp.sysutils.l...

<http://tinyurl.com/kd8xefz>; (at comments.gmane.org)

that I should use    add_field => [ "lonlat", "%{geoip.longitude},%{geoip.latitude}" ] 

but it's not working for me (although people in the thread seem happy about the suggestion)

The field in logstash doesn't substitute the vars

so I have the field "lonlat" with literally the content "%{geoip.longitude},%{geoip.latitude}"

hello

any hint?

i'm having some trouble getting the logstash-index-cleaner.py script to work against an ES cluster with nginx serving as a load balancer

the script works fine if i point the ES host arg to a specific node in the cluster

poiu: why are you using a load balancer? elasticsearch can work in a cluster

the output of elasticsearch_http can only point to one host

it's a single point of failure for the logstash's connection to ES

i have a 3 node cluster

isn't the host that acts as a load balancer a SPOF as well?

possibly, but our ES nodes are more prone to failures than our nginx server :)

anyway, that was the recommendation i got from this chat :) to point to nginx

poiu: also, the output doesn't require a host, afaik you can use discovery

elasticsearch_http plugin?

Is there a script that already exists to close/delete indexes that are older than n days?

poiu: no, elasticsearch

davuxx: yes, i know, but logstash config requires a single host

i cannot specify an array to use discovery; i asked this group

afternoon

poiu: can you use elasticsearch instead of elasticsearch_http?

no, i think we needed elasticsearch_http to use the lastest ES version with logstash

rather than use the embedded one

Does anyone have a date filter for apache logs? Struggling quite a bit with this. This is a sample timestamp: [22/Oct/2013:08:15:44 -0400]

poiu: ok

ronnocol: yes, there is: https://logstash.jira.com/browse/LOGSTASH-211

Title: [LOGSTASH-211] Python script to delete indices older than a set number of days or hours - logstash.jira.com (at logstash.jira.com)

does this look right? match => [ 'log_timestamp', 'dd/MMM/yyyy:HH:mm:ss Z' ], seems to stop all data from being parsed if I use that

inside of a filter block

brooks_: could it be a locale problem?

davuxx: not too sure, what would a locale parameter look like?

brooks_: MMM means it will be a 3-letter month, e.g. Jan

davuxx: the log entry looks like this: [22/Oct/2013:08:15:44 -0400]

to return to my question though, specifying the load-balanced host and port returns with error:

Traceback (most recent call last): File "logstash_index_cleaner.py", line 183, in <module> main() File "logstash_index_cleaner.py", line 148, in main connection = pyes.ES('{0}:{1}'.format(arguments.host, arguments.port), timeout=arguments.timeout) File "/usr/lib/python2.6/site-packages/pyes/es.py", line 216, in __init__ self._check_servers() File "/usr/lib/python2.6/site-packages/pyes/es.py", line 279, in _check_se

brooks_: is logstash running with a "C" or "en" locale?

last line is: RuntimeError: Unable to recognize port-type: "9189"

davuxx: how would I check?

davuxx: (there's nothing specified in the conf file)

brooks_: you could type "lang" in a shell

poiu: thanks

I'm pretty sure it's not a locale problem, but i would check that first anyway, just in case

TomasNunez: http://pastebin.com/Br0dV0Lt

Title: mutate { split => [ "latlong", "," ] } mutate { - Pastebin.com (at pastebin.com)

davuxx: in a linux shell? that doesn't seem to be a command

brooks_: sorry: locale