#logstash

torrancew: you're right there's a type field

gilhoum: IIRC logstash won't re-type an event

if there's a type associated with it, then reading it from an input with "type" wont' change it

torrancew: is that a new behavior , this was working perfectly since months before i try to update logstash

What was your old version?

I only started using LS back in august, myself (1.2.0)

and AFAIK that's been the case since then

torrancew: thanks, maybe it's not new i will just use the type in it as the type in logstash

torrancew: i was just "lucky" to use the same name

ya

now, the redis input "type" would apply to any events that did /not/ already have "type" set

jmartin: yeah i am trying to make an email alert that only fires if certain things a present in the alert

the examples i've found use match, but say you need to use conditional now

don't use match, the 'match' thing was really confusing

on what conditions do you want to send an email?

torrancew: FIXED thanks :-D

gilhoum: np!

one sec, just lost my browser, once i get it back i can give detail

ping again, in case someone has any input; getting 'java.lang.IllegalStateException: field "host" was indexed without position data; cannot run PhraseQuery' when querying by a field. host in this case. running 1.3.3 but I did run 1.3.1 for a week when it came out.

each time i save a kibana dashboard, when i reload it later, the data are not coming, i see a progress bar for pies, etc. what s wrong ?

replaced the template but hasn't fixed it after a day.

is it the template problem... that indexed without position? how do I re-index?

thoht: so when you go to the main default page (without a search), there is no data at all?

jmartin: you still around?

iniazi: if i go to main default, what do you mean ?

YES - Great! We have a prebuilt dashboard: (Logstash Dashboard). ?

if i go there,i can see an histogram with data

jmartin: i havea bunch of hashes coming in from a bunch of hosts specifically in three fields: added, removed, and updated. ideally i'd be able to alert if a hash is seen in one of these 3 fields that is unique across all hosts within the past week.

is that possible?

and if there is a hit, then i'd also like to include a related field, the field "name" which contains the path to the file

whack: do you know if that's possible?

anyone? =)

iniazi: ?

thaht: sorry, so main page has data, but the saved searches don't? sometimes if you don't select fields to show before saving, it will give some timestamp error. You are using the logstash dashboard json to start right? otherwise I'm not sure

if the save dashboard, has fields selected, and no data, what is the error? is it an elasticsearch error? or is kibana saying no data found? it may be refering to an index that is no longer there, etc.

iniazi: do you know if it's possible to do what i described above?

if you go from a working point, then you narrow it down or create chart etc. and save it, and then go go that dashboard later on, it will work.

Was wondering if anyone has outlined preferred methods to collect data and send to Logstash. Ex. Haven't yet found a way to send data from sensors over the network to LS. On one distro I'm using collectd, but need an alternative for a different distro

what is the easiest way to set timezone in LS/ES so that i dont have to keep converting back and forth between utc?

maseda: kibana should use browser based time correction (if you set it up that way) but LS/ES operate on UTC (as it should :P)

ronnocol: fair enough. i dont really use the gui (yet), but i guess i will just set timezone in my scripts that talk to ES over 9200

(ronnocol: i am currently writing an alerting script that must work outside of kibana . the type of join-like queries that i need are not possible in kibana)

maseda: fair enough. But having managed datacenters distributed all over the world, I can assure you that having UTC timestamps for everything will make your life easier. If you want to present a local time, do that at the very end of the chain when you present the information to the user... not when you store it.

ronnocol++

words to live by

having things in ES stored in UTC doesn't mean you can't convert to whatever other timezone you want

so even if you really can't change your times to UTC, or don't want to read times in UTC, the time format logstash uses is parseable by practically every langguage and platform ever

whack: I've been offline for a few hours (meetings); but did you see earlier where I said it looks like (at least with the es output) connections to ES is workers+1?

ronnocol: es output using node client? or transport?

output { elasticsearch {}}

it'll connect to ES, ES might connect to it. Where "ES" is "the cluster" not a single node.

it could use multiple connections to a single node, too