#logstash

hey, I am trying to change the default embedded port of elastic search..

so I have this event hitting logstash, and in kibana I want to create a histogram with two lines - one from the messages.per_second total, and one from connections.per_second -- can I do a single histogram w/ multiple lines from the same events?

I have added the embedded_http_port property in logstash.conf, though I still get an error in the kibana dashboard for not being able to connect to the 9200 port

what else am I missing?

roothacker: do you have something like ufw running (or another firewall)

EyePulp: not sure if that's currently possible but i remember there is or was ( not sure if it was merged ) PR for it to enable that feature. best is to look into the PR list of kibana

electrical: thanks - I'll take a look

np

some care to look at this and point me in the right direction? this on freebsd10 with openjdk6 http://pastebin.com/ADzjPzm6

Title: /usr/local/logstash # java -jar logstash-1.3.3-flatjar.jar agent -f logstash-bro - Pastebin.com (at pastebin.com)

google turned up another user using solaris and seemed to be a prob w/ a 32bit vs 64bit java issue

@Eyepulp: nope, there is no firewall running on my machine that blocks 81 port, also when i do http://localhost:81/_status it displays the Elastic Search index status

roothacker: port 9200 needs to be open in order for you to hit ES, unless you're running ES on a different port, in which case you need to edit kibana to point to the correct host & port

EyePulp: I have firewall in another location which blocks port 9200, which is why I was trying to change the elastic search embedded port

roothacker - that's fine, the key is that you need to have an open line between kibana on the client side and ES on a server.

and kibana needs to be told what host/port to reach ES at

anyone?

EyePulp: how should I tell Kibana that I am running Elastic Search on port 81, when I am running Kibana through Logstash ?

You edit the config.js file that kibana serves up.

EyePulp: how should I override the config.js served by Kibana inside LogStash?

I mean packaged inside LogStash

do I need to specify through Nginx.conf (in my case), to serve different config.js ?

roothacker: I imagine there's a way, but I just followed this route: http://www.elasticsearch.org/overview/kibana/in...

<http://tinyurl.com/otw9dkp>; (at www.elasticsearch.org)

roothacker: trying to futz with the baked in logstash version seemed like more trouble than it was worht when I really just need to serve up some static html/js files

(in my situation, that is)

EyePulp: hmm, I get your point, thanks, will try

roothacker: that being said, are you sure that the kibana files that come with logstash aren't sitting unzipped on your drive someplace in order to be served up?

electrical: I have a multi line message ( json export ) that I would like to log, so if store it in a log then I have to write a complex parser, so I would like to have the option of sending the message to logstash directly

from php

EyePulp the directory in which I run logstash creates a single folder by the name data containing elasticsearch data

roothacker: bite the bullet and download the kibana zip. =) you'll be past this issue in no time

EyePulp: doing the same now, will update if I find a better solution

Dunno if its any use to anyone, but I packaged redis as a rpm: https://github.com/uow-dmurrell/redis-centos - the spec file is a fork of someone else's work, I just updated it to work with the new version + a few little sysadmin type fixes

Title: uow-dmurrell/redis-centos · GitHub (at github.com)

kuku: ah okay. hmm. not sure if its possible. in general we expect in LS the json message to be a single line.( json_lines codec ) not sure if the normal json codec understands a multiline json message

kuku: an other solution would be to send it as plain text, use the multiline filter to make the separate lines into a single event. then use the json filter to translate the content of that into json.

electrical: I think this is the PR that would solve my multiple-plots from multiple event properties in a singel historgram issue. Sounds like it's not settled yet. https://github.com/elasticsearch/kibana/pull/374

Title: Enable histogram panel to plot multiple fields by tvvmb · Pull Request #374 · elasticsearch/kibana · GitHub (at github.com)

EyePulp: ahh okay.

lets hope it will get merged soon

electrical: Is there a way to "unserialize" the message in elasticsearch /

kuku: how do you mean? unserialize the message?

electrical: yeah, it would make for some pretty charts. =)

EyePulp: defo :-)

i ve log with directly the country name in log in a field, can i use geoip ?

thoht: geoip filter only gets the info based on the IP address.

or do you mean the maps in kibana?

yes the map in kibana

so i don t need to use geoip

i can check on the net the latitude/longitude of the 6 town i got as value

ah okay. euhm. if those are the short names ( UK, NL, etc ) it should work i think.

electrical: do you have an example of values expected in the field ?

oh

directly country letter

i believe the short names are used. let me double check

Displays a map of shaded regions using a field containing a 2 letter country , or US state, code. Regions with more hit are shaded darker. Node that this does use the Elasticsearch terms facet, so it is important that you set it to the correct field

so i got FR, DE, IT, UK, BE, ES, NL

should be good

thoht: from the docs "Displays a map of shaded regions using a field containing a 2 letter country name"

if [message] =~ /Support-FR/ { add_field => [ "country", "FR"] } }

should be enough ?

yeah should be good

let s try a shot !

okay :-)

electrical: i ve to reimport my old data (6 months)

i got a uniq _id generated with anonymize

so i won t have duplicata

but will it add the field ?

or should i delete the current data ?

curl -XDELETE 'http://localhost:9200/_all/_query?q=_type:XXXXXXX'

thoht: can you show me the config of LS ?

yes

electrical: http://pastebin.com/aH9h7NbM

Title: input { ###################################### # CSV EXTRACT FOR 1 SHOT - 6 - Pastebin.com (at pastebin.com)

electrical: it is the input siebel_all_country

the type i mean

thoht: i don't see any filters our outputs it would respond to?

oh wait. sorry there are

it is # GEOIP

the first filter

grok/anonymize/data/country

config looks good.

electrical: if i re-inject same data, will it add the new field or should i remove the existing one ?

hmm. it could fail on those yeah... not 100% sure. safest way is to remove the existing stuff

curl -XDELETE 'http://localhost:9200/_all/_query?q=_type:siebel_all_country' should be ok ?