#logstash

you can also cluster redis

indeed

i haven't had any problems with my redis box

though you can cluster ES too

depends how often redis crashes vs ES

logstash/ES ive had problems with

if ES has "issues" but redis doesn't then it makes sense

i've never used redis, so i don't have any experience with it

Dar1us: that's correct, for me i was having problems with logstash/ES reliabiility, redis solved that

Stoosh: OK fair enough

its entirely up to you how to structure it, having a broker in the middle made sense for me

redis is also not your only option, there's also AMPQ, and probably others

Sketch: yea, thats true

vks: its very simple, just set it up as per normal passenger site

move kibana to /var/www/kibana

put document root to /var/wwwkibana/static

and away you go

Stoosh: i am not using Kibana3, my mistake i had to be specific about Kibana version. I am using Kibana2

vks: im talking about kibana2 :)

ill bin my apache virtualhost for you

oh so this will work with Kibana 2 also , ok

Title: DocumentRoot /var/www/kibana/public ServerN - Pastebin.com (at pastebin.com)

that config + a normal passenger/apache install works great for me

Hi Logstash.

subb1: howdy

Using the grok debug app. I'm getting desired results with my grok filter patterns. But from the kibana web interface, the whole log event is stored as in @message field.

my grok patter - %{SYSLOGTIMESTAMP} %{HOSTNAME} %{SYSLOGPROG}: %{GREEDYDATA}

Stoosh: with virualhost in config. will it will be able to use with the I.P

??

remove the servername directive

vks: i just make a dummy domain

foobar.logger

and do a hosts file change

<ip> foobar.logger

that way i don't have to remember the IP

This was my earlier filter - %{SYSLOGTIMESTAMP:date} %{HOSTNAME:host} %{SYSLOGPROG:program}: %{GREEDYDATA:message}

But it kept adding additional fields to Kibana's interface.

no but if i wan to use only ip then, the above setting would be fine???

yes, just remove the ServerName line

hope I was clear..

if i remove the server name line , then i would be able to access it from ip and port??? i guess we need to configure it, suppose i don't want to use port other than 80

sry i want to use port other than 80

vks: you would have to change the apache config

by changing Listen 80, to whatever port

and then changing that in the virtualhost tag *:80

which means in that case i can use apache for only one application?

vks: not necessarily

just depends on how you set it up :)

any hints guys?

subb1: sorry :( all the core guys are all in bed haha

generally by this time

:(

ok Stoosh

does logstash have an archiving to s3 function or something similar?

chendo__: no, what would you be archiving?

uh… logs?

like what happens to old indexes?

old indexes?

logs

thnx Stoosh

in ES?

sure

like generally i wouldn't want to search the entire log history

but i'd like to if i wanted to

in kibana i just load the last 7 days of the index

of indices*

i still keep all the indices in ES just dont use them for searching

so.. nothing to archive?

chendo__: there are some bash scripts floating around that can archive

nothing native to logstash however

hmmm okay

thanks

chendo__:

Title: Backup/restore Elasticsearch index | tech.superhappykittymeow.com (at tech.superhappykittymeow.com)

<http://goo.gl/WuZ61>; (at imperialwicket.com)

i haven't used or looked into them, so i don't know if theyre current.. use at your own peril :D

fair enough

subbl : with ur grok pattern %{SYSLOGTIMESTAMP} %{HOSTNAME} %{SYSLOGPROG}: %{GREEDYDATA}, there is no identifier

probably thats why u re not able to see anything elase

by default u must be seeing some fields in kibana???

vks, thanks .. but it is not.

the semantics are optional to the grok patterns.

semantics/identifies**

I use file as an input for logstash... Does logstash automatically monitor change in the directory?

somehow, it won't automatically send to the elasticsearch when new file is added.

qullnarak: from docs "Files are followed in a manner similar to "tail -0F". File rotation is detected and handled by this input."

qullnarak: sorry misread, new files

what's strange is that I created log.txt with content "logging1" and log2.txt with content "test2"

logstash read log2.txt but it won't read log.txt

so basically, only log2.txt message appear in my elasticsearch

assuming it matches the glob correctly

qullnarak: possibly look at beaver, thats what i use, and monitors new files

much smaller overhead for production servers too :D

I have a custom formatted log text inside a file. How would I convert it to appropriate json_event for logstash input?