#logstash

New news from newjiraissues: Andres created LOGSTASH-1744 - Logstash doesn't change year var after 20131231. <https://logstash.jira.com/browse/LOGSTASH-1744>;

Hi, I have 7servers with ES indexes, I wish to reduce it to six, Is there any easy way to do that?

Tinuviel, is your data distributed over all the nodes ?

Yes, but no shards for indexes

so what I need to do is to move out three logstash indexes.

What I'm trying to do is to move out from server 7 all data write and let there only logstash.

Tinuviel, I think this could help you -> http://www.elasticsearch.org/guide/en/elasticse...

<http://tinyurl.com/polfbs8>; (at www.elasticsearch.org)

I tried to convert es on server number 7 to nodata node, but it's not smart enough to start moving indexes automatially

Thanks vali, may be helpfull

your welcome

and second thing, logstash will always create it's own elasticsearch node?

I created es node well configured and I wish logstash to use it, but it always create it's own one.

Tinuviel, from what I know there is a node for each logstash instance pushing to an ES cluster

the LS node is pushing to a ES data node

So logstash will use this node fro all communication with ES? read and writes right?

yes

That's explein a lot

I have on logstash server also ES DATA node

but now I see I need to configure also logsatsh NODATA node

give it more ram, more workers maybe

why ?

When I'm sending query for big set of data

I'm getting Java heap space error

you can use -w I thinkg to specify the numbers of workers for the started LS agent

or use more indexers/shippers

and this is happening on logstash server

Tinuviel, you need more RAM for that error

what is in fact using this ram? logstash ES node or just data nodes?

If you run out of memory you should also have some dump files in that folder

with the pid that died

logstash node is not a data node

yes I know

ES is separated from LS

so if your LS agent is dying you need more ram for that

not for ES

but when logsatsh is starting it's own ES node, and use it to query ES data node isn't it (ES node created by logstash) require more ram?

hey

what's the ES version required by logstash 1.3 these days ?

vali: he's speaking about embeded ES

:)

I saw it now

thank you

I see that I need more ram for ES, I'm just nor sure whould it be ram for local ES Data node or for no data node.

kick: 0.90.9

at least this I saw there: http://logstash.net/docs/1.3.2/outputs/elastics...

<http://tinyurl.com/lx28k5s>; (at logstash.net)

Tinuviel, are you talking about embedded ES ? :)

vali - no :)

I was using it long time ago

and I thought I know the differenc

but now I'm confused

i thougth that I need to setup separetly in ES nodata node, then just point logstash output to it and thats all.

but logstash is still creating it's own node

Tinuviel, you set the RAM limits for ES in elasticsearch.in.sh

start with an ok value for your server and go up from there if needed

ok, and how it will be split against local ES data node and logstash nodata node?

when I start elasticsearch I will only start data node

that I don't know but you can ask on #elasticsearch

when I start logstash, then I will also start nodata node from logsatsh, almost without any configuration.

ok I will.

Do you familiar with Kibana? Especially queries?

*Are you

Tinuviel, I'm not sure why are you asking so much about starting no data nodes

Quick question; how do I get fqdn in host instead of shortname?

those are just nodes (the actual agent hostname) that join an ES cluster

tobias-, you have to have it in your logs

vali: it's not from syslog, it's from a file input. Am I missing something?

tobias-, if it's in your files that just save it/use it in a variable

it's not from the log-file, it's in the event as "host", the logfile-rows gets in @message

great, then just grok the message and save it in some variable

hello all

can anyone confirm is LOGSTASH-1406 is still an issue in 1.3.2?

Jira issue [LOGSTASH-1406] Logstash dies on email output failure - logstash.jira.com - https://logstash.jira.com/browse/LOGSTASH-1406

I know it's unresolved but Logstash has had some major revisions since

I don't want to update though unless it's fixed

hi all, any help with json filter? getting _jsonparsefailure, trying to parse this: "json" => "{\"gid\":\"69163cccd64aafb91eba8437a2ad844e\"}"

"json" => "{\"gid\": \"69163cccd64aafb91eba8437a2ad844e\"}" to be precise