#logstash

huh

does logstash support storing different log sources in different indices? specifically i'd like to be able to keep different sources for different periods of time, i assume logstash ages out logs by deleting indices, so i'd need different logs in different indices, is this all supported?

catphish: sure

hurrah

i tried graylog first, i liked it, but it doesn't support this partitioning, everything has to go into the same index and must be aged together

catphish: logstash doesn’t delete indices, it only sends the logs to elasticsearch,

catphish: in the output filter, you can do some if else and route the logs where you want

I’m routing my logs based on type. Each type goes to different index

Something along those lines:

elasticsearch {

hosts => ["ES_HOST:9200"]

index => "logs-%{type}-%{+YYYY.MM}"

}

this saves each type to its own index

but be aware, in Kibana, you can only operate on one index at once… at least for now

matejz: that sounds good, does one have to use a separate tool to delete the indices? i think i've seen something to handle this

also, i'm pretty sure kibana can operate on wildcards, i'm sure i've done it

in fact logstash-* is the default config by the look of ir

*it

catphish: that works, but speed can be slow if you are searching for single type over 30 indexes:)

catphish: for deleting indexes, you can use ES API, or even better, 'curator'

catphish: https://github.com/elastic/curator

Title: GitHub - elastic/curator: Curator: Tending your Elasticsearch indices (at github.com)

that makes sense, and yeah curator is the tool i was thinking of :)

it has a great command line and it’s just nice to use

its a shame this isn't handled automagically by logstash, but i'm sure i can configure something

catphish: logstash can’t delete indices by itself

** r4z waits answers **

matejz: that's a shame, but i'm sure this is a common requirement so i'm sure curator can handle it without too much hassle

catphish: yea, curator can handle it no problem

tremendous

catphish: but it’s not a part of logstash

that makes sense

its a separate script, but it does the job:)

I just run it in crontab on one of the ES node

How do you make a line break in a conf file? I have a query that is 219 chars wide. Or should it just be in a separate file (it's a SQL query)? It's not a particularly complex query, just selecting several fields.

Seems like a simple question but I can't seem to get google to answer it for me.

hello is the elasticsearch inplugin suitable to "stream" contents of one ES server to another via logstash?

input plugin*

depends on what you mean by "stream"

continious updates from several outlying ES clusters into a centralized ES cluster

then not.

the input runs a single query and reads the query result to logstash

it does not re-run the query.

so you could cron periodic updates and grab the last 5 mins worth? or so?

Could someone help me debug this?

Title: input { beats { port => 5044 } } filter { mutate { add_fi - Pastebin.com (at pastebin.com)

I want to do a basic thing of creating a new field based on a field called 'source'. Doesn't seem to be working.

I got my mutate command to create a new field called file which is equal to %{source} (filebeat input). But some records have content for 'file' and some don't. ?