#logstash

who was using logstash with hybris again?

I have a question about logstash, can I ask it here?

Go ahead, and we will see if anyone can answer

I am using logstash to fetch data from sql to elastic. And because logstash doesn t nest values from docs with same _id, I am using the aggregate plugin. But the examples that I have all have a flag like "task END" to output the nested field to the output.

how can I get Logstash 5.0?

If you don t have this flag "task end", its possible to do?

guys plz help, i get this error -> :exception=>#<EOFError: End of file reached> any idea?

and it cant deliver an email

We use tab-delimited CSV for our access logs and when our vulnerability scanners hit the applications, they tend to toss a bit of garbage in the fields. One thing being a double quote (written to the log as '\"') which the CSV parser does not like.

I'm trying to replace '\"' with '"""' but it looks like it's getting written to the field as '\"\"\"' when I believe I want '"""'. Any thoughts? mutate { gsub => ["event_message", '\"', '"""'] }

Is this the correct approach, or should I be changing a value on the csv filter?

torrancew: trying to get the bug repro done today

Can I compare a document ID with the previous document ID?

torrancew: hrm... minimal conf can't seem to reproduce the issue

I have some questions about using logstash to ship logs to elasticsearch and I’m not sure if I should ask them here or in the elasticsearch channel. The questions are about the interface between the two products ; right on the boundary. As such, please forgive me if I ask a question in here and it should be in the elasticsearch channel.

The summary is this: I am running Elasticsearch 2.2.1, Logstash 2.2.2, and Kibana 4.4.2. Logs are shipping correctly, but when I view them in Kibana, I note a problem with the mapping. Host name is an analyzed field and is getting split on hyphen. I want it not_analyzed. I understand that I can’t change a current mapping, that I need a new one.

On a related note, I have tried searching for ‘host.raw’, but it’s not being created or I can’t find it; when I try searching for it, kibana returns no results.

Question 1. how do I specify the mapping? Is it a config change to the logstash elasticsearch plugin or a config change to the elasticsearch server? I want to manage the mapping with puppet. Relatedly: is this the most current one that I can copy https://github.com/elastic/logstash/blob/v1.3.1... and make changes to?

<http://tinyurl.com/jo6y89q>; (at github.com)

My research so far shows that for question 1, I can’t put a config file into place on the ES server, I have to use the API. This is firmly into the ES side of things so I’ll take my question there.

GiantEvolving: the elasticsearch puppet module can manage index templates

I don’t see mapping mentioned on https://forge.puppetlabs.com/elasticsearch/elas...

<http://tinyurl.com/oxpxkvg>; (at forge.puppetlabs.com)

because you can't manage mappings indipendently from indices

Ah….

you can only create an index with a given mapping

but there's a feature called index templates

So what I really mange with puppet is the template.

yes

:D

and that can contain a mapping that will be applied to all indices matching the templates selector

well, all newly created indices

Yeah.

OK, much knowledge unlocked. I guess the only thing I’m still concerned about is why I can’t query host.raw now with the default template.

I’m not entirely sure how to read the default template yet, but ‘raw’ is present in it.

i'd have a look at the actual mapping that the index has

you can retrieve that via the api or use a plugin such as kopf to display it

Sure thing, good call, I’ll check that.

Sure enough, no raw versions of the fields…

Makes me wonder what template is actually being used.

Will check...

Is it odd that there’s no template in the server at all?

curl -XGET 'localhost:9200/_template/*' returns nothing

GiantEvolving: I think you'll have to enable logstash's manage_template for it to add the template

I use puppet to manage the templates, so I must admit I'm not really up to date, but I think the default changed to "off"

hmmmm default is true according to https://www.elastic.co/guide/en/logstash/2.2/pl...

<http://tinyurl.com/gt53zca>; (at www.elastic.co)

But I’m happy to manage it in puppet anyway.

quick and possibly dingy question, if I use a csv filter to add columns are these supposed to show up as fields in Kibana/Elasticsearch on output?