#logstash

Any one play with offline installs?

'bin/logstash-plugin install logstash-filter-prune' works but bin/logstash-plugin prepare-offline-pack --overwrite --output ./logstash-filter-prune-5.4.1.zip "logstash-filter-prune" fails with 'Cannot find plugins matching..."

Hi

I am n00b to logstash and I guess I am missing somwthing obvious

I have a simple kafka input to elastic search output to test the creation of a geopoint

I have tro configuration files from pipelines in my conf.d directory but only the main pipeline appears in the logs

I do see two kafka configurations in the startup logs, but I see nothing in the output of elastic search

input {

kafka {

topics => 'analytics-poc-1'

client_id => 'JA-1'

auto_offset_reset => 'earliest'

}

}

filter {

dissect {

mapping => { "message" => "%{org_date_str} %{field1} %{+field1} %{field3} %{field4} %{lat} %{lon}" }

}

date {

match => [ "org_date_str", "UNIX_MS" ]

target => "org_date"

}

mutate {

rename => {

"lon" => "[coordinate][lon]"

"lat" => "[coordinate][lat]"

}

}

}

output {

elasticsearch {

index => "trace-coordinate"

document_type => "log"

hosts => "localhost:9200"

user => "elastic"

password => "secret"

}

}

These are the contents of one of my pipeline conf files

and there is data in the kafka if I go back to the beginning

please use a gist/pastebin/... for pastes

You aren't getting data, or you are getting data but it's not a geo_point?

Xylakant apologies, n00b here

darkmoovt -> nothing at all in the elasticsearch index

jamiguet: auto_offset_reset does not start at the beginning of the kafka topic

it starts at the beginning of the topic, when no previous position is known

so if you ran that pipeline once and haven't added data to kafka (or reset the offset), this will do nothing

i'd also add an stdout output to the pipeline to see if data is actually incoming

Xylakant: I setup a console on the topic no new data, so that may be a posbility

jamiguet: you'll also need to provide a suitable index template. the default logstash templates only apply to indices named logstash-something

on the other side I am having a stack trace in elastic search[2017-11-28T16:17:51,667][ERROR][o.e.x.m.e.l.LocalExporter] failed to set monitoring watch [gUywU2YuTy2a8nPQHviqdQ_xpack_license_expiration]

java.lang.IllegalArgumentException: Rejecting mapping update to [.watches] as the final mapping would have more than 1 type: [watch, doc]

and they expect the geopoint field to have a specific name pattern

hm. that looks watcher-related and like a bug in watcher

index template? I setup the index follownt this https://pastebin.com/VwRiwBSt

ok red haring then

ok, if you specifically provided a mapping when you created the index, that would work fine

it's just fairly uncommon for logstash setups

I guess I started the wrong way around

It looks like there is no new data in Kafka that is why is not being picked up. I will see with my colleagues why there is no data flowing.

you can manually reset the offset for the logstash consumer

oh how?

I don't know about the latest kafka versions, but in 0.8 you could basically remove the consumer group from zookeeper

I'd assume that in the later versions you need to go via the kafka broker, but that's pretty much exactly what the client does: it periodically sends the last offset it processed to kafka/zk

and you can with the cli tooling for kafka send arbitrary values, for example "0" as offset

Many thanks, I will try that