#logstash

TandyUK does source machines are i any clusters? What I mean is if they are in separate datacenters, then I would put kafka as close to those datacenters as possible and then add inter-region replication to another kafka - to have a buffer

so I would suggest writing to kafka from one side with various tools and read fro kafka from other side with logstash

this way you can scale kafka and logstash independently in the future

and it would be possible to create HA setup much more easily

Hi everyone, im new to logstash, and standing infront of installing logstash on a EC2 instance

Anyone have some good pointers when using dynamodb and elastic search with logstash inbetween?

I have this that i want to run: https://github.com/awslabs/logstash-input-dynamodb

im new to elk and i cant to seem to make sence out of it. Why is my nginx body_sent.bytes is parsed as string altough i specify %{NUMBER:[nginx][access][body_sent][bytes]:long} in my grok filter in logstash.conf?

can someone please give me a hand?

It's probably in your index template

If the field is already registered as a string, it won't change until a new index is created

(Or until you reindex)

bjorn_: I haven't specified any templates manually, i'm using those filebeat setup uploaded

bjorn_: I've already tried to delete whole index but it's still shows as string - Bytes in kibana field view

bjorn_: so i cannot actually perform sum aggregation on this field

You can override ES' index template

Also, in logstash, you can try to convert explicitly

convert => [ "[your][field]" => integer ]

bjorn_: Actually somehow only geoip fields get parsed as numbers, other fields like response_code are also strings

Bbl, lunch

bjorn_: Yeah but is it necessary? I thought this one should work out of the box. It actually does work if i choose to send my logs directly to elasticsearch in my filebeats

bjorn_: thank you, have a great lunch