#logstash

which log shipper would people recommend on http://cookbook.logstash.net/recipes/log-shippe...

<http://tinyurl.com/mnk53ay>; (at cookbook.logstash.net)

leev: I just use logstash to "tail" my log files and drop them to redis or RabbitMQ

leev: but that does use quite a lot of RAM

yeah, think i might use beaver shipping to redis

does anyone know if the pdf version of the logstash book has DRM?

only asking so I know if I'm limited to particular readers

when i pass all my logs into redis the host changes from the sender to 0.0.0.0

someone knows why?

Hi all. If all ES instances are down, how long (if at all) would a logstash indexer typically buffer for before the event is lost?

my specific error is ""Failed to flush outgoing items" and wondered if there are some logstash setting to queue to disk or perhaps not read from its input queue unless it knows it has an ES node to write to?

does the syslog input filter do anything smart with the @timestamp? I'm seeing the messages displayed as the timestamp local to that machine (and then converted to UTC), I thought without a date filter @timestamp would be be set to when the message is received

when I look at the debug information it look like a date filter is applied, even if not specified in the config

Hi there. Having a strange time getting two linux openvz ES nodes joined into a cluster. They both have unique node.name entries and share the same cluster.name ... i haven't set any of the discovery options, but have set bootstrap.mlockall: true, indices.memory.index_buffer_size: 50%, index.translog.flush_threshold_ops: 50000

joeblow750 I have never really done this but network.bind_host

I have had issues if they are not in the same network/vlan or vpn

blackmaria: i tried network.bind_host but to no avail ... both hosts are on the same network/vlan and can both telnet to each other's 9200 port

joeblow750 give me a sec...

joeblow750: try unicast instead of multicast zen discovery

<http://tinyurl.com/ol7xppu>; (at www.elasticsearch.org)

joeblow750  you are in same vlan/network and not in ec2?

blackmaria: yup, same vlan/network, and not in ec2

this might be best for #elasticsearch but I am still looking

joeblow750 you mention bootstrap.mlockall does it work with this option false?

joeblow750 just in case you have firewall issues ES also uses 9300 for cluster coms ( iirc )

have currently got iptables accepting for all chains and selinux is disabled ... disabling bootstrap.mlockall didn't help

joeblow750 can you post your logstash config to http://pastebin.com/ or like? ( or is this literally an ES cluster issue )

Title: Pastebin.com - #1 paste tool since 2002! (at pastebin.com)

blackmaria: i've narrowed it down to literally an es cluster as I have stopped all logstash shippers and indexers, cleared out all ES indexes and it still won't join for a vanilla install

that is odd

can you do your telnet magic from es host to es host ?

… and this is linux right?

yup the telnet to each other on 9200 / 9300 works .. linux containers (openvz)

i've never worked with containers…

sugarC_ mentioned http://www.elasticsearch.org/guide/en/elasticse...

<http://tinyurl.com/ol7xppu>; (at www.elasticsearch.org)

I've never needed it SDC/ec2 nor vsphere

but openvz who knows

joeblow750 I have been googling … "There seem to be general problems/needed workarounds with OpenVZ - I am tempted to close this bug" sorry I cant be of more help ( it is getting late in my TZ :)

goodluck!

fwiw I have never had to set the multi/unicast setting is I used a bind host.

anyway, gtg

blackmaria: no probs, thanks for your efforts

cool, I hope you can get it working … check back in #elasticsearch and check their bug tracker to help direct your question… and evidently the ES mailinglist might help too.

l8r all o/