#apache-kafka

/

      • Hoffman joined the channel
      • IRCFrEAK joined the channel
      • IRCFrEAK has left the channel
      • karll has quit
      • tbsf has quit
      • tbsf joined the channel
      • tbsf joined the channel
      • amcm joined the channel
      • karll joined the channel
      • abernstein joined the channel
      • abernstein has quit
      • rthille-ciena joined the channel
      • rthille-ciena joined the channel
      • karll has quit
      • tbsf joined the channel
      • karll joined the channel
      • tobyirc has quit
      • rthille-ciena has quit
      • rthille-ciena joined the channel
      • abernstein joined the channel
      • ottO_ joined the channel
      • Industrial joined the channel
      • stickperson has left the channel
      • rthille-ciena joined the channel
      • Industrial
        Hi.
      • I am using Kafka with Docker and Docker Compose. The problem I'm having is the startup procedure
      • I am using "Dockerize" to be able to wait for a TCP port to be available (it polls the port) before starting a container
      • Kafka opens 0.0.0.0:9092 before it has elected a leader, so my client (kafka-node) fails.
      • If I let kafka and zookeeper finish the leader election first and then start my app manually, it works and it receives all the events.
      • Docker Compose is only for the development stage, not for production (I'd use Docker Swarm or Kubernetes I guess? I haven't gone into production yet with this system/Docker)
      • I'm not sure how to wait for the correct moment to start my kafka client
      • kireevco
        How people are monitoring kafka consumer lag nowdays? I wanted to integrate wiht influxdb or telegraf, and everything seems out of date
      • kafka 0.10.1
      • Industrial
        kireevco: (not related to your question) How are you managing the kafka process? Docker or on the host OS?
      • kireevco
        Industrial: supervisord
      • or systemd
      • Industrial
        ok
      • ottO_ has quit
      • amcm joined the channel
      • tbsf joined the channel
      • hKey joined the channel
      • hKey_ has quit
      • tbsf joined the channel
      • n3o_ has quit
      • n3o` joined the channel
      • tbsf has quit
      • n3o` has quit
      • rthille-ciena
        Industrial: You'll probably have to add retries in your app. We've found that in a production environment you can't always rely on things being there. Retry loops with back-offs are helpful.
      • Industrial
        rthille-ciena: Right. I should do that from the start :-)
      • sood joined the channel
      • n3o` joined the channel
      • Carr0t has quit
      • Carr0t joined the channel
      • oh, great. So I'm using this package 'ascoltatori' that generalizes over several messaging systems (AMQP, ZMQ, Kafka, to name a few) and it uses kafka-node for handling kafka
      • looking at kafka-node I see a LOT of open tickets regarding error handling
      • and even some core Node.js API breakage (callback not called with Error message/string but using arrays or objects as the first argument)
      • that's very bad :-(
      • I just want to get stuff running, working on this for 3 days now, but I'm not going to bet any app I'm making on a client library that you can poke a hole in with your finger >_<
      • I guess there's some other libs worth giving a shot; http://nipstr.com/#kafka
      • but first, to the bedmobile!
      • rthille-ciena
        Yeah, we use Twisted python, so I had to write Afkak ( https://github.com/ciena/afkak ) to support it.
      • karll has quit
      • fracklen has quit
      • fracklen joined the channel
      • Nae`cloud joined the channel
      • fracklen joined the channel
      • aloiscochard joined the channel
      • fracklen joined the channel
      • tbsf joined the channel
      • nathamanath has quit
      • cvanball joined the channel
      • cvanball joined the channel
      • Naresh joined the channel
      • nathamanath joined the channel
      • fede1024 joined the channel
      • rmoff joined the channel
      • cvanball joined the channel
      • hugh_jass joined the channel
      • fracklen joined the channel
      • fracklen joined the channel
      • hugh_jass joined the channel
      • tomaz_b joined the channel
      • hugh_jass joined the channel
      • karll joined the channel
      • fracklen joined the channel
      • cvanball_ joined the channel
      • tbsf joined the channel
      • hugh_jass has quit
      • hugh_jass joined the channel
      • Hoffman has quit
      • fracklen joined the channel
      • hugh_jass joined the channel
      • karll has quit
      • hugh_jass joined the channel
      • sood joined the channel
      • tupty joined the channel
      • mAcR0S joined the channel
      • hugh_jass joined the channel
      • mAcR0S
        Hi guys ! I have a strange question about kafka. I am preparing for a network equipment upgrade in our cluster, and I have some doubts about our kafka 8 nodes cluser. The problem is that we will need racks offline for the network stack upgrade, so up to 4 kafka nodes could be down at the same time (with replication factor 2). What could possibly happen in this case, and how could I avoid issues, if I could, without having to bump replication factor to
      • ? Thank you in advance for help !
      • minimarcel_ joined the channel
      • sood joined the channel
      • karll joined the channel
      • amcm joined the channel