#apache-kafka

/

      • marself has quit
      • karll joined the channel
      • karll has quit
      • cdaley joined the channel
      • daMaestro joined the channel
      • karll joined the channel
      • mgaare has quit
      • karll has quit
      • karll joined the channel
      • mgaare joined the channel
      • cdaley joined the channel
      • karll has quit
      • ldiamond_ joined the channel
      • tasse has quit
      • tasse joined the channel
      • llinguini
        pykafka_user44: Why do you want to do that? That sounds like a problem if your queue is very large
      • karll joined the channel
      • daMaestro joined the channel
      • cupdike has quit
      • cdaley joined the channel
      • ldiamond_ has quit
      • karll has quit
      • cvasilak joined the channel
      • palmerlao joined the channel
      • krphop has quit
      • palmerlao
        hi, I'm working on a customer Converter for Kafka Connect and was wondering how the best way to debug these were
      • I'm trying to get log messages to show up but am not seeing anything when I test my converter with the connect-standalone program
      • repys joined the channel
      • daMaestro has quit
      • repys
        how can I create consumer group?
      • TomBentley joined the channel
      • turlando joined the channel
      • turlando has quit
      • palmerlao has quit
      • Snaps
        pykafka_user44: disable auto commits
      • repys
        ?
      • gregks has quit
      • gregks joined the channel
      • grayjoc joined the channel
      • SkyRocknRoll joined the channel
      • vmukha has quit
      • grayjoc has quit
      • thxffo has quit
      • thxffo joined the channel
      • marself joined the channel
      • marself has quit
      • marself joined the channel
      • karll joined the channel
      • Darcidride joined the channel
      • karll has quit
      • karll joined the channel
      • repys has quit
      • ldiamond_ joined the channel
      • ldiamond_ has quit
      • marself joined the channel
      • karll has quit
      • grayjoc joined the channel
      • TomBentley has quit
      • TomBentley joined the channel
      • grayjoc has quit
      • SkyRocknRoll joined the channel
      • j0xaf joined the channel
      • karll joined the channel
      • marself joined the channel
      • marself has quit
      • karll has quit
      • justanotheruser joined the channel
      • karll joined the channel
      • karll has quit
      • justanotheruser has quit
      • krphop joined the channel
      • pykafka_user44
        llinguini: because I want transactional guarantees. If my program crashes, I want to be able to replay the queue unless the result of the queue is persisted.
      • Carr0t
        I suspect that will require you to do some of your own logic. Even Kafka's exactly-once logic only works while you remain in Kafka. If you're saving state out at any point you need to work out that logic yourself
      • marself joined the channel
      • jscholz joined the channel
      • hazamonzo joined the channel
      • pykafka_user44
        Carr0t: not sure what you mean by saving state out?
      • dijksterhuis joined the channel
      • SkyRocknRoll has quit
      • SkyRocknRoll joined the channel
      • sameerynho joined the channel
      • Sasazuka joined the channel
      • Sasazuka joined the channel
      • TomBentley has quit
      • al-damiri joined the channel
      • ldiamond
        Anyone here know how to deal with zookeeper?
      • I want to replace zookeeper processes "pods in kubernetes"
      • What do I need to do if I replace a given zookeeper server with another that has no data?
      • geekatcmu
        Let me start with the caveat that my k8s experience includes lots of annoyance and no specific Zookeeper-related work.
      • In general, if you replace a ZK, the new node will simply stream all the relevant data from the leader.
      • and then become a full member of the cluster.
      • I rely on this behavior for my own ZK clusters where I have the transaction logs on tmpfs.
      • When a ZK restarts, it comes up with a blank transaction log directory, and the last commited image. It then picks up the relevant transaction logs, applies them, and drives on.
      • That it is in k8s should be irrelevant.
      • Just follow your normal node replacement procedure.
      • llinguini
        Is the best way to poll from a single partition only using multiple consumers or is there a way to use a single consumer but only poll from a single ktp at a time?
      • rayjenkins joined the channel
      • rayjenkins has quit
      • hazamonzo has quit
      • mliang2 joined the channel
      • puppet has quit
      • mliang2 has quit
      • timsmart joined the channel
      • puppet joined the channel
      • timsmart has quit
      • pykafka_user44 has quit
      • puppet has quit
      • puppet joined the channel
      • puppet has quit
      • rayjenkins joined the channel
      • puppet joined the channel
      • rayjenkins has quit
      • jscholz has quit
      • geekatcmu has no idea what you're on about
      • geekatcmu: I have a client application that I want to poll from kafka for records from. I have a setup where one node in the client application needs to poll from n different topic partitions. If I want to have the ability to chose which topic partition I poll from.
      • The current consumer API has a single poll() method that I think will return messages from any of the partitions we were subscribed to
      • ldiamond_ joined the channel
      • The reason I want to have such a method is in order to reactively poll. If the downstream consumer of a certain partition needs more messages to process for that particular partition, then I want to be able to do poll(topicPartition) and get messages for that particular partition
      • One way I can think of is to have n consumers each subscribed to their own topic partition on a client node. But this seems like not a good idea because a) the consumers hog up resources on the client, and b) each consumer creates a new TCP connection to Kafka, which is obviously not scalable