#apache-kafka

/

      • Impakt
        eg: add to cart
      • i was thinking we have a datastore of our workflows
      • in the SO example the workflow would be ineterested in log in events
      • when we receive a log in event from kafka we query the datastore and ask it if there are any workflows interested in a customer logged in event
      • we'd get a hit
      • kick off the workflow in our workflow engine
      • now that workflow would do wahtever then hit an API and say "ok my task is done for now, save me and let me know when this customer who logged in views page X"
      • in the meantime kafka is getting all sorts of http + commerce based events, constantly querying the datastore to see if there are any workflows waiting on the event
      • so later on down the track this customer viewed page X
      • ecommerce server uses a simple API and sends a "page viewed" event with payload to kafka
      • kafka send it to this black box that then queries a datastore and says "who's interested in a page viewed event with this payload?"
      • styles
        That's where you're mistaken
      • You don't want a process to "ok my task is done for now, save me and let me know when this customer who logged in views page X"
      • That's implying that the process is holding state
      • IE: I want to know about user X
      • that's dangerous and error prone
      • What happens when that process crashes?
      • How would you "restart" that same logic
      • Sasazuka_ has quit
      • Check out the general concept of how they break it down to create the same type of events / workflows you're talking about
      • Impakt
      • we wanted to replicate the functionality that this oracle product provides
      • styles
        Yeah, that'd be rolling back on the event log for the last 30 minutes to see if it happened
      • That's literally a background map/reduce ish type job
      • Find me X users that NEVER triggered the purchase event
      • That's what something like Couchbase would do great at
      • It doesn't have to happen exactly on the 30 second dot
      • spuder joined the channel
      • Impakt
        yeah i get that
      • so using the example on that oracle page
      • they have the Logs In part
      • styles
        I 100% would not store workflows in a db though
      • I get what you want really
      • Impakt
        say we have thousands of customers using our site
      • styles
        You want some sort of dynamic workflow system that you can keep building new triggers
      • Impakt
        when someone logs in we need to query something and tell it to not send the email out
      • but let's say the critiera is "when someone logs in and they're male and live in USA" dont worry
      • styles
        That's not true
      • Impakt
        otherwise they get the email
      • styles
        That's just the workflow code
      • If the newer event was a login then .. it shouldn't trigger
      • My point is that's just workflow information
      • I would probably build a database of queryable properties per workflow
      • And then allow myself to daisy chain workflows together to trigger events
      • Sasazuka joined the channel
      • Impakt
        yeah this oracle thing has the ability to build dynamic queries
      • styles
        IE workflow register [date > 10 days] and then also has login [date > 5 days] == create new email event from [template = x]
      • Impakt
        it's limitless
      • styles
        So each event, you'd map out the key at least and the value's data type
      • On the UI you'd generate some sort of event dasiy chain, allowing you to eventually create a new event with w/e propertiees
      • And then the worker would look at all the created work flows and process it based on all the event data or w/e
      • Impakt
        im thinking mongodb, we store the payloads in there alonig with timestamps, event types etc
      • styles
        I wouldn't use mongo, but yeah anything map/reduce is the real goal
      • Impakt
        ok
      • styles
        You might want to checkout Apache Storm
      • It can hookup to kafka
      • You create "spouts" for each event or w/e
      • And it's a real time computation system
      • An example is - spotify using this to process every song play ever
      • And every month they need to calculate how to pay royalities to artists
      • Impakt
        but then there's this
      • so oracle lets you plug in what they call "actions"
      • these actions happen on the spot
      • an action could be to send an email, etc.
      • styles
        If you want to rebuild this system, I'd have to sit down and really dig into all the features
      • Yeah they're basically making simple components that allow them to interact with each other
      • soosfarm joined the channel
      • Event and Actions
      • Impakt
        but then the action could also be to add a free item to their cart, change shipping price to zero, change the promotional content of the site etc
      • styles
        check pm
      • Impakt
        let's say i had a promotion, fill out this survey and get a free item
      • ok thanks
      • timvisher joined the channel
      • Sasazuka has quit
      • Sasazuka joined the channel
      • daMaestro joined the channel
      • Sasazuka has quit
      • SkyRocknRoll joined the channel
      • dnrb has quit
      • champagst has quit
      • champagst joined the channel
      • champagst has quit
      • champagst joined the channel
      • timvisher joined the channel
      • dnrb joined the channel
      • dnrb has quit
      • dnrb joined the channel
      • SkyRocknRoll has quit
      • higgins joined the channel
      • kleptocroc- has quit
      • sood joined the channel
      • sood joined the channel
      • timvisher joined the channel
      • nharkins has quit
      • Impakt has quit
      • Impakt joined the channel
      • timvisher joined the channel
      • dnrb has quit
      • SkyRocknRoll joined the channel
      • Impakt has quit
      • timvisher joined the channel
      • sood joined the channel
      • anechaev joined the channel
      • Riton has quit
      • anechaev has quit
      • Riton joined the channel