#weavenetwork

hi all, is there any way to "containerise" weave, ie, run it in a container? From what I've seen it looks like there is not choice but to have some "host" context, to address the appropriate namespaces and create/link the bridges and interfaces

I've been playing (in my mind sofar :) ) with the idea of combining projects like https://github.com/gliderlabs/registrator and consul with weave. The idea would be to declare IP addresses for a container in the consul KV store, and then have something like registrator (or a fork thereof) watch for changes in docker or consul, and run weave when/where appropriate to "re-attach" interfaces as fast as possible after container start/restart

anyone stumble across something like this yet?

I'm thinking that a containerised Weave (as per moonfish's reply above), would be easily integrated in such a work flow, I could use the Docker API to issue weave commands to the host (as parameters to the weave container)

I've certainly toyed with that idea (I work on Weave)

but it seems like there would be a noticeable lag between starting up the container and its Weave network interface appearing.

See also https://github.com/binocarlos/powerstrip-weave

bryanb: what would introduce the lag?

moonfish: would you say powerstrip could be used for something thats "not prototyping Docker extensions"?

'docker run' -> Docker starts the process -> Docker fires an event -> receive the event -> create the network interface -> attach the interface

stephanbuys: sure

bryanb: I guess "significant" lag is relative :) At the moment I have a saltstack master/minion setup that does the "weave attach" at the end of a run, at least a couple of seconds delay there

Yes, depends what your program can stand.

Cooperating with Docker to do the network config inbetween creating the process and running the container entrypoint would be best.

which is what powerstrip-weave does

for some value of "cooperating" :-)

moonfish: powerstrip becomes the rest frontend for all relevant commands right, in other words, restarts should be handled transparently

and theoretically docker-swarm should work quite well with powerstrip (at least frictionless)

it does

so in a sense the ideal would be to either a) extend powerstrip to do something like registrator does (and watch consul), or b) make something like registrator work with powerstrip (rather than docker direct)

what role does registrator-like-thing play in your setup?

moonfish: my main gripe is with fig, et al. I want a bit more of a loosely couple system. I've prototyped something that a) takes a fig.yml and converts it into a consul KV store representation, then b) a daemon that monitors (using push/long waits) the consul KV store and based on the state of the keys creates and starts docker containers

the problem used to be weave integration, I would have had to monitor the docker events and then fire something on the host to do the weave magic

now it sounds like powerstrip will be my "foe-docker" endpoint, and I just get weave for free :)

stephanbuys: one of my next powerstrip adapters is going to be powerstrip-registrator - i.e. a clone of registrator that works with the docker API rather than the docker events stream

another one is going to be powerstrip-env-injector - i.e. inject ENV vars into containers from consul/etc

binocarlos: why the change?

because I'm trying to add to the library of powerstrip adapters :-)

ah, ok, I think I follow. Powerstrip is node.js right?

also - it means you could do pre-hooks which change the registration values

no the powerstrip server itself it python - we are getting a golang port on the go

individual adapters can be any language - all they need to do is present a HTTP endpoint

ah

so powerstrip-weave is node.js (because thats my main thing)

BUT any lang will do for adapter :-)

"go all the things" I say :)

I am getting into go (slowly) - I like!

(although node is also my strong point atm)

I guess my effort to move "fig" into a daemon boils down to me not wanting to have to manually run a command on each change of my containers layout (which is pretty complex), I would like to define a "state" and then have a little helper keep it there. Consul is nice as I can mess with the KV through so many mechanisms, it also has ACLs and TLS

last question :) in a swarm architecture would it be correct to do: "swarm" -> "powerstrip-weave" -> "docker" (I'm assuming powerstrip-weave needs to be local to each docker host in an environment)

stephanbuys: yes almost - "swarm" -> "powerstrip" -> "powerstrip-weave" -> "docker" -> "powerstrip-weave"

and yes powerstrip + adapters are all local to each host - think of them as replacing/mimicking the docker server on each host

binocarlos: congrats you stumped me :) if you can I would really appreciate if you could elaborate on that (it seems pretty complicated)

stephenbuys: I mean that where there is a docker deamon - there is a powerstrip deamon plus all adapters

the powerstrip deamon "proxies" requests to the docker deamon

swarm would speak to powerstrip - which contacts all adapters

once adapters are finished - powerstrip passed the request to the docker deamon

the docker daemon does its thing and then powerstrip intercepts the response

powerstrip then contacts all adapters again (with a post-hook)

(I missed out the "docker" -> "powerstrip" -> "powerstrip-weave")

this is important because powerstrip-weave triggers "weave attach" on one of the post-hooks

make sense (sorry for spamming) :-)

in terms of architecture, swarm could be running on a seperate host, but it would speak to powerstrip (which in turn will speak to docker) for each of the "member nodes"?

exactly

please spam away :)

think of powerstrip being a front-man for docker - swarm is then choosing which powerstrip (ah-hem docker) to speak to

ok I'm with you

so yes - combine powerstrip-weave with powerstrip-flocker and now swarm (without knowing it) can control portable IP's and portable volumes

so now, powerstrip would also speak to powerstrip-flocker (also installed on each of the "member nodes")?

exactly

powerstrip allows multiple adapters - composition is one of the big things about it

i.e. weave + flocker in the same container

"networking & storage" ftw

ok, pardon my french, but this is f*cking awesome

:-)

and per my earlier discussion with moonfish, my little daemon that does the instantiation, etc can just speak to swarm with the right environment variables, and the rest is magic

(discounting bugs etc, :) )

spot on happy days

weave uses "-e WEAVE_CIDR=..." and flocker uses "-v /flocker/...:/data"

the adapters notice these settings and do their thing

binocarlos: thanks very much for all the help

stephanbuys: welcome :-)

binocarlos: does powerstrip support TLS?

good question not currently - I'm fairly sure this will be part of the roadmap soon though

this is a bit of a problem noted

we've just got it to listen to a unix socket however so you could use a TLS terminator as a temporary hack

hmmm, actually, I'm thinking that maybe the powerstrip endpoint can be exposed inside weave itself? Thats encrypted…

it would take a little bit of handwaving and bootstrapping outside of powerstrip, but if the main use-case for powerstrip is container networking external automation could work

but it will take two cups of coffee to puzzle that one out

stephanbuys: I just put the kettle on! - let me know how you get on :-)

yeah, its almost redundant to bind powerstrip to a local port on the host, it uses a link to talk to the powerstip-weave container (can be replaced with weave or simple bridge on host), but thats not a threat. Then its as simple as putting swarm and powerstrip in the same WEAVE_CIDR…

now I just need to check if weave-powerstrip can use the docker unix socker, or otherwise it would have to go into weave with weave import

ok, scratch the last statement, it uses the socket in your README

in my case I would use my existing saltstack minions on the hosts to do the initial launch and attach

stephanbuys: interesting - so you are talking about getting powerstrip and adapters talking over a weave network? that would be cool...

binocarlos: yeah, TLS seems almost redundant if you can just hide/encrypt the traffic within weave anyway, its just slightly more complex to bootstrap

in my perfect world all IPC for a cluster will be strictly within weave

indeed, provisioning would be harder but outcome awesome :-) - is this an open-source project you are working on (i.e. is there a link where we can see this masterpiece come together)?

yeah, have a look at https://github.com/panoptix-za/HotRod

its about time I fix the README file :)

at the moment all the orchestration is happening from a salt-master (in a container) out to a salt-minion (my powerstrip ;) ) on the host

I'm going to be adding the powerstrip idea over the next couple of days, you should check docs/INSTALL.md to get started

binocarlos: do you have experience with saltstack?

stephanbuys: a little - I know it uses ZeroMQ :-)

ah, well thats all there is to know :P

lol

no, I was just wondering how I can point you at the interesting parts, but its a bit tricky if you dont have some salt experience.

stephanbuys: hotrod watched and starred :-) - give it a go - I've written a bunch of salt states so "may" be able to pick it up

binocarlos: ok, so the interesting parts are https://github.com/panoptix-za/HotRod/tree/mast... (that bootstraps a host and gets weave going), you can use salt-cloud to spin up a cluster but it hasn't been tested in ages

then my "powerstrip" lives in https://github.com/panoptix-za/hotrod-composition

under the "pillar" you will find a fig.yml'esque definition of a cluster, and in cluster you will find my "build, run and network" steps

the hope is to deprecate hotrod-composition and go to a powerstrip model

ahh I see - it's like fig yaml in saltstack pillars

my fascination with moving fig.yml into a KV store ended up in saltstack for now, the point being that it makes the "fig.yml" definition composable (default settings, etc), otherwise it can get quite messy to have overly large fig.yml files. I went this route (including weave) when my fig.yml had 30 containers in with a nightmare of linking dependencies

perhaps salt-stack is still what to use for that - I remember it being useful for composing large complicating configs

and then powerstrip / swarm for the "run" part - perhaps you can save the pillar into the K/V but still use salt for the merging?

binocarlos: exactly, the I also wanted to build custom containers on the actual member nodes of the cluster, in fig I would have to build it all on a master machine and then push/pull to docker on the hosts

i.e. not sure what tool that exists that can do all of the merging of various configs

custom registry (backed by S3)?

binocarlos: i think swarm/powerstrip replacing the "run" part would be step one

stephanbuys: agreed

binocarlos: i have a small prototype that does it for me and shoves it into consul's KV store, but salt can be step 1

my biggest frustration with salt is that its very slow

binocarlos: hi btw, what happens with powerstrip if there was an outage on the host? I'm guessing that the docker daemon will diligently start the containers in a inconsistent state? (sans weave)

stephanbuys: indeed - although I think thats more of a provisioner problem - so in this case salt should restart everything

agreed

ping binocarlos

djx: hey :-) - am gonna get round to testing powerstrip-weave but need to get a few things done first

is there any chance of a Vagrantbox that replicates your OS? (would help me out)

there is probably one already out there is the world of vagrant boxes

^^ in the world

hmm probably, I can try to find it but I'm not a usually Vagrantbox user