#kubevirt

rmohr, morning

fabiand_: hi!

rmohr, I need your support to debug the issues

let me paste: https://paste.fedoraproject.org/paste/69ams9EyR...

fabiand_: 5 minutes

fabiand_: I need the events

fabiand_: make sure to collect them with --all-namespaces included

fabiand_: otherwise I don't see much

aren't they?

I thought those were the events with all-namespaces

no ..

okay

let me re-run

fabiand_: I just saw the events from the default namespace in what you have collected yesterday

fabiand_: also, there is a test replicaset included, which creates three VMs, you can also deploy it and see how long it takes until all three vms are ready

fabiand_: I know now why it fails for you!

oh

fabiand_: I think the issue is that it is pulling down the docker image

tell me :)

here is an up to date paste:

I see a few weird lines:

default 2017-10-05 09:06:09 +0200 CEST 2017-10-05 09:06:09 +0200 CEST 1 haproxy-574c8d574-94w4z.14ea9abee0014edc Pod spec.containers{haproxy} Normal Killing kubelet, master Killing container with id docker://haproxy:Need to kill Pod

since your network connection is slow, it is not fast enough with downloading it

docker://haproxy:Need

which image?

ain't all images present on the vms?

rmohr, ^^

default 2017-10-05 09:06:08 +0200 CEST 2017-10-05 09:06:08 +0200 CEST 7 libvirt.14ea9abe8327a0ad DaemonSet Warning FailedCreate daemon-set Error creating: pods "libvirt-" is forbidden: service account default/kubevirt-infra was not found, retry after the service account is created

fabiand_: I thought that one: kubevirt/cirros-registry-disk-demo:devel

fabiand_: but it says Container image "kubevirt/cirros-registry-disk-demo:devel" already present on machine

fabiand_: so that is not it

fabiand_: the service account is not an issue

kubevirt-test-default 2017-10-05 09:12:04 +0200 CEST 2017-10-05 09:12:04 +0200 CEST 1 virt-launcher-testvm8rm7b-----6ztx4.14ea9b1182a19103 Pod spec.containers{disk0} Warning Unhealthy kubelet, node0 Readiness probe errored: rpc error: code = Unknown desc = container not running (3702c136d527472f0796252ec8c15f332e7d7efbfeef111cca588fe36f958a3b)

fabiand_: it will retry and after a few times it is there

fabiand_: rror: failed to start container "disk0": Error response from daemon: cannot join network of a non running container: 09fc33c58f98565d8311dab910c5705722c9cb0d4009d77f6aa640dbacb35f75

kubevirt-test-default 2017-10-05 09:12:15 +0200 CEST 2017-10-05 09:12:15 +0200 CEST 1 virt-launcher-testvmnmnrr-----2vhg2.14ea9b14074cf33d Pod spec.containers{compute} Warning Unhealthy kubelet, node0 Readiness probe errored: rpc error: code = Unknown desc = container not running (e348e53624151d2521a48f5ae5b2273721482f7ee5617e7d75ce836e338a094a)

fabiand_: rror: failed to start container "disk0": Error response from daemon: cannot join network of a non running container: 09fc33c58f98565d8311dab910c5705722c9cb0d4009d77f6aa640dbacb35f75

that one should be it

other tests were nto related to registry disks

rmohr, did you see the stack trace aet the end of https://paste.fedoraproject.org/paste/sRkKz4vIp...

fabiand_: yes the stack trace is not so much an issue

ok

fabiand_: our tests will try to clean up the namespace for one minute

so is

VirtualMachineReplicaSet

[90m/root/go/src/kubevirt.io/kubevirt/tests/replicaset_test.go:162[0m

A valid VirtualMachineReplicaSet given

[90m/root/go/src/kubevirt.io/kubevirt/tests/replicaset_test.go:161[0m

[91m[1mshould update readyReplicas once VMs are up [It][0m

[90m/root/go/src/kubevirt.io/kubevirt/tests/replicaset_test.go:109[0m

[91mTimed out after 60.000s.

Expected

<int>: 0

to equal

<int>: 2[0m

also using registry disk?

fabiand_: yes

ah

fabiand_: so the stacktrace is printed because ginkgo kills all goroutines

fabiand_: so for some reason, the disk0 container does not start for ages because of this error (but sooner or later it seems to succeed)

fabiand_: this error: cannot join network of a non running container

fabiand_: so our tests time out

hm

I'm reverting to before vossels rework

let's see if they pass there

actually, vossel provided a patch for some issue

fabiand_: btw. they pass for me with vossels change

fabiand_: they also pass on CI

right

then it's really interesting to understand how this issue can be debugged

fabiand_: could be a bug in our code or k8s hard to tell: https://github.com/kubernetes/kubernetes/issues...

fabiand_: others see that too, but not much data there :/

I'm checking now before vossels changes, and then with his + his prs

rmohr, when did you recreat your vagrant env the last time?

fabiand_: yesterday

hm

and NUM_NODES=1 ?

yes

ok

tests https://paste.fedoraproject.org/paste/sKmpCL1VE...

events https://paste.fedoraproject.org/paste/zJXXA2yaV...

two failures

volumes for pod "virt-launcher-testvmpvhn8-----vv29b_kubevirt-test-default(fca2e188-a99c-11e7-a0ad-525400e23a00)": timeout expired waiting for volumes to attach/mount for pod "kubevirt-test-default"/"virt-launcher-testvmpvhn8-----vv29b". list of unattached/unmounted volumes=[sockets default-token-5fjcj]

avctually

kubevirt-test-default 2017-10-05 09:17:45 +0200 CEST 2017-10-05 09:17:45 +0200 CEST 1 virt-launcher-testvmpvhn8-----vv29b.14ea9b60e4265409 Pod Warning FailedMount kubelet, node0 Unable to mount volumes for pod "virt-launcher-testvmpvhn8-----vv29b_kubevirt-test-default(fca2e188-a99c-11e7-a0ad-525400e23a00)": timeout expired waiting for volumes to attach/mount for pod "kubevirt-test-default"/"virt-launcher-testvmpvh

n8-----vv29b". list of unattached/unmounted volumes=[sockets default-token-5fjcj]

rmohr, could you please take a look at the events again?

correlation between test and events is difficult

fabiand_: I see the same errors again

so, how can I fix i?

rmohr,

fabiand: I don't know. Either an error in the registry disk code or in k8s

fabiand: the failed mounts are btw. always there

do you know why?

fabiand: no, but they appear in general with different frequency with different k8s versions, also on ordinary pods

fabiand: so I think the only relevant one is "cannot join network of a non running container", since it will prevent virt-launcher from becoming "ready", and therefor the VM will not be started ...

right

fabiand: I am not sure if that will help you, since you will still have the newer k8s version

are you suggesting to pin an older k8s version?

fabiand: could you try to update weave to the latest version?

currently rebuilding vagrant

fabiand: btw, are you using the docker cache?

not at all

fabiand: too bad

I'd say that it depends on the POV :)

fabiand: yes, if you have plenty of time why not

… or if I want to make sure that the whole chain works.

fabiand: kubectl apply -f https://github.com/weaveworks/weave/releases/do...

fabiand: that will install the latest weave version

fabiand: maybe the errors go away then