#grpc

Hello!

hey :)

Any experience setting the lb_policy for channels in python world @Pixel?

I'm not a python guy, but I can poke one for you

what's your question ?

Hmm we are using

channel = grpc.insecure_channel(endpoint, [('grpc.lb_policy_name', 'round_robin')])

but it isn't obeying the round_robin policy

Hi ted

howdy

And/or how can we compile the python bindings with a debug flag on?

i.e. view the output of gpr_log() calls invoked by the cython wrappers & set the log level thereof

log levels are controlled by environment variables

you can do for instance

GRPC_VERBOSITY=DEBUG GRPC_TRACE=all

to get an insane lot of debugging

like, all the things

that's awesome! thanks.

What about me? =(

I've tried getting attention from python devs

but it's Friday...

THanks buddy!

Is google groups the best for this in reality?

g3p0: Hi, I'm part of gRPC Python Team. :-)

Hooray!

On windows, the following lb_policy isn't obeyed.

channel = grpc.insecure_channel(endpoint, [('grpc.lb_policy_name', 'round_robin')])

Well... maybe hooray; maybe not. gRPC Python shouldn't, and I believe doesn't, do anything with the channel arguments other than pass them to gRPC Core.

nathanielmanista: the behavior we're seeing is that on linux it always round-robins, regardless of which policy it's specified, and on windows, it always picks first

(or it always sends requests to the same endpoint)

Sadly I don't think the keys and values of channel arguments are printed with GRPC_VERBOSITY=debug GRPC_TRACE=api. Maybe with GRPC_TRACE=all?

we set GRPC_TRACE=round_robin

we can see from grpc core log output that the round_robin.cc code path is being invoked

This is sounding more and more like a Core problem. |Pixel|, are you feeling similarly?

g3p0, tedward: how are you specifying "endpoint"?

where cluster.foo.com resolves to 4 A records

Whoa. OK.

endpoing = 'dns:///cluster.foo.com' where cluster.foo.com resolves to four A records.

As for compiling gRPC Python with a debug flag on: we don't particularly support any compilation over any other but if you find us doing something particularly obstructive please let us know about it?

Oh the environment variables were key. Thanks.

g3p0: have you learned something? Something you're able to share?

Hmm here is the output from windows land https://pastebin.com/7sNbEKeF

pastebin including output from both windows & linux runs: https://paste.ee/p/zg44b

here, i've grepped the above logs for the log line where it selects the subchannel: https://paste.ee/p/OYTbs

you can see that in linux it selects pretty randomly from indexs 0-3

and in windows it mostly selects index 1, sometimes 0

btw, we're only making 3 grpc requests in each of these runs, so i'm not sure why it hits the round_robin.cc select code so many times

Poking through the logs, in windows land we are seeing that the subchannel lists and channels are being canceled and unreffed, but the same code on linux land just works.

d3p0, tedward: apologies for passing you along like a hot potato but we're beyond the frontiers of my familiarity with the code. I'm trying to find you another team member to come along. :-)

Thanks!

well, its friday afternoon in the bay, you might need to start a thread in our support forum

Always but who knows who will show up.

I think we traced it down to the round_robin policy doesn't exactly work in windows for some reason.

In linux land it appears to be round_robin, even though it is using pick_first. It just happens quicker so it seems like it is balanced.

rr is pretty balanced, every new rpc gets a new subchannel

well, the next one in the list anyways

Yeah, but on windows land the channels and channel_lists when using the rr policy end up being cancled and unreffed

cancelled because the connection is torn down, or because it was never valid to begin with?

windows and linux logs here https://paste.ee/p/OYTbs