#grpc

is there a channel to ask about google test on freenode?

zeks_: Not that I know of. Probably not. Are you having some problem with it?

hi everyone.

Heyo

hey ejona! how do you do? amazing job on the java side, looking sharp.

Thanks!

It's going alright. Trying to push forward to GA!

yes, i heard there were a few hard thigns to tackle as of recent months.

anyway, i'm using latest release 0.14.0 within a Spring Boot app and i finally got the _time_ to secure everything with tLS.

however, i'm having issues with a Go client.

2016/06/08 16:00:52 transport: http2Client.notifyError got notified that the client transport was broken EOF.

2016/06/08 16:00:53 grpc: Conn.resetTransport failed to create client transport: connection error: desc = "transport: write tcp 10.0.1.3:37334->10.3.247.89:8090: write: broken pipe"; Reconnecting to "dm:8090"

this is basically it. i just can't figure out if it's due to some weird TLS cert issue or incompatible Go client/Java server versions.

I'd expect that to happen if both sides weren't trying to use TLs

the server is running TLS for sure.

i just reconfirmed with openssl s_client -connect

New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-SHA

Server public key is 2048 bit

Secure Renegotiation IS supported

Compression: NONE

Expansion: NONE

No ALPN negotiated

SSL-Session:

Protocol : TLSv1.2

Cipher : ECDHE-RSA-AES256-SHA

sorry if i'm spamming.

Go code looks like:

creds, err := credentials.NewClientTLSFromFile(caFile, "")

opts = append(opts, grpc.WithTransportCredentials(creds))

either i'm doing something wrong here or i'm lost.

Oh, so you are using your own CA. That can lead to more problems, but I'd expect you're still doing it right.

You could try to confirm that ALPN can choose 'h2'. I don't remember the s_client argument to do that off the top of my head.

(It looks like it is "-alpn h2")

checking

openssl s_client -CAfile /etc/tls/ca.pem -tls1_2 -alpn h2 -connect localhost

:8090

it did return OK success

Did "No ALPN negotiated" change?

ejona: nop

Hmm... That could break stuff.

Are you using netty_tcnative or jetty_alpn on the Java side?

ejona: good question. i think netty but confirming.

ejona: i'm building the server with NettyServerBuilder so i assume the first option.

Hmm... you might be using the non-boringssl tcnative, which would use the installed openssl, and which commonly only supports NPN. But Go supports NPN so that should be fine.

It is a runtime dependency. It isn't in the code.

checking

ejona: what's the default? mvn dependency:tree only reveals netty deps

Have you taken a look at SECURITY.md?

btw, i'm running my apps in Alpine 3.4 so OpenSSL 1.0.2h 3 May 2016

reading

Okay. That's a very new OpenSSL, so it supports ALPN (doh, that makes sense, since s_client supported alpn)

:)

my client is built with Go 1.6.2 and is running with same OS.

Using jetty_alpn requires specifying a bootclasspath or a javaagent, so you would typically know if you're using it.

i'm not explicitly using Jetty anywhere.

So my theory is you need to use tcnative and then it all "just works"

But if that turns out to be the case, that raise two questions/bugs:

but openssl s_client can connect to the Java server successfully.

so what am i missing?

But s_client isn't negotiating HTTP/2 (h2) successfully. That is required for gRPC.

1) Why doesn't Java notice ALPN is missing. Maybe you're manually depending on jetty_alpn_api?

2) Why Go doesn't provide a helpful error message.

1 - i'm not explicitly using Jetty or anything related. not a dependency and surely not the appserver.

2 - no idea.

i'm going to try with tcnative

Maybe Jetty ALPN is getting included as a dependency to Spring Boot? (If so, that would probably be a bug)

checking.

What appserver are you using?

Tomcat

If it supports HTTP/2, it may be using Jetty ALPN (you don't have to be using Jetty)

Hmm...

but confirmed with mvn dependency:tree that no deps are imported for jetty.

Oh...

Actually, I just remembered we improved our code, so a stray dependency on jetty_alpn_api won't confuse grpc.

So that means if you look at the 'java' command used to start the server, you should be able to notice if jetty ALPN is being used.

again the question, should i upgrade gRPC on my Go code? since there are no releases, i can only tell you which commit i'm using.

If you see 'alpn' at all, it would be suspicious.

i don't specify any explicit jetty flags

$JAVA $JVM_OPTS -Djava.security.egd=file:/dev/./urandom -Djava.net.preferIPv4Stack=true -Dspring.profiles.active=$ENVIRONMENT -jar /my-app.jar

You can certainly try. But not negotiating h2 _would_ break things.

i'm trying with tcnative

It could be in JVM_OPTS

no, JVM_OPTS are just GC stuff. i set them explicitly as well. no jetty

Great.

I just confirmed that gRPC will prefer tcnative when it is available.

(tehehe, /dev/./urandom I know that trick. Silly Java)

yep, nailed me hard that one.

and DNS caching :gun:

(damn emoji!)

ejona: still no alpn negotiated.

:-/ Okay... Let me try here real quick.

ejona: with tcnative do i need openssl installed?

does Go app needs it too?

If using "netty-tcnative", yes. If using "netty-tcnative-boringssl-static", then no.

it seems i don't have openssl installed when apps boot.

i'm using boringssl-static

Go has its own native (and good!) TLS implementation, so there is no need for OpenSssl

Are you using a 64 bit JVM?

ejona: ejona yes, https://pkgs.alpinelinux.org/package/v3.4/commu....

OpenJDK Runtime Environment (build 1.8.0_92-internal-alpine-r0-b14)

OpenJDK 64-Bit Server VM (build 25.92-b14, mixed mode)

ppires: And you did use osdetector (or are using Fork16 or greater of tcnative)?

interestingly enough, my Go app no longer tries to reconnect now.

ejona: i'm using fork17 for linux.

(I'm still setting up the experiment. Yay for TLS... :( )

since i'm using MacOS for dev

but all of this is running on a Kubernetes cluster, Alpine 3.4 containers.

FYI: osdetector/specifying linux isn't strictly necessary now. https://github.com/grpc/grpc-java/pull/1904

(You still did the right thing, but it is slowly getting easier)

subscribed issue, thanks.

ejona: i tried setting a different flag -nextprotoneg h2

It's basically just a doc change now that newer versions of tcnative have come out, it works today with 0.14.

according to http://serverfault.com/questions/732474/nginx-c...

it should show a different output but no

Are you proxying through nginx?

no

there are some firewall rules for NAT but that's it (kube-proxy)

The Java server is terminating the TLS connection, right?

that's correct.

there's nothing between tcp 8090 and my java app.

I just got "ALPN protocol: h2"

(when specifying -alpn h2)

ppires: So I do think that s_client -alpn h2 is an acurate test.

ejona: ok thank you. let me try a few other things.