#terraform-tool

failshell: it works for us. Is the instance on a private subnet? If so you'll need to set up a NAT.

Also make sure your egress rules aren't blocking relevant ports (HTTP, HTTPS, and anything else your chef setup may need)

thegedge: the subnet has an associated gateway. also have an egress allowing all outbound ports

Do you explicitly specify the chef version? There used to be a bug where it would fail if you left the version attribute empty.

Not sure if that fix got into 0.6.0

the version's specified

Then I'm out of ideas :)

anyway, ive put my instances public for now and will control access with ingress rules for now

has anyone else seen an issue where terraform hangs while setting up provisioners?

when I run plan or refresh in the last day, it never finishes

kingryan: same

I'm rebooting my workstation to see if that's the issue

have you set GOMAXPROCS, and if so, what to?

i have not set it

i rebooted my machine yesterday and that didn’t help

when terraform hangs on me, there are several processes running and it seems that they’re all blocked on some file desciptor, but not sure which

perhaps a deadlock of some kind?

anyone have any times on how to debug this?

it appears they’re running select forever on nothing: select(0x0, 0x0, 0x0, 0x0, 0x208C55F38) = 0 0

I'm having a hard time debugging it

phinze: are you around to lend assistance?

Lefty: yo - what's up - terraform hanging?

TF_LOG=1 and no output on hang?

phinze: same for me

correct, kingryan is having the same issue

strace shows a lot of futexes in FUTEX_WAIT

it appears to stall during setup for provisioners and providers

and a ton of ETIMEDOUT

dtruss shows a lot of select polling (I’m on a mac)

can you open an issue and post debug logs?

so context -

I'll do that, kingryan if you can comment your findings

sure

will post the issue url in a sec

terraform kicks off a bunch of goroutines to handle the graph walk

i'd expect a deadlock to continuously output to logs

i recently added a few timers in there that are supposed to help with that

there's basically an "as wide as we can go" parallel graph walk happening

phinze: perhaps I made my graph too wide recently

whose actual parallelism is limited by GOMAXPROCs

i also just tried setting GOMAXPROCS to a ridiculously high number and that didn’t help

and there's an internal parallelism limit that's currently hardcoded

ah

though some work has been done in a PR to try and expose it as a tunable

not clear how a parallelism limit could cause no progress

I've tried with GOMAXPROCS unset, and GOMAXPROCS=16

should I try setting it higher?

interestingly kingryan started seeing this problem at approximately the same time yesterday

and our setups could not be any less related, heh

Lefty: yeah , makes me feel like one of the services is not responding

whee, bintray is slow today

downloading 0.5.3 to test with

phinze: only problem with 0.5.3 is that I run into 0.5.3 bugs with providers, so my catalog doesn't compile

I'm trying to rip out the 0.6 specific stuff now

so any hopes of ever getting this? https://github.com/hashicorp/terraform/issues/386

would be so nice to have a destroy hook :(

::crickets::

*tumbleweeds*

haha

thegedge: you were right earlier, we have no NAT in place in our VPCs right now

phinze: sorry to be annoying, but you have any idea about the issue above? i’m currently totally blocked on using terraform

kingryan: sorry for the sadness. working to get some eyeballs on it - we're spread sort of thin today

ok. thanks for all the help

failshell: you mentioned having a gateway though, I'm asuming an internet gateway?

thegedge: yeah. thought that would provide NAT'ing :) seems it doesn't. i find that silly. but meh hehe

It should, but you need to set up your route table to go through it

0.0.0.0/0 —> igw-id

that's what i have

Does the instance have a public IP?

that one didnt

If not, that'll be an issue too

Yeah, it needs a public ip for that to work :)

Otherwise, NAT

To talk to the internet, you need something with a public IP

We're working on HA NATs for our own stuff. I wish EC2 provided that.

i thought the internet gateway did that. even for private instances.

yeah. and VPN from VPC to VPC. or cross-region VPC peering

Yep!

i have to deploy VPNs next