#terraform-tool

ah, the challenges of scale.

LinuxNinja: There is also https://github.com/hashicorp/best-practices/

I wouldn't want an outage in one AZ or a single NAT instance going down to take out my whole VPC.

i'm working on figuring out how to get terraform to do my Route53 records, but it doesn't know how to do latency records. So, having TF do the regions records with health checks, and I'm having to add the latency records manually.

dumb question. Do you need to pass the aws creds to every module you pull in, or is it just the main terraform code that needs that information ?

i.e. if I define vars aws_access_key and aws_secret_key, do those need to be passed through everywhere?

I'm still using my environment to specify my AWS crews, so haven't experienced that.

creds not crews - autocorrect...

Oh, okay.

environment vars are probably the way to go...

I have bash aliases that switch me between AWS accounts and chef servers.

I've noticed issues specifying those as command-line arg vars using -var= and switching between accounts.

we don't run terraform apply locally.

out github commits do a webhook back to a jenkins instance that runs a plan on each of our config directories before it will allow us to do a terraform apply. if the plan fails, you have to fix it before you can commit.

Nice!

yea, that sounds like a nice setup

so, all of our changes are pull-requsts to github

since there are so many of us working in terraform, it prevents us from screwing up each others' terraform.tfstate files.

those are commited back to github by jenkins after an apply.

LinuxNinja: Why not use shared remote state?

we need to have the tfstate file local to do plans, so have it in our git repo.

i've asked our guys to come up with a way to do remote state and solve that.

will likely look at moving it to S3

Why is that? If it's set up for remote state, terraform will query S3, or consul, or atlas, or where ever your state is stored.

we don't have consul or atlas. will likely move it to S3.

I understand that. I'm asking "Why do you need local files to run plan?"

If you enable remote state, terraform will look to wherever you have stored that state data.

that way each group can use the shared states if they need to.

don't know what version that was added.

but it didn't work for us when we first set this up.

It's been there for as long as I've been using tf, which is about 6 months.

I wasn't on the team who was evaluating it. We had been experimenting with it for months and had to wait for some bug fixes before we could start using it. I'm still running into bugs with it.

We are looking to use a utility script similar to Pll's runTf.py from the linked repo as well to manage the coordination of state files via S3 buckets and wrapping the remote execution

One additional thing we were discussing is implementing a lock file in the script for the state file bucket to ensure we don't trample each other

The githook sounds interesting as well

jrnt30: my recommendation re the remote state file issue is: Don't manage symlinks unless you want locally version-controlled state files.

Not sure I digested that

Just delete the terraform.tfstate file before each run and copy the correct one down from s3 each time.

right

Want to ensure that mutliple operators couldn't try and mutate that state file simultaneously

my runTF.py manages symlinks.

as even a Plan seems to do that

I don't think that's wise.

which part?

managing symlinks .

I think with remote state, just blowing away the local cache file is best, and have the tf wrapper grab the latest version from S3 directly.

Gotcha, I looked a bit closer at your script and hadn't gotten that far. Still working out the strategy. My expectation would be that the wrapper script would be responsible for 1 - checking the structured s3 bucket to ensure the lockfile doesn't exist

if not, 2 - adding in a lockfile

3 - executing the remote config for the stack

4 - execute

@jrnt30 you'll run into a race condition if you don't put a unique id in the lockfile to ensure it is yours, then read it back after a delay to make sure someone else also isn't writing to the lockfile.

My script was written in haste. I was actively trying to work around terraform's inherent flaws, not really considering the best way to implement a wrapper strategically.

yeah, that is true. It's not a truely atomic "check and set"

my issue with S3 is the delay you run into between the put and get that makes you real unsure if your file is there or not. and with lockfiles, it gets worse.

But it's "a" saftey check

Have either of you looked at the custom HTTP endpoint option

I was going to spin up a server to dump out the information that's sent

Could potentially create a small util server to do something similar to the new Atlas Lock feature

for state as well

we have a separate team who does that type of planning and implementation.

just haven't looked into it

LinuxNinja: yes, that delay, and the "eventually consistent" nature of S3 is a problem. We're looking at consul to replace S3, that should be much better.

we had taken inspiration from https://github.com/hashicorp/terraform/issues/3... around the general structure but hadn't really though through some of the latency issues with S3 to be honest

I'd like a 'token checkout' type system. You ask 'mother may I' to a central auth, and you get back a 'yes you may'.

Consul does have that capability of exclusive locking for keys

then it manages the state file and prevents two people from doing apply changes at the same time.

i just got a failure of our jenkins system to commit the state back to git. now i'm bork'd until it's fixed.....

Yeah, we don't have a "globally available" consul cluster for our mutli-regional deployment currently so was looking for an easier way

and this is due to certain types of changes to r53 resource records not being done properly by tf.

The S3 seemed easiest with the custom HTTP implementation another possibility. Talking through it though, it would probably be easier to simply setup Consul

I had tested changes to our .tf in dev, and then moved the change to qa. i had incrementally made changes in dev until I got the result I wanted. moving it to qa, all I get is that my aws_route53_record command is bad.

well, it is still version < 1

1) can someone take a quick look at my terraform aurora output see if everything looks good? https://paste.ee/p/HlzQz are there any aurora settings i should set instead of using default/leaving empty. 2) what is replication role (writer) vs reader? i am guessing thats like pointing all reads to use reader? 3) what does the "count =" for? isnt 1 instance is enough?

question about consul, do you run a consul cluster per environment (prod/staging/qa) or a single consul cluster

does anyone else get issues with the azure plugin while trying to build terraform from master?

is it possible to only create a resource if a property is not set? for example, I want to be able to specify an elastic IP as an envar, and if that envar isn't set, generate one using aws_eip