~damien/infrastructure

d8565675a1c566d18b5f70509fc937a5695081c4 — Damien Radtke 2 years ago 0adaceb
Testing out running jobs on the new setup
M ca/README.md => ca/README.md +3 -13
@@ 1,14 1,4 @@
This repo demonstrates a simple approach for provisioning servers with
certificates signed by a custom certificate authority.
This folder contains CA files, including their private keys.
They are used at server provisiong time to create certificates for the new servers.

In order to test it, you need to define a `linode_token` variable inside
`secrets.tfvars`, and then run:

```sh
$ ./ca-new consul-agent
$ terraform apply -var-file secrets.tfvars
```

This will provision a server with
`/etc/ssl/consul-agent/server1.dc1.consul.{crt,key}`, which represents a
certificate and key that have been signed by the local `consul-agent` CA.
See the `terraform` folder for invocations of `provision-cert`.

M ca/consul-agent-ca.srl => ca/consul-agent-ca.srl +1 -1
@@ 1,1 1,1 @@
2E741F16F5701C92061B3669C7546E4A1AA2C89A
2E741F16F5701C92061B3669C7546E4A1AA2C8A4

M ca/nomad-agent-ca.srl => ca/nomad-agent-ca.srl +1 -1
@@ 1,1 1,1 @@
3A95C8A7EE198C7858E58366940C5F258426067E
3A95C8A7EE198C7858E58366940C5F258426068C

M ca/provision-cert => ca/provision-cert +3 -3
@@ 38,7 38,7 @@ cn="${hostname:-${basename}}"
# Subject Alt Name is what actually holds the list of valid DNS addresses.
# We always include localhost so that commands can be run locally, and add
# the requested hostname if it was provided.
san="DNS:localhost"
san="IP:${addr},DNS:localhost"
if [[ -n "${hostname:-}" ]]; then
	san="${san},DNS:${hostname}"
fi


@@ 50,7 50,7 @@ openssl req -new -newkey rsa:2048 -nodes -keyout '${basename}.key' -out '${basen
EOF

# Copy the CSR down for cert generation
scp root@"${addr}":"${outdir}/${basename}.csr" "${tmp}/"
scp root@"[${addr}]":"${outdir}/${basename}.csr" "${tmp}/"

# Sign the CSR to get a certificate
if [[ -f "${ca}-ca.srl" ]]; then


@@ 70,7 70,7 @@ openssl x509 -req \
# TODO: add key usages? https://learn.hashicorp.com/tutorials/nomad/security-enable-tls

# Copy the certificate back up
scp "${tmp}/${basename}.crt" root@"${addr}":"${outdir}/${basename}.crt"
scp "${tmp}/${basename}.crt" root@"[${addr}]":"${outdir}/${basename}.crt"

# Clean up the remote CSR
ssh root@"${addr}" "rm ${outdir}/${basename}.csr"

M ca/vault-server-ca.srl => ca/vault-server-ca.srl +1 -1
@@ 1,1 1,1 @@
73BA2463646820941EB0EC0FDCDC6DE86EE14245
73BA2463646820941EB0EC0FDCDC6DE86EE1424F

M jobs/README.md => jobs/README.md +4 -7
@@ 2,20 2,17 @@ This folder contains Nomad job definitions.

## Running (new way)

First, make sure your `.bashrc` is set up correctly with something like
First, make sure you're able to interact with the Nomad cluster:

```bash
export NOMAD_ADDR="https://[2600:3c04::f03c:92ff:fed4:f455]:4646"
export NOMAD_CACERT="/etc/ssl/nomad/ca.pem"
export NOMAD_CLIENT_CERT="${HOME}/nomad-cli.pem"
export NOMAD_CLIENT_KEY="${HOME}/nomad-cli-key.pem"
```sh
$ source terraform/env
```

Then you can submit jobs with the `nomad` command, or use `nomad-compile` from
the tools directory if it requires substiution of secret values:

```bash
$ nomad-compile job.nomad.erb | nomad job run -
$ tools/nomad-compile jobs/job.nomad.erb | nomad job run -
```

## Running (old way)

M jobs/damienradtkecom.nomad.erb => jobs/damienradtkecom.nomad.erb +1 -1
@@ 10,7 10,7 @@ job "damienradtkecom" {
		constraint {
			attribute = "${node.class}"
			operator  = "!="
			value     = "load-balancer"
			value     = "ingress"
		}

		update {

A packer/.gitignore => packer/.gitignore +1 -0
@@ 0,0 1,1 @@
*.pkrvars.hcl

M packer/image.pkr.hcl => packer/image.pkr.hcl +5 -11
@@ 10,7 10,6 @@ packer {

variable "linode_token" {
  type    = string
  default = "443c7fe73bb3179bc748b580f04e0b4b7e00d78c7df74eb84690cc322b9db08d"
}

variable "opensuse_version" {


@@ 20,25 19,21 @@ variable "opensuse_version" {

variable "consul_version" {
  type    = string
  default = "1.10.0"
  default = "1.10.3"
}

variable "nomad_version" {
  type    = string
  default = "1.1.6"
  default = "1.2.0"
}

variable "vault_version" {
  type    = string
  default = "1.7.3"
}

variable "cfssl_version" {
  type = string
  default = "1.6.0"
  default = "1.8.5"
}

locals {
  packages = ["wget", "jq", "firewalld", "moreutils", "git"]
  timestamp = regex_replace(timestamp(), "[- TZ:]", "")
}



@@ 49,7 44,6 @@ openSUSE Leap ${var.opensuse_version}
Consul ${var.consul_version}
Nomad ${var.nomad_version}
Vault ${var.vault_version}
CFSSL ${var.cfssl_version}
EOF
  image_label       = "cluster-image-${local.timestamp}"
  instance_label    = "cluster-imaging-${local.timestamp}"


@@ 65,7 59,7 @@ build {

  provisioner "shell" {
    inline = [
      "zypper --non-interactive install wget jq firewalld moreutils",
      "zypper --non-interactive install ${join(" ", local.packages)}",
      "zypper --non-interactive clean",
      "echo Updating CA certificates",
      "update-ca-certificates --verbose",

M terraform/README.md => terraform/README.md +5 -5
@@ 29,16 29,16 @@ The root token will be needed to run cluster upgrades.

## Upgrading the Cluster

TODO: these steps still need to be tested

Copy the `cluster` module block into a new cluster instance, `cluster-new`. Make the required upgrade changes, including any additional outputs as needed, and then run:
First, create a copy of the module you want to upgrade. For example, to upgrade Consul servers, you could copy `consul-server` as `consul-server-new`.

```sh
$ terraform init
$ terraform apply -var vault_token=<root_token>
$ terraform apply # For Nomad servers, add "-var vault_token=<root_token>"
```

Once the new cluster is up and running, run these commands on the new relevant nodes:
TODO: specify instructions for each server type

Once the new servers are up and running, run these commands on the new relevant nodes:

```sh
# On each new Consul server

A terraform/env => terraform/env +24 -0
@@ 0,0 1,24 @@
#!/usr/bin/env bash
#
# This script sets up the environment for communication with the currently-active cluster.
# It is expected to be run from the root of the repository.
#
#  $ Usage: source terraform/env
#

export NOMAD_ADDR="https://[$(cd terraform && terraform output -json nomad-servers | jq -r '.[0].ip')]:4646"
export NOMAD_CACERT="ca/nomad-agent-ca.crt"
export NOMAD_CLIENT_CERT="ca/nomad-agent-ca.crt"
export NOMAD_CLIENT_KEY="ca/nomad-agent-ca.key"

# TODO: these still need to be verified

export CONSUL_ADDR="https://[$(cd terraform && terraform output -json consul-servers | jq -r '.[0].ip')]:8501"
export CONSUL_CACERT="ca/consul-agent-ca.crt"
export CONSUL_CLIENT_CERT="ca/consul-agent-ca.crt"
export CONSUL_CLIENT_KEY="ca/consul-agent-ca.key"

export VAULT_ADDR="https://[$(cd terraform && terraform output -json vault-servers | jq -r '.[0].ip')]:8200"
export VAULT_CACERT="ca/vault-server-ca.crt"
export VAULT_CLIENT_CERT="ca/vault-server-ca.crt"
export VAULT_CLIENT_KEY="ca/vault-server-ca.key"

M terraform/main.tf => terraform/main.tf +8 -6
@@ 9,8 9,7 @@ terraform {

locals {
  datacenter       = "ca-central"
  image            = "private/13315378"
  // image            = "private/14625896"
  image            = "private/15166213"
  authorized_users = [data.linode_profile.me.username]
  instance_type    = "g6-nanode-1"
}


@@ 79,18 78,16 @@ module "nomad-client" {
  authorized_users = local.authorized_users
}

// TODO: group by class

output "nomad-clients" {
  description = "Nomad clients"
  value       = module.nomad-client.instances
}

module "nomad-client-load-balancer" {
module "nomad-client-ingress" {
  source = "./nomad-client"

  clients           = 1
  node_class        = "load-balancer"
  node_class        = "ingress"
  consul_server_ips = module.consul-server.instances[*].ip

  datacenter       = local.datacenter


@@ 99,6 96,11 @@ module "nomad-client-load-balancer" {
  authorized_users = local.authorized_users
}

output "nomad-clients-ingress" {
  description = "Nomad ingress clients"
  value       = module.nomad-client-ingress.instances
}

module "vault-server" {
  source = "./vault-server"


A terraform/ssh => terraform/ssh +1 -0
@@ 0,0 1,1 @@
#!/usr/bin/env bash