~bsprague/k8s-restart

#K8s Restart

This is a simple Twilio webhook receiver that restarts K8s deployments based on received messages.

#Why?

Because recently the disks have been acting up on my server, taking down a number of services when I needed them, and didn't have quick access to the cluster. This extremely hacky solution allows me to manually reboot a service from my phone, using one of the least secure messaging protocols available (SMS).

#A Word of Caution

This server does verify the Twilio requests to the extent possible, and verifies the number came from a single allowed number, but this is still likely subject to SIM spoofing attacks if someone knows both your phone number and your Twilio number, so don't use this on any real production deployments.

#Usage

First, configure a service-mapping like:

{
  "svc1": {
    "namespace": "ns1",
    "deployment_name": "svc1-deployment"
  },
  "svc2": {
    "namespace": "ns2",
    "deployment_name": "svc2-deployment"
  },
  "svc3": {
    "namespace": "ns3",
    "deployment_name": "svc3-deployment"
  }
}

The JSON keys are the names you'll use when rebooting the service, by texting reboot: <svc name>.

Running the server looks something like:

./k8s-restart \
  --service_mapping=service-mapping.json \
  --allowed_number=<your phone number> \
  --twilio_auth_token=<your Twilio auth token> \
  --addr=:8080 \ # Run the webhook HTTP server :8080
  --in_cluster=false # Set to true this if k8s-restart will be running in the cluster it is actuating on. This controls where the K8s API config comes from.

This configuration can also be passed in via env vars, just capitalize the flag names.

#Host + Scheme Overrides

There are also --host_override and --scheme_override flags, which can be useful in environments where the HTTP request might be proxied and lose the original host information. The host/scheme in the webhook handler needs to match the URL entered in the Twilio console, because that info is encoded in the webhook signature. So if your webhook URL is https://mywebhook.example.com/, but your handler is proxied and just receives /, you'd want to set --host_override=mywebhook.example.com and --scheme_override=https.

Note that these overrides only take place if there's no host info in the request.

#Deploying

There's a Dockerfile in the repo, you can build an image with:

docker build -t <registry>/k8s-restart .

For deploying in a K8s cluster, you can use something like:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: k8s-restart
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: k8s-restart
rules:
- apiGroups: ["apps"]
  resources: ["deployments"]
  verbs:
  - get
  - list
  - patch
  - watch
- apiGroups: ["apps"]
  resources: ["deployments/scale"]
  verbs:
  - update
  - patch
- apiGroups: [""]
  resources: ["pods"]
  verbs:
  - list
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: k8s-restart
subjects:
- kind: ServiceAccount
  name: k8s-restart
roleRef:
  kind: Role
  name: k8s-restart
  apiGroup: rbac.authorization.k8s.io
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: k8s-restart-config
data:
  service-mapping.json: |
    {
      "svc1": {
        "namespace": "ns1",
        "deployment_name": "svc1-deployment"
      },
      "svc2": {
        "namespace": "ns2",
        "deployment_name": "svc2-deployment"
      },
      "svc3": {
        "namespace": "ns3",
        "deployment_name": "svc3-deployment"
      }
    }
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: k8s-restart-deployment
  labels:
    app: k8s-restart
spec:
  selector:
    matchLabels:
      app: k8s-restart
  # See https://kubernetes.io/docs/tasks/run-application/run-single-instance-stateful-application/
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: k8s-restart
    spec:
        serviceAccountName: k8s-restart
        containers:
        - image: <registry>/k8s-restart
          name: k8s-restart
          env:
          - name: HOST_OVERRIDE
            value: <your host>
          - name: SCHEME_OVERRIDE
            value: https
          - name: SERVICE_MAPPING
            value: /config/service-mapping.json
          - name: ALLOWED_NUMBER
            value: <your phone number>
          - name: ADDR
            value: ":8080"
          - name: IN_CLUSTER
            value: "true"
          - name: TWILIO_AUTH_TOKEN
            # This secret needs to exist
            valueFrom:
              secretKeyRef:
                name: twilio
                key: auth_token
          volumeMounts:
          - mountPath: /config
            name: config
          ports:
            - containerPort: 8080
              name: web
        volumes:
        - name: config
          configMap:
            name: k8s-restart-config
---
apiVersion: v1
kind: Service
metadata:
  name: k8s-restart
spec:
  selector:
    app: k8s-restart
  ports:
    - name: web
      protocol: TCP
      port: 8080
      targetPort: 8080

Make sure to set the namespace as appropriate.

#TODO

[ ] Investigate if watch is doing what we want here, it seems to return awfully fast.
[x] Use TwiML responses to let user know if we actually rebooted or not, as opposed to just guessing
[x] Consider waiting/polling for deployment to finish updating
[x] Add example K8s configuration for deploying this
- That should include a minimal Dockerfile/image for packaging this

refs