Deploying memcached in a StatefulSet with OpenShift

Over the past few months at Red Hat, I’ve been working with my team on streamlining our CI/CD process and migrating some of our applications into OpenShift. As we’ve been slowly moving apps, it’s been a great opportunity to revisit some of the basics of our architecture and look at ways we can better use OpenShift to . What may have worked well in a VM-based deployment doesn’t necessarily translate well into a container-based deployment. For the sake of this post, I’ll be showing how we use a recently stable feature of OpenShift (and Kubernetes) to deploy memcached for one of our Ruby apps on the Red Hat Customer Portal.

The Problem

In our VM-based deployment, we co-located an instance of memcached on each VM that ran our app (that’s the textbook use-case!). Our app was then configured to connect to each app server and shard keys across the memcached instances. This worked fine for us in an environment where hostnames didn’t change all-too-often.

If you’ve ever deployed an app in OpenShift or Kubernetes, you’ve probably noticed that whenever you rollout a new deployment you end up with a bunch of pods with funky hostnames like app-8-m6t5v. You probably never even cared, since resources like routes and services abstract us from needing to care about those hostnames. In an OpenShift deployment like this where instances of our app come and go fairly often, it’s not feasible to configure our app to connect to memcached instances by pod name.

Potential Solutions

One Big Pod

Memcached was designed to take advantage of unused memory resources distributed across web servers. We don’t quite have that same problem now that OpenShift will . Instead of using, say, (8) 256MB instances of memcached, we can use just use one 2GB instance, right?

It's a trap!

Well, you could, but having only one replica of anything is usually a bad idea. In our environment, our nodes get rebuilt at-minimum once a week due to things like updates and autoscaling. If our pod is on a node getting rebuilt, there will at least be a minute or two where it will be unavailable while it’s being rescheduled on a new node. Losing all of our cache each time that happens would be less than optimal. While most objects aren’t cached very long and our app gracefully handles cache being unavailable, it’s still not super great for performance or our backend services. Let’s see if we can find a better way to deploy this while still keeping it distributed.

Creating multiple services and deployments

One way to approach this would be to create a deployment and service for each instance of memcached we want. A deployment ensures that we have a pod running and a service provides a static hostname we can use for accessing the pod. We would ultimately need to create a deployment and service for each instance of memcached (e.g. memcached-a, memcached-b, memcached-c, etc).

This would work fine,  but it causes some management overhead since we have to configure each individual instance instead of configuring one resource to define them all.

StatefulSets

Building on top of the previous approach, a newer OpenShift feature called StatefulSets allows us to use a single controller to manage all of our memcached pods. The Kubernetes docs give a good overview of all the things you can do with them, but since memcached doesn’t need any persistent storage or have any dependencies on which pods come up first, we’ll mainly take advantage of the stable pod identities provided by StatefulSets.

The StatefulSet resource will allow us to specify a desired number of replicas, and create those pods with stable names (e.g. memcached-0, memcached-1, memcached-3, etc). Whenever a pod is terminated, a new one will replace it with the same pod name. This means we can configure our app with a list of memcached shards and expect it work even when pods come and go.

Let’s Deploy It

Before we get started, I’m going to assume to you’re logged into an OpenShift or Kubernetes cluster and have the oc or kubectl CLI tools installed. Though I’ll probably mention OpenShift more often, all the concepts in this article will translate over to a vanilla Kubernetes cluster as well.

Create the service

Since our apps shard keys across all memcached instances in the cluster, the typical load-balanced service isn’t all that useful. Instead, we’ll create a headless service where no load balancing or proxying takes place. This service simply allows endpoints to be looked up via DNS or via the API.

cat <<EOF | oc apply -f -
apiVersion: v1
kind: Service
metadata:
  labels:
    component: memcached
  name: memcached
spec:
  type: ClusterIP
  clusterIP: None
  selector:
    component: memcached
  ports:
    - name: memcached
      port: 11211
      protocol: TCP
      targetPort: 11211
EOF
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Service
metadata:
  labels:
    component: memcached
  name: memcached
spec:
  type: ClusterIP
  clusterIP: None
  selector:
    component: memcached
  ports:
    - name: memcached
      port: 11211
      protocol: TCP
      targetPort: 11211
EOF

Create the StatefulSet

Now that we have the service in place, let’s spin up some memcached instances. For this example, lets create 3 shards/replicas with 64MB of memory each.

Since we don’t care about what order pods spin up, we’re also specifying the parallel podManagementPolicy. This still ensures our pods get their unique names, but doesn’t limit us to spinning up one pod at a time.

By specifying serviceName here, we also get a unique hostname for each pod based on the pod name and service name (e.g. memcached-0.memcached.default.svc.cluster.local, memcached-1.memcached.default.svc.cluster.local, etc)

cat <<EOF | oc apply -f -
apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
  labels:
    component: memcached
  name: memcached
spec:
  replicas: 3
  revisionHistoryLimit: 5
  selector:
    matchLabels:
      component: memcached
  serviceName: memcached
  podManagementPolicy: Parallel
  template:
    metadata:
      labels:
        component: memcached
    spec:
      containers:
        - name: memcached
          args: ["memcached", "-m", "64"]
          image: memcached:1.5
          ports:
            - containerPort: 11211
              name: memcached
              protocol: TCP
          livenessProbe:
            tcpSocket:
              port: memcached
          readinessProbe:
            tcpSocket:
              port: memcached
          resources:
            limits:
              cpu: "1"
              memory: 96Mi
            requests:
              memory: 64Mi
EOF
cat <<EOF | kubectl apply -f -
apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
  labels:
    component: memcached
  name: memcached
spec:
  replicas: 3
  revisionHistoryLimit: 5
  selector:
    matchLabels:
      component: memcached
  serviceName: memcached
  podManagementPolicy: Parallel
  template:
    metadata:
      labels:
        component: memcached
    spec:
      containers:
        - name: memcached
          args: ["memcached", "-m", "64"]
          image: memcached:1.5
          ports:
            - containerPort: 11211
              name: memcached
              protocol: TCP
          livenessProbe:
            tcpSocket:
              port: memcached
          readinessProbe:
            tcpSocket:
              port: memcached
          resources:
            limits:
              cpu: '1'
              memory: 96Mi
            requests:
              memory: 64Mi
EOF

At this point, we should have a ready-to-go memcached cluster. Let’s check things out and make sure we see the pods and that they’re discoverable. Feel free to skip ahead.

# Confirm we see 3 memcached pods running
$ oc get po -l component=memcached
NAME READY STATUS RESTARTS AGE
memcached-0 1/1 Running 0 1d
memcached-1 1/1 Running 0 1d
memcached-2 1/1 Running 0 1d


# Let's spin up a pod with a container running digso we can confirm DNS entries
$ oc run net-utils --restart=Never --image=patrickeasters/net-utils
pod "net-utils" created

# The service hostname should return with a list of pods
# Don't forget to replace default with the name of your OpenShift project
$ oc exec net-utils dig +short memcached.default.svc.cluster.local
10.244.0.17
10.244.0.15
10.244.0.16

# Pod IPs can still change, so it's best to configure any apps with hostnames instead
# Looking up SRV records for the service hostname should give us the pod FQDNs
$ oc exec net-utils dig +short srv memcached.default.svc.cluster.local
10 33 0 memcached-2.memcached.default.svc.cluster.local.
10 33 0 memcached-1.memcached.default.svc.cluster.local.
10 33 0 memcached-0.memcached.default.svc.cluster.local.

# Let's clean up this pod now that we're done with our validation
# (Or you can keep running queries... that's cool too)
$ oc delete po net-utils
pod "net-utils" deleted
# Confirm we see 3 memcached pods running
$ kubectl get po -l component=memcached
NAME READY STATUS RESTARTS AGE
memcached-0 1/1 Running 0 1d
memcached-1 1/1 Running 0 1d
memcached-2 1/1 Running 0 1d


# Let's spin up a pod with a container running digso we can confirm DNS entries
$ kubectl run net-utils --restart=Never --image=patrickeasters/net-utils
pod "net-utils" created

# The service hostname should return with a list of pods
# Don't forget to replace default with the name of your Kubernetes namespace
$ kubectl exec net-utils dig +short memcached.default.svc.cluster.local
10.244.0.17
10.244.0.15
10.244.0.16

# Pod IPs can still change, so it's best to configure any apps with hostnames instead
# Looking up SRV records for the service hostname should give us the pod FQDNs
$ kubectl exec net-utils dig +short srv memcached.default.svc.cluster.local
10 33 0 memcached-2.memcached.default.svc.cluster.local.
10 33 0 memcached-1.memcached.default.svc.cluster.local.
10 33 0 memcached-0.memcached.default.svc.cluster.local.

# Let's clean up this pod now that we're done with our validation
# (Or you can keep running queries... that's cool too)
$ kubectl delete po net-utils
pod "net-utils" deleted

Connecting Our App

Now it’s time to point our app to our memcached cluster. Instead of configuring a static list of pods, we’ll take advantage of the built-in DNS service discovery. All we have to do is provide the hostname of our service, and our app can discover all the pods in the memcached cluster on its own.

If you speak Ruby (or at least pretend to, like me), feel free to glean from the below example from one my team’s apps.

memcached_hosts = []
Resolv::DNS.new.each_resource('memcached.default.svc.cluster.local', Resolv::DNS::Resource::IN::SRV) { |rr|
  memcached_hosts << rr.target.to_s
}
config.cache_store = :dalli_store, memcached_hosts

Closing Thoughts

Hopefully now you have a working memcached cluster and are on your way to configuring your app to take advantage of some of service discovery greatness we get for free from OpenShift and Kubernetes. Let me know in the comments or on Twitter if you were able to try this out for yourself. Happy caching!

Integrating existing home security sensors with MQTT

When my wife and I bought a house a couple years back, I knew it would only be a matter of time before I started getting into home automation. My house, like many built in the late 90s, was pre-wired for an alarm system. While I had no desire to revive a 20-year-old alarm panel, it did mean all my exterior doors were pre-wired with inconspicuous sensors.

I already run Home Assistant on a Raspberry Pi, so I was looking for a way to integrate these hard-wired door sensors with what I already have. I had read about these cheap WiFi-enabled ESP8266 boards, so I decided this would be a simple project to try it out with.

Breaking into the existing alarm system

The brains of my old alarm system were tucked away in a wall-mounted cabinet in my laundry room.

The old alarm system (with bonus lead-acid battery that I need to get rid of)

All the sensor wires were cut back, meaning I had to strip each one and trace what they did. It was decently quick work with my multimeter and a wife to open/close doors for me. Since the existing sensors were just simple reed switches, the two wires from each door would short when the door was closed, and open when the door opened.

I guessed correctly that the small 2-conductor wires went to my three exterior doors, but honestly, I’m still not sure where the rest of them go.

While I typically don’t use wire nuts in my projects, it made the most sense here since I was dealing with pre-existing structured wiring from the old security system.

Making it work on a breadboard

Now the fun part: building it. This is a super simple circuit, with just a microcontroller and 3 switches. I used 3 of the GPIO pins to connect to the 3 door switches and connected the other side of the switches to ground. One nice feature of this board was that most of the pins had a built-in weak pull-up resistor. This meant that when a switch was open, the pin would be pulled high (3.3V).

The components labeled S1-S3 represent the reed switches already installed in the doors. When a door opens, the magnet in the door causes the switch to open (or disconnect).

Adding some code

All of my source code is on GitHub, so feel free to check out that repo and see how it works there. This is my first time using Lua, so I’m sure there are things that can be optimized in this code. Feel free to submit an issue or PR if you find anything that stands out!

  • init.lua: the simple startup script. Best practice is to keep this file relatively simple and give yourself an opportunity to break out of the boot cycle should your app have an issue.
  • secrets.lua: secret variables such as WiFi credentials
  • config.lua: just some simple configuration variables
  • sensor.lua: the brains of the project. All of the real logic lives in this file.

I won’t get too detailed here, but after establishing the initial WiFi and MQTT connections, the application code is triggered asynchronously by an interrupt when a change on one of the door sensors is detected.

Preparing the NodeMCU board

Before I could run my code on the NodeMCU board, I needed to flash it with updated firmware that contained the modules I needed. I used this handy Cloud Build Service to build a firmware image. For my needs, I simply needed the following modules: file, gpio, mqtt, net, node, tmr, uart, wifi.

To actually flash the firmware, I used an open-source Python tool called nodemcu-pyflasher. It worked great on my Mac and should on most other platforms as well.

Now that I was ready to upload my code, I used a tool called ESPlorer to handle that portion. It provides a pretty handy console that lets you run commands ad-hoc as well as edit your code. I personally found it easiest to just use my usual text editor and the “Upload” button in the left pane.

Integrating with Home Assistant

Like I mentioned before, my home automation platform of choice is an open-source app called Home Assistant. It has a strong community backing and the maintainers are constantly pushing new releases.

While I could have used the built-in REST API, I opted to use the MQTT protocol for a couple of reasons: first, MQTT is super lightweight and perfect for the modest compute resources of the NodeMCU board; and second, it’s a more universal protocol that can be used outside of the Home Assistant ecosystem without changes.

If you use Home Assistant, I configured my door sensors as binary_sensors. The snippet for one sensor is below, but you can look at the rest of them in the Github repo for my config.

- platform: mqtt
  state_topic: "pat/alarm/Front Door"
  name: "Front Door"
  payload_on: "open"
  payload_off: "closed"
  device_class: opening

Wrapping it up

Thanks for reading this far! Hopefully this project inspired you to build something of your own or saved you a bit of time as you tackle something similar. I’m happy to clarify anything or answer questions, so feel free to reach out here or on GitHub.

Resources

NodeMCU Documentation (super thorough and helpful)
Project Code
Home Assistant Config

Using Traefik with TLS on Kubernetes

Over the past few months, I’ve been working with Kubernetes a lot as Ayetier has been making the shift towards container orchestration. As easy as it was to create and scale services, it was a bit frustrating to see how most reverse proxy solutions seemed kludgy at best.

That’s why I was pretty intrigued when I first read about Traefik — a modern reverse proxy supporting dynamic configuration from several orchestration and service discovery backends, including Kubernetes.

Traefik is still relatively new and doesn’t fully support Kubernetes’ TLS configuration for the ingress, so it took a bit of manual configuration. Much trial-and-error was involved, so I thought I’d share the process here.

First Things First

I’m going to assume you already have a working Kubernetes cluster and have the kubectl tool installed to manage your cluster. I’m also on Google Container Engine (GKE), so depending on your cloud provider, a few things like LoadBalancer service types may be different.

Any code I use is also in this GitHub repo, so feel free to clone it and follow along. (It’s more fun that way)

git clone https://github.com/patrickeasters/traefik-k8s-tls-example.git

Deploy Backend Services

For the purposes of this post, I made a pretty simple web service in Go that will aid in testing. You could also make your own service to display cat facts if you really want. Let’s go ahead and deploy 3 replication controllers and services from the backend.yaml file.

kubectl create -f backend.yaml

Secure All The Things

We’re just going to generate a self-signed certificate for this tutorial, but any certificate/key pair will work. Run the following command to generate your certificate and dump the certificate and private key.

openssl req 
        -newkey rsa:2048 -nodes -keyout tls.key 
        -x509 -days 365 -out tls.crt

Now that we have the certificate, we’ll use kubectl to store it as a secret. We’ll use this so our pods running Traefik can access it.

kubectl create secret generic traefik-cert 
        --from-file=tls.crt 
        --from-file=tls.key

Configure Traefik

As I mentioned earlier, due to lack of native support for TLS with Kubernetes ingresses, we’ll have to do a bit of manual configuration on the pods running Traefik.The only real points of interest here are setting up the HTTP to HTTPS redirect and then setting the certificate to be used for TLS.

# traefik.toml
defaultEntryPoints = ["http","https"]
[entryPoints]
  [entryPoints.http]
  address = ":80"
    [entryPoints.http.redirect]
      entryPoint = "https"
  [entryPoints.https]
  address = ":443"
    [entryPoints.https.tls]
      [[entryPoints.https.tls.certificates]]
      CertFile = "/ssl/tls.crt"
      KeyFile = "/ssl/tls.key"

Now let’s take this configuration and store it in a ConfigMap to be mounted as a volume in the Traefik pods.

kubectl create configmap traefik-conf --from-file=traefik.toml

Deploy Traefik

Now we finally get to deploy the replication controller for Traefik. I’m going to use 2 pods, but this can be scaled out as desired. I’m also using a LoadBalancer service, but you can change it to NodePort and configure your external load balancer as required if your cloud provider doesn’t natively support it.

Here are a few things to note in the pod spec from traefik.yaml, which contains the RC and service.

  • The traefik-cert secret is mounted as a volume to /ssl, which allows the tls.crt and tls.key files to be read by the pod
  • The traefik-conf ConfigMap is mounted as a volume to /config, which lets Traefik read the traefik.conf file
  • The log level is set to debug, which is great when you’re troubleshooting or getting started, but it may be more manageable if you set it to something less chatty before going into production with it.
spec:
      terminationGracePeriodSeconds: 60
      volumes:
      - name: ssl
        secret:
          secretName: traefik-cert
      - name: config
        configMap:
          name: traefik-conf
      containers:
      - image: traefik
        name: traefik-ingress-lb
        imagePullPolicy: Always
        volumeMounts:
        - mountPath: "/ssl"
          name: "ssl"
        - mountPath: "/config"
          name: "config"
        ports:
        - containerPort: 80
        - containerPort: 443
        args:
        - --configfile=/config/traefik.toml
        - --kubernetes
        - --logLevel=DEBUG

Now let’s go ahead and create this RC and service in the cluster.

kubectl create -f traefik.yaml

Configuring the Ingress

Now that Traefik is up and running, we need to configure an ingress so it has actual rules. We’re going to set up 2 route prefixes, /s1 and /s2, pointing to svc1 and svc2 respectively. Anything else will be sent to svc3. The pods running Traefik are watching the API for any changes made to the ingress configuration

A note for any GKE users: To prevent the default L7 load balancer ingress controller from picking up this configuration, I set the kubernetes.io/ingress.class annotation to traefik. Google’s ingress controller will ignore any ingresses whose class is not set to gcp.

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: example-web-app
  annotations:
    kubernetes.io/ingress.class: "traefik"
spec:
  tls:
    - secretName: traefik-cert
  rules:
  - host:
    http:
      paths:
      - path: /s1
        backend:
          serviceName: svc1
          servicePort: 8080
      - path: /s2
        backend:
          serviceName: svc2
          servicePort: 8080
      - path: /
        backend:
          serviceName: svc3
          servicePort: 8080

Behold, our last configuration.

kubectl create -f ingress.yaml

Testing It Out

At this point, you can test your routes and see your shiny new config doing its magic. In my output below, you can see each service identify itself.

$ curl -k https://myhost/s1

Hi, I'm the svc1 service!
Hostname: svc1-on0mm

$ curl -k https://myhost/s2

Hi, I'm the svc2 service!
Hostname: svc2-i485q

$ curl -k https://myhost/

Hi, I'm the svc3 service!
Hostname: svc3-iict9

$ curl -k https://myhost/lolcat

Hi, I'm the svc3 service!
Hostname: svc3-iict9

Final Thoughts

Hopefully at this point you have a working Traefik reverse proxy setup. Hit me up in the comments or on Twitter if you have any questions.