Achieving Mutual TLS

Ted Hahn Ted Hahn, TCB Technologies, Inc.

Mark Hahn Mark Hahn, Ciber Global

tcbtech.com/kubetls

Achieving Mutual TLS

Secure Pod-to-Pod communication without the hassle

Every Kubernetes pod should include a SSL Certificate, verifying its
identity. This should be signed by the kubernetes master, and be
specific to each pod.

Certificate Lifecycle

  • Create a private key - public key pair
  • Put the public key in a Certificate Signing Request (CSR)
  • Certificate Authority(CA) signs the CSR to create a Certificate
    • I.e. The CA computes a hash of the CRS's public key using the CA's private key
  • The CA returns the Certifcate to the requester

A client without prior knowledge of the server but with prior trust in the Certificate Authority, can communicate with the Server.

Manual Creation of Certificates

cfssl

# Create a Certificate Authority
cd ca
cfssl genkey -initca ca-csr.json  | cfssljson -bare

# Create and then sign the server certificate
cfssl genkey server-csr.json | cfssljson -bare
cfssl sign -ca ../ca/cert.pem -ca-key ../ca/cert-key.pem cert.csr  | cfssljson -bare
openssl verify -CAfile ../ca/cert.pem cert.pem

# Create and then sign the client certificate
cd ../client/
cfssl genkey client-csr.json | cfssljson -bare
cfssl sign -ca ../ca/cert.pem -ca-key ../ca/cert-key.pem cert.csr  | cfssljson -bare

Manual Creation of Certificates

openssl

# create openssl.cnf, tweak details
rm index.txt
touch index.txt
echo 1000 > serial

openssl ecparam -name prime256v1 -genkey > private/ca.key.pem
openssl req -config openssl.cnf -key private/ca.key.pem -new -x509 -extensions v3_ca -out certs/ca.cert.pem -subj "/CN=CA"
openssl x509 -in certs/ca.cert.pem -text

openssl ecparam -name prime256v1 -genkey > dev-key.pem
openssl req -new -key dev-key.pem -out dev.csr -config req.conf
openssl ca -config openssl.cnf -extensions server_cert -in dev.csr -batch -out dev-cert.pem

How are Certificates used

Developers need to include code in thier programs/microservices to do the configuration and initialization to serve requests through TLS.

Golang Applications without TLS

var (
        listenAddr = flag.String("listenAddr", ":8080", "Address to listen on")
)

func main() {
	flag.Parse()
	goApi := goApi.NewGoApi()
	http.HandleFunc("/go/api", goApi.endPoint)

	http.HandleFunc("/webhook", sslmutator.AcceptWebhook)
	log.Printf("Listening on %s", *listenAddr)
	log.Fatal(http.ListenAndServe(*listenAddr, nil))
}

Golang Applications using TLS

var (
        listenAddr = flag.String("listenAddr", ":8443", "Address to listen on")
+       certFile   = flag.String("certFile", "/var/tls/tls.crt", "TLS Certificate")
+       keyFile    = flag.String("keyFile", "/var/tls/tls.key", "TLS Key")
 )
 
func main() {
@@ -16,5 +17,5 @@ func main() {
 
        http.HandleFunc("/webhook", sslmutator.AcceptWebhook)
        log.Printf("Listening on %s", *listenAddr)
-       log.Fatal(http.ListenAndServe(*listenAddr, nil))
+       log.Fatal(http.ListenAndServeTLS(*listenAddr, *certFile, *keyFile, nil))
}

Java Applications without TLS

public static void main(String[] args) throws IOException {
    SpringApplication.run(ApiDemonstration.class, args);
}

@RestController
@RequestMapping("/api")
public class ApiEndpointController {
    @GetMapping("/testinfo")
    public TestInfo getTestInfo() {
        return new TestInfo();
    }
}

Java Applications using TLS

   public static void main(String[] args) throws IOException {
+     System.setProperty("server.ssl.enabled", "true");
+     System.setProperty("server.ssl.key-store", "/var/tls/tls.p12");
+     System.setProperty("server.ssl.key-store-password", "abc123");
     SpringApplication.run(ApiDemonstration.class, args);
  }

  @RestController
  @RequestMapping("/api")
  public class ApiEndpointController {
      @GetMapping("/testinfo")
      public TestInfo getTestInfo() {
          return new TestInfo();
      }
  }

Mutual TLS in your services

  • So far configuring the certificates is easy, but getting the certifcate is the hard part

  • Generating and storing the certificates correctly is fiddly

  • Error reporting from certificate tools, and programming languages is poor

  • How can we make this easier?

TLS keys with KubeTLS

Kubernetes to the rescue!

  • Kubernetes already has a built-in Certificate Authority
  • One could make the case that a Kubernetes cluster is defined by it's master certificate authority
  • Kubernetes should create TLS certificates for all pods/services/deployments
  • Similar to Borg/ALTS

The plan:

  • Create a TLS cert that is compatible with the way you use Kubernetes services already
  • Generate a certificate and key for every serivce (endpoint)
  • Provide that certifcate and key to every pod that matches that service
  • All certificates are automatically generated by a central (Kubernetes) CA, and trusted by everything else in the cluster

The details:

Mutating Controller

  • Kubernetes provides a webhook interface which for verifying and modifying objects as they are created
  • KubeTLS is our service which implements a webhook watching for pod creation
  • Pod create requests are modified to inclued a TLS secret with that that POD's certificate and key

KubeTLS in Action

Demo Time!

Mutating Webook Request Body

{ "kind": "AdmissionReview", "apiVersion": "admission.k8s.io/v1beta1",
  "request": {
    "uid": "43a278cc-6ee3-4f6c-a1bd-7c2ca21d7df4",
    "kind": {            "group": "", "version": "v1", "kind": "Pod" },
    "resource": {        "group": "", "version": "v1", "resource": "pods" },
    "requestKind": {     "group": "", "version": "v1", "kind": "Pod" },
    "requestResource": { "group": "", "version": "v1", "resource": "pods" },
    "namespace": "default",
    "operation": "CREATE",
    "userInfo": {
      "username": "system:serviceaccount:kube-system:replicaset-controller",
      "uid": "f80d5e0e-e75f-4c42-adb5-f49231d03278",
      "groups": [  "system:serviceaccounts",
                   "system:serviceaccounts:kube-system",
                   "system:authenticated"  ]
    },
    "object": {
      "kind": "Pod", "apiVersion": "v1",
	  . . .

Mutating Webook Response Body

  },
  "response": {
    "uid": "43a278cc-6ee3-4f6c-a1bd-7c2ca21d7df4",
    "allowed": true,
    "status": {
      "metadata": {}, "status": "Success"
    },
    "patch": "W3sib3AiOiJhZ . . . HMtYmMxYzhhMWYiLCJzZWNyZXQiOnsic2VjcmV0TmFtZSI6InRscy1iYzFjOGExZiJ9fX1d",
    "patchType": "JSONPatch"
  }
}
 - - - - - - - - - - - - - -
[ { "op": "add",
    "path": "/spec/containers/0/volumeMounts/-",
    "value": {
      "mountPath": "/var/run/secrets/gauntletwizard.net/tls",
      "name": "tls-94d67ccf" }  },
  { "op": "add",
    "path": "/spec/volumes/-",
    "value": {
      "name": "tls-94d67ccf",
      "secret": {
        "secretName": "tls-94d67ccf"
      } } }
]

Webhook Controller

func (s TLSController) AcceptWebhook(w http.ResponseWriter, r *http.Request) {
	body, err := ioutil.ReadAll(r.Body)
	admissionRequest, err := parseAdmissionRequest(body)
	pod := &corev1.Pod{}
	json.Unmarshal(admissionRequest.Request.Object.Raw, pod)
	pod.Namespace = admissionRequest.Request.Namespace
	services, err := s.ss.MatchingServices(r.Context(), pod)
	secret, err := s.tlss.SecretForServices(r.Context(), *pod, services)
	review := Mutate(*admissionRequest, secret.Name)
	response, err := json.Marshal(review)
	w.WriteHeader(200)
	w.Write(response)
}

Create And Upload Csr

	key, err := rsa.GenerateKey(rand.Reader, 1024)
	subj := pkix.Name{
		CommonName: info.ServiceAccount(),
		// we can add more here, but it's not necessary
	}
	template := x509.CertificateRequest{
		Subject:  subj, SignatureAlgorithm: x509.SHA256WithRSA, DNSNames: info.DNSNames(),
	}
	csrBytes, _ := x509.CreateCertificateRequest(rand.Reader, &template, key)
	 . . .  Convert to PEM format . . . 
	csrObject := certsv1.CertificateSigningRequest{}
	csrObject.Name = info.Name()
	csrObject.Annotations = info.Annotations()
	csrObject.Labels = t.labels
	csrObject.Spec.SignerName = &SignerName
	csrObject.Spec.Request = csr
	// Allowed usages of our cert.
	csrObject.Spec.Usages = []certsv1.KeyUsage{certsv1.UsageDigitalSignature, certsv1.UsageKeyEncipherment, certsv1.UsageServerAuth, certsv1.UsageClientAuth}
	createdCsrObject, err := client.Create(ctx, &csrObject, metav1.CreateOptions{})
	// This is the stamp we assign to the CSR object to let the cluster master sign it.
	approval := certsv1.CertificateSigningRequest{
		Status: certsv1.CertificateSigningRequestStatus{
			Conditions: []certsv1.CertificateSigningRequestCondition{{
				Type: certsv1.CertificateApproved,
				Reason:  "Approved by TLS Service",	Message: "KubeTLS Approved",
				LastUpdateTime: metav1.Now(),
			}},
		},
	}
	approval.ObjectMeta = createdCsrObject.ObjectMeta
	approved, err := client.UpdateApproval(ctx, &approval, metav1.UpdateOptions{})

Create Secret

	secretToPass := &corev1.Secret{}
	secretToPass.Name = info.Name()
	secretToPass.Annotations = info.Annotations()
	secretToPass.Labels = t.labels
	secretToPass.Type = corev1.SecretTypeTLS
	keybytes, key, csr := csrForServices(info)
	cert, err := t.uploadAndApproveCSR(ctx, csr, info)
	pkcs12store, pkcs12passord, err := t.convertToPkcs12(cert, &keybytes)
	if err != nil {
		fmt.Println("Failed to convert to Pkcs12 store, ", err)
		return nil, err
	}
	secretToPass.Data = map[string][]byte{
		"tls.key": key,
		"tls.csr": csr,
		"tls.crt": cert,
		"pkcs12.certStore": pkcs12store,
		"pkcs12.password": pkcs12passord,
	}

Deploy Go App with MTLS

  • Simple golang grpc
  • Job communicating via mutual tls
  • Pods have the correct keys applied automatically

Using KubeTLS

const ( 
        KubeTLSSecretLocation = "/var/run/secrets/gauntletwizard.net/tls/"
        KubeTLSKeyLocation    = KubeTLSSecretLocation + "tls.key"
        KubeTLSCertLocation   = KubeTLSSecretLocation + "tls.crt"
        KubeTLSCALocation     = "/var/run/secrets/kubernetes.io/serviceaccount/ca.crt"
)

// NewKubeTLS constructs a TLS Configuration for use with KubeTLS, suitable for use in both client and server
func NewKubeTLS() (*tls.Config, error) {
        certificate, err := tls.LoadX509KeyPair(KubeTLSCertLocation, KubeTLSKeyLocation)
	    ...
        serverpool, err := x509.SystemCertPool()
        serverpool.AppendCertsFromPEM(rootCertData)
        return &tls.Config{
                Certificates: []tls.Certificate{certificate},
                // Server settings
                ClientAuth:   tls.RequireAndVerifyClientCert,
                ClientCAs:    clientpool,
                // Client Settings
                RootCAs: serverpool,
}

Using KubeTLS

  • libraries in golang
  • libraries in java
  • more to come . . .

Compare to Service Mesh

  • This is service Mesh
  • Istio adds two contact edges - Client < - > Mesh, Mesh < - > Server
  • Debugging is hard - Harder even than TLS
  • Abstracts too much from Developers

Future Directions

  • More languages supported
  • Ideally this would be in the Kubelet
  • Changes in 1.19 alter they way that this works

Repository

https://gitlab.com/gauntletwizard_net/kubetls

Presentation

tcbtech.com/kubetls

Questions?

TLS Secret Creation

TLS Secret Creation

Use Kubernetes as a signing server manually

# Certificate authority is in the Kubernetes API
cfssl genkey server-csr.json |cfssljson -bare
echo {} | jq --rawfile csr cert.csr '{apiVersion: "certificates.k8s.io/v1beta1", kind: "CertificateSigningRequest", metadata: {name: "kubetls-bootstrap"}, spec: {request: $csr| @base64, usages: ["server auth", "client auth", "digital signature", "key encipherment"]}}' | kubectl create -f -
kubectl certificate approve kubetls-bootstrap
kubectl get csr kubetls-bootstrap -o json | jq -r  '.status.certificate | @base64d' > cert.crt

Ted Hahn is an SRE for hire working on planet-scale distributed systems. His clients include Epic Games and startups in Seattle and New York.

Mark Hahn is Director of Cloud Strategies and DevOps for Ciber Global, a consulting firm. In this role he is responsible for all things related to software velocity.

we are father son team that put this idea and demonstration together

Your core infrastructure should included opinoonated web server which includes middleware for metrics, logging, security

We propose a Kubenetes controller (and libraries for using) which allow you to easily stand up secure pod-pod commuications

The controller creates certificates for each pod allowing pods to easily setup TLS, and support mTLS

Using certificates and TLS is all goodness and light, but getting the certificate setup and distirbuted is hard work

a developer can easily follow these steps to create certificates for development, and used them for testing but getting certificates for test environments or production can be much harder

A common example is to use CFSSL, the Cloud Flair SSL tool to operate your own certificate authority

this is painful process this particully when you are running the CA and are the client of the CA; it is easy to get the role mixed up and distirbute the wrong parts to the wrong people

This a pain in CFSSL, as there's a bunch of moving parts and the two roles get confused. This example lives in branch ted/mtlsexample /docs/manual-ca/

You will find more detailed examples in our repository

This is a pain in OpenSSL. There's: - config files in Windows old INI format - cobbled support for batch mode - support for newer x509 extenstions is wonky (Netscape anyone?)

we use TLS through out, call us out if we accidentialy say SSL

by the way we should always say TLS and not SSL. call us out if we forget to call it TLS

this is a simpe golanger server which . . .

but all servers or serivvces should use TLS and we will show how that is done

these are the modifications needed to enable the server to use TLS

This code shouldn't be copy+pasted throughout your repositories: It should be in one simple central place

But creating the PKCS.12 Java Key Store is antiquated, and it is encrypted with DES.

introduce the topic of MTLS and movtivate difficultly with certs

Reader's note: The view that the cluster's CA is the definition of the cluster is shared by older documentation, but changes in 1.19 seem to disagree

ALTS = Application Layer Transport Security

this is how kubelets already join the cluster; they generate a private key and CSR which is sent to the master

the CSR is turned into a certificate which is then used to secure all furhter control plane communication

So, certificates are already distributed to each node, so why shouldn't they be for each pod

How we have done this is we have created a mutating webhook service which is called by k8s whenever a pod is created the webhook dynmaincly creates and attaches a secret to your pod's container. The secret contains the necessary certificate and private key necessary for inter pod communication.

demo : two tabs in terminal: one in infocoinatiner base ; one in grpc branch

Demo prep: Working KubeTLS server, then: Delete the mutatingwebhookconfigurations (have a backup!), delete secret (and matching csr) infocontainers are using, restart infocontainers (so they have no secret). Remove grpc pods

Show pods; show running KubeTLS server

Show a pod with no cert (infocontainer)

install KubeTLS server as the mutating webhook controller

watchthe KubeTLS logs

kill pod

look at pod with new secret (infocontainer)

This is the body of the request to the mutating webhook controller, but hand pretty-printed to get it to fit on the slide. The full pod objet is provided in the object: field.

this is the main Go code for the logic of the KubeTLS Mutating Webhook Controller. All the error checking has been deleted so it fits on one screen. This follows the bullet point plan we outlined earlier in our presentation.

Demo

In ted/mtlsexample: cd kubessl/tlslib/example/golang-grpc kubectl apply -f greeter_server.yaml kubectl create -f greeter_client.yaml (copy the created job name) kubectl logs CTRL+V

RootCAs is the certificate pool used for verifying servers as a client. Clienct CAs is the certificate pool used for verifying clients as a server

There's example code in our repository, as well as libraries. The libraries provide features like authentication based on the requesting pod's ServiceAccount

the magic here is not in the libraries but in the distriution of the certificates to every place that needs them

Serivce Meshes: Ishtio, linkerd, Consul, etc. <!-- Ted makes the point that this is service mesh done Kubernetes native

Service mesh adds too many point of contact

Article on itshtio and wireshare

Mark makes the point that service mesh takes the responsibility away from developers

Developers throw their requests into the stream, rather than understand and coordinate with teams they are in direct contact with

This complements tools like [KubeResolver](https://github.com/everflow-io/kuberesolver), which allow load-balancing by using the Kubernetes API Why not Istio? I recently read [an article](https://shrayk.medium.com/ten-tips-for-running-istio-in-production-4ea2b158440a) in which the first piece of advice is "Use Wireshark to debug". This is pretty much the opposite of one of the most stated purposes of Istio - To prevent to sniffing or interception traffic.

# FAQ: ## What does the KubeTLS Certificate look like? The Certificate generated is unique to the set of services that the pod matches, as well as the serviceAccount the pod is running as. The relevant bits are such: *SANs*: A list of services matching the pod, i.e. if a pod can be found in the "nginx" and "web-frontend" endpoints objects in the "default" namespace, it will have ["nginx", "web-frontend"]. Support for namespace specific suffixes and cluster-specific suffixes is planned. *CN*: ServiceAccount, in the format "system:serviceaccount:Namespace:ServiceAccount", i.e. "system:serviceaccount:default:nginx" The commonName of the certificate is inteded for use as the client's name. Use of CommonName as an alternate DNS validation has long been deprecated, and starting in Go 1.15 the system no longer accepts Server certificates without a SubjectAlertnativeName field. ## Future of the KubeTLS Certificate: The KubeTLS certificate should be unique per-pod. Generating a unique certificate per-pod is a future goal of KubeTLS, and the current method will be deprecated. The following changes will be made: *SANs*: Will include a pod-specific identifier. Will *not* include the pod's IP address, for both security (IP Addresses may be reused) and practical (KubeTLS does not know the Pod IP at webhook time) reasons. *CN*: Will stay as is, representing the ServiceAccount. *Subject*: Will include some specific OU or other structure specifying the pod ID. ## Future of KubeTLS: Kubernetes 1.19 deprecated the `certificates.k8s.io/v1beta1` API. Included in the new `certificates.k8s.io/v1` API are Signing Profiles, specified by the `signerName` field. This represents a significant change to the way certificates are generated and stored. KubeTLS will no longer be able to create dual-use "client auth" and "server-auth" certificates, so those functionalities may be split into separate files. In the farther future, we believe that this functionality belongs in the kubelet - That the Kubelet should be responsible for the generation of the certificate, and it's presentation to the Kubernetes APIserver for signing.

Kubernetes simplifes this a lot: You don't need to keep your highly-sensitive CA Certificate in an accessible place or have a special signing computer. You simply call the k8s API This example follows from our k8s/bootstrap directory