Setup a Kubernetes cluster using
k3d
running in GitHub Codespaces
This repo is to demonstrate the use of Kubernetes in Codespaces in a production-like scenario where an ingress
is set up for routing.
Refer to the original
Kubernetes in Codespaces
repository to understand more about the setup and usage.
When a repository is created from Kubernetes in Codespaces template, it pulls a dotnet
based application, IMDB App, from here. While starting the codespace, this application is built and hosted in a k3d
cluster running in Codespaces.
-
Use the template here and create a Github repository.
-
Create a codespace on the
main
branch or a new branch frommain
. -
Once the codespace is created and configured, the IMDB App should be running. Check this either in the container creation logs or by running
kic pods
from azsh
terminal.Wait for
onCreate
,postCreate
andpostStart
commands to finish before checking. -
Make sure that the IMDB App is running in Codespace by navigating to the nodeport exposed through Codespaces port
30080
. Swagger documentation should be visible on the homepage. -
Navigate to
/api/movies
and check if returns a list of movies.
The following sections describe how another application can be added to this environment.
The first scenario is talking about IMDB UI, which consumes one of the APIs in IMDB App. In this scenario, the other app is being developed in a separate repository.
All changes mentioned in the below steps are available as part of commit Onboard meavk/imdb-ui application for reference.
-
The first step is to clone that repository to Kubernetes in Codespaces repository. This can be done by adding a few lines of code to
.devcontainer/on-create.sh
to cloneimdb-ui
and restore packages.# clone repos # -- clone other repos -- git clone https://github.com/meavk/imdb-ui /workspaces/imdb-ui # restore the repos # -- restore other projects -- dotnet restore /workspaces/imdb-ui/src/Imdb.BlazorWasm/Imdb.BlazorWasm.csproj
-
You'd also need to keep it up to date. Modify
.devcontainer/post-create.sh
for this.# update the repos # -- update other repos -- git -C /workspaces/imdb-ui pull
-
As the new app is already dockerized, the next step is to create a YAML file
imdb-ui.yaml
for IMDB UIdeployment
andservice
in a new folderdeploy/apps/imdb-ui
for the app. Note that the service listens onnode port
30090
.apiVersion: apps/v1 kind: Deployment metadata: name: imdb-ui namespace: imdb labels: app.kubernetes.io/name: imdb-ui spec: replicas: 1 selector: matchLabels: app: imdb-ui template: metadata: labels: app: imdb-ui spec: containers: - name: app image: k3d-registry.localhost:5500/imdb-ui:local imagePullPolicy: Always ports: - name: http containerPort: 80 protocol: TCP resources: limits: cpu: 1000m memory: 256Mi requests: cpu: 200m memory: 64Mi --- apiVersion: v1 kind: Service metadata: name: imdb-ui namespace: imdb spec: type: NodePort ports: - port: 9080 nodePort: 30090 targetPort: http protocol: TCP name: http selector: app: imdb-ui
-
The
kic
CLI hasbuild
commands for building and deploying the apps. Add one to build and deploy IMDB UI by creating a new fileimdb-ui
incli/.kic/commands/build
folder. It can be seen that it's mostly a copy-paste of the same forimdb
except for a few changes.#!/bin/bash #name: imdb-ui #short: Build and deploy the IMDb UI to the local cluster cd "$REPO_BASE" || exit # validate directories if [ ! -d /workspaces/imdb-ui ]; then echo "/workspaces/imdb-ui directory not found. Please clone the imdb-ui repo to /workspaces"; exit 1; fi if [ ! -d deploy/apps/imdb-ui ]; then echo "$REPO_BASE/deploy/apps/imdb-ui directory not found. Please cd to an appropriate directory"; exit 1; fi # delete webv and imdb kubectl delete -f deploy/apps/imdb-ui --ignore-not-found=true --wait=false # build and push the local image for imdb-ui docker build /workspaces/imdb-ui/src/Imdb.BlazorWasm -t k3d-registry.localhost:5500/imdb-ui:local docker push k3d-registry.localhost:5500/imdb-ui:local # wait for delete to finish kubectl wait pod -l app=imdb-ui -n imdb --for delete --timeout=30s # deploy local app and re-deploy webv kubectl apply -f deploy/apps/imdb-ui kubectl wait pod -l app=imdb-ui -n imdb --for condition=ready --timeout=30s
-
This command needs to be added
kic build
scripts list by adding the following lines toroot.yaml
incli/.kic
folder.- name: imdb-ui short: Build the IMDb UI app path: build/imdb-ui
-
Add the following lines to
.devcontainer/on-create.sh
, right afterkic build imdb
, to invokeimdb-ui
build command so that IMDB UI is built as the container gets created.echo "building IMDb UI" chmod +x /workspaces/k8s-in-codespaces-routing/cli/.kic/commands/build/imdb-ui kic build imdb-ui
-
The app needs to be exposed to the internet. Codespaces has port-forwarding to expose services that are running in Codespaces. In an earlier step, while creating a
service
for IMDB-UI, you might have noticed anodePort
being assigned. Since this is ak3d
node, first step is to configurek3d
to expose it. This can be done by adding a new entry to theports
section in.devcontainer/k3d.yaml
.- port: 30090:30090 nodeFilters: - server[0]
-
Lastly, enable port forwarding by adding port
30090
to respective sections in.devcontainer/devcontainer.json
."forwardPorts": [ 30000, 30080, 30090, 31080, 32000 ], "portsAttributes": { "30000": { "label": "Prometheus" }, "30080": { "label": "IMDb App" }, "30090": { "label": "IMDb UI" }, "31080": { "label": "Heartbeat" }, "32000": { "label": "Grafana" } },
-
Rebuild codespace by using the command
Codespaces: Rebuild Container
from Command Palette (Ctrl+Shift+P
) for the changes to take effect. -
Once the codespace is built and pods are running,
Ports
section should show a new entryIMDb UI (30090)
. Click on the link to browse the application.Use
kic pods
command or check creation logs as did earlier to verify pods, includingimdb-ui
, are running.
The UI for IMDB onboarded in the previous section is a Blazor WebAssembly application that runs completely in the browser. Navigate to the 'Movies' tab to see a list of movies.
You might remember from the earlier step that the response from /api/movies
endpoint had a large number of movies in the list. However, the app is displaying only 2 movies.
This is because the UI is showing a hardcoded list of movies as the request to /api/movies
failed.
And, application fetched a hard-coded list of movies from a .json
file
try
{
movies = await Http.GetFromJsonAsync<Movie[]>("api/movies");
}
catch (System.Exception)
{
movies = await Http.GetFromJsonAsync<Movie[]>("sample-data/movies.json");
}
This is a possible real-world scenario. In the microservices world, subdomains and different paths on the same domain can be served by different applications. Kubernetes and other microservices frameworks provide such capabilities. It's only fair to expect Kubernetes in Codespaces also to provide such capability. The next section shows how Codespaces port-forwarding and a Kubernetes ingress
can be leveraged to implement the same.
For codespaces, the host
part of the URL will be dynamically generated by GitHub and it changes from Codespace to Codespace, and even for different ports in the same codespace. Look at the 'Ports' tab screenshot above, you'll see the port number being appended to the host address. For the scenario described in the previous section, requests to /api/
should be routed to 'IMDB App' and the rest to 'IMDB UI`. Usually, this is easily doable by creating a Kubernetes Ingress and adding respective rules. This alone wouldn't solve the problem here since the cluster is running on a virtual node inside Codespace. The ingress-managed loadbalancer is not directly bound to the Codespace host IP. This is where Codespaces' port-forwarding comes in handy.
All changes mentioned in the below steps are available as part of commit Configure routing through ingress. for reference.
-
The first requirement for the solution is an ingress controller.
k3d
comes with Traefik Ingress Controller. Since Kubernetes in Codespaces has disabled this, a new ingress needs to be set up. Add the following code to.devcontainer/on-create.sh
afterkic cluster deploy
to install the latest version of Traefik.This sample uses Traefik. Any other ingress, Nginx for example, would also work. Just have to configure the rest accordingly
echo "installing traefik" helm repo add traefik https://containous.github.io/traefik-helm-chart helm install traefik traefik/traefik
-
Next add a port mapping to
ports
section in.devcontainer/k3d.yaml
to map port8081
to Traefikloadbalancer
port.- port: 8081:80 nodeFilters: - loadbalancer
The above configuration maps port
8081
from the host to port80
on the container that matches the nodefilterloadbalancer
. And it matchestraefik
LoadBalancer service.@meavk ➜ /workspaces/k8s-in-codespaces-routing (main) $ k get service traefik NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE traefik LoadBalancer 10.43.161.1 172.18.0.2 80:31244/TCP,443:30201/TCP 5d5h
-
Add port-forwarding to
.devcontainer/devcontainer.json
.// forward ports for the app "forwardPorts": [ 30000, 30080, 31080, 32000, 30090, 8081 ], // add labels "portsAttributes": { "30000": { "label": "Prometheus" }, "30080": { "label": "IMDb App" }, "31080": { "label": "Heartbeat" }, "32000": { "label": "Grafana" }, "30090": { "label": "IMDb UI" }, "8081": { "label": "Traefik Ingress" } },
-
Lastly, to route the requests as described earlier, create
imdb-ingress.yaml
to create an ingress resource in a new folderdeploy/apps/ingress
.apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: imdb-ingress namespace: imdb annotations: ingress.kubernetes.io/ssl-redirect: "false" spec: rules: - http: paths: - path: / pathType: Prefix backend: service: name: imdb-ui port: number: 9080 - path: /api pathType: Prefix backend: service: name: imdb port: number: 8080
-
Rebuild Codespace to apply the changes.
-
After the rebuild, once commands are run and the applications are up and running, navigate to 'Traefik Ingress' to load the UI. This time, the 'Movies' tab should load the full list of movies.