We'll be deploying 2 microservices on Kubernetes. Our first microservice will be a httpd server
& our second microservice will be running php-fpm8.2
. The php-fpm server will be using TCP socket at port 9000
. The user request will interact with httpd server & httpd server is going to act as a proxy server which is going to proxy forward the request to fpm server.
.
├── ...
├── resources # Additional Resources
│ ├── www.conf # FPM server pool.d conf
│ ├── apache.vhost.conf # httpd server conf
│ ├── migration-script.sh # migration script
├── kubernetes # Kubernetes manifests
│ ├── namespace.yaml
│ ├── httpd-deployment.yaml
│ ├── php-fpm-deployment.yaml
│ └── php-fpm-vpa.yaml
├── Dockerfile.httpd # Apache Dockerfile
├── Dockerfile.php-fpm # PHP-FPM Dockerfile
├── README.md # Updated README
└── ...
You can skip this step if you're using some other cluster of your own.
minikube start
git clone https://github.com/nautiyaldeepak/demo.git
git clone https://github.com/kubernetes/autoscaler.git
cd demo/
docker build -t httpd:v1 -f Dockerfile.httpd .
docker build -t php-fpm:v1 -f Dockerfile.php-fpm .
The below example is to push docker images to a Container Registry
. You don't have to do this step, if you'll be using pre-built public images (mentioned below).
docker tag httpd:v1 <YOUR_REPOSITORY_URL>:httpd-v1
docker tag php-fpm:v1 <YOUR_REPOSITORY_URL>:php-fpm-v1
docker push <YOUR_REPOSITORY_URL>:httpd-v1
docker push <YOUR_REPOSITORY_URL>:php-fpm-v1
Check your kubernetes nodes CPU architecture. Based on that architecture use the pre-built
images
.
kubectl describe nodes | grep arch=
Pre Built Public Images
Images | arm64 | amd64 |
---|---|---|
php-fpm | public.ecr.aws/o6a6b5p9/php-fpm:arm64-v1 |
public.ecr.aws/o6a6b5p9/php-fpm:amd64-v1 |
httpd | public.ecr.aws/o6a6b5p9/httpd:arm64-v1 |
public.ecr.aws/o6a6b5p9/httpd:amd64-v1 |
Use the below commands to replace images in kubernetes manifests.
sed -i '' 's|REPLACE_IMAGE|<YOUR_HTTPD_IMAGE_HERE>|g' kubernetes/httpd-deployment.yaml # httpd image
sed -i '' 's|REPLACE_IMAGE|<YOUR_PHP_FPM_IMAGE_HERE>|g' kubernetes/php-fpm-deployment.yaml # php-fpm image
grep -r image kubernetes/ # Check if images have been replaced in kubernetes manifests correctly
The below commands will create a namespace testing
& all the resources will be deployed in that namespace. I already have prebuild images which are being used in these deployments.
kubectl apply -f kubernetes/namespace.yaml
kubectl apply -f kubernetes/httpd-deployment.yaml
kubectl apply -f kubernetes/php-fpm-deployment.yaml
NOTE:
- If images fail to get deployed, please double check the architecture of your nodes or contact me.
We'll use port-forwading to test our task.
kubectl port-forward svc/httpd-server --address 0.0.0.0 8080:80 -n testing
Now, access the URL http://0.0.0.0:8080
on your browser. Make sure it is http
& not https
, since there are no certificates. Use the URL to add/edit some content on the website (We're going to use this in Task 2).
NOTE: When accessing the URL http://0.0.0.0:8080
, there are some APIs which are not working. It'll give you 403 AccessDenied
error. This is an application level issue, therefore I haven't addressed them. When you're logging into the app make sure you use ADMIN_USER
and not the REGULAR_USER
.
- There is a Sqlite database running on directory
/app/data/
. There is no Persistent Volume attached to the pod, the storage is ephemeral. So, basically here we need to migrate/app/data/database.sqlite
. - In order to successfully migrate our data from current pod to the new pod we're using
postStart
lifeCycle hook on ourphp-fpm
deployment. The lifecycle hook will execute a migration script which is going to copy/app/data/database.sqlite
from the old container to the new container. - This migration script has already been made part of
task 1
. So when you deployed task 1 themigration script
was also deployed with thephp-fpm
deployment.
Our kubernetes deployments are already running from Task 1. We just need to redeploy our php-fpm
pod. Use the command below to rollout a new pod.
kubectl rollout restart deployment php-fpm -n testing
This command will create a new pod and this new pod will copy the /app/data/database.sqlite
from the previous running pod to this new pod & only after that this new pod will get into Running
state & old pod will be Terminated
.
Now, lets check the content of this new pod.
kubectl port-forward svc/httpd-server --address 0.0.0.0 8080:80 -n testing
Again access the URL http://0.0.0.0:8080
. Check if the updated content from Task 1 still exists.
NOTE: The migration-script is written in such a manner that if the deployment if happening for the first time then it'll know that there is no migration necessary.
- Currently our php-fpm microservice is stateful, as the database is within the microservice itself.
- Due to this nature of the application it is impossible to scale the application horizontally, as each microservice will have its own database. So we're going to scale this application vertically.
Deploying metrics-server for VPA. VPA will use data from metrics-server to analyse the metrics of our microservices. If you're running on minikube then use the following command
minikube addons enable metrics-server
OR
kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml
After deploying metrics-server, we'll deploy Vertical Pod Autoscaler custom resource from the autoscaler repository which we cloned earlier.
cd ../autoscaler/vertical-pod-autoscaler/ # Change directory
./hack/vpa-up.sh
NOTE: VPA
& metrics-server
resources will be create in kube-system
namespace
Now we'll deploy VPA resource for php-fpm
deployment.
cd ../../demo
kubectl apply -f kubernetes/php-fpm-vpa.yaml
kubectl get vpa -n testing
kubectl describe vpa php-fpm-vpa -n testing # See scaling recommendations made by VPA
NOTE: Metrics can take some time to show up.
- The VPA only supports scaling on metrics cpu & memory.
- The above VPA is using
standard
vpa recommender. - App will vertically scale up if CPU/Memory usage crosses 90% threshold.
- The main issue with the current architecture is that, database and the app are running on the same container.
- Our app is now acting as a stateful application which is not fault tolerant & also not highly scalable.
- We cannot deploy horizontal pod autoscaling for our app, as we cannot have each replica running its own database.
- The same problem goes with database as well. If our pod is deleted due to some reason all our data is lost.
- Also, the scaling of database is also very difficult as the storage in this setup is ephemeral.
- In the diagram above we're segregating the database from the app.
- The app is running independently of the database & now our
php-fpm
is stateless. If our app wants to interact with database it'll make the request to the database. - Since our
php-fpm
app in this setup is stateless we can apply horizontal pod autoscaling on this deployment. HPA also provides us with the flexibility to use custom-metrics. We can use metrics likerequest rate
ornetwork throughput
to horizontally scale our application. - With horizontal pod autoscaling we can deploy multiple replicas for our app if the resource utilization crosses a certain threshold. This kind of setup will make our app fault tolerant & highly scalable.
- In this new setup, we'll manage our database independently. Our database can be scaled vertically & a database replica is also deployed which is going to act as a backup and also as failover DB.
- This entire process can be implemented on a CI/CD pipeline.
- I could have reduced docker image size by using multistage build and lighter base images.
- In place of kubernetes manifests, I could have implemented helm-charts.
- There can be improvements made on security, by adding securityContext for our pods.
- Horizontal Scaling can also be applied on
httpd-server
. - In a proper
prod
environment, I would've used proper branches to manage my work. Currently I've directly commited my work onmaster
.