diff --git a/.gitignore b/.gitignore index 7d9713f..a4d3524 100644 --- a/.gitignore +++ b/.gitignore @@ -2,4 +2,5 @@ .vscode !.gitignore scripts/__pycache__ -output* \ No newline at end of file +output* +.idea \ No newline at end of file diff --git a/README.md b/README.md index 625186c..4018323 100644 --- a/README.md +++ b/README.md @@ -126,6 +126,8 @@ make build-node-image OS_TARGET=photon-3 KUBERNETES_VERSION=v1.23.15+vmware.1 TK Sample customization examples can be found [here](docs/examples/README.md) +For Windows support you may refer to [Windows tutorial](docs/windows.md) + ## Debugging - To enable debugging for the [make file scripts](hack/make-helpers/) export `DEBUGGING=true`. diff --git a/docs/examples/README.md b/docs/examples/README.md index b77e82b..502e911 100644 --- a/docs/examples/README.md +++ b/docs/examples/README.md @@ -1,8 +1,8 @@ # Examples -This document gives information about different customization examples to customize a Node image for Tanzu on vSphere. +This document gives information about how to build Kubernetes Release on vSphere Node image and different customization examples to customize the Node image. -- [Changing VM Hardware Version(VMX version)](changing_hardware_version.md) -- [Adding new OS packages and configuring the repositories or sources](adding_os_pkg_repos.md). -- [Running Prometheus node exporter service on the nodes](prometheus_node_exporter.md) - [Tutorial for Using the vSphere Tanzu Kubernetes Grid Image Builder](tutorial_building_an_image.md) +- [Changing VM Hardware Version(VMX version)](./customizations/changing_hardware_version.md) +- [Adding new OS packages and configuring the repositories or sources](./customizations/adding_os_pkg_repos.md). +- [Running Prometheus node exporter service on the nodes](./customizations/prometheus_node_exporter.md) diff --git a/docs/examples/adding_os_pkg_repos.md b/docs/examples/adding_os_pkg_repos.md deleted file mode 100644 index 2cf5f3f..0000000 --- a/docs/examples/adding_os_pkg_repos.md +++ /dev/null @@ -1,109 +0,0 @@ -# Adding new OS packages and configuring the repositories or sources - -## Use case - -As a user I want to - -1. Add new OS packages such as `pkg1` and `pkg2`. -2. Configure sources/repositories to an internal mirror to build the node images in an air-gapped scenario. Configuring new sources/repositories is also useful when we want to install internally built software. - -## Customization - -Configuration of OS packages and sources/repositories is exposed as packer variables. To view a list of packer variables exposed by [kubernetes image builder][kubernetes-image-builder] please refer to this [page][customizations-doc]. - -### Adding new packages - -To add new packages [kubernetes image builder][kubernetes-image-builder] provides `extra_rpms` and `extra_debs` packer variables for Photon and Ubuntu respectively. To add new packages, add the packages to the packer variables in the [default-args.j2][default-args] file as shown below. - -_**Note**: If you are building just the Ubuntu OS node image you will not have to modify the photon OS block._ - -```jinja - {% if os_type == "photon-3" %} - "distro_version": "3.0", - "extra_rpms": "existing_packages pkg1 pkg2" - {% elif os_type == "ubuntu-2004-efi" %} - "extra_debs": "existing_packages pkg1 pkg2", - "boot_disable_ipv6": "1" - {% endif %} -``` - -### Configuring new sources/repositories - -[kubernetes image builder][kubernetes-image-builder] provides `extra_repos` packer variables through which sources/repositories can be configured for both Photon and Ubuntu. As there is a difference in how Ubuntu/Photon sources are configured we need to have separate source files for Photon/Ubuntu. - -- Create new folder `repos` in [ansible files][ansible-files] folder -- Create a new file for Photon sources called `photon.repo` in the `repos` folder. Refer below for sample content and refer to the official Photon [document][photon-repo-doc] for more information - -```text -[photon] -name=VMware Photon Linux $releasever ($basearch) -baseurl=/$releasever/photon_release_$releasever_$basearch -gpgkey= -gpgcheck=1 -enabled=1 - -[photon-updates] -name=VMware Photon Linux $releasever ($basearch) Updates -baseurl=/$releasever/photon_updates_$releasever_$basearch -gpgkey= -gpgcheck=1 -enabled=1 - -[photon-extras] -name=VMware Photon Extras $releasever ($basearch) -baseurl=/$releasever/photon_extras_$releasever_$basearch -gpgkey= -gpgcheck=1 -enabled=1 -``` - -- Create a new file for Ubuntu sources called `ubuntu.list` in the `repos` folder. Refer to the official ubuntu [documentation][ubuntu-sources-doc] for more information. - - _**Note**: `focal` is for ubuntu 20.04 so this needs to be changed if the ubuntu version is also changed example for ubuntu 22.04 it is `jammy`_ - -```text -deb focal main restricted universe -deb focal-security main restricted -deb focal-updates main restricted -``` - -- Create a new file `repos.j2` or `repos.json` in [packer-variables](./../../packer-variables/) folder for configuring the `extra_repos` packer variable folder. (Uses [jinja][jinja] templating) - -```jinja -{ - {% if os_type == "photon-3" %} - "extra_repos": "/image-builder/images/capi/image/ansible/files/repos/photon.repo" - {% elif os_type == "ubuntu-2004-efi" %} - "extra_repos": "/image-builder/images/capi/image/ansible/files/repos/ubuntu.list" - {% endif %} -} -``` - -### Disabling public repositories/sources - -For disabling public repos set the `disable_public_repos` packer variable to `true` in the `repos.j2` file. - -### Removing the extra repositories/sources - -To remove the extra repositories/sources that were configured during the image build process set the `remove_extra_repos` packer variable to `true`. - -```jinja -{ - "disable_public_repos": true, - "remove_extra_repos": true - {% if os_type == "photon-3" %} - "extra_repos": "/image-builder/images/capi/image/ansible/files/repos/photon.repo" - {% elif os_type == "ubuntu-2004-efi" %} - "extra_repos": "/image-builder/images/capi/image/ansible/files/repos/ubuntu.list" - {% endif %} -} -``` - -[//]: Links - -[ansible-files]: ./../../ansible/files/ -[customizations-doc]: https://image-builder.sigs.k8s.io/capi/capi.html#customization -[default-args]: ./../../packer-variables/default-args.j2 -[jinja]: https://jinja.palletsprojects.com/en/3.1.x/ -[kubernetes-image-builder]: https://github.com/kubernetes-sigs/image-builder/ -[photon-repo-doc]: https://vmware.github.io/photon/assets/files/html/3.0/photon_admin/adding-a-new-repository.html -[ubuntu-sources-doc]: https://manpages.ubuntu.com/manpages/focal/man5/sources.list.5.html diff --git a/docs/examples/customizations/README.md b/docs/examples/customizations/README.md new file mode 100644 index 0000000..2c19931 --- /dev/null +++ b/docs/examples/customizations/README.md @@ -0,0 +1,12 @@ +# Examples + +This document gives information about different customization examples to customize a Kubernetes Release on vSphere Node image. + +- [Changing VM Hardware Version(VMX version)][hardware-version] +- [Adding new OS packages and configuring the repositories or sources][custom-package-repos]. +- [Running Prometheus node exporter service on the nodes][prometheus-node-exporter] + +[//]: Links +[hardware-version]: changing_hardware_version.md +[custom-package-repos]: adding_os_pkg_repos.md +[prometheus-node-exporter]: prometheus_node_exporter.md diff --git a/docs/examples/customizations/adding_os_pkg_repos.md b/docs/examples/customizations/adding_os_pkg_repos.md new file mode 100644 index 0000000..d320e41 --- /dev/null +++ b/docs/examples/customizations/adding_os_pkg_repos.md @@ -0,0 +1,114 @@ +# Adding new OS packages and configuring the repositories or sources + +## Use case + +As a user I want to + +1. Add new OS packages such as `pkg1` and `pkg2`. +2. Configure sources/repositories to an internal mirror to build the node images in an air-gapped scenario. Configuring new sources/repositories is also useful when we want to install internally built software. + +## Customization + +Configuration of OS packages and sources/repositories is exposed as packer variables. To view a list of packer variables exposed by [kubernetes image builder][kubernetes-image-builder] please refer to this [page][customizations-doc]. + +### Adding new packages + +To add new packages [kubernetes image builder][kubernetes-image-builder] provides `extra_rpms` and `extra_debs` packer variables for Photon and Ubuntu respectively. + +To add new packages to Ubuntu 22.04, add the packages to the packer variables in the [default-args-ubuntu-2204-efi.j2][default-args-ubuntu-2204-efi] file as shown below. + +```jinja + "extra_debs": "existing_packages pkg1 pkg2" +``` + +To add new packages to Photon 5, add the packages to the packer variables in the [default-args-photon-5.j2][default-args-photon-5] file as shown below. + +```jinja + "extra_rpms": "existing_packages pkg1 pkg2" +``` + +_**Note**: The location of default configuration specific to OS follows the path nomenclature, `packer-variables/-/default-args--.j2`. Include `-efi` suffix as well along with the version for Ubuntu._ + +### Configuring new sources/repositories + +[kubernetes image builder][kubernetes-image-builder] provides `extra_repos` packer variables through which sources/repositories can be configured for both Photon and Ubuntu. As there is a difference in how Ubuntu/Photon sources are configured we need to have separate source files for Photon/Ubuntu. + +- Create new folder `repos` in [ansible files][ansible-files] folder +- Depending upon the Linux OS flavour, use either of the below steps + - For **Photon** sources, create a new file called `photon.repo` in the `repos` folder. Refer below for sample content and refer to the official Photon [document][photon-repo-doc] for more information. + + ```text + [photon] + name=VMware Photon Linux $releasever ($basearch) + baseurl=/$releasever/photon_release_$releasever_$basearch + gpgkey= + gpgcheck=1 + enabled=1 + + [photon-updates] + name=VMware Photon Linux $releasever ($basearch) Updates + baseurl=/$releasever/photon_updates_$releasever_$basearch + gpgkey= + gpgcheck=1 + enabled=1 + + [photon-extras] + name=VMware Photon Extras $releasever ($basearch) + baseurl=/$releasever/photon_extras_$releasever_$basearch + gpgkey= + gpgcheck=1 + enabled=1 + ``` + + - For **Ubuntu** sources, create a new file called `ubuntu.list` in the `repos` folder. Refer to the official ubuntu [documentation][ubuntu-sources-doc] for more information. + - _**Note**: `jammy` is for ubuntu 22.04 so this needs to be changed if the ubuntu version is also changed, example for ubuntu 20.04 it is `focal`_ + +```text +deb jammy main restricted universe +deb jammy-security main restricted +deb jammy-updates main restricted +``` + +- Create a new file `repos.j2` in [packer-variables][packer-variables] folder for configuring the `extra_repos` packer variable folder. (Uses [jinja][jinja] templating) + +```jinja +{ + {% if os_type == "photon-5" %} + "extra_repos": "/image-builder/images/capi/image/ansible/files/repos/photon.repo" + {% elif os_type == "ubuntu-2204-efi" %} + "extra_repos": "/image-builder/images/capi/image/ansible/files/repos/ubuntu.list" + {% endif %} +} +``` + +### Disabling public repositories/sources + +For disabling public repos set the `disable_public_repos` packer variable to `true` in the `repos.j2` file. + +### Removing the extra repositories/sources + +To remove the extra repositories/sources that were configured during the image build process set the `remove_extra_repos` packer variable to `true`. + +```jinja +{ + "disable_public_repos": true, + "remove_extra_repos": true + {% if os_type == "photon-5" %} + "extra_repos": "/image-builder/images/capi/image/ansible/files/repos/photon.repo" + {% elif os_type == "ubuntu-2204-efi" %} + "extra_repos": "/image-builder/images/capi/image/ansible/files/repos/ubuntu.list" + {% endif %} +} +``` + +[//]: Links + +[ansible-files]: ./../../ansible/files/ +[customizations-doc]: https://image-builder.sigs.k8s.io/capi/capi.html#customization +[jinja]: https://jinja.palletsprojects.com/en/3.1.x/ +[kubernetes-image-builder]: https://github.com/kubernetes-sigs/image-builder/ +[photon-repo-doc]: https://vmware.github.io/photon/assets/files/html/3.0/photon_admin/adding-a-new-repository.html +[ubuntu-sources-doc]: https://manpages.ubuntu.com/manpages/focal/man5/sources.list.5.html +[default-args-ubuntu-2204-efi]: ../../packer-variables/ubuntu-2204-efi/default-args-ubuntu-2204-efi.j2 +[default-args-photon-5]: ../../packer-variables/photon-5/default-args-photon-5.j2 +[packer-variables]: ./../../packer-variables/ diff --git a/docs/examples/changing_hardware_version.md b/docs/examples/customizations/changing_hardware_version.md similarity index 66% rename from docs/examples/changing_hardware_version.md rename to docs/examples/customizations/changing_hardware_version.md index 9de5298..348040d 100644 --- a/docs/examples/changing_hardware_version.md +++ b/docs/examples/customizations/changing_hardware_version.md @@ -6,7 +6,7 @@ As a customer, I want to change the Hardware version of the node image to use th ## Background -By default, node images use the hardware version(`VMX`) 17. Please refer to the below documents to learn more about the hardware version and its compatibility with the vSphere environment. +By default, node images use the hardware version(`VMX`) 17 defined in [default-args.j2][default-args](Windows uses hardware version 18 by default defined in [default-args-windows.j2][default-args-windows]). Please refer to the below documents to learn more about the hardware version and its compatibility with the vSphere environment. - [Hardware Features Available with Virtual Machine Compatibility Settings](https://docs.vmware.com/en/VMware-vSphere/8.0/vsphere-vm-administration/GUID-789C3913-1053-4850-A0F0-E29C3D32B6DA.html) - [ESXi/ESX hosts and compatible virtual machine hardware versions list](https://kb.vmware.com/s/article/2007240) @@ -19,7 +19,10 @@ By default, node images use the hardware version(`VMX`) 17. Please refer to the "vmx_version": "17", ``` +- _**Note**: For Windows, update the `vmx_version` in the [default-args-windows.j2][default-args-windows] file._ + [//]: Links [default-args]: [./../../../packer-variables/default-args.j2] [kubernetes-image-builder]: https://github.com/kubernetes-sigs/image-builder/ +[default-args-windows]: ../../packer-variables/windows/default-args-windows.j2 diff --git a/docs/examples/prometheus_node_exporter.md b/docs/examples/customizations/prometheus_node_exporter.md similarity index 100% rename from docs/examples/prometheus_node_exporter.md rename to docs/examples/customizations/prometheus_node_exporter.md diff --git a/docs/examples/tutorial_building_an_image.md b/docs/examples/tutorial_building_an_image.md index 24173d4..a71c336 100644 --- a/docs/examples/tutorial_building_an_image.md +++ b/docs/examples/tutorial_building_an_image.md @@ -1,15 +1,14 @@ # Tutorial for Using the vSphere Tanzu Kubernetes Grid Image Builder -This tutorial describes how to use the vSphere Tanzu Kubernetes Grid Image Builder to build a custom TKR for use with TKG 2 on Supervisor in the vSphere with Tanzu environment. +This tutorial describes how to use the vSphere Tanzu Kubernetes Grid Image Builder to build a custom TKR for use on Supervisor in the vSphere. The vSphere Tanzu Kubernetes Grid Image Builder uses Hashicorp Packer to generate images. Packer invokes vCenter APIs to create a VM from a TKR. ## Requirements -- Check the [prerequisites](tkg#prerequisites) -- vCenter Server 8, which can be any vCenter 8 instance, it does not have to be the same vCenter managing your vSphere with Tanzu environment +- vCenter Server 8, which can be any vCenter 8 instance, it does not have to be the same vCenter managing your vSphere - Packer requires the vSphere environment to have DHCP configured; you cannot use static IP address management -- Tutorial uses Ubuntu v1.22 to generate the image +- Tutorial uses Ubuntu 22.04 based Linux VM to generate the image ## Clone the Repository @@ -23,43 +22,7 @@ git clone https://github.com/vmware-tanzu/vsphere-tanzu-kubernetes-grid-image-bu The vSphere Tanzu Kubernetes Grid Image Builder runs components as Docker images to generate VMs. -Docker requires a 64-bit system with a Linux kernel having version 3.10 or newer. - -Use the `uname` command to check the version of your kernel: - -```bash -uname -r -``` - -Docker provides a convenience script to install the latest version of Docker for testing, dev and lab purposes. - -```bash -wget -qO- https://get.docker.com/ | sh -``` - -Remove the need for sudo execution of docker commands. - -```bash -sudo usermod -aG docker $(whoami) -``` - -Log out of the shell and log back in. - -```bash -exit -``` - -Check if you can invoke the Docker client without sudo. - -```bash -docker --version -``` - -Verify that the Docker daemon (dockerd engine) is running using the system service interface system. - -```bash -systemctl status --full docker -``` +Refer to the [Docker installation guide][docker-installation] for setting up Docker Engine. ## Install JQ @@ -126,19 +89,23 @@ For example: {# TO create a clone of the Packer VM after customization#} "linked_clone": "true", {# To create a snapshot of the Packer VM after customization #} - "create_snapshot": "true" + "create_snapshot": "true", + {# To destroy Packer VM after Image Build is completed #} + "destroy": "true" } ``` ## Select Kubernetes version -From the `vsphere-tanzu-kubernetes-grid-image-builder` directory where the Makefile is located run: +To identify the version of Kubernetes Release to be used, from the `vsphere-tanzu-kubernetes-grid-image-builder` directory where the Makefile is located run: ```bash make list-versions ``` -## Run the Artifacts Container for the Selected Kubernetes Version +## Run the Artifacts Server Container for the Selected Kubernetes Version + +Running the `run-artifacts-container` Makefile target, will pull the Artifacts Server container image corresponding to the selected Kubernetes Release. Usage: @@ -149,7 +116,7 @@ make run-artifacts-container KUBERNETES_VERSION= Example: ```bash -make run-artifacts-container KUBERNETES_VERSION=v1.24.9+vmware.1 +make run-artifacts-container KUBERNETES_VERSION=v1.31.1+vmware.2-fips ``` ## Run the Image Builder Application @@ -157,7 +124,7 @@ make run-artifacts-container KUBERNETES_VERSION=v1.24.9+vmware.1 Usage: ```bash -make build-node-image OS_TARGET= KUBERNETES_VERSION=v1.24.9+vmware.1 TKR_SUFFIX= HOST_IP= IMAGE_ARTIFACTS_PATH= ARTIFACTS_CONTAINER_PORT=8081 +make build-node-image OS_TARGET= KUBERNETES_VERSION=v1.31.1+vmware.2-fips TKR_SUFFIX= HOST_IP= IMAGE_ARTIFACTS_PATH= ARTIFACTS_CONTAINER_PORT=8081 ``` NOTE: The HOST_IP must be reachable from the vCenter. @@ -165,7 +132,7 @@ NOTE: The HOST_IP must be reachable from the vCenter. Example: ```bash -make build-node-image OS_TARGET=ubuntu-2004-efi KUBERNETES_VERSION=v1.24.9+vmware.1 TKR_SUFFIX=byoi HOST_IP=192.2.2.3 IMAGE_ARTIFACTS_PATH=/home/ubuntu/image ARTIFACTS_CONTAINER_PORT=8081 +make build-node-image OS_TARGET=ubuntu-2004-efi KUBERNETES_VERSION=v1.31.1+vmware.2-fips TKR_SUFFIX=byoi HOST_IP=192.2.2.3 IMAGE_ARTIFACTS_PATH=/home/ubuntu/image ARTIFACTS_CONTAINER_PORT=8081 ``` ## Verify the Custom Image @@ -180,14 +147,19 @@ You should see the image being built in the datacenter, cluster, folder that you ## Customize the image -Refer to the customization examples. +Refer to the [customization examples][customizations]. -## Upload the Image to the TKG 2 with Supervisor Environment +## Upload the Image to generate custom Kubernetes Release Download the custom image from local storage or from the vCenter Server. -In your vSphere with Tanzu environment, create a local content library and upload the custom image there. +In your vSphere environment, create a local content library and upload the custom image there. -Refer to the documentation for [creating a local content library](https://docs.vmware.com/en/VMware-vSphere/8.0/vsphere-with-tanzu-tkg/GUID-19E8E034-5256-4EFC-BEBF-D4F17A8ED021.html) for use with TKG with Supervisor. +Refer to the documentation for [creating a local content library][create-local-content-library] for use with Supervisor. To use the custom TKR, configure the vSphere Namespace to use the local content library. + +[//]: Links +[docker-installation]: https://docs.docker.com/engine/install/ +[create-local-content-library]: https://docs.vmware.com/en/VMware-vSphere/8.0/vsphere-with-tanzu-tkg/GUID-19E8E034-5256-4EFC-BEBF-D4F17A8ED021.html +[customizations]: ./customizations/README.md diff --git a/docs/windows.md b/docs/windows.md new file mode 100644 index 0000000..25c5582 --- /dev/null +++ b/docs/windows.md @@ -0,0 +1,266 @@ +# Building Windows Image Using the vSphere Tanzu Kubernetes Grid Image Builder + +This tutorial describes how to use the vSphere Tanzu Kubernetes Grid Image Builder to build Windows OVA image for use with [vSphere Kubernetes Service 3.2](https://docs.vmware.com/en/VMware-vSphere/8/rn/vmware-tanzu-kubernetes-grid-service-release-notes/index.html) and above. + +## Use case + +I want to build a Windows Node Image to deploy a node pool for Windows container workloads in my guest cluster. + +## Requirements + +- Check the [prerequisites](https://docs.vmware.com/en/VMware-vSphere/8.0/vsphere-with-tanzu-installation-configuration/GUID-EE236215-DA4D-4579-8BEB-A693D1882C77.html) +- vCenter Server 8, which can be any vCenter 8 instance, it does not have to be the same vCenter managing your vSphere with Tanzu environment +- Packer requires the vSphere environment to have DHCP configured; you cannot use static IP address management +- A recent Windows Server 22H2 ISO image. Download through your Microsoft Developer Network (MSDN) or Volume Licensing (VL) account. The use of evaluation media is not supported or recommended. +- VMware Tools ISO Image +- A datastore on your vCenter that can accommodate your custom Windows VM template, which can have a starting size greater than 10GB (thin provisioned). + +## Prepare for Image Builder + +Follow the [standard tutorial](examples/tutorial_building_an_image.md) to prepare the environment for vSphere Tanzu Kubernetes Grid Image Builder. + +## Get Windows Server and VMware Tools ISO + +Windows Server 22H2 ISO image can be downloaded from Microsoft through your Microsoft Developer Network (MSDN) or Volume Licensing (VL) account. + +VMware Tools can be downloaded via the [Broadcom Knowledge Base](https://knowledge.broadcom.com/external/article/315363/how-to-install-vmware-tools.html). + +In this tutorial, we use Windows Server 22H2 `en-us_windows_server_2022_x64_dvd_620d7eac.iso` and `VMware-tools-windows-12.5.0-23800621.iso`. + +### Install govc (Optional) + +You may follow the [govc documentation](https://github.com/vmware/govmomi/blob/main/govc/README.md) to install govc on the Linux VM you're building the image. + +You may use the below example bash commands to upload Windows Server 22H2 ISO and the VMware Tools Windows ISO to your vCenter instance. + +```bash +export GOVC_URL=[VC_URL] +export GOVC_USERNAME=[VC_USERNAME] +export GOVC_PASSWORD=[VC_PASSWORD] +export GOVC_INSECURE=1 +export GOVC_DATACENTER=Datacenter +export GOVC_CLUSTER=Management-Cluster +export GOVC_DATASTORE=datastore22 + +govc datastore.upload --ds="$GOVC_DATASTORE" --dc="$GOVC_DATACENTER" en-us_windows_server_2022_x64_dvd_620d7eac.iso windows2022.iso +govc datastore.upload --ds="$GOVC_DATASTORE" --dc="$GOVC_DATACENTER" VMware-tools-windows-12.5.0-23800621.iso vmtools.iso +``` + +Alternatively, you may use the vCenter UI to upload the ISOs to the datastore. + +## Prepare Windows setup answer file + +You may customize the Windows Node Image with a [Windows setup answer file](https://learn.microsoft.com/en-us/windows-hardware/manufacture/desktop/update-windows-settings-and-scripts-create-your-own-answer-file-sxs?view=windows-11) + +The upstream Windows setup answer file can be found at [image builder](https://raw.githubusercontent.com/kubernetes-sigs/image-builder/refs/heads/main/images/capi/packer/ova/windows/windows-2022-efi/autounattend.xml). + +### Provision Administrative Account for Log Collection + +In order for the Windows nodes to work with the [vSphere Kubernetes Service support bundle tool](https://knowledge.broadcom.com/external/article/345464/gathering-logs-for-vsphere-with-tanzu.html), you need to add an administrative account in the answer file. + +The following snippet shows how to add an administrative account in the answer file. + +```bash +curl https://raw.githubusercontent.com/kubernetes-sigs/image-builder/refs/heads/main/images/capi/packer/ova/windows/windows-2022-efi/autounattend.xml -o /home/image-builder/windows_autounattend.xml +vi /home/image-builder/windows_autounattend.xml +``` + +Locate the `LocalAccounts` in the xml and add a new `LocalAccount` to this section. + +```xml + + + Administrator + Administrator + Administrators + Administrator + + + + MyAdminPassw0rd + true</PlainText> + </Password> + <Description>For log collection</Description> + <DisplayName>Admin Account</DisplayName> + <Name>WindowsAdmin</Name> + <Group>Administrators</Group> + </LocalAccount> +</LocalAccounts> +``` + +You should alter the user name and password to comform to organizational policies. + +## Update vsphere.j2 with vSphere Environment Details + +The `vsphere.j2` file is a packer configuration file with vSphere environment details. + +CD to the `vsphere-tanzu-kubernetes-grid-image-builder/packer-variables/` directory. + +Update the `vsphere.j2` and `packer-variables/windows/vsphere-windows.j2` environment variables with details for your vCenter 8 instance. + +```bash +$ vi vsphere.j2 + +{ + {# vCenter server IP or FQDN #} + "vcenter_server":"192.2.2.2", + {# vCenter username #} + "username":"user@vsphere.local", + {# vCenter user password #} + "password":"ADMIN-PASSWORD", + {# Datacenter name where packer creates the VM for customization #} + "datacenter":"Datacenter", + {# Datastore name for the VM #} + "datastore":"datastore22", + {# [Optional] Folder name #} + "folder":"", + {# Cluster name where packer creates the VM for customization #} + "cluster": "Management-Cluster", + {# Packer VM network #} + "network": "PG-MGMT-VLAN-1050", + {# To use insecure connection with vCenter #} + "insecure_connection": "true", + {# TO create a clone of the Packer VM after customization#} + "linked_clone": "true", + {# To create a snapshot of the Packer VM after customization #} + "create_snapshot": "true", + {# To destroy Packer VM after Image Build is completed #} + "destroy": "true" +} +``` + +```bash +vi packer-variables/windows/vsphere-windows.j2 + +{ + "os_iso_path": "[datastore22] windows2022.iso", + "vmtools_iso_path": "[datastore22] vmtools.iso" +} +``` + +NOTE: You need to match the ISO image file names in the datastore. + +## Select Kubernetes version + +From the `vsphere-tanzu-kubernetes-grid-image-builder` directory where the Makefile is located run: + +```bash +make list-versions +``` + +Windows container workload support is only available in Kubernetes release v1.31 and above. + +## Run the Artifacts Container for the Selected Kubernetes Version + +Usage: + +```bash +make run-artifacts-container KUBERNETES_VERSION=<version> +``` + +Example: + +```bash +make run-artifacts-container KUBERNETES_VERSION=v1.31.1+vmware.2-fips +``` + +## Run the Image Builder Application + +Usage: + +```bash +make build-node-image OS_TARGET=<os_target> KUBERNETES_VERSION=v1.31.1+vmware.2-fips TKR_SUFFIX=<tkr_suffix> HOST_IP=<host_ip> IMAGE_ARTIFACTS_PATH=<image_artifacts_path> ARTIFACTS_CONTAINER_PORT=8081 +``` + +NOTE: + +- The HOST_IP must be reachable from the vCenter. + +- You need to use the Kubernetes version which supports Windows. You may check Windows is in the Supported OS from +the result from `make list-versions`. + +- You may list the Kubernetes in your Supervisor cluster to get the version suffix. + +```bash +$ kubectl get kr + +NAME VERSION READY COMPATIBLE CREATED TYPE +kubernetesrelease.kubernetes.vmware.com/v1.31.1---vmware.2-fips-vkr.2 v1.31.1+vmware.2-fips-vkr.2 True True 3h8m +``` + +Example: + +```bash +make build-node-image OS_TARGET=windows-2022-efi KUBERNETES_VERSION=v1.31.1+vmware.2-fips TKR_SUFFIX=vkr.2 HOST_IP=192.2.2.3 IMAGE_ARTIFACTS_PATH=/home/image-builder/image ARTIFACTS_CONTAINER_PORT=8081 AUTO_UNATTEND_ANSWER_FILE_PATH=/home/image-builder/windows_autounattend.xml +``` + +## Verify the Custom Image + +Locally the image is stored in the `/image/ovas` directory. For example, `/home/image-builder/image/ovas`. + +The `/image/logs` directory contains the `packer-xxxx.log` file that you can use to troubleshoot image building errors. + +To verify that image is built successfully, check vCenter Server. + +You should see the image being built in the datacenter, cluster, folder that you specified in the vsphere.j2 file. + +## Upload the Image to the vSphere Kubernetes Service Environment + +Download the custom image from local storage or from the vCenter Server. + +In your vSphere with Tanzu environment, create a local content library and upload the custom image there. + +Refer to the documentation for [creating a local content library](https://docs.vmware.com/en/VMware-vSphere/8.0/vsphere-with-tanzu-tkg/GUID-19E8E034-5256-4EFC-BEBF-D4F17A8ED021.html) for use with vSphere Kubernetes Service. + +You need to upload both Linux and Windows node images to the local content library as the Linux node image will be +used to deploy VMs for Kubernetes Control Plane and Linux node pools (if any). + +Note: You should disable Security Policy for this content library for Windows image. + +## Create a cluster with Windows Node Pool + +You may refer to [vSphere Kubernetes Service 3.2 documentation]() for more information on how to deploy a cluster with Windows Node Pool with vSphere Kubernetes Service 3.2 and above. + +## Known Issues for Windows Container Workload Support + +### Kubernetes Release v1.31 + +- The minimum vmclass should be best-effort-large for Windows Worker Node + + When a windows worker node is configured with a vm-class which has resource configuration lower than best-effort-large, some of the management pods may not run due to loss of network connectivity. + + Resolution: + Switch to a vmclass configured with more resources. + +- Some Pods on Window Node's networking don't work correctly + + Some Pods on Window Node's networking don’t work correctly, which makes the Pod is not reachable, or the Pod can’t access other network peers. + + If searching in antrea-agent log from the corresponding antrea-agent Pod which locates on the same Node as the bad Pod, some logs records shall be found with the key words “Failed to execute postInterfaceCreateHook”, e.g., + + ```bash + kubectl logs -n kube-system -c antrea-agent antrea-agent-windows-stvgp | grep "Failed to execute postInterfaceCreateHook" + + E0921 04:38:39.289057 4364 interface_configuration_windows.go:488] "Failed to execute postInterfaceCreateHook" err="timed out: \"wait\" timed out after 5012 ms" interface="vEthernet (vsphere--ab7002)" + ``` + + If checking OVS port status, we may observe that a port is configured with error “could not add network device xxxxx to ofproto (Invalid argument)”, e.g., + + ```bash + kubectl exec -it -n kube-system antrea-agent-windows-znncw -- cmd.exe /c "C:\openvswitch\usr\bin\ovs-vsctl.exe show" + 3cc1a6d5-adc1-45d3-a336-c3c2b8203e77 + Bridge br-int + datapath_type: system + Port vsphere--c546b0 + Interface vsphere--c546b0 + type: internal + error: "could not add network device vsphere--c546b0 to ofproto (Invalid argument)" + ovs_version: "3.0.1.60555" + ``` + + Workaround: + Kill the bad Pod using kubectl command to reschedule the Pod so that antrea-agent can re-program the networking for the Pod. + +### Generic Known Issues + +You may refer to [this link](https://docs.vmware.com/en/VMware-Tanzu-Kubernetes-releases/services/rn/vmware-tanzu-kubernetes-releases-release-notes/index.html) for generic known issues for vSphere Kubernetes Service. diff --git a/supported-versions.json b/supported-versions.json index 44353f2..f72ff66 100644 --- a/supported-versions.json +++ b/supported-versions.json @@ -48,5 +48,16 @@ "docker_build_args": { "IMAGE_BUILDER_COMMIT_ID": "ed30bc3e7b71ea41d2f79a16789d2b291d9f43d1" } + }, + "v1.31.1+vmware.2-fips": { + "supported_os": [ + "photon-5", + "ubuntu-2204-efi", + "windows-2022-efi" + ], + "artifacts_image": "projects.packages.broadcom.com/vsphere/iaas/kubernetes-release/1.31.1/vkr-artifact-server:v1.31.1_vmware.2-fips-vkr.2", + "docker_build_args": { + "IMAGE_BUILDER_COMMIT_ID": "ed30bc3e7b71ea41d2f79a16789d2b291d9f43d1" + } } }