From 106787d9a081bfea257146bcb1d6716325dbcaf0 Mon Sep 17 00:00:00 2001 From: Sriraman S Date: Thu, 3 Oct 2024 12:13:47 +0530 Subject: [PATCH 1/4] =?UTF-8?q?feat:=20=F0=9F=8E=B8=20Add=20support=20for?= =?UTF-8?q?=20v1.31.1=20Kubernetes=20Release?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- supported-versions.json | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/supported-versions.json b/supported-versions.json index 44353f2..f72ff66 100644 --- a/supported-versions.json +++ b/supported-versions.json @@ -48,5 +48,16 @@ "docker_build_args": { "IMAGE_BUILDER_COMMIT_ID": "ed30bc3e7b71ea41d2f79a16789d2b291d9f43d1" } + }, + "v1.31.1+vmware.2-fips": { + "supported_os": [ + "photon-5", + "ubuntu-2204-efi", + "windows-2022-efi" + ], + "artifacts_image": "projects.packages.broadcom.com/vsphere/iaas/kubernetes-release/1.31.1/vkr-artifact-server:v1.31.1_vmware.2-fips-vkr.2", + "docker_build_args": { + "IMAGE_BUILDER_COMMIT_ID": "ed30bc3e7b71ea41d2f79a16789d2b291d9f43d1" + } } } From e33510abdef5f9c93dbeec682e8ae2e389e7655b Mon Sep 17 00:00:00 2001 From: Sriraman S Date: Thu, 3 Oct 2024 12:14:24 +0530 Subject: [PATCH 2/4] Add BYOI doc for Windows --- .gitignore | 3 +- README.md | 2 + docs/windows.md | 262 ++++++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 266 insertions(+), 1 deletion(-) create mode 100644 docs/windows.md diff --git a/.gitignore b/.gitignore index 7d9713f..a4d3524 100644 --- a/.gitignore +++ b/.gitignore @@ -2,4 +2,5 @@ .vscode !.gitignore scripts/__pycache__ -output* \ No newline at end of file +output* +.idea \ No newline at end of file diff --git a/README.md b/README.md index 625186c..4018323 100644 --- a/README.md +++ b/README.md @@ -126,6 +126,8 @@ make build-node-image OS_TARGET=photon-3 KUBERNETES_VERSION=v1.23.15+vmware.1 TK Sample customization examples can be found [here](docs/examples/README.md) +For Windows support you may refer to [Windows tutorial](docs/windows.md) + ## Debugging - To enable debugging for the [make file scripts](hack/make-helpers/) export `DEBUGGING=true`. diff --git a/docs/windows.md b/docs/windows.md new file mode 100644 index 0000000..5fb6e8b --- /dev/null +++ b/docs/windows.md @@ -0,0 +1,262 @@ +# Building Windows Image Using the vSphere Tanzu Kubernetes Grid Image Builder + +This tutorial describes how to use the vSphere Tanzu Kubernetes Grid Image Builder to build Windows OVA image for use with [vSphere Kubernetes Service 3.2](https://docs.vmware.com/en/VMware-vSphere/8/rn/vmware-tanzu-kubernetes-grid-service-release-notes/index.html) and above. + +## Use case + +I want to build a Windows Node Image to deploy a node pool for Windows container workloads in my guest cluster. + +## Requirements + +- Check the [prerequisites](https://docs.vmware.com/en/VMware-vSphere/8.0/vsphere-with-tanzu-installation-configuration/GUID-EE236215-DA4D-4579-8BEB-A693D1882C77.html) +- vCenter Server 8, which can be any vCenter 8 instance, it does not have to be the same vCenter managing your vSphere with Tanzu environment +- Packer requires the vSphere environment to have DHCP configured; you cannot use static IP address management +- A recent Windows Server 22H2 ISO image. Download through your Microsoft Developer Network (MSDN) or Volume Licensing (VL) account. The use of evaluation media is not supported or recommended. +- VMware Tools ISO Image +- A datastore on your vCenter that can accommodate your custom Windows VM template, which can have a starting size greater than 10GB (thin provisioned). + +## Prepare for Image Builder + +Follow the [standard tutorial](examples/tutorial_building_an_image.md) to prepare the environment for vSphere Tanzu Kubernetes Grid Image Builder. + +## Get Windows Server and VMware Tools ISO + +Windows Server 22H2 ISO image can be downloaded from Microsoft through your Microsoft Developer Network (MSDN) or Volume Licensing (VL) account. + +VMware Tools can be downloaded via the [Broadcom Knowledge Base](https://knowledge.broadcom.com/external/article/315363/how-to-install-vmware-tools.html). + +In this tutorial, we use Windows Server 22H2 `en-us_windows_server_2022_x64_dvd_620d7eac.iso` and `VMware-tools-windows-12.5.0-23800621.iso`. + +### Install govc (Optional) + +You may follow the [govc documentation](https://github.com/vmware/govmomi/blob/main/govc/README.md) +to install govc on the Linux VM you're building the image. + +```bash +$ export GOVC_URL=[VC_URL] +$ export GOVC_USERNAME=[VC_USERNAME] +$ export GOVC_PASSWORD=[VC_PASSWORD] +$ export GOVC_INSECURE=1 +$ export GOVC_DATACENTER=Datacenter +$ export GOVC_CLUSTER=Management-Cluster +$ export GOVC_DATASTORE=datastore22 + +$ govc datastore.upload --ds="$GOVC_DATASTORE" --dc="$GOVC_DATACENTER" en-us_windows_server_2022_x64_dvd_620d7eac.iso windows2022.iso +$ govc datastore.upload --ds="$GOVC_DATASTORE" --dc="$GOVC_DATACENTER" VMware-tools-windows-12.5.0-23800621.iso vmtools.iso +``` + +Alternatively, you may use the vCenter UI to upload the ISOs to the datastore. + +## Prepare Windows setup answer file + +You may customize the Windows Node Image with a [Windows setup answer file](https://learn.microsoft.com/en-us/windows-hardware/manufacture/desktop/update-windows-settings-and-scripts-create-your-own-answer-file-sxs?view=windows-11) + +The upstream Windows setup answer file can be found at [image builder](https://raw.githubusercontent.com/kubernetes-sigs/image-builder/refs/heads/main/images/capi/packer/ova/windows/windows-2022-efi/autounattend.xml). + +### Provision Administrative Account for Log Collection + +In order for the Windows nodes to work with the [vSphere Kubernetes Service support bundle tool](https://knowledge.broadcom.com/external/article/345464/gathering-logs-for-vsphere-with-tanzu.html), you need to add an administrative account in the answer file. + +The following snippet shows how to add an administrative account in the answer file. + +```bash +$ curl https://raw.githubusercontent.com/kubernetes-sigs/image-builder/refs/heads/main/images/capi/packer/ova/windows/windows-2022-efi/autounattend.xml -o /home/image-builder/windows_autounattend.xml +$ vi /home/image-builder/windows_autounattend.xml +``` + +Locate the `LocalAccounts` in the xml and add a new `LocalAccount` to this section. + +```xml + + + Administrator + Administrator + Administrators + Administrator + + + + MyAdminPassw0rd + true</PlainText> + </Password> + <Description>For log collection</Description> + <DisplayName>Admin Account</DisplayName> + <Name>WindowsAdmin</Name> + <Group>Administrators</Group> + </LocalAccount> +</LocalAccounts> +``` + +You should alter the user name and password to comform to organizational policies. + +## Update vsphere.j2 with vSphere Environment Details + +The `vsphere.j2` file is a packer configuration file with vSphere environment details. + +CD to the `vsphere-tanzu-kubernetes-grid-image-builder/packer-variables/` directory. + +Update the `vsphere.j2` and `packer-variables/windows/vsphere-windows.j2` environment variables with details for your vCenter 8 instance. + +```bash +$ vi vsphere.j2 + +{ + {# vCenter server IP or FQDN #} + "vcenter_server":"192.2.2.2", + {# vCenter username #} + "username":"user@vsphere.local", + {# vCenter user password #} + "password":"ADMIN-PASSWORD", + {# Datacenter name where packer creates the VM for customization #} + "datacenter":"Datacenter", + {# Datastore name for the VM #} + "datastore":"datastore22", + {# [Optional] Folder name #} + "folder":"", + {# Cluster name where packer creates the VM for customization #} + "cluster": "Management-Cluster", + {# Packer VM network #} + "network": "PG-MGMT-VLAN-1050", + {# To use insecure connection with vCenter #} + "insecure_connection": "true", + {# TO create a clone of the Packer VM after customization#} + "linked_clone": "true", + {# To create a snapshot of the Packer VM after customization #} + "create_snapshot": "true", + {# To destroy Packer VM after Image Build is completed #} + "destroy": "true" +} +``` + +```bash +vi packer-variables/windows/vsphere-windows.j2 + +{ + "os_iso_path": "[datastore22] windows2022.iso", + "vmtools_iso_path": "[datastore22] vmtools.iso" +} +``` + +NOTE: You need to match the ISO image file names in the datastore. + +## Select Kubernetes version + +From the `vsphere-tanzu-kubernetes-grid-image-builder` directory where the Makefile is located run: + +```bash +$ make list-versions +``` + +Windows container workload support is only available in Kubernetes release v1.31 and above. + +## Run the Artifacts Container for the Selected Kubernetes Version + +Usage: + +```bash +$ make run-artifacts-container KUBERNETES_VERSION=<version> +``` + +Example: + +```bash +$ make run-artifacts-container KUBERNETES_VERSION=v1.31.1+vmware.2-fips +``` + +## Run the Image Builder Application + +Usage: + +```bash +$ make build-node-image OS_TARGET=<os_target> KUBERNETES_VERSION=v1.31.1+vmware.2-fips TKR_SUFFIX=<tkr_suffix> HOST_IP=<host_ip> IMAGE_ARTIFACTS_PATH=<image_artifacts_path> ARTIFACTS_CONTAINER_PORT=8081 +``` + +NOTE: +* The HOST_IP must be reachable from the vCenter. +* You need to use the Kubernetes version which supports Windows. You may check Windows is in the Supported OS from +the result from `make list-versions`. +* You may list the Kubernetes in your Supervisor cluster to get the version suffix. + +```bash +$ kubectl get kr + +NAME VERSION READY COMPATIBLE CREATED TYPE +kubernetesrelease.kubernetes.vmware.com/v1.31.1---vmware.2-fips-vkr.2 v1.31.1+vmware.2-fips-vkr.2 True True 3h8m +``` + +Example: + +```bash +$ make build-node-image OS_TARGET=windows-2022-efi KUBERNETES_VERSION=v1.31.1+vmware.2-fips TKR_SUFFIX=vkr.2 HOST_IP=192.2.2.3 IMAGE_ARTIFACTS_PATH=/home/image-builder/image ARTIFACTS_CONTAINER_PORT=8081 AUTO_UNATTEND_ANSWER_FILE_PATH=/home/image-builder/windows_autounattend.xml +``` + +## Verify the Custom Image + +Locally the image is stored in the `/image/ovas` directory. For example, `/home/image-builder/image/ovas`. + +The `/image/logs` directory contains the `packer-xxxx.log` file that you can use to troubleshoot image building errors. + +To verify that image is built successfully, check vCenter Server. + +You should see the image being built in the datacenter, cluster, folder that you specified in the vsphere.j2 file. + +## Upload the Image to the vSphere Kubernetes Service Environment + +Download the custom image from local storage or from the vCenter Server. + +In your vSphere with Tanzu environment, create a local content library and upload the custom image there. + +Refer to the documentation for [creating a local content library](https://docs.vmware.com/en/VMware-vSphere/8.0/vsphere-with-tanzu-tkg/GUID-19E8E034-5256-4EFC-BEBF-D4F17A8ED021.html) for use with vSphere Kubernetes Service. + +You need to upload both Linux and Windows node images to the local content library as the Linux node image will be +used to deploy VMs for Kubernetes Control Plane and Linux node pools (if any). + +Note: You should disable Security Policy for this content library for Windows image. + +## Create a cluster with Windows Node Pool + +You may refer to [vSphere Kubernetes Service 3.2 documentation]() for more information on how to deploy a cluster with Windows Node Pool with vSphere Kubernetes Service 3.2 and above. + +## Known Issues for Windows Container Workload Support + +### Kubernetes Release v1.31 + +* The minimum vmclass should be best-effort-large for Windows Worker Node + + When a windows worker node is configured with a vm-class which has resource configuration lower than best-effort-large, some of the management pods may not run due to loss of network connectivity. + + Resolution: + Switch to a vmclass configured with more resources. + +* Some Pods on Window Node's networking don't work correctly + + Some Pods on Window Node's networking don’t work correctly, which makes the Pod is not reachable, or the Pod can’t access other network peers. + + If searching in antrea-agent log from the corresponding antrea-agent Pod which locates on the same Node as the bad Pod, some logs records shall be found with the key words “Failed to execute postInterfaceCreateHook”, e.g., + + ```bash + kubectl logs -n kube-system -c antrea-agent antrea-agent-windows-stvgp | grep "Failed to execute postInterfaceCreateHook" + + E0921 04:38:39.289057 4364 interface_configuration_windows.go:488] "Failed to execute postInterfaceCreateHook" err="timed out: \"wait\" timed out after 5012 ms" interface="vEthernet (vsphere--ab7002)" + ``` + + If checking OVS port status, we may observe that a port is configured with error “could not add network device xxxxx to ofproto (Invalid argument)”, e.g., + + ```bash + kubectl exec -it -n kube-system antrea-agent-windows-znncw -- cmd.exe /c "C:\openvswitch\usr\bin\ovs-vsctl.exe show" + 3cc1a6d5-adc1-45d3-a336-c3c2b8203e77 + Bridge br-int + datapath_type: system + Port vsphere--c546b0 + Interface vsphere--c546b0 + type: internal + error: "could not add network device vsphere--c546b0 to ofproto (Invalid argument)" + ovs_version: "3.0.1.60555" + ``` + + Workaround: + Kill the bad Pod using kubectl command to reschedule the Pod so that antrea-agent can re-program the networking for the Pod. + +### Generic Known Issues + +You may refer to [this link](https://docs.vmware.com/en/VMware-Tanzu-Kubernetes-releases/services/rn/vmware-tanzu-kubernetes-releases-release-notes/index.html) for generic known issues for vSphere Kubernetes Service. From 5e2492c7bbb98c03358cbe1fc0bab428e2d8e6f6 Mon Sep 17 00:00:00 2001 From: Sriraman S <sriraman.dev@gmail.com> Date: Thu, 3 Oct 2024 15:12:59 +0530 Subject: [PATCH 3/4] =?UTF-8?q?docs:=20=E2=9C=8F=EF=B8=8F=20Update=20docs?= =?UTF-8?q?=20to=20incorporate=20newly=20introduced=20changes?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docs/examples/adding_os_pkg_repos.md | 93 ++++++++++++---------- docs/examples/changing_hardware_version.md | 5 +- docs/windows.md | 54 +++++++------ 3 files changed, 82 insertions(+), 70 deletions(-) diff --git a/docs/examples/adding_os_pkg_repos.md b/docs/examples/adding_os_pkg_repos.md index 2cf5f3f..3d7b01b 100644 --- a/docs/examples/adding_os_pkg_repos.md +++ b/docs/examples/adding_os_pkg_repos.md @@ -13,66 +13,69 @@ Configuration of OS packages and sources/repositories is exposed as packer varia ### Adding new packages -To add new packages [kubernetes image builder][kubernetes-image-builder] provides `extra_rpms` and `extra_debs` packer variables for Photon and Ubuntu respectively. To add new packages, add the packages to the packer variables in the [default-args.j2][default-args] file as shown below. +To add new packages [kubernetes image builder][kubernetes-image-builder] provides `extra_rpms` and `extra_debs` packer variables for Photon and Ubuntu respectively. -_**Note**: If you are building just the Ubuntu OS node image you will not have to modify the photon OS block._ +To add new packages to Ubuntu 22.04, add the packages to the packer variables in the [default-args-ubuntu-2204-efi.j2][default-args-ubuntu-2204-efi] file as shown below. + +```jinja + "extra_debs": "existing_packages pkg1 pkg2" +``` + +To add new packages to Photon 5, add the packages to the packer variables in the [default-args-photon-5.j2][default-args-photon-5] file as shown below. ```jinja - {% if os_type == "photon-3" %} - "distro_version": "3.0", "extra_rpms": "existing_packages pkg1 pkg2" - {% elif os_type == "ubuntu-2004-efi" %} - "extra_debs": "existing_packages pkg1 pkg2", - "boot_disable_ipv6": "1" - {% endif %} ``` +_**Note**: The location of default configuration specific to OS follows the path nomenclature, `packer-variables/<OS>-<Version>/default-args-<OS>-<version>.j2`. Include `-efi` suffix as well along with the version for Ubuntu._ + ### Configuring new sources/repositories [kubernetes image builder][kubernetes-image-builder] provides `extra_repos` packer variables through which sources/repositories can be configured for both Photon and Ubuntu. As there is a difference in how Ubuntu/Photon sources are configured we need to have separate source files for Photon/Ubuntu. - Create new folder `repos` in [ansible files][ansible-files] folder -- Create a new file for Photon sources called `photon.repo` in the `repos` folder. Refer below for sample content and refer to the official Photon [document][photon-repo-doc] for more information - -```text -[photon] -name=VMware Photon Linux $releasever ($basearch) -baseurl=<internal_mirror_url>/$releasever/photon_release_$releasever_$basearch -gpgkey=<gpg_key> -gpgcheck=1 -enabled=1 - -[photon-updates] -name=VMware Photon Linux $releasever ($basearch) Updates -baseurl=<internal_mirror_url>/$releasever/photon_updates_$releasever_$basearch -gpgkey=<gpg_key> -gpgcheck=1 -enabled=1 - -[photon-extras] -name=VMware Photon Extras $releasever ($basearch) -baseurl=<internal_mirror_url>/$releasever/photon_extras_$releasever_$basearch -gpgkey=<gpg_key> -gpgcheck=1 -enabled=1 -``` - -- Create a new file for Ubuntu sources called `ubuntu.list` in the `repos` folder. Refer to the official ubuntu [documentation][ubuntu-sources-doc] for more information. - - _**Note**: `focal` is for ubuntu 20.04 so this needs to be changed if the ubuntu version is also changed example for ubuntu 22.04 it is `jammy`_ +- Depending upon the Linux OS flavour, use either of the below steps + - For Photon sources, create a new file called `photon.repo` in the `repos` folder. Refer below for sample content and refer to the official Photon [document][photon-repo-doc] for more information. + + ```text + [photon] + name=VMware Photon Linux $releasever ($basearch) + baseurl=<internal_mirror_url>/$releasever/photon_release_$releasever_$basearch + gpgkey=<gpg_key> + gpgcheck=1 + enabled=1 + + [photon-updates] + name=VMware Photon Linux $releasever ($basearch) Updates + baseurl=<internal_mirror_url>/$releasever/photon_updates_$releasever_$basearch + gpgkey=<gpg_key> + gpgcheck=1 + enabled=1 + + [photon-extras] + name=VMware Photon Extras $releasever ($basearch) + baseurl=<internal_mirror_url>/$releasever/photon_extras_$releasever_$basearch + gpgkey=<gpg_key> + gpgcheck=1 + enabled=1 + ``` + + - For Ubuntu sources, create a new file called `ubuntu.list` in the `repos` folder. Refer to the official ubuntu [documentation][ubuntu-sources-doc] for more information. + - _**Note**: `jammy` is for ubuntu 22.04 so this needs to be changed if the ubuntu version is also changed, example for ubuntu 20.04 it is `focal`_ ```text -deb <internal_source_url> focal main restricted universe -deb <internal_source_url> focal-security main restricted -deb <internal_source_url> focal-updates main restricted +deb <internal_source_url> jammy main restricted universe +deb <internal_source_url> jammy-security main restricted +deb <internal_source_url> jammy-updates main restricted ``` -- Create a new file `repos.j2` or `repos.json` in [packer-variables](./../../packer-variables/) folder for configuring the `extra_repos` packer variable folder. (Uses [jinja][jinja] templating) +- Create a new file `repos.j2` in [packer-variables][packer-variables] folder for configuring the `extra_repos` packer variable folder. (Uses [jinja][jinja] templating) ```jinja { - {% if os_type == "photon-3" %} + {% if os_type == "photon-5" %} "extra_repos": "/image-builder/images/capi/image/ansible/files/repos/photon.repo" - {% elif os_type == "ubuntu-2004-efi" %} + {% elif os_type == "ubuntu-2204-efi" %} "extra_repos": "/image-builder/images/capi/image/ansible/files/repos/ubuntu.list" {% endif %} } @@ -90,9 +93,9 @@ To remove the extra repositories/sources that were configured during the image b { "disable_public_repos": true, "remove_extra_repos": true - {% if os_type == "photon-3" %} + {% if os_type == "photon-5" %} "extra_repos": "/image-builder/images/capi/image/ansible/files/repos/photon.repo" - {% elif os_type == "ubuntu-2004-efi" %} + {% elif os_type == "ubuntu-2204-efi" %} "extra_repos": "/image-builder/images/capi/image/ansible/files/repos/ubuntu.list" {% endif %} } @@ -102,8 +105,10 @@ To remove the extra repositories/sources that were configured during the image b [ansible-files]: ./../../ansible/files/ [customizations-doc]: https://image-builder.sigs.k8s.io/capi/capi.html#customization -[default-args]: ./../../packer-variables/default-args.j2 [jinja]: https://jinja.palletsprojects.com/en/3.1.x/ [kubernetes-image-builder]: https://github.com/kubernetes-sigs/image-builder/ [photon-repo-doc]: https://vmware.github.io/photon/assets/files/html/3.0/photon_admin/adding-a-new-repository.html [ubuntu-sources-doc]: https://manpages.ubuntu.com/manpages/focal/man5/sources.list.5.html +[default-args-ubuntu-2204-efi]: ../../packer-variables/ubuntu-2204-efi/default-args-ubuntu-2204-efi.j2 +[default-args-photon-5]: ../../packer-variables/photon-5/default-args-photon-5.j2 +[packer-variables]: ./../../packer-variables/ diff --git a/docs/examples/changing_hardware_version.md b/docs/examples/changing_hardware_version.md index 9de5298..348040d 100644 --- a/docs/examples/changing_hardware_version.md +++ b/docs/examples/changing_hardware_version.md @@ -6,7 +6,7 @@ As a customer, I want to change the Hardware version of the node image to use th ## Background -By default, node images use the hardware version(`VMX`) 17. Please refer to the below documents to learn more about the hardware version and its compatibility with the vSphere environment. +By default, node images use the hardware version(`VMX`) 17 defined in [default-args.j2][default-args](Windows uses hardware version 18 by default defined in [default-args-windows.j2][default-args-windows]). Please refer to the below documents to learn more about the hardware version and its compatibility with the vSphere environment. - [Hardware Features Available with Virtual Machine Compatibility Settings](https://docs.vmware.com/en/VMware-vSphere/8.0/vsphere-vm-administration/GUID-789C3913-1053-4850-A0F0-E29C3D32B6DA.html) - [ESXi/ESX hosts and compatible virtual machine hardware versions list](https://kb.vmware.com/s/article/2007240) @@ -19,7 +19,10 @@ By default, node images use the hardware version(`VMX`) 17. Please refer to the "vmx_version": "17", ``` +- _**Note**: For Windows, update the `vmx_version` in the [default-args-windows.j2][default-args-windows] file._ + [//]: Links [default-args]: [./../../../packer-variables/default-args.j2] [kubernetes-image-builder]: https://github.com/kubernetes-sigs/image-builder/ +[default-args-windows]: ../../packer-variables/windows/default-args-windows.j2 diff --git a/docs/windows.md b/docs/windows.md index 5fb6e8b..25c5582 100644 --- a/docs/windows.md +++ b/docs/windows.md @@ -29,20 +29,21 @@ In this tutorial, we use Windows Server 22H2 `en-us_windows_server_2022_x64_dvd_ ### Install govc (Optional) -You may follow the [govc documentation](https://github.com/vmware/govmomi/blob/main/govc/README.md) -to install govc on the Linux VM you're building the image. +You may follow the [govc documentation](https://github.com/vmware/govmomi/blob/main/govc/README.md) to install govc on the Linux VM you're building the image. + +You may use the below example bash commands to upload Windows Server 22H2 ISO and the VMware Tools Windows ISO to your vCenter instance. ```bash -$ export GOVC_URL=[VC_URL] -$ export GOVC_USERNAME=[VC_USERNAME] -$ export GOVC_PASSWORD=[VC_PASSWORD] -$ export GOVC_INSECURE=1 -$ export GOVC_DATACENTER=Datacenter -$ export GOVC_CLUSTER=Management-Cluster -$ export GOVC_DATASTORE=datastore22 - -$ govc datastore.upload --ds="$GOVC_DATASTORE" --dc="$GOVC_DATACENTER" en-us_windows_server_2022_x64_dvd_620d7eac.iso windows2022.iso -$ govc datastore.upload --ds="$GOVC_DATASTORE" --dc="$GOVC_DATACENTER" VMware-tools-windows-12.5.0-23800621.iso vmtools.iso +export GOVC_URL=[VC_URL] +export GOVC_USERNAME=[VC_USERNAME] +export GOVC_PASSWORD=[VC_PASSWORD] +export GOVC_INSECURE=1 +export GOVC_DATACENTER=Datacenter +export GOVC_CLUSTER=Management-Cluster +export GOVC_DATASTORE=datastore22 + +govc datastore.upload --ds="$GOVC_DATASTORE" --dc="$GOVC_DATACENTER" en-us_windows_server_2022_x64_dvd_620d7eac.iso windows2022.iso +govc datastore.upload --ds="$GOVC_DATASTORE" --dc="$GOVC_DATACENTER" VMware-tools-windows-12.5.0-23800621.iso vmtools.iso ``` Alternatively, you may use the vCenter UI to upload the ISOs to the datastore. @@ -60,8 +61,8 @@ In order for the Windows nodes to work with the [vSphere Kubernetes Service supp The following snippet shows how to add an administrative account in the answer file. ```bash -$ curl https://raw.githubusercontent.com/kubernetes-sigs/image-builder/refs/heads/main/images/capi/packer/ova/windows/windows-2022-efi/autounattend.xml -o /home/image-builder/windows_autounattend.xml -$ vi /home/image-builder/windows_autounattend.xml +curl https://raw.githubusercontent.com/kubernetes-sigs/image-builder/refs/heads/main/images/capi/packer/ova/windows/windows-2022-efi/autounattend.xml -o /home/image-builder/windows_autounattend.xml +vi /home/image-builder/windows_autounattend.xml ``` Locate the `LocalAccounts` in the xml and add a new `LocalAccount` to this section. @@ -144,7 +145,7 @@ NOTE: You need to match the ISO image file names in the datastore. From the `vsphere-tanzu-kubernetes-grid-image-builder` directory where the Makefile is located run: ```bash -$ make list-versions +make list-versions ``` Windows container workload support is only available in Kubernetes release v1.31 and above. @@ -154,13 +155,13 @@ Windows container workload support is only available in Kubernetes release v1.31 Usage: ```bash -$ make run-artifacts-container KUBERNETES_VERSION=<version> +make run-artifacts-container KUBERNETES_VERSION=<version> ``` Example: ```bash -$ make run-artifacts-container KUBERNETES_VERSION=v1.31.1+vmware.2-fips +make run-artifacts-container KUBERNETES_VERSION=v1.31.1+vmware.2-fips ``` ## Run the Image Builder Application @@ -168,14 +169,17 @@ $ make run-artifacts-container KUBERNETES_VERSION=v1.31.1+vmware.2-fips Usage: ```bash -$ make build-node-image OS_TARGET=<os_target> KUBERNETES_VERSION=v1.31.1+vmware.2-fips TKR_SUFFIX=<tkr_suffix> HOST_IP=<host_ip> IMAGE_ARTIFACTS_PATH=<image_artifacts_path> ARTIFACTS_CONTAINER_PORT=8081 +make build-node-image OS_TARGET=<os_target> KUBERNETES_VERSION=v1.31.1+vmware.2-fips TKR_SUFFIX=<tkr_suffix> HOST_IP=<host_ip> IMAGE_ARTIFACTS_PATH=<image_artifacts_path> ARTIFACTS_CONTAINER_PORT=8081 ``` NOTE: -* The HOST_IP must be reachable from the vCenter. -* You need to use the Kubernetes version which supports Windows. You may check Windows is in the Supported OS from + +- The HOST_IP must be reachable from the vCenter. + +- You need to use the Kubernetes version which supports Windows. You may check Windows is in the Supported OS from the result from `make list-versions`. -* You may list the Kubernetes in your Supervisor cluster to get the version suffix. + +- You may list the Kubernetes in your Supervisor cluster to get the version suffix. ```bash $ kubectl get kr @@ -187,7 +191,7 @@ kubernetesrelease.kubernetes.vmware.com/v1.31.1---vmware.2-fips-vkr.2 v1.31.1+ Example: ```bash -$ make build-node-image OS_TARGET=windows-2022-efi KUBERNETES_VERSION=v1.31.1+vmware.2-fips TKR_SUFFIX=vkr.2 HOST_IP=192.2.2.3 IMAGE_ARTIFACTS_PATH=/home/image-builder/image ARTIFACTS_CONTAINER_PORT=8081 AUTO_UNATTEND_ANSWER_FILE_PATH=/home/image-builder/windows_autounattend.xml +make build-node-image OS_TARGET=windows-2022-efi KUBERNETES_VERSION=v1.31.1+vmware.2-fips TKR_SUFFIX=vkr.2 HOST_IP=192.2.2.3 IMAGE_ARTIFACTS_PATH=/home/image-builder/image ARTIFACTS_CONTAINER_PORT=8081 AUTO_UNATTEND_ANSWER_FILE_PATH=/home/image-builder/windows_autounattend.xml ``` ## Verify the Custom Image @@ -221,14 +225,14 @@ You may refer to [vSphere Kubernetes Service 3.2 documentation]() for more infor ### Kubernetes Release v1.31 -* The minimum vmclass should be best-effort-large for Windows Worker Node +- The minimum vmclass should be best-effort-large for Windows Worker Node When a windows worker node is configured with a vm-class which has resource configuration lower than best-effort-large, some of the management pods may not run due to loss of network connectivity. - Resolution: + Resolution: Switch to a vmclass configured with more resources. -* Some Pods on Window Node's networking don't work correctly +- Some Pods on Window Node's networking don't work correctly Some Pods on Window Node's networking don’t work correctly, which makes the Pod is not reachable, or the Pod can’t access other network peers. From 98c2dcb531e03f8cbb8e14ac20357237c07618f7 Mon Sep 17 00:00:00 2001 From: Sriraman S <sriraman.dev@gmail.com> Date: Fri, 4 Oct 2024 12:06:59 +0530 Subject: [PATCH 4/4] =?UTF-8?q?docs:=20=E2=9C=8F=EF=B8=8F=20Update=20docs?= =?UTF-8?q?=20to=20reflect=20on=20changes=20to=20make=20K8s=20release?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docs/examples/README.md | 8 +- docs/examples/customizations/README.md | 12 +++ .../adding_os_pkg_repos.md | 4 +- .../changing_hardware_version.md | 0 .../prometheus_node_exporter.md | 0 docs/examples/tutorial_building_an_image.md | 74 ++++++------------- 6 files changed, 41 insertions(+), 57 deletions(-) create mode 100644 docs/examples/customizations/README.md rename docs/examples/{ => customizations}/adding_os_pkg_repos.md (92%) rename docs/examples/{ => customizations}/changing_hardware_version.md (100%) rename docs/examples/{ => customizations}/prometheus_node_exporter.md (100%) diff --git a/docs/examples/README.md b/docs/examples/README.md index b77e82b..502e911 100644 --- a/docs/examples/README.md +++ b/docs/examples/README.md @@ -1,8 +1,8 @@ # Examples -This document gives information about different customization examples to customize a Node image for Tanzu on vSphere. +This document gives information about how to build Kubernetes Release on vSphere Node image and different customization examples to customize the Node image. -- [Changing VM Hardware Version(VMX version)](changing_hardware_version.md) -- [Adding new OS packages and configuring the repositories or sources](adding_os_pkg_repos.md). -- [Running Prometheus node exporter service on the nodes](prometheus_node_exporter.md) - [Tutorial for Using the vSphere Tanzu Kubernetes Grid Image Builder](tutorial_building_an_image.md) +- [Changing VM Hardware Version(VMX version)](./customizations/changing_hardware_version.md) +- [Adding new OS packages and configuring the repositories or sources](./customizations/adding_os_pkg_repos.md). +- [Running Prometheus node exporter service on the nodes](./customizations/prometheus_node_exporter.md) diff --git a/docs/examples/customizations/README.md b/docs/examples/customizations/README.md new file mode 100644 index 0000000..2c19931 --- /dev/null +++ b/docs/examples/customizations/README.md @@ -0,0 +1,12 @@ +# Examples + +This document gives information about different customization examples to customize a Kubernetes Release on vSphere Node image. + +- [Changing VM Hardware Version(VMX version)][hardware-version] +- [Adding new OS packages and configuring the repositories or sources][custom-package-repos]. +- [Running Prometheus node exporter service on the nodes][prometheus-node-exporter] + +[//]: Links +[hardware-version]: changing_hardware_version.md +[custom-package-repos]: adding_os_pkg_repos.md +[prometheus-node-exporter]: prometheus_node_exporter.md diff --git a/docs/examples/adding_os_pkg_repos.md b/docs/examples/customizations/adding_os_pkg_repos.md similarity index 92% rename from docs/examples/adding_os_pkg_repos.md rename to docs/examples/customizations/adding_os_pkg_repos.md index 3d7b01b..d320e41 100644 --- a/docs/examples/adding_os_pkg_repos.md +++ b/docs/examples/customizations/adding_os_pkg_repos.md @@ -35,7 +35,7 @@ _**Note**: The location of default configuration specific to OS follows the path - Create new folder `repos` in [ansible files][ansible-files] folder - Depending upon the Linux OS flavour, use either of the below steps - - For Photon sources, create a new file called `photon.repo` in the `repos` folder. Refer below for sample content and refer to the official Photon [document][photon-repo-doc] for more information. + - For **Photon** sources, create a new file called `photon.repo` in the `repos` folder. Refer below for sample content and refer to the official Photon [document][photon-repo-doc] for more information. ```text [photon] @@ -60,7 +60,7 @@ _**Note**: The location of default configuration specific to OS follows the path enabled=1 ``` - - For Ubuntu sources, create a new file called `ubuntu.list` in the `repos` folder. Refer to the official ubuntu [documentation][ubuntu-sources-doc] for more information. + - For **Ubuntu** sources, create a new file called `ubuntu.list` in the `repos` folder. Refer to the official ubuntu [documentation][ubuntu-sources-doc] for more information. - _**Note**: `jammy` is for ubuntu 22.04 so this needs to be changed if the ubuntu version is also changed, example for ubuntu 20.04 it is `focal`_ ```text diff --git a/docs/examples/changing_hardware_version.md b/docs/examples/customizations/changing_hardware_version.md similarity index 100% rename from docs/examples/changing_hardware_version.md rename to docs/examples/customizations/changing_hardware_version.md diff --git a/docs/examples/prometheus_node_exporter.md b/docs/examples/customizations/prometheus_node_exporter.md similarity index 100% rename from docs/examples/prometheus_node_exporter.md rename to docs/examples/customizations/prometheus_node_exporter.md diff --git a/docs/examples/tutorial_building_an_image.md b/docs/examples/tutorial_building_an_image.md index 24173d4..a71c336 100644 --- a/docs/examples/tutorial_building_an_image.md +++ b/docs/examples/tutorial_building_an_image.md @@ -1,15 +1,14 @@ # Tutorial for Using the vSphere Tanzu Kubernetes Grid Image Builder -This tutorial describes how to use the vSphere Tanzu Kubernetes Grid Image Builder to build a custom TKR for use with TKG 2 on Supervisor in the vSphere with Tanzu environment. +This tutorial describes how to use the vSphere Tanzu Kubernetes Grid Image Builder to build a custom TKR for use on Supervisor in the vSphere. The vSphere Tanzu Kubernetes Grid Image Builder uses Hashicorp Packer to generate images. Packer invokes vCenter APIs to create a VM from a TKR. ## Requirements -- Check the [prerequisites](tkg#prerequisites) -- vCenter Server 8, which can be any vCenter 8 instance, it does not have to be the same vCenter managing your vSphere with Tanzu environment +- vCenter Server 8, which can be any vCenter 8 instance, it does not have to be the same vCenter managing your vSphere - Packer requires the vSphere environment to have DHCP configured; you cannot use static IP address management -- Tutorial uses Ubuntu v1.22 to generate the image +- Tutorial uses Ubuntu 22.04 based Linux VM to generate the image ## Clone the Repository @@ -23,43 +22,7 @@ git clone https://github.com/vmware-tanzu/vsphere-tanzu-kubernetes-grid-image-bu The vSphere Tanzu Kubernetes Grid Image Builder runs components as Docker images to generate VMs. -Docker requires a 64-bit system with a Linux kernel having version 3.10 or newer. - -Use the `uname` command to check the version of your kernel: - -```bash -uname -r -``` - -Docker provides a convenience script to install the latest version of Docker for testing, dev and lab purposes. - -```bash -wget -qO- https://get.docker.com/ | sh -``` - -Remove the need for sudo execution of docker commands. - -```bash -sudo usermod -aG docker $(whoami) -``` - -Log out of the shell and log back in. - -```bash -exit -``` - -Check if you can invoke the Docker client without sudo. - -```bash -docker --version -``` - -Verify that the Docker daemon (dockerd engine) is running using the system service interface system. - -```bash -systemctl status --full docker -``` +Refer to the [Docker installation guide][docker-installation] for setting up Docker Engine. ## Install JQ @@ -126,19 +89,23 @@ For example: {# TO create a clone of the Packer VM after customization#} "linked_clone": "true", {# To create a snapshot of the Packer VM after customization #} - "create_snapshot": "true" + "create_snapshot": "true", + {# To destroy Packer VM after Image Build is completed #} + "destroy": "true" } ``` ## Select Kubernetes version -From the `vsphere-tanzu-kubernetes-grid-image-builder` directory where the Makefile is located run: +To identify the version of Kubernetes Release to be used, from the `vsphere-tanzu-kubernetes-grid-image-builder` directory where the Makefile is located run: ```bash make list-versions ``` -## Run the Artifacts Container for the Selected Kubernetes Version +## Run the Artifacts Server Container for the Selected Kubernetes Version + +Running the `run-artifacts-container` Makefile target, will pull the Artifacts Server container image corresponding to the selected Kubernetes Release. Usage: @@ -149,7 +116,7 @@ make run-artifacts-container KUBERNETES_VERSION=<version> Example: ```bash -make run-artifacts-container KUBERNETES_VERSION=v1.24.9+vmware.1 +make run-artifacts-container KUBERNETES_VERSION=v1.31.1+vmware.2-fips ``` ## Run the Image Builder Application @@ -157,7 +124,7 @@ make run-artifacts-container KUBERNETES_VERSION=v1.24.9+vmware.1 Usage: ```bash -make build-node-image OS_TARGET=<os_target> KUBERNETES_VERSION=v1.24.9+vmware.1 TKR_SUFFIX=<tkr_suffix> HOST_IP=<host_ip> IMAGE_ARTIFACTS_PATH=<image_artifacts_path> ARTIFACTS_CONTAINER_PORT=8081 +make build-node-image OS_TARGET=<os_target> KUBERNETES_VERSION=v1.31.1+vmware.2-fips TKR_SUFFIX=<tkr_suffix> HOST_IP=<host_ip> IMAGE_ARTIFACTS_PATH=<image_artifacts_path> ARTIFACTS_CONTAINER_PORT=8081 ``` NOTE: The HOST_IP must be reachable from the vCenter. @@ -165,7 +132,7 @@ NOTE: The HOST_IP must be reachable from the vCenter. Example: ```bash -make build-node-image OS_TARGET=ubuntu-2004-efi KUBERNETES_VERSION=v1.24.9+vmware.1 TKR_SUFFIX=byoi HOST_IP=192.2.2.3 IMAGE_ARTIFACTS_PATH=/home/ubuntu/image ARTIFACTS_CONTAINER_PORT=8081 +make build-node-image OS_TARGET=ubuntu-2004-efi KUBERNETES_VERSION=v1.31.1+vmware.2-fips TKR_SUFFIX=byoi HOST_IP=192.2.2.3 IMAGE_ARTIFACTS_PATH=/home/ubuntu/image ARTIFACTS_CONTAINER_PORT=8081 ``` ## Verify the Custom Image @@ -180,14 +147,19 @@ You should see the image being built in the datacenter, cluster, folder that you ## Customize the image -Refer to the customization examples. +Refer to the [customization examples][customizations]. -## Upload the Image to the TKG 2 with Supervisor Environment +## Upload the Image to generate custom Kubernetes Release Download the custom image from local storage or from the vCenter Server. -In your vSphere with Tanzu environment, create a local content library and upload the custom image there. +In your vSphere environment, create a local content library and upload the custom image there. -Refer to the documentation for [creating a local content library](https://docs.vmware.com/en/VMware-vSphere/8.0/vsphere-with-tanzu-tkg/GUID-19E8E034-5256-4EFC-BEBF-D4F17A8ED021.html) for use with TKG with Supervisor. +Refer to the documentation for [creating a local content library][create-local-content-library] for use with Supervisor. To use the custom TKR, configure the vSphere Namespace to use the local content library. + +[//]: Links +[docker-installation]: https://docs.docker.com/engine/install/ +[create-local-content-library]: https://docs.vmware.com/en/VMware-vSphere/8.0/vsphere-with-tanzu-tkg/GUID-19E8E034-5256-4EFC-BEBF-D4F17A8ED021.html +[customizations]: ./customizations/README.md