diff --git a/doc/cephadm/administration.rst b/doc/cephadm/administration.rst deleted file mode 100644 index afc1dd21cfec0..0000000000000 --- a/doc/cephadm/administration.rst +++ /dev/null @@ -1,265 +0,0 @@ -.. _cephadm-administration: - -====================== -cephadm Administration -====================== - - -Configuration -============= - -The cephadm orchestrator can be configured to use an SSH configuration file. This is -useful for specifying private keys and other SSH connection options. - -:: - - # ceph config set mgr mgr/cephadm/ssh_config_file /path/to/config - -An SSH configuration file can be provided without requiring an accessible file -system path as the method above does. - -:: - - # ceph cephadm set-ssh-config -i /path/to/config - -To clear this value use the command: - -:: - - # ceph cephadm clear-ssh-config - -Health checks -============= - -CEPHADM_PAUSED --------------- - -Cephadm background work has been paused with ``ceph orch pause``. Cephadm -will continue to perform passive monitoring activities (like checking -host and daemon status), but it will not make any changes (like deploying -or removing daemons). - -You can resume cephadm work with:: - - ceph orch resume - -CEPHADM_STRAY_HOST ------------------- - -One or more hosts have running Ceph daemons but are not registered as -hosts managed by *cephadm*. This means that those services cannot -currently be managed by cephadm (e.g., restarted, upgraded, included -in `ceph orch ps`). - -You can manage the host(s) with:: - - ceph orch host add ** - -Note that you may need to configure SSH access to the remote host -before this will work. - -Alternatively, you can manually connect to the host and ensure that -services on that host are removed and/or migrated to a host that is -managed by *cephadm*. - -You can also disable this warning entirely with:: - - ceph config set mgr mgr/cephadm/warn_on_stray_hosts false - -CEPHADM_STRAY_DAEMON --------------------- - -One or more Ceph daemons are running but not are not managed by -*cephadm*, perhaps because they were deploy using a different tool, or -were started manually. This means that those services cannot -currently be managed by cephadm (e.g., restarted, upgraded, included -in `ceph orch ps`). - -**FIXME:** We need to implement and document an adopt procedure here. - -You can also disable this warning entirely with:: - - ceph config set mgr mgr/cephadm/warn_on_stray_daemons false - -CEPHADM_HOST_CHECK_FAILED -------------------------- - -One or more hosts have failed the basic cephadm host check, which verifies -that (1) the host is reachable and cephadm can be executed there, and (2) -that the host satisfies basic prerequisites, like a working container -runtime (podman or docker) and working time synchronization. -If this test fails, cephadm will no be able to manage services on that host. - -You can manually run this check with:: - - ceph cephadm check-host ** - -You can remove a broken host from management with:: - - ceph orch host rm ** - -You can disable this health warning with:: - - ceph config set mgr mgr/cephadm/warn_on_failed_host_check false - - -Converting an existing cluster to cephadm -========================================= - -Cephadm allows you to (pretty) easily convert an existing Ceph cluster that -has been deployed with ceph-deploy, ceph-ansible, DeepSea, or similar tools. - -Limitations ------------ - -* Cephadm only works with BlueStore OSDs. If there are FileStore OSDs - in your cluster you cannot manage them. - -Adoption Process ----------------- - -#. Get the ``cephadm`` command line too on each host. You can do this with curl or by installing the package. The simplest approach is:: - - [each host] # curl --silent --remote-name --location https://github.com/ceph/ceph/raw/master/src/cephadm/cephadm - [each host] # chmod +x cephadm - -#. Prepare each host for use by ``cephadm``:: - - [each host] # ./cephadm prepare-host - -#. List all Ceph daemons on the current host:: - - # ./cephadm ls - - You should see that all existing daemons have a type of ``legacy`` - in the resulting output. - -#. Determine which Ceph version you will use. You can use any Octopus - release or later. For example, ``docker.io/ceph/ceph:v15.2.0``. The default - will be the latest stable release, but if you are upgrading from an earlier - release at the same time be sure to refer to the upgrade notes for any - special steps to take while upgrading. - - The image is passed to cephadm with:: - - # ./cephadm --image $IMAGE - -#. Adopt each monitor:: - - # ./cephadm adopt --style legacy --name mon. - -#. Adopt each manager:: - - # ./cephadm adopt --style legacy --name mgr. - -#. Enable cephadm:: - - # ceph mgr module enable cephadm - # ceph orch set backend cephadm - -#. Generate an SSH key:: - - # ceph cephadm generate-key - # ceph cephadm get-pub-key - -#. Install the SSH key on each host to be managed:: - - # echo | sudo tee /root/.ssh/authorized_keys - - Note that ``/root/.ssh/authorized_keys`` should have mode ``0600`` and - ``/root/.ssh`` should have mode ``0700``. - -#. Tell cephadm which hosts to manage:: - - # ceph orch host add [ip-address] - - This will perform a ``cephadm check-host`` on each host before - adding it to ensure it is working. The IP address argument is only - required if DNS doesn't allow you to connect to each host by it's - short name. - -#. Verify that the monitor and manager daemons are visible:: - - # ceph orch ps - -#. Adopt all remainingg daemons:: - - # ./cephadm adopt --style legacy --name - # ./cephadm adopt --style legacy --name - # ./cephadm adopt --style legacy --name - - Repeat for each host and daemon. - -#. Check the ``ceph health detail`` output for cephadm warnings about - stray cluster daemons or hosts that are not yet managed. - -Troubleshooting -=============== - -Sometimes there is a need to investigate why a cephadm command failed or why -a specific service no longer runs properly. - -As cephadm deploys daemons as containers, troubleshooting daemons is slightly -different. Here are a few tools and commands to help investigating issues. - -Gathering log files -------------------- - -Use journalctl to gather the log files of all daemons: - -.. note:: By default cephadm now stores logs in journald. This means - that you will no longer find daemon logs in ``/var/log/ceph/``. - -To read the log file of one specific daemon, run:: - - cephadm logs --name - -Note: this only works when run on the same host where the daemon is running. To -get logs of a daemon running on a different host, give the ``--fsid`` option:: - - cephadm logs --fsid --name - -Where the ```` corresponds to the cluster id printed by ``ceph status``. - -To fetch all log files of all daemons on a given host, run:: - - for name in $(cephadm ls | jq -r '.[].name') ; do - cephadm logs --fsid --name "$name" > $name; - done - -Collecting systemd status -------------------------- - -To print the state of a systemd unit, run:: - - systemctl status "ceph-$(cephadm shell ceph fsid)@.service"; - - -To fetch all state of all daemons of a given host, run:: - - fsid="$(cephadm shell ceph fsid)" - for name in $(cephadm ls | jq -r '.[].name') ; do - systemctl status "ceph-$fsid@$name.service" > $name; - done - - -List all downloaded container images ------------------------------------- - -To list all container images that are downloaded on a host: - -.. note:: ``Image`` might also be called `ImageID` - -:: - - podman ps -a --format json | jq '.[].Image' - "docker.io/library/centos:8" - "registry.opensuse.org/opensuse/leap:15.2" - - -Manually running containers ---------------------------- - -cephadm writes small wrappers that run a containers. Refer to -``/var/lib/ceph///unit.run`` for the container execution command. -to execute a container. diff --git a/doc/cephadm/adoption.rst b/doc/cephadm/adoption.rst new file mode 100644 index 0000000000000..f94e46325fde9 --- /dev/null +++ b/doc/cephadm/adoption.rst @@ -0,0 +1,142 @@ +.. _cephadm-adoption: + +Converting an existing cluster to cephadm +========================================= + +Cephadm allows you to convert an existing Ceph cluster that +has been deployed with ceph-deploy, ceph-ansible, DeepSea, or similar tools. + +Limitations +----------- + +* Cephadm only works with BlueStore OSDs. If there are FileStore OSDs + in your cluster you cannot manage them. + +Preparation +----------- + +#. Get the ``cephadm`` command line tool on each host in the existing + cluster. See :ref:`get-cephadm`. + +#. Prepare each host for use by ``cephadm``:: + + # cephadm prepare-host + +#. Determine which Ceph version you will use. You can use any Octopus (15.2.z) + release or later. For example, ``docker.io/ceph/ceph:v15.2.0``. The default + will be the latest stable release, but if you are upgrading from an earlier + release at the same time be sure to refer to the upgrade notes for any + special steps to take while upgrading. + + The image is passed to cephadm with:: + + # cephadm --image $IMAGE + +#. Cephadm can provide a list of all Ceph daemons on the current host:: + + # cephadm ls + + Before starting, you should see that all existing daemons have a + style of ``legacy`` in the resulting output. As the adoption + process progresses, adopted daemons will appear as style + ``cephadm:v1``. + + +Adoption process +---------------- + +#. Ensure the ceph configuration is migrated to use the cluster config database. + If the ``/etc/ceph/ceph.conf`` is identical on each host, then on one host:: + + # ceph config assimilate-conf -i /etc/ceph/ceph.conf + + If there are config variations on each host, you may need to repeat + this command on each host. You can view the cluster's + configuration to confirm that it is complete with:: + + # ceph config dump + +#. Adopt each monitor:: + + # cephadm adopt --style legacy --name mon. + + Each legacy monitor should stop, quickly restart as a cephadm + container, and rejoin the quorum. + +#. Adopt each manager:: + + # cephadm adopt --style legacy --name mgr. + +#. Enable cephadm:: + + # ceph mgr module enable cephadm + # ceph orch set backend cephadm + +#. Generate an SSH key:: + + # ceph cephadm generate-key + # ceph cephadm get-pub-key > ceph.pub + +#. Install the cluster SSH key on each host in the cluster:: + + # ssh-copy-id -f -i ceph.pub root@ + +#. Tell cephadm which hosts to manage:: + + # ceph orch host add [ip-address] + + This will perform a ``cephadm check-host`` on each host before + adding it to ensure it is working. The IP address argument is only + required if DNS does not allow you to connect to each host by its + short name. + +#. Verify that the adopted monitor and manager daemons are visible:: + + # ceph orch ps + +#. Adopt all OSDs in the cluster:: + + # cephadm adopt --style legacy --name + + For example:: + + # cephadm adopt --style legacy --name osd.1 + # cephadm adopt --style legacy --name osd.2 + +#. Redeploy MDS daemons by telling cephadm how many daemons to run for + each file system. You can list file systems by name with ``ceph fs + ls``. For each file system:: + + # ceph orch apply mds + + For example, in a cluster with a single file system called `foo`:: + + # ceph fs ls + name: foo, metadata pool: foo_metadata, data pools: [foo_data ] + # ceph orch apply mds foo 2 + + Wait for the new MDS daemons to start with:: + + # ceph orch ps --daemon-type mds + + Finally, stop and remove the legacy MDS daemons:: + + # systemctl stop ceph-mds.target + # rm -rf /var/lib/ceph/mds/ceph-* + +#. Redeploy RGW daemons. Cephadm manages RGW daemons by zone. For each + zone, deploy new RGW daemons with cephadm:: + + # ceph orch apply rgw [--port ] [--ssl] + + where ** can be a simple daemon count, or a list of + specific hosts (see :ref:`orchestrator-cli-placement-spec`). + + Once the daemons have started and you have confirmed they are functioning, + stop and remove the old legacy daemons:: + + # systemctl stop ceph-rgw.target + # rm -rf /var/lib/ceph/radosgw/ceph-* + +#. Check the ``ceph health detail`` output for cephadm warnings about + stray cluster daemons or hosts that are not yet managed. diff --git a/doc/cephadm/drivegroups.rst b/doc/cephadm/drivegroups.rst index b0032bfa3dcbc..8497559f486eb 100644 --- a/doc/cephadm/drivegroups.rst +++ b/doc/cephadm/drivegroups.rst @@ -1,3 +1,5 @@ +.. _drivegroups: + =========== DriveGroups =========== diff --git a/doc/cephadm/index.rst b/doc/cephadm/index.rst index ddce3b3302564..4a0a9230fbd78 100644 --- a/doc/cephadm/index.rst +++ b/doc/cephadm/index.rst @@ -1,297 +1,32 @@ -.. _cephadm-bootstrap: +.. _cephadm: -======================== - Installation (cephadm) -======================== +======= +Cephadm +======= -.. note:: The *cephadm* bootstrap feature is first introduced in Octopus, and is not yet recommended for production deployments. +Cephadm deploys and manages a Ceph cluster by connection to hosts from the +manager daemon via SSH to add, remove, or update Ceph daemon containers. It +does not rely on external configuration or orchestration tools like Ansible, +Rook, or Salt. -cephadm manages nodes in a cluster by establishing an SSH connection -and issues explicit management commands. It does not rely on -separate systems such as Rook or Ansible. +Cephadm starts by bootstrapping a tiny Ceph cluster on a single node +(one monitor and one manager) and then uses the orchestration +interface ("day 2" commands) to expand the cluster to include all +hosts and to provision all Ceph daemons and services. This can be +performed via the Ceph command-line interface (CLI) or dashboard +(GUI). -A new Ceph cluster is deployed by bootstrapping a cluster on a single -node, and then adding additional nodes and daemons via the CLI or GUI -dashboard. - -The following example installs a basic three-node cluster. Each -node will be identified by its prompt. For example, "[monitor 1]" -identifies the first monitor, "[monitor 2]" identifies the second -monitor, and "[monitor 3]" identifies the third monitor. This -information is provided in order to make clear which commands -should be issued on which systems. - -"[any node]" identifies any Ceph node, and in the context -of this installation guide means that the associated command -can be run on any node. - -Requirements -============ - -- Podman or Docker -- LVM2 - -.. highlight:: console - -Get cephadm -=========== - -The ``cephadm`` utility is used to bootstrap a new Ceph Cluster. - -Use curl to fetch the standalone script:: - - [monitor 1] # curl --silent --remote-name --location https://github.com/ceph/ceph/raw/master/src/cephadm/cephadm - [monitor 1] # chmod +x cephadm - -You can also get the utility by installing a package provided by -your Linux distribution:: - - [monitor 1] # apt install -y cephadm # or - [monitor 1] # dnf install -y cephadm # or - [monitor 1] # yum install -y cephadm # or - [monitor 1] # zypper install -y cephadm - - -Bootstrap a new cluster -======================= - -To create a new cluster, you need to know which *IP address* to use -for the cluster's first monitor. This is normally just the IP for the -first cluster node. If there are multiple networks and interfaces, be -sure to choose one that will be accessible by any hosts accessing the -Ceph cluster. - -To bootstrap the cluster run the following command:: - - [node 1] $ sudo ./cephadm bootstrap --mon-ip ** - -This command does a few things: - -* Creates a monitor and manager daemon for the new cluster on the - local host. A minimal configuration file needed to communicate with - the new cluster is written to ``ceph.conf`` in the local directory. -* A copy of the ``client.admin`` administrative (privileged!) secret - key is written to ``ceph.client.admin.keyring`` in the local directory. -* Generates a new SSH key, and adds the public key to the local root user's - ``/root/.ssh/authorized_keys`` file. A copy of the public key is written - to ``ceph.pub`` in the current directory. - -Interacting with the cluster -============================ - -To interact with your cluster, start up a container that has all of -the Ceph packages installed:: - - [any node] $ sudo ./cephadm shell --config ceph.conf --keyring ceph.client.admin.keyring - -The ``--config`` and ``--keyring`` arguments will bind those local -files to the default locations in ``/etc/ceph`` inside the container -to allow the ``ceph`` CLI utility to work without additional -arguments. Inside the container, you can check the cluster status with:: - - [ceph: root@monitor_1_hostname /]# ceph status - -In order to interact with the Ceph cluster outside of a container -(that is, from the command line), install the Ceph -client packages and install the configuration and privileged -administrator key in a global location:: - - [any node] $ sudo apt install -y ceph-common # or, - [any node] $ sudo dnf install -y ceph-common # or, - [any node] $ sudo yum install -y ceph-common - - [any node] $ sudo install -m 0644 ceph.conf /etc/ceph/ceph.conf - [any node] $ sudo install -m 0600 ceph.keyring /etc/ceph/ceph.keyring - -Watching cephadm log messages -============================= - -Cephadm logs to the ``cephadm`` cluster log channel, which means you can monitor progress in realtime with:: - - # ceph -W cephadm - -By default it will show info-level events and above. To see -debug-level messages too:: - - # ceph config set mgr mgr/cephadm/log_to_cluster_level debug - # ceph -W cephadm --watch-debug - -Be careful: the debug messages are very verbose! - -You can see recent events with:: - - # ceph log last cephadm - -These events are also logged to the ``ceph.cephadm.log`` file on -monitor hosts and/or to the monitor-daemon stderr. - -Adding hosts to the cluster -=========================== - -For each new host you'd like to add to the cluster, you need to do two things: - -#. Install the cluster's public SSH key in the new host's root user's - ``authorized_keys`` file. This is easy with the ``ssh-copy-id`` script:: - - [monitor 1] # ssh-copy-id -f -i ceph.pub root@*newhost* - -#. Tell Ceph that the new node is part of the cluster:: - - # ceph orch host add *newhost* - -Deploying additional monitors -============================= - -Normally a Ceph cluster has three or five monitor daemons spread -across different hosts. As a rule of thumb, you should deploy five -monitors if there are five or more nodes in your cluster. - -.. _CIDR: https://en.wikipedia.org/wiki/Classless_Inter-Domain_Routing#CIDR_notation - -If all of your monitors will exist on the same IP subnet, cephadm can -automatically scale the number of monitors. This subnet should be -specified in `CIDR`_ format (e.g., ``10.1.2.0/24``). (If you do not -specify a subnet, you will need to manually specify an IP or subnet -when creating each monitor.):: - - # ceph config set mon public_network ** - -For example:: - - # ceph config set mon public_network 10.1.2.0/24 - -There are several ways to add additional monitors: - -* You can simply tell cephadm how many monitors you want, and it will pick the - hosts (randomly):: - - # ceph orch apply mon ** - - For example, if you have 5 or more hosts added to the cluster,:: - - # ceph orch apply mon 5 - -* You can explicitly specify which hosts to deploy on. Be sure to include - the first monitor host in this list.:: - - # ceph orch apply mon ** - - For example,:: - - # ceph orch apply mon host1,host2,host3 - -* You can control which hosts the monitors run on by adding the ``mon`` label - to the appropriate hosts:: - - # ceph orch host label add ** mon - - To view the current hosts and labels,:: - - # ceph orch host ls - - For example:: - - # ceph orch host label add host1 mon - # ceph orch host label add host2 mon - # ceph orch host label add host3 mon - # ceph orch host ls - HOST ADDR LABELS STATUS - host1 mon - host2 mon - host3 mon - host4 - host5 - - Then tell cephadm to deploy monitors based on the label:: - - # ceph orch apply mon label:mon - -* You can explicitly specify the IP address or CIDR for each monitor - and control where it is placed. This is the only supported method - if you did not specify the CIDR monitor network above. - - To deploy additional monitors,:: - - # ceph orch daemon add mon * [...]* - - For example, to deploy a second monitor on ``newhost1`` using an IP - address ``10.1.2.123`` and a third monitor on ``newhost2`` in - network ``10.1.2.0/24``,:: - - # ceph orch daemon add mon newhost1:10.1.2.123 - # ceph orch daemon add mon newhost2:10.1.2.0/24 - -Deploying OSDs -============== - -To add OSDs to the cluster, you have two options: - -#. You need to know the device name for the block device (hard disk or -SSD) that will be used. Then,:: - - # ceph orch osd create **:** - - For example, to deploy an OSD on host *newhost*'s SSD,:: - - # ceph orch osd create newhost:/dev/disk/by-id/ata-WDC_WDS200T2B0A-00SM50_182294800028 - - -#. You need to describe your disk setup by it's properties (Drive Groups) - - Link to DriveGroup docs.:: - - # ceph orch osd create -i my_drivegroups.yml - - -.. _drivegroups: drivegroups:: - -Deploying manager daemons -========================= - -It is a good idea to have at least one backup manager daemon. To -deploy one or more new manager daemons,:: - - # ceph orch apply mgr ** [** ...] - -Deploying MDSs -============== - -One or more MDS daemons is required to use the CephFS file system. -These are created automatically if the newer ``ceph fs volume`` -interface is used to create a new file system. For more information, -see :ref:`fs-volumes-and-subvolumes`. - -To deploy metadata servers,:: - - # ceph orch apply mds ** ** [** ...] - -Deploying RGWs -============== - -Cephadm deploys radosgw as a collection of daemons that manage a -particular *realm* and *zone*. (For more information about realms and -zones, see :ref:`multisite`.) To deploy a set of radosgw daemons for -a particular realm and zone,:: - - # ceph orch apply rgw ** ** ** [** ...] - -Note that with cephadm, radosgw daemons are configured via the monitor -configuration database instead of via a `ceph.conf` or the command line. If -that confiruation isn't already in place (usually in the -``client.rgw..`` section), then the radosgw -daemons will start up with default settings (e.g., binding to port -80). - - -Further Reading -=============== +Cephadm is new in the Octopus v15.2.0 release and does not support older +versions of Ceph. .. toctree:: :maxdepth: 2 - Cephadm administration + install + adoption + upgrade + Cephadm operations Cephadm monitoring Cephadm CLI <../mgr/orchestrator> DriveGroups - OS recommendations <../start/os-recommendations> - + troubleshooting diff --git a/doc/cephadm/install.rst b/doc/cephadm/install.rst new file mode 100644 index 0000000000000..42c135e55560e --- /dev/null +++ b/doc/cephadm/install.rst @@ -0,0 +1,322 @@ +============================ +Deploying a new Ceph cluster +============================ + +Cephadm creates a new Ceph cluster by "bootstrapping" on a single +host, expanding the cluster to encompass any additional hosts, and +then deploying the needed services. + +.. highlight:: console + +Requirements +============ + +- Systemd +- Podman or Docker for running containers +- Time synchronization (such as chrony or NTP) +- LVM2 for provisioning storage devices + +Any modern Linux distribution should be sufficient. Dependencies +are installed automatically by the bootstrap process below. + +.. _get-cephadm: + +Install cephadm +=============== + +The ``cephadm`` command can (1) bootstrap a new cluster, (2) +launch a containerized shell with a working Ceph CLI, and (3) aid in +debugging containerized Ceph daemons. + +There are a few ways to install cephadm: + +* Use ``curl`` to fetch the most recent version of the + standalone script:: + + # curl --silent --remote-name --location https://github.com/ceph/ceph/raw/octopus/src/cephadm/cephadm + # chmod +x cephadm + + This script can be run directly from the current directory with:: + + # ./cephadm + +* Although the standalone script is sufficient to get a cluster started, it is + convenient to have the ``cephadm`` command installed on the host. To install + these packages for the current Octopus release:: + + # ./cephadm add-repo --release octopus + # ./cephadm install + + Confirm that ``cephadm`` is now in your PATH with:: + + # which cephadm + +* Some commercial Linux distributions (e.g., RHEL, SLE) may already + include up-to-date Ceph packages. In that case, you can install + cephadm directly. For example:: + + # dnf install -y cephadm # or + # zypper install -y cephadm + + + +Bootstrap a new cluster +======================= + +You need to know which *IP address* to use for the cluster's first +monitor daemon. This is normally just the IP for the first host. If there +are multiple networks and interfaces, be sure to choose one that will +be accessible by any host accessing the Ceph cluster. + +To bootstrap the cluster:: + + # mkdir -p /etc/ceph + # cephadm bootstrap --mon-ip ** + +This command will: + +* Create a monitor and manager daemon for the new cluster on the local + host. +* Generate a new SSH key for the Ceph cluster and adds it to the root + user's ``/root/.ssh/authorized_keys`` file. +* Write a minimal configuration file needed to communicate with the + new cluster to ``/etc/ceph/ceph.conf``. +* Write a copy of the ``client.admin`` administrative (privileged!) + secret key to ``/etc/ceph/ceph.client.admin.keyring``. +* Write a copy of the public key to + ``/etc/ceph/ceph.pub``. + +The default bootstrap behavior will work for the vast majority of +users. See below for a few options that may be useful for some users, +or run ``cephadm bootstrap -h`` to see all available options: + +* Bootstrap writes the files needed to access the new cluster to + ``/etc/ceph`` for convenience, so that any Ceph packages installed + on the host itself (e.g., to access the command line interface) can + easily find them. + + Daemon containers deployed with cephadm, however, do not need + ``/etc/ceph`` at all. Use the ``--output-dir **`` option + to put them in a different directory (like ``.``), avoiding any + potential conflicts with existing Ceph configuration (cephadm or + otherwise) on the same host. + +* You can pass any initial Ceph configuration options to the new + cluster by putting them in a standard ini-style configuration file + and using the ``--config **`` option. + + +Enable Ceph CLI +=============== + +Cephadm does not require any Ceph packages to be installed on the +host. However, we recommend enabling easy access to the the ``ceph`` +command. There are several ways to do this: + +* The ``cephadm shell`` command launches a bash shell in a container + with all of the Ceph packages installed. By default, if + configuration and keyring files are found in ``/etc/ceph`` on the + host, they are passed into the container environment so that the + shell is fully functional:: + + # cephadm shell + +* It may be helpful to create an alias:: + + # alias ceph='cephadm shell --' + +* You can install the ``ceph-common`` package, which contains all of the + ceph commands, including ``ceph``, ``rbd``, ``mount.ceph`` (for mounting + CephFS file systems), etc.:: + + # cephadm add-repo --release octopus + # cephadm install ceph-common + +Confirm that the ``ceph`` command is accessible with:: + + # ceph -v + +Confirm that the ``ceph`` command can connect to the cluster and also +its status with:: + + # ceph status + + +Add hosts to the cluster +======================== + +To add each new host to the cluster, perform two steps: + +#. Install the cluster's public SSH key in the new host's root user's + ``authorized_keys`` file:: + + # ssh-copy-id -f -i ceph.pub root@** + + For example:: + + # ssh-copy-id -f -i ceph.pub root@host2 + # ssh-copy-id -f -i ceph.pub root@host3 + +#. Tell Ceph that the new node is part of the cluster:: + + # ceph orch host add *newhost* + + For example:: + + # ceph orch host add host2 + # ceph orch host add host3 + + +Deploy additional monitors (optional) +===================================== + +A typical Ceph cluster has three or five monitor daemons spread +across different hosts. We recommend deploying five +monitors if there are five or more nodes in your cluster. + +.. _CIDR: https://en.wikipedia.org/wiki/Classless_Inter-Domain_Routing#CIDR_notation + +When Ceph knows what IP subnet the monitors should use it can automatically +deploy and scale monitors as the cluster grows (or contracts). By default, +Ceph assumes that other monitors should use the same subnet as the first +monitor's IP. + +If your Ceph monitors (or the entire cluster) live on a single subnet, +then by default cephadm automatically adds up to 5 monitors as you add new +hosts to the cluster. No further steps are necessary. + +* If there is a specific IP subnet that should be used by monitors, you + can configure that in `CIDR`_ format (e.g., ``10.1.2.0/24``) with:: + + # ceph config set mon public_network ** + + For example:: + + # ceph config set mon public_network 10.1.2.0/24 + + Cephadm only deploys new monitor daemons on hosts that have IPs + configured in the configured subnet. + +* If you want to adjust the default of 5 monitors:: + + # ceph orch apply mon ** + +* To deploy monitors on a specific set of hosts:: + + # ceph orch apply mon ** + + Be sure to include the first (bootstrap) host in this list. + +* You can control which hosts the monitors run on by making use of + host labels. To set the ``mon`` label to the appropriate + hosts:: + + # ceph orch host label add ** mon + + To view the current hosts and labels:: + + # ceph orch host ls + + For example:: + + # ceph orch host label add host1 mon + # ceph orch host label add host2 mon + # ceph orch host label add host3 mon + # ceph orch host ls + HOST ADDR LABELS STATUS + host1 mon + host2 mon + host3 mon + host4 + host5 + + Tell cephadm to deploy monitors based on the label:: + + # ceph orch apply mon label:mon + +* You can explicitly specify the IP address or CIDR network for each monitor + and control where it is placed. To disable automated monitor deployment:: + + # ceph orch apply mon --unmanaged + + To deploy each additional monitor:: + + # ceph orch daemon add mon * [...]* + + For example, to deploy a second monitor on ``newhost1`` using an IP + address ``10.1.2.123`` and a third monitor on ``newhost2`` in + network ``10.1.2.0/24``:: + + # ceph orch apply mon --unmanaged + # ceph orch daemon add mon newhost1:10.1.2.123 + # ceph orch daemon add mon newhost2:10.1.2.0/24 + + +Deploy OSDs +=========== + +An inventory of storage devices on all cluster hosts can be displayed with:: + + # ceph orch device ls + +A storage device is considered *available* if all of the following +conditions are met: + +* The device must have no partitions. +* The device must not have any LVM state. +* The device must not be mounted. +* The device must not contain a file system. +* The device must not contain a Ceph BlueStore OSD. +* The device must be larger than 5 GB. + +Ceph refuses to provision an OSD on a device that is not available. + +There are a few ways to create new OSDs: + +* Tell Ceph to consume any available and unused storage device:: + + # ceph orch apply osd --all-available-devices + +* Create an OSD from a specific device on a specific host:: + + # ceph orch daemon add osd **:** + + For example:: + + # ceph orch daemon add osd host1:/dev/sdb + +* Use :ref:`drivegroups` to describe device(s) to consume + based on their properties, such device type (SSD or HDD), device + model names, size, or the hosts on which the devices exist:: + + # ceph orch osd create -i spec.yml + + +Deploy MDSs +=========== + +One or more MDS daemons is required to use the CephFS file system. +These are created automatically if the newer ``ceph fs volume`` +interface is used to create a new file system. For more information, +see :ref:`fs-volumes-and-subvolumes`. + +To deploy metadata servers:: + + # ceph orch apply mds ** ** [** ...] + +Deploy RGWs +=========== + +Cephadm deploys radosgw as a collection of daemons that manage a +particular *realm* and *zone*. (For more information about realms and +zones, see :ref:`multisite`.) To deploy a set of radosgw daemons for +a particular realm and zone:: + + # ceph orch apply rgw ** ** ** [** ...] + +Note that with cephadm, radosgw daemons are configured via the monitor +configuration database instead of via a `ceph.conf` or the command line. If +that confiruation isn't already in place (usually in the +``client.rgw..`` section), then the radosgw +daemons will start up with default settings (e.g., binding to port +80). diff --git a/doc/cephadm/monitoring.rst b/doc/cephadm/monitoring.rst index 0d4a16e0a5df0..31c93a3cce59b 100644 --- a/doc/cephadm/monitoring.rst +++ b/doc/cephadm/monitoring.rst @@ -5,7 +5,9 @@ The Ceph dashboard makes use of prometheus, grafana, and related tools to store and visualize detailed metrics on cluster utilization and performance. Ceph users have three options: -#. Have cephadm deploy and configure these services. +#. Have cephadm deploy and configure these services. This is the default + when bootstrapping a new cluster unless the ``--skip-monitoring-stack`` + option is used. #. Deploy and configure these services manually. This is recommended for users with existing prometheus services in their environment (and in cases where Ceph is running in Kubernetes with Rook). @@ -15,7 +17,10 @@ performance. Ceph users have three options: Deploying monitoring with cephadm --------------------------------- -To deploy a basic monitoring stack: +By default, bootstrap will deploy a basic monitoring stack. If you +did not do this (by passing ``--skip-monitoring-stack``, or if you +converted an existing cluster to cephadm management, you can set up +monitoring by following the steps below. #. Enable the prometheus module in the ceph-mgr daemon. This exposes the internal Ceph metrics so that prometheus can scrape them.:: @@ -52,6 +57,18 @@ completed, you should see something like this from ``ceph orch ls``:: node-exporter 2/2 6s ago docker.io/prom/node-exporter:latest e5a616e4b9cf present prometheus 1/1 6s ago docker.io/prom/prometheus:latest e935122ab143 present +Disabling monitoring +-------------------- + +If you have deployed monitoring and would like to remove it, you can do +so with:: + + ceph orch rm grafana + ceph orch rm prometheus --force # this will delete metrics data collected so far + ceph orch rm node-exporter + ceph orch rm alertmanager + ceph mgr module disable prometheus + Deploying monitoring manually ----------------------------- diff --git a/doc/cephadm/operations.rst b/doc/cephadm/operations.rst new file mode 100644 index 0000000000000..2e25874aabdf8 --- /dev/null +++ b/doc/cephadm/operations.rst @@ -0,0 +1,254 @@ +================== +Cephadm Operations +================== + +Watching cephadm log messages +============================= + +Cephadm logs to the ``cephadm`` cluster log channel, meaning you can +monitor progress in realtime with:: + + # ceph -W cephadm + +By default it will show info-level events and above. To see +debug-level messages too:: + + # ceph config set mgr mgr/cephadm/log_to_cluster_level debug + # ceph -W cephadm --watch-debug + +Be careful: the debug messages are very verbose! + +You can see recent events with:: + + # ceph log last cephadm + +These events are also logged to the ``ceph.cephadm.log`` file on +monitor hosts and to the monitor daemons' stderr. + + +Ceph daemon logs +================ + +Logging to stdout +----------------- + +Traditionally, Ceph daemons have logged to ``/var/log/ceph``. By +default, cephadm daemons log to stderr and the logs are +captured by the container runtime environment. For most systems, by +default, these logs are sent to journald and accessible via +``journalctl``. + +For example, to view the logs for the daemon ``mon.foo`` for a cluster +with ID ``5c5a50ae-272a-455d-99e9-32c6a013e694``, the command would be +something like:: + + journalctl -u ceph-5c5a50ae-272a-455d-99e9-32c6a013e694@mon.foo + +This works well for normal operations when logging levels are low. + +To disable logging to stderr:: + + ceph config set global log_to_stderr false + ceph config set global mon_cluster_log_to_stderr false + +Logging to files +---------------- + +You can also configure Ceph daemons to log to files instead of stderr, +just like they have in the past. When logging to files, Ceph logs appear +in ``/var/log/ceph/``. + +To enable logging to files:: + + ceph config set global log_to_file true + ceph config set global mon_cluster_log_to_file true + +We recommend disabling logging to stderr (see above) or else everything +will be logged twice:: + + ceph config set global log_to_stderr false + ceph config set global mon_cluster_log_to_stderr false + +By default, cephadm sets up log rotation on each host to rotate these +files. You can configure the logging retention schedule by modifying +``/etc/logrotate.d/ceph.``. + + +Data location +============= + +Cephadm daemon data and logs in slightly different locations than older +versions of ceph: + +* ``/var/log/ceph/`` contains all cluster logs. Note + that by default cephadm logs via stderr and the container runtime, + so these logs are normally not present. +* ``/var/lib/ceph/`` contains all cluster daemon data + (besides logs). +* ``/var/lib/ceph//`` contains all data for + an individual daemon. +* ``/var/lib/ceph//crash`` contains crash reports for + the cluster. +* ``/var/lib/ceph//removed`` contains old daemon + data directories for stateful daemons (e.g., monitor, prometheus) + that have been removed by cephadm. + +Disk usage +---------- + +Because a few Ceph daemons may store a significant amount of data in +``/var/lib/ceph`` (notably, the monitors and prometheus), we recommend +moving this directory to its own disk, partition, or logical volume so +that it does not fill up the root file system. + + + +SSH Configuration +================= + +Cephadm uses SSH to connect to remote hosts. SSH uses a key to authenticate +with those hosts in a secure way. + + +Default behavior +---------------- + +Cephadm stores an SSH key in the monitor that is used to +connect to remote hosts. When the cluster is bootstrapped, this SSH +key is generated automatically and no additional configuration +is necessary. + +A *new* SSH key can be generated with:: + + ceph cephadm generate-key + +The public portion of the SSH key can be retrieved with:: + + ceph cephadm get-pub-key + +The currently stored SSH key can be deleted with:: + + ceph cephadm clear-key + +You can make use of an existing key by directly importing it with:: + + ceph config-key set mgr/cephadm/ssh_identity_key -i + ceph config-key set mgr/cephadm/ssh_identity_pub -i + +You will then need to restart the mgr daemon to reload the configuration with:: + + ceph mgr fail + + +Customizing the SSH configuration +--------------------------------- + +Cephadm generates an appropriate ``ssh_config`` file that is +used for connecting to remote hosts. This configuration looks +something like this:: + + Host * + User root + StrictHostKeyChecking no + UserKnownHostsFile /dev/null + +There are two ways to customize this configuration for your environment: + +#. Import a customized configuration file that will be stored + by the monitor with:: + + ceph cephadm set-ssh-config -i + + To remove a customized SSH config and revert back to the default behavior:: + + ceph cephadm clear-ssh-config + +#. You can configure a file location for the SSH configuration file with:: + + ceph config set mgr mgr/cephadm/ssh_config_file + + We do *not recommend* this approach. The path name must be + visible to *any* mgr daemon, and cephadm runs all daemons as + containers. That means that the file either need to be placed + inside a customized container image for your deployment, or + manually distributed to the mgr data directory + (``/var/lib/ceph//mgr.`` on the host, visible at + ``/var/lib/ceph/mgr/ceph-`` from inside the container). + + +Health checks +============= + +CEPHADM_PAUSED +-------------- + +Cephadm background work has been paused with ``ceph orch pause``. Cephadm +continues to perform passive monitoring activities (like checking +host and daemon status), but it will not make any changes (like deploying +or removing daemons). + +Resume cephadm work with:: + + ceph orch resume + +CEPHADM_STRAY_HOST +------------------ + +One or more hosts have running Ceph daemons but are not registered as +hosts managed by *cephadm*. This means that those services cannot +currently be managed by cephadm (e.g., restarted, upgraded, included +in `ceph orch ps`). + +You can manage the host(s) with:: + + ceph orch host add ** + +Note that you may need to configure SSH access to the remote host +before this will work. + +Alternatively, you can manually connect to the host and ensure that +services on that host are removed or migrated to a host that is +managed by *cephadm*. + +You can also disable this warning entirely with:: + + ceph config set mgr mgr/cephadm/warn_on_stray_hosts false + +CEPHADM_STRAY_DAEMON +-------------------- + +One or more Ceph daemons are running but not are not managed by +*cephadm*. This may be because they were deployed using a different +tool, or because they were started manually. Those +services cannot currently be managed by cephadm (e.g., restarted, +upgraded, or included in `ceph orch ps`). + +If the daemon is a stateful one (monitor or OSD), it should be adopted +by cephadm; see :ref:`cephadm-adoption`. For stateless daemons, it is +usually easiest to provision a new daemon with the ``ceph orch apply`` +command and then stop the unmanaged daemon. + +This warning can be disabled entirely with:: + + ceph config set mgr mgr/cephadm/warn_on_stray_daemons false + +CEPHADM_HOST_CHECK_FAILED +------------------------- + +One or more hosts have failed the basic cephadm host check, which verifies +that (1) the host is reachable and cephadm can be executed there, and (2) +that the host satisfies basic prerequisites, like a working container +runtime (podman or docker) and working time synchronization. +If this test fails, cephadm will no be able to manage services on that host. + +You can manually run this check with:: + + ceph cephadm check-host ** + +You can remove a broken host from management with:: + + ceph orch host rm ** + +You can disable this health warning with:: + + ceph config set mgr mgr/cephadm/warn_on_failed_host_check false diff --git a/doc/cephadm/troubleshooting.rst b/doc/cephadm/troubleshooting.rst new file mode 100644 index 0000000000000..375420ad18282 --- /dev/null +++ b/doc/cephadm/troubleshooting.rst @@ -0,0 +1,71 @@ + +Troubleshooting +=============== + +Sometimes there is a need to investigate why a cephadm command failed or why +a specific service no longer runs properly. + +As cephadm deploys daemons as containers, troubleshooting daemons is slightly +different. Here are a few tools and commands to help investigating issues. + +Gathering log files +------------------- + +Use journalctl to gather the log files of all daemons: + +.. note:: By default cephadm now stores logs in journald. This means + that you will no longer find daemon logs in ``/var/log/ceph/``. + +To read the log file of one specific daemon, run:: + + cephadm logs --name + +Note: this only works when run on the same host where the daemon is running. To +get logs of a daemon running on a different host, give the ``--fsid`` option:: + + cephadm logs --fsid --name + +where the ```` corresponds to the cluster ID printed by ``ceph status``. + +To fetch all log files of all daemons on a given host, run:: + + for name in $(cephadm ls | jq -r '.[].name') ; do + cephadm logs --fsid --name "$name" > $name; + done + +Collecting systemd status +------------------------- + +To print the state of a systemd unit, run:: + + systemctl status "ceph-$(cephadm shell ceph fsid)@.service"; + + +To fetch all state of all daemons of a given host, run:: + + fsid="$(cephadm shell ceph fsid)" + for name in $(cephadm ls | jq -r '.[].name') ; do + systemctl status "ceph-$fsid@$name.service" > $name; + done + + +List all downloaded container images +------------------------------------ + +To list all container images that are downloaded on a host: + +.. note:: ``Image`` might also be called `ImageID` + +:: + + podman ps -a --format json | jq '.[].Image' + "docker.io/library/centos:8" + "registry.opensuse.org/opensuse/leap:15.2" + + +Manually running containers +--------------------------- + +Cephadm writes small wrappers that run a containers. Refer to +``/var/lib/ceph///unit.run`` for the +container execution command. diff --git a/doc/cephadm/upgrade.rst b/doc/cephadm/upgrade.rst new file mode 100644 index 0000000000000..9ffa68f65df93 --- /dev/null +++ b/doc/cephadm/upgrade.rst @@ -0,0 +1,118 @@ +============== +Upgrading Ceph +============== + +Cephadm is capable of safely upgrading Ceph from one bugfix release to +another. For example, you can upgrade from v15.2.0 (the first Octopus +release) to the next point release v15.2.1. + +The automated upgrade process follows Ceph best practices. For example: + +* The upgrade order starts with managers, monitors, then other daemons. +* Each daemon is restarted only after Ceph indicates that the cluster + will remain available. + +Keep in mind that the Ceph cluster health status is likely to switch to +`HEALTH_WARNING` during the upgrade. + + +Starting the upgrade +==================== + +Before you start, you should verify that all hosts are currently online +and your cluster is healthy. + +:: + + # ceph -s + +To upgrade (or downgrade) to a specific release:: + + # ceph upgrade start --version + +For example, to upgrade to v15.2.1:: + + # ceph upgrade start --version 15.2.1 + + +Monitoring the upgrade +====================== + +Determine whether an upgrade is in process and what version the cluster is +upgrading to with:: + + # ceph upgrade status + +While the upgrade is underway, you will see a progress bar in the ceph +status output. For example:: + + # ceph -s + [...] + progress: + Upgrade to docker.io/ceph/ceph:v15.2.1 (00h 20m 12s) + [=======.....................] (time remaining: 01h 43m 31s) + +You can also watch the cephadm log with:: + + # ceph -W cephadm + + +Canceling an upgrade +==================== + +You can stop the upgrade process at any time with:: + + # ceph upgrade stop + + +Potential problems +================== + +There are a few health alerts that can arise during the upgrade process. + +UPGRADE_NO_STANDBY_MGR +---------------------- + +Ceph requires an active and standby manager daemon in order to proceed, but +there is currently no standby. + +You can ensure that Cephadm is configured to run 2 (or more) managers with:: + + # ceph orch apply mgr 2 # or more + +You can check the status of existing mgr daemons with:: + + # ceph orch ps --daemon-type mgr + +If an existing mgr daemon has stopped, you can try restarting it with:: + + # ceph orch daemon restart + +UPGRADE_FAILED_PULL +------------------- + +Ceph was unable to pull the container image for the target version. +This can happen if you specify an version or container image that does +not exist (e.g., 1.2.3), or if the container registry is not reachable from +one or more hosts in the cluster. + +You can cancel the existing upgrade and specify a different target version with:: + + # ceph upgrade stop + # ceph upgrade start --version + + +Using customized container images +================================= + +For most users, simplify specifying the Ceph version is sufficient. +Cephadm will locate the specific Ceph container image to use by +combining the ``container_image_base`` configuration option (default: +``docker.io/ceph/ceph``) with a tag of ``vX.Y.Z``. + +You can also upgrade to an arbitrary container image. For example, to +upgrade to a development build:: + + # ceph upgrade start --image quay.io/ceph-ci/ceph:recent-git-branch-name + +For more information about available container images, see :ref:`containers`. diff --git a/doc/cephfs/fs-volumes.rst b/doc/cephfs/fs-volumes.rst index 052f4f5d89e76..62eb4ec0e5b74 100644 --- a/doc/cephfs/fs-volumes.rst +++ b/doc/cephfs/fs-volumes.rst @@ -258,5 +258,28 @@ only unprotected snapshots can be removed. This guarantees that a snapshot canno .. note:: Cloning only synchronizes directories, regular files and symbolic links. Also, inode timestamps (access and modification times) are synchronized upto seconds granularity. +An `in-progress` or a `pending` clone operation can be canceled. To cancel a clone operation use the `clone cancel` command:: + + $ ceph fs clone cancel [--group_name ] + +On successful cancelation, the cloned subvolume is moved to `canceled` state:: + + $ ceph fs subvolume snapshot protect cephfs subvol1 snap1 + $ ceph fs subvolume snapshot clone cephfs subvol1 snap1 clone1 + $ ceph fs clone cancel cephfs clone1 + $ ceph fs clone status cephfs clone1 + { + "status": { + "state": "canceled", + "source": { + "volume": "cephfs", + "subvolume": "subvol1", + "snapshot": "snap1" + } + } + } + +.. note:: The canceled cloned can be deleted by using --force option in `fs subvolume rm` command. + .. _manila: https://github.com/openstack/manila .. _CSI: https://github.com/ceph/ceph-csi diff --git a/doc/index.rst b/doc/index.rst index af3b49c6c33cf..fe10a9f288c96 100644 --- a/doc/index.rst +++ b/doc/index.rst @@ -92,8 +92,8 @@ about Ceph, see our `Architecture`_ section. :hidden: start/intro - cephadm/index install/index + cephadm/index rados/index cephfs/index rbd/index diff --git a/doc/install/containers.rst b/doc/install/containers.rst index ab29c0e3ad260..c819c646dd0c5 100644 --- a/doc/install/containers.rst +++ b/doc/install/containers.rst @@ -1,3 +1,5 @@ +.. _containers: + Ceph Container Images ===================== diff --git a/doc/install/index.rst b/doc/install/index.rst index 69a48a6d003c1..507247ca90b26 100644 --- a/doc/install/index.rst +++ b/doc/install/index.rst @@ -4,109 +4,66 @@ Installing Ceph =============== -There are various options for installing Ceph. Review the documention for each method before choosing the one that best serves your needs. +There are several different ways to install Ceph. Choose the +method that best suites your needs. -We recommend the following installation methods: +Recommended methods +~~~~~~~~~~~~~~~~~~~ - * cephadm - * Rook +:ref:`Cephadm ` installs and manages a Ceph cluster using containers and +systemd, with tight integration with the CLI and dashboard GUI. +* cephadm only supports only Octopus and newer releases. +* cephadm is fully integrated with the new orchestration API and + fully supports the new CLI and dashboard features to manage + cluster deployment. +* cephadm requires container support (podman or docker) and + Python 3. -We offer these other methods of installation in addition to the ones we recommend: +`Rook `_ deploys and manages Ceph clusters running +in Kubernetes, while also enabling management of storage resources and +provisioning via Kubernetes APIs. We recommend Rook as the way to run Ceph in +Kubernetes or to connect an existing Ceph storage cluster to Kubernetes. - * ceph-ansible - * ceph-deploy (no longer actively maintained) - * Deepsea (Salt) - * Juju - * Manual installation (using packages) - * Puppet +* Rook only supports Nautilus and newer releases of Ceph. +* Rook is the preferred method for running Ceph on Kubernetes, or for + connecting a Kubernetes cluster to an existing (external) Ceph + cluster. +* Rook supports the new orchestrator API. New management features + in the CLI and dashboard are fully supported. +Other methods +~~~~~~~~~~~~~ -Recommended Methods of Ceph Installation -======================================== +`ceph-ansible `_ deploys and manages +Ceph clusters using Ansible. -cephadm -------- +* ceph-ansible is widely deployed. +* ceph-ansible is not integrated with the new orchestrator APIs, + introduced in Nautlius and Octopus, which means that newer + management features and dashboard integration are not available. -Installs Ceph using containers and systemd. -* :ref:`cephadm-bootstrap` - - * cephadm is supported only on Octopus and newer releases. - * cephadm is fully integrated with the new orcehstration API and fully supports the new CLI and dashboard features to manage cluster deployment. - * cephadm requires container support (podman or docker) and Python 3. - -Rook ----- - -Installs Ceph in Kubernetes. - -* `rook.io `_ - - * Rook supports only Nautilus and newer releases of Ceph. - * Rook is the preferred deployment method for Ceph with Kubernetes. - * Rook fully suports the new orchestrator API. New management features in the CLI and dashboard are fully supported. - -Other Methods of Ceph Installation -================================== - -ceph-ansible ------------- - -Installs Ceph using Ansible. - -* `docs.ceph.com/ceph-ansible `_ - -ceph-deploy ------------ - -Install ceph using ceph-deploy - -* :ref:`ceph-deploy-index` +:ref:`ceph-deploy ` is a tool for quickly deploying clusters. .. IMPORTANT:: - ceph-deploy is no longer actively maintained. It is not tested on versions of Ceph newer than Nautilus. It does not support RHEL8, CentOS 8, or newer operating systems. -.. toctree:: - :hidden: - - ceph-deploy/index - - -DeepSea -------- - -Install Ceph using Salt - -* `github.com/SUSE/DeepSea `_ - -Juju ----- +`DeepSea `_ installs Ceph using Salt. -Installs Ceph using Juju. +`jaas.ai/ceph-mon `_ installs Ceph using Juju. -* `jaas.ai/ceph-mon `_ +`github.com/openstack/puppet-ceph `_ installs Ceph via Puppet. +Ceph can also be :ref:`installed manually `. -Manual ------- - -Manually install Ceph using packages. - -* :ref:`install-manual` .. toctree:: :hidden: - - index_manual -Puppet ------- - -Installs Ceph using Puppet + index_manual + ceph-deploy/index -* `github.com/openstack/puppet-ceph `_ diff --git a/doc/releases/octopus.rst b/doc/releases/octopus.rst index c9968502d2ec4..b8dff086e8fea 100644 --- a/doc/releases/octopus.rst +++ b/doc/releases/octopus.rst @@ -14,7 +14,7 @@ General * A new deployment tool called **cephadm** has been introduced that integrates Ceph daemon deployment and management via containers into the orchestration layer. For more information see - :ref:`cephadm-bootstrap`. + :ref:`cephadm`. * Health alerts can now be muted, either temporarily or permanently. * A simple 'alerts' capability has been introduced to send email health alerts for clusters deployed without the benefit of an @@ -113,13 +113,21 @@ RADOS **RBD** block storage ~~~~~~~~~~~~~~~~~~~~~ +* Mirroring now supports a new snapshot-based mode that no longer requires + the journaling feature and its related impacts in exchange for the loss + of point-in-time consistency (it remains crash consistent). * Clone operations now preserve the sparseness of the underlying RBD image. * The trash feature has been improved to (optionally) automatically move old parent images to the trash when their children are all deleted or flattened. +* The trash can be configured to automatically purge on a defined schedule. +* Images can be online re-sparsified to reduce the usage of zeroed extents. * The ``rbd-nbd`` tool has been improved to use more modern kernel interfaces. * Caching has been improved to be more efficient and performant. - +* ``rbd-mirror`` automatically adjusts its per-image memory usage based + upon its memory target. +* A new persistent read-only caching daemon is available to offload reads from + shared parent images. **RGW** object storage ~~~~~~~~~~~~~~~~~~~~~~ diff --git a/qa/suites/rados/cephadm/upgrade/1-start.yaml b/qa/suites/rados/cephadm/upgrade/1-start.yaml index 25e6ee90e8502..6e974cef32f51 100644 --- a/qa/suites/rados/cephadm/upgrade/1-start.yaml +++ b/qa/suites/rados/cephadm/upgrade/1-start.yaml @@ -1,4 +1,4 @@ tasks: - cephadm: - image: quay.io/ceph-ci/ceph:wip-sage3-testing-2020-03-14-0747 - cephadm_branch: wip-sage3-testing-2020-03-14-0747 + image: quay.io/ceph-ci/ceph:wip-sage-testing-2020-03-16-1740 + cephadm_branch: wip-sage-testing-2020-03-16-1740 diff --git a/qa/suites/rados/verify/ceph.yaml b/qa/suites/rados/verify/ceph.yaml index ed4f5a5bc9112..fc5ce350ad6a4 100644 --- a/qa/suites/rados/verify/ceph.yaml +++ b/qa/suites/rados/verify/ceph.yaml @@ -1,8 +1,6 @@ overrides: ceph: conf: - global: - osd heartbeat grace: 60 mon: mon min osdmap epochs: 50 paxos service trim min: 10 @@ -12,8 +10,6 @@ overrides: mon osdmap full prune txsize: 2 osd: debug monc: 20 - debug ms: 1 - debug osd: 20 tasks: - install: - ceph: diff --git a/qa/suites/rados/verify/tasks/rados_api_tests.yaml b/qa/suites/rados/verify/tasks/rados_api_tests.yaml index d61c6f8b1297d..79f24479ae382 100644 --- a/qa/suites/rados/verify/tasks/rados_api_tests.yaml +++ b/qa/suites/rados/verify/tasks/rados_api_tests.yaml @@ -23,6 +23,8 @@ overrides: tasks: - workunit: timeout: 6h + env: + ALLOW_TIMEOUTS: "1" clients: client.0: - rados/test.sh diff --git a/qa/suites/rados/verify/validater/valgrind.yaml b/qa/suites/rados/verify/validater/valgrind.yaml index fe8e0e1cac045..83eb2add54300 100644 --- a/qa/suites/rados/verify/validater/valgrind.yaml +++ b/qa/suites/rados/verify/validater/valgrind.yaml @@ -9,18 +9,21 @@ overrides: ceph: conf: global: - osd heartbeat grace: 40 - debug refs: 5 + osd heartbeat grace: 80 mon: mon osd crush smoke test: false osd: osd fast shutdown: false + debug bluestore: 1 + debug bluefs: 1 log-whitelist: - overall HEALTH_ # valgrind is slow.. we might get PGs stuck peering etc - \(PG_ # mons sometimes are left off of initial quorum due to valgrind slowness. ok to whitelist here because we'll still catch an actual crash due to the core - \(MON_DOWN\) + - \(SLOW_OPS\) + - slow request valgrind: mon: [--tool=memcheck, --leak-check=full, --show-reachable=yes] osd: [--tool=memcheck] diff --git a/qa/tasks/cephadm.py b/qa/tasks/cephadm.py index 8f7947571fc6e..2b076053a784c 100644 --- a/qa/tasks/cephadm.py +++ b/qa/tasks/cephadm.py @@ -353,11 +353,13 @@ def ceph_bootstrap(ctx, config): 'sudo', ctx.cephadm, '--image', ctx.ceph[cluster_name].image, + '-v', 'bootstrap', '--fsid', fsid, '--mon-id', first_mon, '--mgr-id', first_mgr, '--orphan-initial-daemons', # we will do it explicitly! + '--skip-monitoring-stack', # we'll provision these explicitly '--config', '{}/seed.{}.conf'.format(testdir, cluster_name), '--output-config', '/etc/ceph/{}.conf'.format(cluster_name), '--output-keyring', @@ -411,6 +413,10 @@ def ceph_bootstrap(ctx, config): 'sudo', 'chmod', '0600', '/root/.ssh/authorized_keys', ]) + # set options + _shell(ctx, cluster_name, bootstrap_remote, + ['ceph', 'config', 'set', 'mgr', 'mgr/cephadm/allow_ptrace', 'true']) + # add other hosts for remote in ctx.cluster.remotes.keys(): if remote == bootstrap_remote: diff --git a/qa/tasks/cephfs/test_volumes.py b/qa/tasks/cephfs/test_volumes.py index ea8bc937eb204..6f5bdec237a3c 100644 --- a/qa/tasks/cephfs/test_volumes.py +++ b/qa/tasks/cephfs/test_volumes.py @@ -49,6 +49,9 @@ def _wait_for_clone_to_complete(self, clone, clone_group=None, timo=120): def _wait_for_clone_to_fail(self, clone, clone_group=None, timo=120): self.__check_clone_state("failed", clone, clone_group, timo) + def _check_clone_canceled(self, clone, clone_group=None): + self.__check_clone_state("canceled", clone, clone_group, timo=1) + def _verify_clone_attrs(self, subvolume, clone, source_group=None, clone_group=None): path1 = self._get_subvolume_path(self.volname, subvolume, group_name=source_group) path2 = self._get_subvolume_path(self.volname, clone, group_name=clone_group) @@ -2126,3 +2129,115 @@ def test_subvolume_snapshot_attr_clone(self): # verify trash dir is clean self._wait_for_trash_empty() + + def test_subvolume_snapshot_clone_cancel_in_progress(self): + subvolume = self._generate_random_subvolume_name() + snapshot = self._generate_random_snapshot_name() + clone = self._generate_random_clone_name() + + # create subvolume + self._fs_cmd("subvolume", "create", self.volname, subvolume) + + # do some IO + self._do_subvolume_io(subvolume, number_of_files=128) + + # snapshot subvolume + self._fs_cmd("subvolume", "snapshot", "create", self.volname, subvolume, snapshot) + + # now, protect snapshot + self._fs_cmd("subvolume", "snapshot", "protect", self.volname, subvolume, snapshot) + + # schedule a clone + self._fs_cmd("subvolume", "snapshot", "clone", self.volname, subvolume, snapshot, clone) + + # cancel on-going clone + self._fs_cmd("clone", "cancel", self.volname, clone) + + # verify canceled state + self._check_clone_canceled(clone) + + # now, unprotect snapshot + self._fs_cmd("subvolume", "snapshot", "unprotect", self.volname, subvolume, snapshot) + + # remove snapshot + self._fs_cmd("subvolume", "snapshot", "rm", self.volname, subvolume, snapshot) + + # remove subvolumes + self._fs_cmd("subvolume", "rm", self.volname, subvolume) + self._fs_cmd("subvolume", "rm", self.volname, clone, "--force") + + # verify trash dir is clean + self._wait_for_trash_empty() + + def test_subvolume_snapshot_clone_cancel_pending(self): + """ + this test is a bit more involved compared to canceling an in-progress clone. + we'd need to ensure that a to-be canceled clone has still not been picked up + by cloner threads. exploit the fact that clones are picked up in an FCFS + fashion and there are four (4) cloner threads by default. When the number of + cloner threads increase, this test _may_ start tripping -- so, the number of + clone operations would need to be jacked up. + """ + # default number of clone threads + NR_THREADS = 4 + # good enough for 4 threads + NR_CLONES = 5 + # yeh, 1gig -- we need the clone to run for sometime + FILE_SIZE_MB = 1024 + + subvolume = self._generate_random_subvolume_name() + snapshot = self._generate_random_snapshot_name() + clones = self._generate_random_clone_name(NR_CLONES) + + # create subvolume + self._fs_cmd("subvolume", "create", self.volname, subvolume) + + # do some IO + self._do_subvolume_io(subvolume, number_of_files=4, file_size=FILE_SIZE_MB) + + # snapshot subvolume + self._fs_cmd("subvolume", "snapshot", "create", self.volname, subvolume, snapshot) + + # now, protect snapshot + self._fs_cmd("subvolume", "snapshot", "protect", self.volname, subvolume, snapshot) + + # schedule clones + for clone in clones: + self._fs_cmd("subvolume", "snapshot", "clone", self.volname, subvolume, snapshot, clone) + + to_wait = clones[0:NR_THREADS] + to_cancel = clones[NR_THREADS:] + + # cancel pending clones and verify + for clone in to_cancel: + status = json.loads(self._fs_cmd("clone", "status", self.volname, clone)) + self.assertEqual(status["status"]["state"], "pending") + self._fs_cmd("clone", "cancel", self.volname, clone) + self._check_clone_canceled(clone) + + # let's cancel on-going clones. handle the case where some of the clones + # _just_ complete + for clone in list(to_wait): + try: + self._fs_cmd("clone", "cancel", self.volname, clone) + to_cancel.append(clone) + to_wait.remove(clone) + except CommandFailedError as ce: + if ce.exitstatus != errno.EINVAL: + raise RuntimeError("invalid error code when cancelling on-going clone") + + # now, unprotect snapshot + self._fs_cmd("subvolume", "snapshot", "unprotect", self.volname, subvolume, snapshot) + + # remove snapshot + self._fs_cmd("subvolume", "snapshot", "rm", self.volname, subvolume, snapshot) + + # remove subvolumes + self._fs_cmd("subvolume", "rm", self.volname, subvolume) + for clone in to_wait: + self._fs_cmd("subvolume", "rm", self.volname, clone) + for clone in to_cancel: + self._fs_cmd("subvolume", "rm", self.volname, clone, "--force") + + # verify trash dir is clean + self._wait_for_trash_empty() diff --git a/qa/tasks/mgr/dashboard/test_osd.py b/qa/tasks/mgr/dashboard/test_osd.py index 0c17be4f6a104..1bd75e4b09472 100644 --- a/qa/tasks/mgr/dashboard/test_osd.py +++ b/qa/tasks/mgr/dashboard/test_osd.py @@ -111,8 +111,10 @@ def test_create_lost_destroy_remove(self): def test_create_with_drive_group(self): data = { 'method': 'drive_groups', - 'data': { - 'test': { + 'data': [ + { + 'service_type': 'osd', + 'service_id': 'test', 'host_pattern': '*', 'data_devices': { 'vendor': 'abc', @@ -136,7 +138,7 @@ def test_create_with_drive_group(self): 'db_slots': 5, 'encrypted': True } - }, + ], 'tracking_id': 'test' } self._post('/api/osd', data) diff --git a/qa/workunits/cephadm/test_cephadm.sh b/qa/workunits/cephadm/test_cephadm.sh index 7ce4eaf1e534f..b6c6cad306cfb 100755 --- a/qa/workunits/cephadm/test_cephadm.sh +++ b/qa/workunits/cephadm/test_cephadm.sh @@ -6,7 +6,7 @@ SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )" FSID='00000000-0000-0000-0000-0000deadbeef' # images that are used -IMAGE_MASTER=${IMAGE_MASTER:-'docker.io/ceph/daemon-base:latest-master-devel'} +IMAGE_MASTER=${IMAGE_MASTER:-'quay.io/ceph-ci/ceph:octopus'} # octopus for octopus branch IMAGE_NAUTILUS=${IMAGE_NAUTILUS:-'docker.io/ceph/daemon-base:latest-nautilus'} IMAGE_MIMIC=${IMAGE_MIMIC:-'docker.io/ceph/daemon-base:latest-mimic'} @@ -190,7 +190,10 @@ $CEPHADM bootstrap \ --config $ORIG_CONFIG \ --output-config $CONFIG \ --output-keyring $KEYRING \ - --allow-overwrite + --output-pub-ssh-key $TMPDIR/ceph.pub \ + --allow-overwrite \ + --skip-mon-network \ + --skip-monitoring-stack test -e $CONFIG test -e $KEYRING rm -f $ORIG_CONFIG @@ -211,6 +214,11 @@ systemctl | grep system-ceph | grep -q .slice # naming is escaped and annoying $CEPHADM shell --fsid $FSID --config $CONFIG --keyring $KEYRING -- \ ceph -s | grep $FSID +for t in mon mgr node-exporter prometheus grafana; do + $CEPHADM shell --fsid $FSID --config $CONFIG --keyring $KEYRING -- \ + ceph orch apply $t --unmanaged +done + ## ls $CEPHADM ls | jq '.[]' | jq 'select(.name == "mon.a").fsid' \ | grep $FSID @@ -299,6 +307,8 @@ $CEPHADM shell --fsid $FSID --config $CONFIG --keyring $KEYRING -- \ ceph osd pool create $nfs_rados_pool 64 $CEPHADM shell --fsid $FSID --config $CONFIG --keyring $KEYRING -- \ rados --pool nfs-ganesha --namespace nfs-ns create conf-nfs.a +$CEPHADM shell --fsid $FSID --config $CONFIG --keyring $KEYRING -- \ + ceph orch pause $CEPHADM deploy --name nfs.a \ --fsid $FSID \ --keyring $KEYRING \ @@ -306,6 +316,8 @@ $CEPHADM deploy --name nfs.a \ --config-json ${CEPHADM_SAMPLES_DIR}/nfs.json cond="$SUDO ss -tlnp '( sport = :nfs )' | grep 'ganesha.nfsd'" is_available "nfs" "$cond" 10 +$CEPHADM shell --fsid $FSID --config $CONFIG --keyring $KEYRING -- \ + ceph orch resume ## run # WRITE ME diff --git a/qa/workunits/rbd/permissions.sh b/qa/workunits/rbd/permissions.sh index 68144d2b9402e..f8a9aaa7128ef 100755 --- a/qa/workunits/rbd/permissions.sh +++ b/qa/workunits/rbd/permissions.sh @@ -168,12 +168,11 @@ create_self_managed_snapshot() { cat << EOF | CEPH_ARGS="-k $KEYRING" python3 import rados -cluster = rados.Rados(conffile="", rados_id="${ID}") -cluster.connect() -ioctx = cluster.open_ioctx("${POOL}") +with rados.Rados(conffile="", rados_id="${ID}") as cluster: + ioctx = cluster.open_ioctx("${POOL}") -snap_id = ioctx.create_self_managed_snap() -print ("Created snap id {}".format(snap_id)) + snap_id = ioctx.create_self_managed_snap() + print ("Created snap id {}".format(snap_id)) EOF } @@ -184,19 +183,17 @@ remove_self_managed_snapshot() { cat << EOF | CEPH_ARGS="-k $KEYRING" python3 import rados -cluster1 = rados.Rados(conffile="", rados_id="mon_write") -cluster1.connect() -ioctx1 = cluster1.open_ioctx("${POOL}") +with rados.Rados(conffile="", rados_id="mon_write") as cluster1, \ + rados.Rados(conffile="", rados_id="${ID}") as cluster2: + ioctx1 = cluster1.open_ioctx("${POOL}") -snap_id = ioctx1.create_self_managed_snap() -print ("Created snap id {}".format(snap_id)) + snap_id = ioctx1.create_self_managed_snap() + print ("Created snap id {}".format(snap_id)) -cluster2 = rados.Rados(conffile="", rados_id="${ID}") -cluster2.connect() -ioctx2 = cluster2.open_ioctx("${POOL}") + ioctx2 = cluster2.open_ioctx("${POOL}") -ioctx2.remove_self_managed_snap(snap_id) -print ("Removed snap id {}".format(snap_id)) + ioctx2.remove_self_managed_snap(snap_id) + print ("Removed snap id {}".format(snap_id)) EOF } diff --git a/src/cephadm/cephadm b/src/cephadm/cephadm index 1357f1e210a63..af145a459bb9a 100755 --- a/src/cephadm/cephadm +++ b/src/cephadm/cephadm @@ -1,6 +1,7 @@ #!/usr/bin/python3 -DEFAULT_IMAGE='docker.io/ceph/daemon-base:latest-master-devel' # FIXME when octopus is ready!!! +#DEFAULT_IMAGE='docker.io/ceph/ceph:v15.2' +DEFAULT_IMAGE='quay.io/ceph-ci/ceph:octopus' DATA_DIR='/var/lib/ceph' LOG_DIR='/var/log/ceph' LOCK_DIR='/run/cephadm' @@ -1422,9 +1423,11 @@ def get_container_mounts(fsid, daemon_type, daemon_id, return mounts -def get_container(fsid, daemon_type, daemon_id, privileged=False, +def get_container(fsid, daemon_type, daemon_id, + privileged=False, + ptrace=False, container_args=[]): - # type: (str, str, Union[int, str], bool, List[str]) -> CephContainer + # type: (str, str, Union[int, str], bool, bool, List[str]) -> CephContainer if daemon_type in ['mon', 'osd']: # mon and osd need privileged in order for libudev to query devices privileged = True @@ -1484,6 +1487,7 @@ def get_container(fsid, daemon_type, daemon_id, privileged=False, cname='ceph-%s-%s.%s' % (fsid, daemon_type, daemon_id), envs=envs, privileged=privileged, + ptrace=ptrace, ) def extract_uid_gid(img='', file_path='/var/lib/ceph'): @@ -1838,16 +1842,18 @@ class CephContainer: cname='', container_args=[], envs=None, - privileged=False): - # type: (str, str, List[str], Dict[str, str], str, List[str], Optional[List[str]], Optional[bool]) -> None + privileged=False, + ptrace=False): + # type: (str, str, List[str], Dict[str, str], str, List[str], Optional[List[str]], bool, bool) -> None self.image = image self.entrypoint = entrypoint self.args = args self.volume_mounts = volume_mounts self.cname = cname self.container_args = container_args - self.privileged = privileged self.envs = envs + self.privileged = privileged + self.ptrace = ptrace def run_cmd(self): # type: () -> List[str] @@ -1863,6 +1869,8 @@ class CephContainer: priv = ['--privileged', # let OSD etc read block devs that haven't been chowned '--group-add=disk'] + if self.ptrace: + priv.append('--cap-add=SYS_PTRACE') vols = sum( [['-v', '%s:%s' % (host_dir, container_dir)] for host_dir, container_dir in self.volume_mounts.items()], []) @@ -2010,6 +2018,7 @@ def command_bootstrap(): # ip r = re.compile(r':(\d+)$') + base_ip = None if args.mon_ip: hasport = r.findall(args.mon_ip) if hasport: @@ -2022,8 +2031,10 @@ def command_bootstrap(): logger.warning('Using msgr2 protocol for unrecognized port %d' % port) addr_arg = '[v2:%s]' % args.mon_ip - check_ip_port(args.mon_ip[0:-(len(str(port)))-1], port) + base_ip = args.mon_ip[0:-(len(str(port)))-1] + check_ip_port(base_ip, port) else: + base_ip = args.mon_ip addr_arg = '[v2:%s:3300,v1:%s:6789]' % (args.mon_ip, args.mon_ip) check_ip_port(args.mon_ip, 3300) check_ip_port(args.mon_ip, 6789) @@ -2040,10 +2051,25 @@ def command_bootstrap(): port = int(hasport[0]) # strip off v1: or v2: prefix addr = re.sub(r'^\w+:', '', addr) - check_ip_port(addr[0:-(len(str(port)))-1], port) + base_ip = addr[0:-(len(str(port)))-1] + check_ip_port(base_ip, port) else: raise Error('must specify --mon-ip or --mon-addrv') - logger.debug('Final addrv is %s' % addr_arg) + logger.debug('Base mon IP is %s, final addrv is %s' % (base_ip, addr_arg)) + + mon_network = None + if not args.skip_mon_network: + # make sure IP is configured locally, and then figure out the + # CIDR network + for net, ips in list_networks().items(): + if base_ip in ips: + mon_network = net + logger.info('Mon IP %s is in CIDR network %s' % (base_ip, + mon_network)) + break + if not mon_network: + raise Error('Failed to infer CIDR network for mon ip %s; pass ' + '--skip-mon-network to configure it later' % base_ip) # config cp = read_config(args.config) @@ -2226,6 +2252,10 @@ def command_bootstrap(): get_unit_name(fsid, 'mon', mon_id) ]) + if mon_network: + logger.info('Setting mon public_network...') + cli(['config', 'set', 'mon', 'public_network', mon_network]) + # create mgr logger.info('Creating mgr...') mgr_keyring = '[mgr.%s]\n\tkey = %s\n' % (mgr_id, mgr_key) @@ -2317,6 +2347,13 @@ def command_bootstrap(): logger.info('Deploying %s service with default placement...' % t) cli(['orch', 'apply', t]) + if not args.skip_monitoring_stack: + logger.info('Enabling mgr prometheus module...') + cli(['mgr', 'module', 'enable', 'prometheus']) + for t in ['prometheus', 'grafana', 'node-exporter', 'alertmanager']: + logger.info('Deploying %s service with default placement...' % t) + cli(['orch', 'apply', t]) + if not args.skip_dashboard: logger.info('Enabling the dashboard module...') cli(['mgr', 'module', 'enable', 'dashboard']) @@ -2399,7 +2436,8 @@ def command_deploy(): (config, keyring) = get_config_and_keyring() (uid, gid) = extract_uid_gid() make_var_run(args.fsid, uid, gid) - c = get_container(args.fsid, daemon_type, daemon_id) + c = get_container(args.fsid, daemon_type, daemon_id, + ptrace=args.allow_ptrace) deploy_daemon(args.fsid, daemon_type, daemon_id, c, uid, gid, config=config, keyring=keyring, osd_fsid=args.osd_fsid, @@ -2624,6 +2662,39 @@ def command_logs(): ################################## +def list_networks(): + # type: () -> Dict[str,List[str]] + + ## sadly, 18.04's iproute2 4.15.0-2ubun doesn't support the -j flag, + ## so we'll need to use a regex to parse 'ip' command output. + #out, _, _ = call_throws(['ip', '-j', 'route', 'ls']) + #j = json.loads(out) + #for x in j: + + out, _, _ = call_throws([find_executable('ip'), 'route', 'ls']) + return _parse_ip_route(out) + +def _parse_ip_route(out): + r = {} # type: Dict[str,List[str]] + p = re.compile(r'^(\S+) (.*)scope link (.*)src (\S+)') + for line in out.splitlines(): + m = p.findall(line) + if not m: + continue + net = m[0][0] + ip = m[0][3] + if net not in r: + r[net] = [] + r[net].append(ip) + return r + +def command_list_networks(): + # type: () -> None + r = list_networks() + print(json.dumps(r, indent=4)) + +################################## + def command_ls(): # type: () -> None ls = list_daemons(detail=not args.no_detail, @@ -3247,9 +3318,9 @@ class CustomValidation(argparse.Action): ################################## def get_distro(): - id_ = None - version = None - codename = None + distro = None + distro_version = None + distro_codename = None with open('/etc/os-release', 'r') as f: for line in f.readlines(): line = line.strip() @@ -3259,20 +3330,21 @@ def get_distro(): if val[0] == '"' and val[-1] == '"': val = val[1:-1] if var == 'ID': - id_ = val.lower() + distro = val.lower() elif var == 'VERSION_ID': - version = val.lower() + distro_version = val.lower() elif var == 'VERSION_CODENAME': - codename = val.lower() - return id_, version, codename + distro_codename = val.lower() + return distro, distro_version, distro_codename class Packager(object): - def __init__(self, stable=None, branch=None, commit=None): + def __init__(self, stable=None, version=None, branch=None, commit=None): assert \ (stable and not branch and not commit) or \ - (not stable and branch) or \ - (not stable and not branch and not commit) + (not stable and not version and branch) or \ + (not stable and not version and not branch and not commit) self.stable = stable + self.version = version self.branch = branch self.commit = commit @@ -3282,13 +3354,13 @@ class Packager(object): def rm_repo(self): raise NotImplementedError - def query_shaman(self, distro, version, branch, commit): + def query_shaman(self, distro, distro_version, branch, commit): # query shaman logging.info('Fetching repo metadata from shaman and chacra...') - shaman_url = 'https://shaman.ceph.com/api/repos/ceph/{version}/{sha1}/{distro}/{distro_version}/repo/?arch={arch}'.format( + shaman_url = 'https://shaman.ceph.com/api/repos/ceph/{branch}/{sha1}/{distro}/{distro_version}/repo/?arch={arch}'.format( distro=distro, - distro_version=version, - version=branch, + distro_version=distro_version, + branch=branch, sha1=commit or 'latest', arch=get_arch() ) @@ -3326,11 +3398,12 @@ class Apt(Packager): 'debian': 'debian', } - def __init__(self, stable, branch, commit, - distro, version, codename): - super(Apt, self).__init__(stable=stable, branch=branch, commit=commit) + def __init__(self, stable, version, branch, commit, + distro, distro_version, distro_codename): + super(Apt, self).__init__(stable=stable, version=version, + branch=branch, commit=commit) self.distro = self.DISTRO_NAMES[distro] - self.codename = codename + self.distro_codename = distro_codename def repo_path(self): return '/etc/apt/sources.list.d/ceph.list' @@ -3349,10 +3422,15 @@ class Apt(Packager): f.write(key) if self.stable: - content = 'deb %s/debian-%s/ %s main\n' % ( - args.repo_url, self.stable, self.codename) + if self.version: + content = 'deb %s/debian-%s-%s/ %s main\n' % ( + args.repo_url, self.stable, self.version, + self.distro_codename) + else: + content = 'deb %s/debian-%s/ %s main\n' % ( + args.repo_url, self.stable, self.distro_codename) else: - content = self.query_shaman(self.distro, self.codename, self.branch, + content = self.query_shaman(self.distro, self.distro_codename, self.branch, self.commit) logging.info('Installing repo file at %s...' % self.repo_path()) @@ -3395,10 +3473,11 @@ class YumDnf(Packager): 'fedora': ('fedora', 'fc'), } - def __init__(self, stable, branch, commit, - distro, version): - super(YumDnf, self).__init__(stable=stable, branch=branch, commit=commit) - self.major = int(version.split('.')[0]) + def __init__(self, stable, version, branch, commit, + distro, distro_version): + super(YumDnf, self).__init__(stable=stable, version=version, + branch=branch, commit=commit) + self.major = int(distro_version.split('.')[0]) self.distro_normalized = self.DISTRO_NAMES[distro][0] self.distro_code = self.DISTRO_NAMES[distro][1] + str(self.major) if (self.distro_code == 'fc' and self.major >= 30) or \ @@ -3468,7 +3547,13 @@ class YumDnf(Packager): def repo_baseurl(self): assert self.stable - return '%s/rpm-%s/%s' % (args.repo_url, self.stable, self.distro_code) + if self.version: + return '%s/rpm-%s-%s/%s' % (args.repo_url, self.stable, + self.version, + self.distro_code) + else: + return '%s/rpm-%s/%s' % (args.repo_url, self.stable, + self.distro_code) def add_repo(self): if self.stable: @@ -3525,14 +3610,15 @@ class Zypper(Packager): 'opensuse-leap' ] - def __init__(self, stable, branch, commit, - distro, version): - super(Zypper, self).__init__(stable=stable, branch=branch, commit=commit) + def __init__(self, stable, version, branch, commit, + distro, distro_version): + super(Zypper, self).__init__(stable=stable, version=version, + branch=branch, commit=commit) self.tool = 'zypper' self.distro = 'opensuse' - self.version = '15.1' - if 'tumbleweed' not in distro and version is not None: - self.version = version + self.distro_version = '15.1' + if 'tumbleweed' not in distro and distro_version is not None: + self.distro_version = distro_version def custom_repo(self, **kw): """ @@ -3568,6 +3654,9 @@ class Zypper(Packager): def repo_baseurl(self): assert self.stable + if self.version: + return '%s/rpm-%s-%s/%s' % (args.repo_url, self.stable, + self.version, self.distro) return '%s/rpm-%s/%s' % (args.repo_url, self.stable, self.distro) def add_repo(self): @@ -3587,7 +3676,7 @@ class Zypper(Packager): ) content += '\n\n' else: - content = self.query_shaman(self.distro, self.version, + content = self.query_shaman(self.distro, self.distro_version, self.branch, self.commit) @@ -3607,22 +3696,49 @@ class Zypper(Packager): self.install(['podman']) -def create_packager(stable=None, branch=None, commit=None): - distro, version, codename = get_distro() +def create_packager(stable=None, version=None, branch=None, commit=None): + distro, distro_version, distro_codename = get_distro() if distro in YumDnf.DISTRO_NAMES: - return YumDnf(stable=stable, branch=branch, commit=commit, - distro=distro, version=version) + return YumDnf(stable=stable, version=version, + branch=branch, commit=commit, + distro=distro, distro_version=distro_version) elif distro in Apt.DISTRO_NAMES: - return Apt(stable=stable, branch=branch, commit=commit, - distro=distro, version=version, codename=codename) + return Apt(stable=stable, version=version, + branch=branch, commit=commit, + distro=distro, distro_version=distro_version, + distro_codename=distro_codename) elif distro in Zypper.DISTRO_NAMES: - return Zypper(stable=stable, branch=branch, commit=commit, - distro=distro, version=version) - raise Error('Distro %s version %s not supported' % (distro, version)) + return Zypper(stable=stable, version=version, + branch=branch, commit=commit, + distro=distro, distro_version=distro_version) + raise Error('Distro %s version %s not supported' % (distro, distro_version)) def command_add_repo(): - pkg = create_packager(stable=args.release, branch=args.dev, + if args.version and args.release: + raise Error('you can specify either --release or --version but not both') + if args.version: + try: + (x, y, z) = args.version.split('.') + except Exception as e: + raise Error('version must be in the form x.y.z (e.g., 15.2.0)') + relnames = { + '16': 'pacific', + '15': 'octopus', + '14': 'nautilus', + '13': 'mimic', + '12': 'luminous', + '11': 'kraken', + '10': 'jewel', + } + args.release = relnames.get(x, None) + if not args.release: + raise Error('unknown release %s (not in %s)' % ( + x, ' '.join(relnames.values()))) + + pkg = create_packager(stable=args.release, + version=args.version, + branch=args.dev, commit=args.dev_commit) pkg.add_repo() @@ -3706,6 +3822,10 @@ def _get_parser(): default='/', help='base directory for legacy daemon data') + parser_list_networks = subparsers.add_parser( + 'list-networks', help='list IP networks') + parser_list_networks.set_defaults(func=command_list_networks) + parser_adopt = subparsers.add_parser( 'adopt', help='adopt daemon deployed with a different tool') parser_adopt.set_defaults(func=command_adopt) @@ -3916,6 +4036,10 @@ def _get_parser(): '--dashboard-crt', help='Dashboard certificate') + parser_bootstrap.add_argument( + '--skip-mon-network', + action='store_true', + help='set mon public_network based on bootstrap mon ip') parser_bootstrap.add_argument( '--skip-dashboard', action='store_true', @@ -3956,6 +4080,10 @@ def _get_parser(): '--orphan-initial-daemons', action='store_true', help='Do not create initial mon, mgr, and crash service specs') + parser_bootstrap.add_argument( + '--skip-monitoring-stack', + action='store_true', + help='Do not automatically provision monitoring stack (prometheus, grafana, alertmanager, node-exporter)') parser_deploy = subparsers.add_parser( 'deploy', help='deploy a daemon') @@ -3992,6 +4120,10 @@ def _get_parser(): '--reconfig', action='store_true', help='Reconfigure a previously deployed daemon') + parser_deploy.add_argument( + '--allow-ptrace', + action='store_true', + help='Allow SYS_PTRACE on daemon container') parser_check_host = subparsers.add_parser( 'check-host', help='check host configuration') @@ -4012,7 +4144,10 @@ def _get_parser(): parser_add_repo.set_defaults(func=command_add_repo) parser_add_repo.add_argument( '--release', - help='use specified upstream release') + help='use latest version of a named release (e.g., octopus)') + parser_add_repo.add_argument( + '--version', + help='use specific upstream version (x.y.z)') parser_add_repo.add_argument( '--dev', help='use specified bleeding edge build from git branch or tag') @@ -4024,7 +4159,7 @@ def _get_parser(): help='specify alternative GPG key location') parser_add_repo.add_argument( '--repo-url', - default='https://download.ceph.com/', + default='https://download.ceph.com', help='specify alternative repo location') # TODO: proxy? diff --git a/src/cephadm/tests/test_cephadm.py b/src/cephadm/tests/test_cephadm.py index b0dad3cd42d67..311abbe39583e 100644 --- a/src/cephadm/tests/test_cephadm.py +++ b/src/cephadm/tests/test_cephadm.py @@ -43,3 +43,56 @@ def test_parse_podman_version_invalid(self): with pytest.raises(ValueError) as res: cd._parse_podman_version('podman version inval.id') assert 'inval' in str(res.value) + + @pytest.mark.parametrize("test_input, expected", [ + ( +""" +default via 192.168.178.1 dev enxd89ef3f34260 proto dhcp metric 100 +10.0.0.0/8 via 10.4.0.1 dev tun0 proto static metric 50 +10.3.0.0/21 via 10.4.0.1 dev tun0 proto static metric 50 +10.4.0.1 dev tun0 proto kernel scope link src 10.4.0.2 metric 50 +137.1.0.0/16 via 10.4.0.1 dev tun0 proto static metric 50 +138.1.0.0/16 via 10.4.0.1 dev tun0 proto static metric 50 +139.1.0.0/16 via 10.4.0.1 dev tun0 proto static metric 50 +140.1.0.0/17 via 10.4.0.1 dev tun0 proto static metric 50 +141.1.0.0/16 via 10.4.0.1 dev tun0 proto static metric 50 +169.254.0.0/16 dev docker0 scope link metric 1000 +172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 +192.168.39.0/24 dev virbr1 proto kernel scope link src 192.168.39.1 linkdown +192.168.122.0/24 dev virbr0 proto kernel scope link src 192.168.122.1 linkdown +192.168.178.0/24 dev enxd89ef3f34260 proto kernel scope link src 192.168.178.28 metric 100 +192.168.178.1 dev enxd89ef3f34260 proto static scope link metric 100 +195.135.221.12 via 192.168.178.1 dev enxd89ef3f34260 proto static metric 100 +""", + { + '10.4.0.1': ['10.4.0.2'], + '172.17.0.0/16': ['172.17.0.1'], + '192.168.39.0/24': ['192.168.39.1'], + '192.168.122.0/24': ['192.168.122.1'], + '192.168.178.0/24': ['192.168.178.28'] + } + ), ( +""" +default via 10.3.64.1 dev eno1 proto static metric 100 +10.3.64.0/24 dev eno1 proto kernel scope link src 10.3.64.23 metric 100 +10.3.64.0/24 dev eno1 proto kernel scope link src 10.3.64.27 metric 100 +10.88.0.0/16 dev cni-podman0 proto kernel scope link src 10.88.0.1 linkdown +172.21.0.0/20 via 172.21.3.189 dev tun0 +172.21.1.0/20 via 172.21.3.189 dev tun0 +172.21.2.1 via 172.21.3.189 dev tun0 +172.21.3.1 dev tun0 proto kernel scope link src 172.21.3.2 +172.21.4.0/24 via 172.21.3.1 dev tun0 +172.21.5.0/24 via 172.21.3.1 dev tun0 +172.21.6.0/24 via 172.21.3.1 dev tun0 +172.21.7.0/24 via 172.21.3.1 dev tun0 +192.168.122.0/24 dev virbr0 proto kernel scope link src 192.168.122.1 linkdown +""", + { + '10.3.64.0/24': ['10.3.64.23', '10.3.64.27'], + '10.88.0.0/16': ['10.88.0.1'], + '172.21.3.1': ['172.21.3.2'], + '192.168.122.0/24': ['192.168.122.1']} + ), + ]) + def test_parse_ip_route(self, test_input, expected): + assert cd._parse_ip_route(test_input) == expected diff --git a/src/common/options.cc b/src/common/options.cc index a9162ebe20e13..50d123f9085bc 100644 --- a/src/common/options.cc +++ b/src/common/options.cc @@ -439,7 +439,8 @@ std::vector