Team wise storage

NOTICE: This tool has been deprecated, please refer to Setup Kubernetes Persistent Volumes as Storage on PAI.

A tool to manage external storage in PAI.

Index

• What is team wise storage
• Team wise storage usages
  • Setup server
  • Create storage server in PAI
  • Create storage config in PAI
  • Set storage config access for group
  • Use Storage in PAI
  • Example
• Storage data structure
  • Server data structure
    • Nfs Server data structure
    • Samba Server data structure
    • Azurefile Server data structure
    • Azureblob Server data structure
    • Hdfs Server data structure
  • Config data structure
  • Config in group data

What is team wise storage

Team wise storage is a solution that helps admin to manage NAS(network attached storage) by team/group. After admin configured team wise storage settings, users can easily use NAS in their jobs.
Team wise storage solution offers:

• Multiple NAS support, including NFS, Samba, Azurefile, Azureblob and HDFS
• Configurable mount structure settings
• Mixed usage for different NAS
• Configuration for Team/Group scope

Team wise storage usages

Setup server

• NFS

Edit /etc/exports, export /root/path/to/share

/root/path/to/share (rw, sync, no_root_squash)

no_root_squash is needed for storage plugin to creae folders.

• Samba

After create samba server, create user for PAI to use samba.

useradd paismb
smbpasswd -a paismb
#Input password for paismb

• Azurefile

Create Azurefile share through azure web portal.

• Azureblob

Create Azureblob share through azure web portal.

Create storage server in PAI

In PAI dev-box, swith to folder pai/contrib/storage-plugin

Create server config using command:

• NFS:

python storagectl.py server set NAME nfs ADDRESS ROOTPATH

• Samba:

python storagectl.py server set NAME samba ADDRESS ROOTPATH USERNAME PASSWORD DOMAIN

• Azurefile:

python storagectl.py server set NAME azurefile DATASTORE FILESHARE ACCOUNTNAME KEY

• Azureblob:

python storagectl.py server set NAME azureblob DATASTORE CONTAINERNAME ACCOUNTNAME KEY

• HDFS:

python storagectl.py server set NAME hdfs NAMENODE PORT

Create storage config in PAI

In PAI dev-box, swith to folder pai/contrib/storage-plugin

Create config using command:

python storagectl.py config set CONFIG_NAME GROUP_NAME [-s SERVER_NAME_1 SERVER_NAME_2 ...] [-m MOUNT_POINT SERVER PATH]... [-d]

Set storage config access for group

In PAI dev-box, swith to folder pai/contrib/storage-plugin

Set storage config access for group using command:

python storagectl.py groupsc add GROUP_NAME CONFIG_NAME

Use Storage info in job container

User can use team wise storage through job submit page. Please refer to related page for details.

Example

Suppose admin has set up a new samba server "smbserver" on "10.0.0.0", created PAI account "paismb" with password "paipwd". The structure of samba server is as follows:

-- root
    -- data
    -- users
        -- user1
        -- user2
        ...

Now we want all members of "paigroup" mount server's data folder to /data, and user's data (e.g user1) to /user by default. The admin should setup storage config in PAI using:

python storagectl.py server set smbserver samba 10.0.0.1 root paismb paipwd local
python storagectl.py config set configsmb -s smbserver -m /data smbserver data -m /user smbserver 'users/${PAI_USER_NAME}' -d
python storagectl.py groupsc add paigroup configsmb

Then when "paiuser" from "paigroup" uses job submit page, the configsmb will be shown and user can choose whether to use it

Team wise storage data structures

Server data structure

{
	"spn": "servername",
	"type": "nfs|samba|azurefile|azureblob"
}

Nfs Server data structure

{
	"spn": "servername",
	"type": "nfs",
	"address": "server/address",
	"rootPath": "server/root/path"
}

Samba Server data structure

{
	"spn": "servername",
	"type": "samba",
	"address": "server/address",
	"rootPath": "server/root/path",
	"userName": "username",
	"password": "password",
	"domain": "userdomain"
}

Azurefile Server data structure

{
	"spn": "servername",
	"type": "azurefile",
	"dataStore": "datastore",
	"fileShare": "fileshare",
	"accountName": "accountname",
	"key": "key"
}

Azureblob Server data structure

{
	"spn": "servername",
	"type": "azureblob",
	"dataStore": "datastore",
	"containerName": "containername",
	"accountName": "accountname",
	"key": "key"
}

Hdfs Server data structure

{
	"spn": "servername",
	"type": "hdfs",
	"namenode": "namenode",
	"port": "port",
}

Config data structure

{
	"name": "configname",
	"gpn": "groupname",
	"default": false,
	"servers": [
		"servername",
	],
	"mountInfos": [
	    {
	        "mountpoint": "local/mount/point",
	        "server": "servername",
	        "path": "server/sub/path"
	    },
	]
}

• MountInfo: How user mount server path to local.

{
	"mountpoint": "local/mount/point",
	"server": "servername",
	"path": "server/sub/path"
}

Config in group data

• Which storage configs that a group can access is stored in group data's extension field. For example, a group that can access STORAGE_CONFIG is like following:

{
	"groupname": "groupname",
	"externalName": "externalName",
	"description": "description",
	"extension": {
		"acls": {
			"admin": false,
			"virtualClusters": [],
			"storageConfigs": ["STORAGE_CONFIG"]
		}
	}
}