Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

refactor(cli): split cli in actions files #889

Merged
merged 18 commits into from
Apr 5, 2023
Merged
Show file tree
Hide file tree
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
130 changes: 130 additions & 0 deletions javascript/packages/cli/src/actions/convert.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,130 @@
import type {
NodeConfig,
PL_ConfigType,
ParachainConfig,
PolkadotLaunchConfig,
} from "@zombienet/orchestrator";
import { decorators, getFilePathNameExt } from "@zombienet/utils";
import fs from "fs";
import path from "path";
import { DEFAULT_BALANCE } from "src/constants";

export async function convert(param: string) {
try {
const filePath = param;

if (!filePath) {
throw Error("Path of configuration file was not provided");
}

// Read through the JSON and write to stream sample
await convertInput(filePath);
} catch (err) {
console.log(
`\n ${decorators.red("Error: ")} \t ${decorators.bright(err)}\n`,
);
}
}

// Convert functions
// Read the input file
async function readInputFile(
ext: string,
fPath: string,
): Promise<PL_ConfigType> {
let json: object;
if (ext === "json" || ext === "js") {
json =
ext === "json"
? JSON.parse(fs.readFileSync(`${fPath}`, "utf8"))
: await import(path.resolve(fPath));
} else {
throw Error("No valid extension was found.");
}
return json;
pepoviola marked this conversation as resolved.
Show resolved Hide resolved
}

async function convertInput(filePath: string) {
const { fullPath, fileName, extension } = getFilePathNameExt(filePath);

const convertedJson = await readInputFile(extension, filePath);

const { relaychain, parachains, simpleParachains, hrmpChannels, types } =
convertedJson;
pepoviola marked this conversation as resolved.
Show resolved Hide resolved

let jsonOutput: PolkadotLaunchConfig;
const nodes: NodeConfig[] = [];
const paras: ParachainConfig[] = [];
pepoviola marked this conversation as resolved.
Show resolved Hide resolved
let collators: NodeConfig[] = [];

const DEFAULT_NODE_VALUES = {
validator: true,
invulnerable: true,
balance: DEFAULT_BALANCE,
};

parachains &&
pepoviola marked this conversation as resolved.
Show resolved Hide resolved
parachains.forEach((parachain: any) => {
collators = [];
parachain.nodes.forEach((n: any) => {
collators.push({
name: n.name,
command: "adder-collator",
...DEFAULT_NODE_VALUES,
});
});
paras.push({
id: parachain.id,
collators,
});
});
Copy link
Contributor

@l0r1s l0r1s Apr 5, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
parachains &&
parachains.forEach((parachain: any) => {
collators = [];
parachain.nodes.forEach((n: any) => {
collators.push({
name: n.name,
command: "adder-collator",
...DEFAULT_NODE_VALUES,
});
});
paras.push({
id: parachain.id,
collators,
});
});
paras = paras.concat(
parachains.map(({ id, nodes }) => ({
id,
collators: (nodes || []).map(({ name }) => ({
name,
command: "adder-collator",
...DEFAULT_NODE_VALUES,
})),
})),
);

What do you think, using destructuring and functionnal programming ?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds good 👍


collators = [];
pepoviola marked this conversation as resolved.
Show resolved Hide resolved

simpleParachains &&
pepoviola marked this conversation as resolved.
Show resolved Hide resolved
simpleParachains.forEach((sp: any) => {
collators.push({
name: sp.name,
command: "adder-collator",
...DEFAULT_NODE_VALUES,
});
paras.push({
id: sp.id,
collators,
});
});
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
simpleParachains &&
simpleParachains.forEach((sp: any) => {
collators.push({
name: sp.name,
command: "adder-collator",
...DEFAULT_NODE_VALUES,
});
paras.push({
id: sp.id,
collators,
});
});
paras = paras.concat(
simpleParachains.map(({ id, name }) => ({
id,
collators: [{ name, command: "adder-collator", ...DEFAULT_NODE_VALUES }],
})),
);

Destructuring and functionnal programming part 2

Copy link
Contributor

@l0r1s l0r1s Apr 5, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My question here is, because we don't reset the collators array in the old code, do we really want to have a parachain B to have the previous collator of parachain A ?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, we don't want. This is a bug and the suggested code is ok.
Thanks!


if (relaychain?.nodes) {
relaychain.nodes.forEach((n: any) => {
nodes.push({
name: `"${n.name}"`,
...DEFAULT_NODE_VALUES,
});
});
}

jsonOutput = {
relaychain: {
default_image: "docker.io/paritypr/polkadot-debug:master",
default_command: "polkadot",
default_args: ["-lparachain=debug"],
chain: relaychain?.chain || "",
nodes,
genesis: relaychain?.genesis,
},
types,
hrmp_channels: hrmpChannels || [],
pepoviola marked this conversation as resolved.
Show resolved Hide resolved
parachains: paras,
};

fs.writeFile(
`${fullPath}/${fileName}-zombienet.json`,
JSON.stringify(jsonOutput),
(error: any) => {
if (error) throw error;
},
);
Comment on lines +108 to +114
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you use fs/promises instead of fs, you can simplify to:

Suggested change
fs.writeFile(
`${fullPath}/${fileName}-zombienet.json`,
JSON.stringify(jsonOutput),
(error: any) => {
if (error) throw error;
},
);
await fs.writeFile(
`${fullPath}/${fileName}-zombienet.json`,
JSON.stringify(jsonOutput)
);

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should change the import to use fs.promises here.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, the exact import is import {...} from "fs/promises

console.log(
`Converted JSON config exists now under: ${fullPath}/${fileName}-zombienet.json`,
);
}
235 changes: 235 additions & 0 deletions javascript/packages/cli/src/actions/setup.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,235 @@
import { askQuestion, convertBytes, decorators } from "@zombienet/utils";
import cliProgress from "cli-progress";
import fs from "fs";
import path from "path";

interface OptIf {
[key: string]: { name: string; url?: string; size?: string };
}

const options: OptIf = {};
/**
* Setup - easily download latest artifacts and make them executablein order to use them with zombienet
* Read more here: https://paritytech.github.io/zombienet/cli/setup.html
* @param params binaries that willbe downloaded and set up. Possible values: `polkadot` `polkadot-parachain`
* @returns
*/
export async function setup(params: any) {
const POSSIBLE_BINARIES = ["polkadot", "polkadot-parachain"];

console.log(decorators.green("\n\n🧟🧟🧟 ZombieNet Setup 🧟🧟🧟\n\n"));
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Have you heard about the colors library ?

It could enable to simplify the code like this (it extends the String.constructor):

Suggested change
console.log(decorators.green("\n\n🧟🧟🧟 ZombieNet Setup 🧟🧟🧟\n\n"));
console.log("\n\n🧟🧟🧟 ZombieNet Setup 🧟🧟🧟\n\n".green);

Simpler and better because we don't have to manage our own "colors" system in utils.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I remember to look at colors but I just went to make an small library for a few colors we use. We can iterate over this decision.
Thanks!

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Have you heard about the colors library ?

It could enable to simplify the code like this (it extends the String.constructor):

Simpler and better because we don't have to manage our own "colors" system in utils.

I think we should reduce npm dependencies as much as possible

if (
["aix", "freebsd", "openbsd", "sunos", "win32"].includes(process.platform)
) {
console.log(
"Zombienet currently supports linux and MacOS. \n Alternative, you can use k8s or podman. For more read here: https://github.com/paritytech/zombienet#requirements-by-provider",
);
return;
}
pepoviola marked this conversation as resolved.
Show resolved Hide resolved

console.log(decorators.green("Gathering latest releases' versions...\n"));
await new Promise<void>((resolve) => {
latestPolkadotReleaseURL("polkadot", "polkadot").then(
(res: [string, string]) => {
options.polkadot = {
name: "polkadot",
url: res[0],
size: res[1],
};
resolve();
},
);
});

await new Promise<void>((resolve) => {
latestPolkadotReleaseURL("cumulus", "polkadot-parachain").then(
(res: [string, string]) => {
options["polkadot-parachain"] = {
name: "polkadot-parachain",
url: res[0],
size: res[1],
};
resolve();
},
);
});

// If the platform is MacOS then the polkadot repo needs to be cloned and run locally by the user
// as polkadot do not release a binary for MacOS
if (process.platform === "darwin" && params.includes("polkadot")) {
console.log(
`${decorators.yellow(
"Note: ",
)} You are using MacOS. Please, clone the polkadot repo ` +
decorators.cyan("(https://github.com/paritytech/polkadot)") +
` and run it locally.\n At the moment there is no polkadot binary for MacOs.\n\n`,
);
const index = params.indexOf("polkadot");
if (index !== -1) {
params.splice(index, 1);
}
pepoviola marked this conversation as resolved.
Show resolved Hide resolved
}

if (params.length === 0) {
console.log(decorators.green("No binaries to download. Exiting..."));
return;
}
let count = 0;
console.log("Setup will start to download binaries:");
params.forEach((a: any) => {
if (!POSSIBLE_BINARIES.includes(a)) {
const index = params.indexOf(a);
index > -1 && params.splice(index, 1);
pepoviola marked this conversation as resolved.
Show resolved Hide resolved
console.log(
decorators.red(
`"${a}" is not one of the possible options for this setup and will be skipped;`,
),
decorators.green(` Valid options: polkadot polkadot-parachain`),
);
return;
}
const size = parseInt(options[a]?.size || "0", 10);
count += size;
console.log("-", a, "\t Approx. size ", size, " MB");
});
console.log("Total approx. size: ", count, "MB");
const response = await askQuestion(
decorators.yellow("\nDo you want to continue? (y/n)"),
);
if (response.toLowerCase() !== "n" && response.toLowerCase() !== "y") {
console.log("Invalid input. Exiting...");
return;
}
if (response.toLowerCase() === "n") {
return;
}
downloadBinaries(params);
return;
}

// helper fns
// Download the binaries
const downloadBinaries = async (binaries: string[]): Promise<void> => {
try {
console.log(decorators.yellow("\nStart download...\n"));
const promises = [];

const multibar = new cliProgress.MultiBar(
{
clearOnComplete: false,
hideCursor: true,
format:
decorators.yellow("{bar} - {percentage}%") +
" | " +
decorators.cyan("Binary name:") +
" {filename}",
},
cliProgress.Presets.shades_grey,
);

for (let binary of binaries) {
promises.push(
new Promise<void>(async (resolve, reject) => {
let result = options[binary];
if (!result) {
console.log("options", options, "binary", binary);
throw new Error("Binary is not defined");
}
const { url, name } = result;

if (!url) throw new Error("No url for downloading, was provided");

const response = await fetch(url);

if (!response.ok)
throw Error(response.status + " " + response.statusText);

const contentLength = response.headers.get(
"content-length",
) as string;
let loaded = 0;

const progressBar = multibar.create(parseInt(contentLength, 10), 0);
const reader = response.body?.getReader();
const writer = fs.createWriteStream(path.resolve(name));

while (true) {
const read = await reader?.read()!;
if (read?.done) {
writer.close();
resolve();
break;
}

loaded += read.value.length;
progressBar.increment();
progressBar.update(loaded, {
filename: name,
});
writer.write(read.value);
}
}),
);
}

await Promise.all(promises);
multibar.stop();
console.log(
decorators.cyan(
`\n\nPlease add the current dir to your $PATH by running the command:\n`,
),
decorators.blue(`export PATH=${process.cwd()}:$PATH\n\n`),
);
} catch (err) {
console.log(
`\n ${decorators.red("Unexpected error: ")} \t ${decorators.bright(
err,
)}\n`,
);
}
};

// Retrieve the latest release for polkadot
const latestPolkadotReleaseURL = async (
repo: string,
name: string,
): Promise<[string, string]> => {
try {
const releases = await fetch(
`https://api.github.com/repos/paritytech/${repo}/releases`,
);

let obj: any;
let tag_name;

const allReleases = await releases.json();
const release = allReleases.find((r: any) => {
obj = r?.assets?.find((a: any) => a.name === name);
return Boolean(obj);
});

tag_name = release.tag_name;

if (!tag_name) {
throw new Error(
"Should never come to this point. Tag_name should never be undefined!",
);
}

return [
`https://github.com/paritytech/${repo}/releases/download/${tag_name}/${name}`,
pepoviola marked this conversation as resolved.
Show resolved Hide resolved
convertBytes(obj.size),
];
} catch (err: any) {
if (err.code === "ENOTFOUND") {
throw new Error("Network error.");
} else if (err.response && err.response.status === 404) {
throw new Error(
"Could not find a release. Error 404 (not found) detected",
);
}
throw new Error(
`Error status: ${err?.response?.status}. Error message: ${err?.response}`,
);
}
};
pepoviola marked this conversation as resolved.
Show resolved Hide resolved
Loading