Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fix dedup logs #977

Merged
merged 2 commits into from
May 1, 2023
Merged

fix dedup logs #977

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
e72ef03
fix dedup logs
pepoviola Apr 28, 2023
0f61961
fmt
pepoviola Apr 28, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 0 additions & 1 deletion javascript/packages/orchestrator/src/network.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -370,7 +370,6 @@ export class Network {

replaceWithNetworInfo(placeholder: string): string {
return placeholder.replace(
///{{ZOMBIE:(.*?):(.*?)}}/gi,
TOKEN_PLACEHOLDER,
(_substring, nodeName, key: keyof NetworkNode) => {
const node = this.getNodeByName(nodeName);
Expand Down
32 changes: 19 additions & 13 deletions javascript/packages/orchestrator/src/networkNode.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,8 +40,6 @@ export class NetworkNode implements NetworkNodeInterface {
userDefinedTypes: any;
para?: PARA;
parachainId?: number;
lastLogLineCheckedTimestamp?: string;
lastLogLineCheckedIndex?: number;
group?: string;

constructor(
Expand Down Expand Up @@ -415,6 +413,8 @@ export class NetworkNode implements NetworkNodeInterface {
timeout: number = DEFAULT_INDIVIDUAL_TEST_TIMEOUT,
): Promise<boolean> {
try {
let lastLogLineCheckedTimestamp: string;
let lastLogLineCheckedIndex: number;
const re = isGlob ? makeRe(pattern) : new RegExp(pattern, "ig");
if (!re) throw new Error(`Invalid glob pattern: ${pattern} `);
const client = getClient();
Expand All @@ -425,7 +425,10 @@ export class NetworkNode implements NetworkNodeInterface {
const dedupedLogs = this._dedupLogs(
logs.split("\n"),
client.providerName === "native",
lastLogLineCheckedTimestamp,
lastLogLineCheckedIndex,
Comment on lines +428 to +429
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It won't cause issue if it's not initialized ?

);

const index = dedupedLogs.findIndex((line) => {
if (client.providerName !== "native") {
// remove the extra timestamp
Expand All @@ -436,11 +439,9 @@ export class NetworkNode implements NetworkNodeInterface {

if (index >= 0) {
done = true;
this.lastLogLineCheckedTimestamp = dedupedLogs[index];
this.lastLogLineCheckedIndex = index;
debug(
this.lastLogLineCheckedTimestamp.split(" ").slice(1).join(" "),
);
lastLogLineCheckedTimestamp = dedupedLogs[index];
lastLogLineCheckedIndex = index;
debug(lastLogLineCheckedTimestamp.split(" ").slice(1).join(" "));
} else {
await new Promise((resolve) => setTimeout(resolve, 1000));
logs = await client.getNodeLogs(this.name, 2, true);
Expand Down Expand Up @@ -538,12 +539,17 @@ export class NetworkNode implements NetworkNodeInterface {
return spanNames;
}

// prevent to seach in the same log line twice.
_dedupLogs(logs: string[], useIndex = false): string[] {
if (!this.lastLogLineCheckedTimestamp) return logs;
if (useIndex) return logs.slice(this.lastLogLineCheckedIndex);

const lastLineTs = this.lastLogLineCheckedTimestamp.split(" ")[0];
// prevent to search in the same log line twice.
_dedupLogs(
logs: string[],
useIndex = false,
lastLogLineCheckedTimestamp: string,
lastLogLineCheckedIndex: number,
): string[] {
if (!lastLogLineCheckedTimestamp) return logs;
if (useIndex) return logs.slice(lastLogLineCheckedIndex);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here, lastLogLineCheckedIndex is undefined when entering the above loop

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well, those two variables work in tandem to represent the different providers features. The invariant here is that lastLogLineCheckedIndex will be undefined at the same time that lastLogLineCheckedTimestamp is undefinded, so the this if (https://github.com/paritytech/zombienet/pull/977/files#diff-fc4624d6bb61da0712b154b5018f73681adaf8edadb4236a0b5293ea915ef41eR549) cover this case.

I think there is room to improving here, but we can do it in a follow up pr.

Thanks!


const lastLineTs = lastLogLineCheckedTimestamp.split(" ")[0];
const index = logs.findIndex((logLine) => {
const thisLineTs = logLine.split(" ")[0];
return thisLineTs > lastLineTs;
Expand Down
5 changes: 4 additions & 1 deletion javascript/packages/orchestrator/src/test-runner/assertions.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@ import {
validateRuntimeCode,
} from "../jsapi-helpers";
import { Network } from "../network";
import { NetworkNode } from "../networkNode";
import { FnArgs } from "../types";
const utilCrypto = require("@polkadot/util-crypto");

Expand Down Expand Up @@ -127,7 +128,9 @@ const LogMatch = ({ node_name, pattern, match_type, timeout }: FnArgs) => {
return async (network: Network) => {
const nodes = network.getNodes(node_name!);
const results = await Promise.all(
nodes.map((node: any) => node.findPattern(pattern!, isGlob, timeout)),
nodes.map((node: NetworkNode) =>
node.findPattern(pattern!, isGlob, timeout),
),
);

const found = results.every(Boolean);
Expand Down