Reduce $totalSchedulerThroughputPods in large clusters

[mborsz]

In large clusters we create 2x more latency pods than #nodes. We don't think it's needed, so let's try to remove that.

I will run 5k test to see what happens in that scale.
Presubmit should validate 100 nodes scale.
/hold

[mm4tt]

/lgtm

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: mborsz, mm4tt

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• clusterloader2/OWNERS [mborsz,mm4tt]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[mm4tt]

TBH, I'm more worried about the 100 node tests than the 5k tests. This change may increase the flakiness of the scheduling throughput measurement and to detect it a single presubmit run won't be enough. That said, I think we should just submit this and keep an eye on the periodic tests

[mborsz]

Matt, correct me if I'm wrong, but I don't think this affects 100 nodes scale at all:

• before this change we had totalSchedulerThroughputPods = max(1000, 2 * nodes)
• after this change we will have max(1000, nodes)

This means, that up to 500 nodes, 1000 value is used in both cases.

[mm4tt]

@mborsz, you're right. In that case I don't see any reason for not merging this. I'd be surprised if it affects 5k tests in any way. Creating 5k pods seems more than enough to accurately measure scheduling throughput.

[mborsz]

/hold cancel

Thanks!