Policy: create dynamic-pools policy for cri-rm #969

Yugar-1 · 2022-12-28T07:34:09Z

This PR creates a new policy for cri-resource-manager named dynamic-pools policy.
This policy can put the workload in different dynamic pools, and the CPUs of dynamic pools do not overlap with each other. These dynamic pools can be dynamically resized based on the requests of the containers and the CPU utilization of the workload in the dynamic pools.
Users can define and configure dynamic pools. A dynamic pool has a set of CPUs and a set of containers running on the CPUs. CPUs in dynamic pools can be configured, for example, by setting min and max frequencies on CPU cores and uncore.
The code and documentation of the dynamic-pools policy are included in this PR.

jukkar · 2022-12-28T08:00:27Z

The commits should be squashed together and commit message enhanced.
How this differs from balloons policy as the code seems to be copied from balloons policy?

Yugar-1 · 2022-12-28T10:20:48Z

The commits should be squashed together and commit message enhanced. How this differs from balloons policy as the code seems to be copied from balloons policy?

The commits have been squashed together.
The dynamic-pools policy allocates CPUs to pools differently than the balloons policy. The dynamic-pools policy dynamically allocates CPUs based on the requests of the containers and the CPU utilization of the workload in the dynamic pools, and can reallocate CPUs at regular intervals to make CPU allocation more balanced.
The dynamic-pools policy cannot be merged into the balloons policy, because they have conficts, which will modify the behavior of the balloon policy, and the characteristics of the balloons cannot be maintained.

askervin

Thank you very much for this impressive enhancement PR! This is a very nice add-on to CRI-RM features.

I added some comments to help you polishing the code. I would also ask you to include end-to-end tests (most importantly test/e2e/policies.test-suite/dynamic-pools/n4c16/testNN-TESTNAME/code.var.sh files) to ensure functional quality now and for the future.

docs/policy/dynamic-pools.md

pkg/cri/resource-manager/policy/builtin/dynamic-pools/cpu.go

pkg/cri/resource-manager/policy/builtin/dynamic-pools/dyp.go

askervin · 2022-12-28T14:07:15Z

pkg/cri/resource-manager/policy/builtin/dynamic-pools/dyp.go

+
+// updatePoolCpuset updates the cpuset of the dynamicPools.
+func (p *dynamicPools) updatePoolCpuset() error {
+	requestCpu, remainFree := p.calculateAllPoolRequests()


A situations to consider and include in e2e tests: none of the containers in a dynamic pool request any CPUs. (The containers may be BestEffort or they may have only resource limit.) It looks to me that in that case the container might get an empty cpuset, which will mean "can run on any CPU".

If a container has no requests, it will be allocated to the shared dynamic-pool, which has cpuset.
We will include this case in the e2e tests later.

askervin · 2022-12-28T14:20:08Z

pkg/cri/resource-manager/policy/builtin/dynamic-pools/dyp.go

+	}
+
+	for _, dp := range p.dynamicPools {
+		if dp.Def.Name == "reserved" {


Reallocation of CPUs can severely harm performance of a workload. If the workload has been running on CPU package0 (socket0) and is suddenly pinned to CPU cores on package1 (socket1), then suddenly memory accesses become very expensive.

In order to avoid unnecessary CPU reallocations, I would suggest:

First release only the necessary amount of CPUs from a pool (current number of CPUs minus requestCpu[dp], if this is a positive number).

Then allocate CPUs only to those pools that really need them.

The current balloons policy implements topology-aware CPU allocation, that is, it allocates CPU cores that are topologically as close to the currently allocated CPUs as possible. This is a bigger enhancement and can be left to be done in future PRs.

Thanks for your suggestions!
This is indeed where the dynamic-pools policy needs to be improved. We will seriously consider your suggestions, and we will improve the performance of dynamic-pools policy in the future.

docs/policy/dynamic-pools.md

The CPU will always belong to a dynamic-pool, so the IdleCPUClass configuration option is not needed.

1. Change `defaultDynamicPoolDefName` to `sharedDynamicPoolDefName`, and use `reservedDynamicPoolDefName` and `sharedDynamicPoolDefName` instead of "reserved" or "shared" strings in all codes of dynamic-pool policy. 2. Call `getCpuUtilization()` once in the beginning of `calculateAllPoolWeights()`, and pass the same cpuInfo to all `updateRealCpuUsed()` calls as a parameter. 3. use `ToSliceNoSort()` instead of `ToSlice()`. 4. Let `updateRealCpuUsed()` use `DynamicPool` instead of `dynamicPools` as the receiver. 5. Modify the return value type of `updateRealCpuUsed` to `float64` instead of `int`. 6. Fix typo: s/dyanmic/dynamic/g in this file. 7. Update the information that needs to be included in `log.Info`, and put the rest in `log.Debug()`.

…de cleanup 1. Rename `formatStatLine` to `parseStatLine`. 2. Return `nil` instead of a slice that includes an empty string. 3. Check if cpuLine is less than 8, if so, do error handling. 4. Function `parseStatLine` does not process total cpu line, and add corresponding comments. 5. Remove `clocksPerSecond`, no need to scale the data (user, system, ...). 6. Document the return value of `calculateOneCpuUtilization`.

Yugar-1 · 2023-01-05T08:00:52Z

Thank you very much for this impressive enhancement PR! This is a very nice add-on to CRI-RM features.

I added some comments to help you polishing the code. I would also ask you to include end-to-end tests (most importantly test/e2e/policies.test-suite/dynamic-pools/n4c16/testNN-TESTNAME/code.var.sh files) to ensure functional quality now and for the future.

We are going to doing end-to-end tests for dynamic-pools policy. We read the end-to-end tests of the balloons policy, but we didn't understand it very well. Do you have relevant documents that we can learn from? Thanks！

askervin · 2023-01-05T13:29:27Z

Hi @Yugar-1,

I wrote you a documented example tests that introduce how to use the e2e test framework for testing the dynamic-pools policy. Please see the e2e commit in my e2e_dynamic_pools_example branch.

I would advice you to start by installing e2e framework dependencies: docker and govm. You can find details in the CRI Resource Manager Developer's guide / End-to-end tests documentation.

Then, if you read through test01 code first, test02 code second, I'm sure you will get the idea and will be able to extend the test set further.

Feel free to take the contents of this commit as a starting point for your e2e tests.

1. Split out the two functions `isNeedReallocate()` and `calculatePoolCpuset()` from the function `updatePoolCpuset()`. 2. Added unit tests for two functions `isNeedReallocate()` and `calculatePoolCpuset()`.

…tats instead of new environment variable `HOST_PROC`

Yugar-1 · 2023-01-12T09:15:03Z

Hi @Yugar-1,

I wrote you a documented example tests that introduce how to use the e2e test framework for testing the dynamic-pools policy. Please see the e2e commit in my e2e_dynamic_pools_example branch.

I would advice you to start by installing e2e framework dependencies: docker and govm. You can find details in the CRI Resource Manager Developer's guide / End-to-end tests documentation.

Then, if you read through test01 code first, test02 code second, I'm sure you will get the idea and will be able to extend the test set further.

Feel free to take the contents of this commit as a starting point for your e2e tests.

Hi, @askervin , thank you very much for your support!
With your help, I have pushed the code of dynamic-pools policy's e2e tests to the latest PR, thank you!

1. If there is no container in a dynamic pool, there is no need to calculate its weight, that is, there is no need to allocate CPUs to it. 2. If there are containers in the shared dynamic pool, allocate at least one CPU to it, otherwise there is no need to allocate CPUs to it. 3. Update data types in log.debug. 4. Delete the `IdleCPUClass` of example configuration in the documentation of dynamic-pools policy.

codecov-commenter · 2023-01-23T08:50:52Z

Codecov Report

Merging #969 (69bac75) into master (01cf8c2) will decrease coverage by 1.72%.
The diff coverage is 8.57%.

@@            Coverage Diff             @@
##           master     #969      +/-   ##
==========================================
- Coverage   33.68%   31.96%   -1.72%     
==========================================
  Files          61       65       +4     
  Lines        9156     9833     +677     
==========================================
+ Hits         3084     3143      +59     
- Misses       5789     6402     +613     
- Partials      283      288       +5

Impacted Files	Coverage Δ
...source-manager/policy/builtin/dynamic-pools/cpu.go	`0.00% <0.00%> (ø)`
...ce-manager/policy/builtin/dynamic-pools/metrics.go	`0.00% <0.00%> (ø)`
...source-manager/policy/builtin/dynamic-pools/dyp.go	`9.35% <9.35%> (ø)`
...urce-manager/policy/builtin/dynamic-pools/flags.go	`46.42% <46.42%> (ø)`
pkg/cpuallocator/allocator.go	`48.32% <0.00%> (+0.15%)`	⬆️

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

askervin · 2023-01-23T11:45:07Z

@Yugar-1 , how do you see, has this PR stabilized now for next review round, or are you working on some changes still?

Yugar-1 · 2023-01-25T05:25:00Z

@Yugar-1 , how do you see, has this PR stabilized now for next review round, or are you working on some changes still?

Hi @askervin , this PR is stable and the function has been completed, can it be merged first?
Aiming at the problem of low unit test coverage, we plan to create a new PR to improve the unit test coverage. What do you think?

askervin · 2023-01-25T07:36:04Z

Have you experienced flakiness in the e2e tests? I got a failure from the first run of test03-rebalancing.

...test/e2e$ reinstall_cri_resmgr=1 distro=debian-sid ./run_tests.sh policies.test-suite/dynamic-pools
...
# VM name        : n4c16-debian-sid-crirm-containerd
...
PASS dynamic-pools/n4c16/test01-basic-placement
PASS dynamic-pools/n4c16/test02-prometheus-metrics
PASS dynamic-pools/n4c16/test02-rebalancing
FAIL dynamic-pools/n4c16/test03-rebalancing
PASS dynamic-pools/n4c16/test04-reserved
PASS dynamic-pools/n4c16/test05-namespace
PASS dynamic-pools/n4c16/test06-update-configmap
PASS dynamic-pools/n4c16/test07-numa

Having a deeper look:

...test/e2e$ less policies.test-suite/dynamic-pools/n4c16/test03-rebalancing/output/run.sh.output
...
root@vm> nohup kubectl exec pod2 -- /bin/sh -c 'gzip </dev/zero >/dev/null' </dev/null >&/dev/null &
host> sleep 3
package0 die0 node0 core0 thread0 cpu00
                          thread1 cpu01
                    core1 thread0 cpu02 pod0c0/3247541
                          thread1 cpu03 pod2c0/3248925
                    mem   node0   2G    pod0c0/3247541
                                        pod2c0/3248925
              node1 core2 thread0 cpu04 pod1c0/3248637
                          thread1 cpu05 pod1c0/3248637
                    core3 thread0 cpu06 pod1c0/3248637
                          thread1 cpu07 pod1c0/3248637
                    mem   node1   2G    pod1c0/3248637
package1 die0 node2 core0 thread0 cpu08 pod1c0/3248637
                          thread1 cpu09 pod1c0/3248637
                    core1 thread0 cpu10 pod1c0/3248637
                          thread1 cpu11 pod1c0/3248637
                    mem   node2   2G    pod1c0/3248637
              node3 core2 thread0 cpu12 pod1c0/3248637
                          thread1 cpu13 pod1c0/3248637
                    core3 thread0 cpu14 pod1c0/3248637
                          thread1 cpu15 pod1c0/3248637
                    mem   node3   2G    pod1c0/3248637
### Verifying assertion 'len(cpus["pod0c0"]) == 1'

### The assertion holds.

### Verifying assertion 'len(cpus["pod1c0"]) == 1'

--> this assertion fails, len(["pod1c0"]) equals 12 at this point.

...in the cri-resmgr perspective the last rebalancing looked like this:

...test/e2e$ less policies.test-suite/dynamic-pools/n4c16/test03-rebalancing/output/cri-resmgr.output.txt
...
I: [resource-manager] rebalancing (reallocating) containers...
D: [     policy     ] rebalancing containers...
D: [     policy     ] dynamic pool shared{Cpus:2, Mems:0} request cpu 1
D: [     policy     ] dynamic pool pool1{Cpus:4-15, Mems:1,2,3} request cpu 1
D: [     policy     ] dynamic pool pool2{Cpus:3, Mems:0} request cpu 1
D: [     policy     ] sum remain free cpu 11
D: [     policy     ] dynamic pool shared cpuset: 2,  cpu utilization: 2
D: [     policy     ] dynamic pool: shared{Cpus:2, Mems:0}, weight: 2
D: [     policy     ] dynamic pool pool1 cpuset: 4-15,  cpu utilization: 46.02699261522791
D: [     policy     ] dynamic pool: pool1{Cpus:4-15, Mems:1,2,3}, weight: 46.02699261522791
D: [     policy     ] dynamic pool pool2 cpuset: 3,  cpu utilization: 3.8834951456310676
D: [     policy     ] dynamic pool: pool2{Cpus:3, Mems:0}, weight: 3.8834951456310676
D: [     policy     ] sum weight: 51.91048776085898

The same test failed when I ran with distro=ubuntu-22.04, too. Yet it passed the above verification, it failed very soon after it:

### Verifying assertion 'len(cpus["pod1c0"]) == 7'

--> failed, because len(cpus["pod1c0"]) was 6 instead of 7.

Would you check if these tests could be made more reliable? (Allow running a couple of rebalance rounds to let cpu load stabilize? Maybe assert 6 <= len(...) <= 8 instead of requiring exactly 7?)

The coverage in e2e tests looks pretty good, so I'm ok with not increasing unit test coverage in this PR.

Yugar-1 · 2023-01-26T05:21:46Z

Have you experienced flakiness in the e2e tests? I got a failure from the first run of test03-rebalancing.

...test/e2e$ reinstall_cri_resmgr=1 distro=debian-sid ./run_tests.sh policies.test-suite/dynamic-pools
...
# VM name        : n4c16-debian-sid-crirm-containerd
...
PASS dynamic-pools/n4c16/test01-basic-placement
PASS dynamic-pools/n4c16/test02-prometheus-metrics
PASS dynamic-pools/n4c16/test02-rebalancing
FAIL dynamic-pools/n4c16/test03-rebalancing
PASS dynamic-pools/n4c16/test04-reserved
PASS dynamic-pools/n4c16/test05-namespace
PASS dynamic-pools/n4c16/test06-update-configmap
PASS dynamic-pools/n4c16/test07-numa

Having a deeper look:

...test/e2e$ less policies.test-suite/dynamic-pools/n4c16/test03-rebalancing/output/run.sh.output
...
root@vm> nohup kubectl exec pod2 -- /bin/sh -c 'gzip </dev/zero >/dev/null' </dev/null >&/dev/null &
host> sleep 3
package0 die0 node0 core0 thread0 cpu00
                          thread1 cpu01
                    core1 thread0 cpu02 pod0c0/3247541
                          thread1 cpu03 pod2c0/3248925
                    mem   node0   2G    pod0c0/3247541
                                        pod2c0/3248925
              node1 core2 thread0 cpu04 pod1c0/3248637
                          thread1 cpu05 pod1c0/3248637
                    core3 thread0 cpu06 pod1c0/3248637
                          thread1 cpu07 pod1c0/3248637
                    mem   node1   2G    pod1c0/3248637
package1 die0 node2 core0 thread0 cpu08 pod1c0/3248637
                          thread1 cpu09 pod1c0/3248637
                    core1 thread0 cpu10 pod1c0/3248637
                          thread1 cpu11 pod1c0/3248637
                    mem   node2   2G    pod1c0/3248637
              node3 core2 thread0 cpu12 pod1c0/3248637
                          thread1 cpu13 pod1c0/3248637
                    core3 thread0 cpu14 pod1c0/3248637
                          thread1 cpu15 pod1c0/3248637
                    mem   node3   2G    pod1c0/3248637
### Verifying assertion 'len(cpus["pod0c0"]) == 1'

### The assertion holds.

### Verifying assertion 'len(cpus["pod1c0"]) == 1'

--> this assertion fails, len(["pod1c0"]) equals 12 at this point.

...in the cri-resmgr perspective the last rebalancing looked like this:

...test/e2e$ less policies.test-suite/dynamic-pools/n4c16/test03-rebalancing/output/cri-resmgr.output.txt
...
I: [resource-manager] rebalancing (reallocating) containers...
D: [     policy     ] rebalancing containers...
D: [     policy     ] dynamic pool shared{Cpus:2, Mems:0} request cpu 1
D: [     policy     ] dynamic pool pool1{Cpus:4-15, Mems:1,2,3} request cpu 1
D: [     policy     ] dynamic pool pool2{Cpus:3, Mems:0} request cpu 1
D: [     policy     ] sum remain free cpu 11
D: [     policy     ] dynamic pool shared cpuset: 2,  cpu utilization: 2
D: [     policy     ] dynamic pool: shared{Cpus:2, Mems:0}, weight: 2
D: [     policy     ] dynamic pool pool1 cpuset: 4-15,  cpu utilization: 46.02699261522791
D: [     policy     ] dynamic pool: pool1{Cpus:4-15, Mems:1,2,3}, weight: 46.02699261522791
D: [     policy     ] dynamic pool pool2 cpuset: 3,  cpu utilization: 3.8834951456310676
D: [     policy     ] dynamic pool: pool2{Cpus:3, Mems:0}, weight: 3.8834951456310676
D: [     policy     ] sum weight: 51.91048776085898

The same test failed when I ran with distro=ubuntu-22.04, too. Yet it passed the above verification, it failed very soon after it:

### Verifying assertion 'len(cpus["pod1c0"]) == 7'

--> failed, because len(cpus["pod1c0"]) was 6 instead of 7.

Would you check if these tests could be made more reliable? (Allow running a couple of rebalance rounds to let cpu load stabilize? Maybe assert 6 <= len(...) <= 8 instead of requiring exactly 7?)

The coverage in e2e tests looks pretty good, so I'm ok with not increasing unit test coverage in this PR.

Hi, @askervin , thank you very much for your suggestion! I have fixed the e2e tests of dynamic-pools policy. Please try again and contact me if you have any questions.

askervin

Hi @Yugar-1, Thank you for all these improvements!

I think this large PR is very close to getting merged now.

I added a few mostly cosmetic comments. My wish is that anyone will be able to try out the dynamic-pools policy without errors just by copy-pasting the configuration samples.

All E2E tests passed with distro={debian-sid,ubuntu-22.04,fedora-36}.

docs/policy/dynamic-pools.md

askervin · 2023-01-30T07:38:05Z

docs/policy/dynamic-pools.md

+  HTTPEndpoint::8891
+  PrometheusExport:true
+logger:
+  Debug:policy


Copy-pasting this to the end of a working configuration file results in yaml parsing errors. You will need to add a space between : that separates a key from a value. (Please test sample configuration snippets.)

This version still results in configuration failure:

$ cri-resmgr -force-config doc-test.cfg ... F0130 09:52:03.497887 36423 main.go:85] failed to create resource manager instance: resource-manager: failed to load forced configuration doc-test.cfg: config error: failed to apply configuration from file: config error: failed to load configuration from file "doc-test.cfg": error converting YAML to JSON: yaml: line 31: could not find expected ':'

Two more key-value fixes needed. Please test.

pkg/cri/resource-manager/policy/builtin/dynamic-pools/cpu.go

pkg/cri/resource-manager/policy/builtin/dynamic-pools/dyp_test.go

Yugar-1 · 2023-01-30T09:41:24Z

Hi @Yugar-1, Thank you for all these improvements!

I think this large PR is very close to getting merged now.

I added a few mostly cosmetic comments. My wish is that anyone will be able to try out the dynamic-pools policy without errors just by copy-pasting the configuration samples.

All E2E tests passed with distro={debian-sid,ubuntu-22.04,fedora-36}.

Hi, @askervin, thank you so much for your comments, you noticed a lot of things that I had overlooked!
I've fixed all the bugs, thanks!

askervin

Also, would you remove the ConfigMap header and related indentation from the "Example configuration that runs all pods in dynamic-pools." yaml in the documentation.

Please make sure that it runs fine with cri-resmgr -force-config doc-test.cfg. This will make it easy for readers to try out.

askervin · 2023-01-30T09:54:09Z

docs/policy/dynamic-pools.md

+  HTTPEndpoint::8891
+  PrometheusExport:true
+logger:
+  Debug:policy


This version still results in configuration failure:

$ cri-resmgr -force-config doc-test.cfg ... F0130 09:52:03.497887 36423 main.go:85] failed to create resource manager instance: resource-manager: failed to load forced configuration doc-test.cfg: config error: failed to apply configuration from file: config error: failed to load configuration from file "doc-test.cfg": error converting YAML to JSON: yaml: line 31: could not find expected ':'

Two more key-value fixes needed. Please test.

Yugar-1 · 2023-01-30T11:48:41Z

Also, would you remove the ConfigMap header and related indentation from the "Example configuration that runs all pods in dynamic-pools." yaml in the documentation.

Please make sure that it runs fine with cri-resmgr -force-config doc-test.cfg. This will make it easy for readers to try out.

I'm so sorry for my carelessness, I've fixed it, please check again.

docs/policy/dynamic-pools.md

askervin

Thank you, @Yugar-1!

LGTM.

askervin · 2023-01-30T13:57:36Z

Approved for CI.

@klihub , @marquiz , @jukkar , @fmuyassarov , would you like to have a look, too? :)

…/cri-resource-manager into add-dynamic-pools-policy

klihub

Let's merge this. Thanks Antti ! You've done a great job of reviewing it. And I apologize that I haven't had time to look into this.

I have one general comment which might be irrelevant for this PR. Since I understood so that this was forked off originally from an existing policy, it's good to eventually make sure that there is no unnecessary verbatim code duplication. If there is any substantial duplication of code which is not directly related to the actual policy logic but for instance metrics collection, or something other similar 'utility' functionality, then those maybe could be separated/exported out (in a subsequent PR) in the original policy into a subdirectory and then imported from there to this new policy. But as I said, it might not apply to this PR, if such code duplication does not exist...

marquiz · 2023-02-10T09:35:51Z

Yeah, lgtm this now and address issues have further enhancements in subsequent PRs. And thanks @askervin for all the reviews! I was really useless here 🙄

Yugar-1 requested review from kad, klihub, marquiz, mythi, askervin, jukkar and fmuyassarov as code owners December 28, 2022 07:34

Yugar-1 force-pushed the add-dynamic-pools-policy branch from b0a91a5 to 9125219 Compare December 28, 2022 09:57

askervin requested changes Dec 28, 2022

View reviewed changes

create dynamic-pools policy for cri-rm and corresponding document

9125219

changzhi1990 reviewed Jan 3, 2023

View reviewed changes

docs/policy/dynamic-pools.md Show resolved Hide resolved

Yugar-1 force-pushed the add-dynamic-pools-policy branch from 522df2e to 9125219 Compare January 3, 2023 02:06

Yugar-1 added 3 commits January 3, 2023 09:48

remove IdleCPUClass configuration option

4ed4d78

The CPU will always belong to a dynamic-pool, so the IdleCPUClass configuration option is not needed.

Yugar-1 added 4 commits January 5, 2023 10:25

Fix: code tidy and add unit tests

d11c8f8

1. Split out the two functions `isNeedReallocate()` and `calculatePoolCpuset()` from the function `updatePoolCpuset()`. 2. Added unit tests for two functions `isNeedReallocate()` and `calculatePoolCpuset()`.

Merge branch 'master' into add-dynamic-pools-policy

1939b01

Fix: use joining that -host-root and "/proc/stat" to read the CPU s…

85df06b

…tats instead of new environment variable `HOST_PROC`

Update dynamic-pools policy documentation

ab482fe

Yugar-1 closed this Jan 12, 2023

Yugar-1 reopened this Jan 12, 2023

Yugar-1 added 4 commits January 12, 2023 11:25

Test: add e2e tests for dynamic-pools policy

aeb67c9

Fix CI

3f2fe15

Fix: dynamic-pools tests bugs

69bac75

Yugar-1 added 2 commits January 26, 2023 07:55

Fix: dynamic-pools e2e tests bugs

3949877

Merge branch 'master' into add-dynamic-pools-policy

3755c1a

askervin requested changes Jan 30, 2023

View reviewed changes

docs/policy/dynamic-pools.md Outdated Show resolved Hide resolved

askervin approved these changes Jan 30, 2023

View reviewed changes

Yugar-1 added 4 commits January 30, 2023 12:17

Fix: polish dynamic-pools policy documentation and code

92e730b

Merge branch 'add-dynamic-pools-policy' of https://github.com/Yugar-1…

df922a3

…/cri-resource-manager into add-dynamic-pools-policy

Fix: dynamic-pools policy documentation update

95601db

Fix: dynamic-pools policy documentation yaml update

69b1425

klihub approved these changes Feb 10, 2023

View reviewed changes

askervin merged commit 4674927 into intel:master Feb 10, 2023

marquiz mentioned this pull request Jan 8, 2024

Release v0.9.0 #1075

Closed

18 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Policy: create dynamic-pools policy for cri-rm #969

Policy: create dynamic-pools policy for cri-rm #969

Yugar-1 commented Dec 28, 2022

jukkar commented Dec 28, 2022

Yugar-1 commented Dec 28, 2022

askervin left a comment

askervin Dec 28, 2022

Yugar-1 Jan 3, 2023

askervin Dec 28, 2022

Yugar-1 Jan 3, 2023

Yugar-1 commented Jan 5, 2023 •

edited

Loading

askervin commented Jan 5, 2023 •

edited

Loading

Yugar-1 commented Jan 12, 2023

codecov-commenter commented Jan 23, 2023

askervin commented Jan 23, 2023

Yugar-1 commented Jan 25, 2023

askervin commented Jan 25, 2023

Yugar-1 commented Jan 26, 2023

askervin left a comment

askervin Jan 30, 2023

askervin Jan 30, 2023

Yugar-1 commented Jan 30, 2023

askervin left a comment

askervin Jan 30, 2023

Yugar-1 commented Jan 30, 2023

askervin left a comment

askervin commented Jan 30, 2023

klihub left a comment

marquiz commented Feb 10, 2023

Policy: create dynamic-pools policy for cri-rm #969

Policy: create dynamic-pools policy for cri-rm #969

Conversation

Yugar-1 commented Dec 28, 2022

jukkar commented Dec 28, 2022

Yugar-1 commented Dec 28, 2022

askervin left a comment

Choose a reason for hiding this comment

askervin Dec 28, 2022

Choose a reason for hiding this comment

Yugar-1 Jan 3, 2023

Choose a reason for hiding this comment

askervin Dec 28, 2022

Choose a reason for hiding this comment

Yugar-1 Jan 3, 2023

Choose a reason for hiding this comment

Yugar-1 commented Jan 5, 2023 • edited Loading

askervin commented Jan 5, 2023 • edited Loading

Yugar-1 commented Jan 12, 2023

codecov-commenter commented Jan 23, 2023

Codecov Report

askervin commented Jan 23, 2023

Yugar-1 commented Jan 25, 2023

askervin commented Jan 25, 2023

Yugar-1 commented Jan 26, 2023

askervin left a comment

Choose a reason for hiding this comment

askervin Jan 30, 2023

Choose a reason for hiding this comment

askervin Jan 30, 2023

Choose a reason for hiding this comment

Yugar-1 commented Jan 30, 2023

askervin left a comment

Choose a reason for hiding this comment

askervin Jan 30, 2023

Choose a reason for hiding this comment

Yugar-1 commented Jan 30, 2023

askervin left a comment

Choose a reason for hiding this comment

askervin commented Jan 30, 2023

klihub left a comment

Choose a reason for hiding this comment

marquiz commented Feb 10, 2023

Yugar-1 commented Jan 5, 2023 •

edited

Loading

askervin commented Jan 5, 2023 •

edited

Loading