Improve task scheduler rate limiter

[yycptt]

What changed?

• Fix task scheduler rate limiter disabling. Previously when disabling, if a task channel is throttled, the throttled flag will never be cleared.
• Use Allow() instead of Reserve(), to minimize the impact of idle task channel when there's large number of empty namespaces. Keep doing Reserve() & CancelAt() may cause some rps token to be wasted.
• Add a shadow mode for this rate limiter to emit throttled metrics but no actually block task scheduling.
• Emit metrics upon throttling.

Why?

How did you test it?

Potential risks

Is hotfix candidate?

[MichaelSnowden]

Can we use a decorator pattern here instead of a flag argument? That would follow the open-closed principle and reduce branching/entanglement. For example:

fx.Decorate(func(rateLimiter, cfg) RateLimiter {
  return &rateLimiterShadow{rateLimiter, cfg.EnableRateLimiterShadowMode}
}

type rateLimiterShadow {
  RateLimiter
  UseShadow dynamicconfig.BoolPropertyFn
 }

func (t *rateLimiterShadow) Wait(...) ... {
  if t.UseShadow {
  } else {
  }
}

[yycptt]

Can we use a decorator pattern here instead of a flag argument? That would follow the open-closed principle and reduce branching/entanglement. For example:

fx.Decorate(func(rateLimiter, cfg) RateLimiter {
  return &rateLimiterShadow{rateLimiter, cfg.EnableRateLimiterShadowMode}
}

type rateLimiterShadow {
  RateLimiter
  UseShadow dynamicconfig.BoolPropertyFn
 }

func (t *rateLimiterShadow) Wait(...) ... {
  if t.UseShadow {
  } else {
  }
}

Yeah that's a good idea. Not that easy though since different rate limiter requests need different metrics tag (and we need a way to pass those information in)

But for this change, consider it as a debug build, and all shadow related code will be deleted in next release after the issue was fixed.

[MichaelSnowden]

It'd be better if this PR were split up into individual changes and we didn't use flag arguments, but given that this will be removed next release, LGTM.

[MichaelSnowden]

It's difficult for me to review the changes in common/tasks/interleaved_weighted_round_robin.go because the loop involved is so immense. If there's enough time, I'd encourage you to do some preparatory refactoring before adding this change. In particular, there's a complicated mix of blocking, non-blocking checks, usage of loop labels, and synchronization in one function. I'd encourage a more idiomatic Go approach where we use goroutines and select channel loops as opposed to nested loops with non-blocking ready checks