[DML EP] Add DML implementation for BiasGelu

[PatriceVignola]

Description

Add DML implementation for BiasGelu

[sumitsays]

ADD1 only optimizes Relu and PRelu activations at the shader level. For rest other operators it will dispatch a separate shader which will be equivalent to execute decomposed form of BiasGelu. So, I think it won't add any benefit in performance.

[PatriceVignola]

ADD1 only optimizes Relu and PRelu activations at the shader level. For rest other operators it will dispatch a separate shader which will be equivalent to execute decomposed form of BiasGelu. So, I think it won't add any benefit in performance.

Sure, but it still won't be worse than calling both operators separately. And the current behavior is that it falls back to the CPU, which is very bad. We could also optimize the Gelu activation into ADD1 if necessary in the future.

[sumitsays]

Thank you.

[fdwr]

Pat: Do you know why it now falls back to CPU, given (I thought anyway, the last time we looked Sumit) there was a functional decomposition to Add and Gelu, which called DML? Maybe I misremember, but this is a general concern that if DML is the priority number #1 in the execution provider list, that any decomposable operators go to it first, rather than a fused version of the CPU one. I had some emails with Scott McKay long ago about this that I'll dredge up. Maybe there's a bug elsewhere to fix, and we can delete this temporary kernel later.

[PatriceVignola]

Pat: Do you know why it now falls back to CPU, given (I thought anyway, the last time we looked Sumit) there was a functional decomposition to Add and Gelu, which called DML? Maybe I misremember, but this is a general concern that if DML is the priority number #1 in the execution provider list, that any decomposable operators go to it first, rather than a fused version of the CPU one. I had some emails with Scott McKay long ago about this that I'll dredge up. Maybe there's a bug elsewhere to fix, and we can delete this temporary kernel later.

I'm not familiar with how op decomposition works. In this case, it's not even a "fusion": BiasGelu is hardcoded in the ONNX model. Does decomposition work in this case? And where would I find the logic in the code?

[sumitsays]

Pat: Do you know why it now falls back to CPU, given (I thought anyway, the last time we looked Sumit) there was a functional decomposition to Add and Gelu, which called DML? Maybe I misremember, but this is a general concern that if DML is the priority number #1 in the execution provider list, that any decomposable operators go to it first, rather than a fused version of the CPU one. I had some emails with Scott McKay long ago about this that I'll dredge up. Maybe there's a bug elsewhere to fix, and we can delete this temporary kernel later.

I'm not familiar with how op decomposition works. In this case, it's not even a "fusion": BiasGelu is hardcoded in the ONNX model. Does decomposition work in this case? And where would I find the logic in the code?

There is no such decomposition happens. If BiasGelu` is hardcoded, then in that case we need dedicated BiasGelu registration to make sure that it won't fallback to CPU.

[fdwr]

There is no such decomposition happens. If BiasGelu` is hardcoded, then in that case we need dedicated BiasGelu registration to make sure that it won't fallback to CPU.

Alrighty. I must have been thinking of another one of the many other *elu's then, like Selu.

Function	Since version	Function version
Celu	12	12
Elu	6, 1	18
PRelu	16, 9, 7, 6, 1	16
Relu	14, 13, 6, 1	18
Selu	6, 1	18