StyleTransfer_UE4

realtime style transfer in unreal engine

Steps for Release build

• Make Sure only enable one Intel iGPU

• Right click StyleTransfer.uproject, click generate visual studio project file, then StyleTransfer.sln is appeared

• Open StyleTransfer.sln, select Development Editor to build

• Open StyleTransfer.uproject, go to menu File-> Project Launcher -> Windows(64-bit)

• Choose folder\for\Release project, waiting for the compiling result

• Go to folder\for\Release project, run StyleTransfer.exe -WINDOWED -ResX=1920 -ResY=1080 -ExecCmds="r.OVST.Width 1920, r.OVST.Height 1080"

• Press ~ to change mode, for example, r.OVST.Enabled 2.
  

  Performance

model_v5 inference

	FPS	Inference time	copy buffer from CPU to GPU	Loading time
No style	200	0	0	0
GPU_OCL	9	105ms	0	3.8s
GPU1	6	150ms	17ms	2.8s
GPU0	1.89	502ms	18ms	3.1s

model_v9 inference

	FPS	Inference time	copy buffer from CPU to GPU	Loading time
No style	200	0	0	0
GPU_OCL	24	38ms	0	3.2s
GPU1	9.6	84ms	19ms	2.4s
GPU0	3.4	256ms	23ms	2.0s

model_v9_int8 inference

	FPS	Inference time	copy buffer from CPU to GPU	Loading time
No style	200	0	0	0
GPU_OCL	31	28ms	0	2.2s
GPU1	10	76ms	19ms	1.7s
GPU0	5	156ms	23ms	1.1s

• GPU 0: Intel(R) UHD Graphics
• GPU 1: Intel(R) Arc(TM) A730M Graphics
• There are two times of CPU/GPU copy during inference in GPU.1/GPU.0 mode
• int8 model got 1.3x faster on GPU.1 and 2x faster on GPU.0

To-Do list

• Change model to int8 precision for 30 fps target
• GPU管线和openvino 推理 时间统计，加到屏幕统计信息
• 用openvino D3D api 拿到DirectX的数据做推理
• 把UE渲染数据通过ID3D11Device的方式拿到
• 把style transfer的结果放到主窗口显示

Step to build OpenVINO Wrapper

• mkdir build
• cd build
• cmake ..
• open OpenVinoWrapper.sln project properties -> C/C++ -> preprocessor -> preprocessor definition -> join NOMINMAX

Step to import OpenVINO plugin into another UE project

• make sure your project is C++ project. If not, directly new c++ class(left top UI), it will automatically convert the project into C++ project
• copy Folder Content\Intel and Plugin into new project folder
• open *uproject
• create Blueprint file by refering BP ThirdPersonActor under folder ThridPersonBP
• Drop the BP into Main scene for connecting OpenVINO plugin
• From now, you can use command line r.OVST.Enabled 2 to turn on style transfer