Skip to content
/ nemo Public
forked from kaist-ina/nemo

This is an official GitHub repository for the paper, "NEMO: Enabling Neural-enhanced Video Streaming on Commodity Mobile Devices".

Notifications You must be signed in to change notification settings

oximi123/nemo

 
 

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

93 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

NEMO (MobiCom'20)

This is an official Github repository for the MobiCom paper "NEMO: Enabling Neural-enhanced Video Streaming on Commodity Mobile Devices". This project is built upon Google libvpx, Android Exoplayer, and Qualcomm SNPE and consists of C/C++/Java/Python.
[Project homepage] [Paper] [Video]

If you use our work for research, please cite it.

@inproceedings{yeo2020nemo,
  title={NEMO: enabling neural-enhanced video streaming on commodity mobile devices},
  author={Yeo, Hyunho and Chong, Chan Ju and Jung, Youngmok and Ye, Juncheol and Han, Dongsu},
  booktitle={Proceedings of the 26th Annual International Conference on Mobile Computing and Networking},
  pages={1--14},
  year={2020}
}

Lastly, NEMO is currently protected under the patent and is retricted to be used for the commercial usage.

Project structure

./nemo
├── video                  # Python: Video downloader/encoder
├── dnn                    # Python: DNN trainer/converter
├── cache_profile          # Python: Anchor point selector
├── player                 # Java, C/C++: Android video player built upon Exoplayer and the SR-integrated codec
./third_party
├── libvpx                 # C/C++: SR-integrated codec

Prerequisites

Guide

We provide a step-by-step guide with a single video (which content is product review).
All the folloiwing commands must be executed inside the docker.

1. Setup

  • Clone the NEMO docker repository
git clone https://github.com/chaos5958/nemo-docker.git
  • Build the docker image
cd ${HOME}/nemo-docker
./build.sh
  • Run & Attach to the docker
cd ${HOME}/nemo-docker
./run.sh
  • Clone the NEMO main repository
git clone --recurse-submodules https://github.com/kaist-ina/nemo.git ${NEMO_CODE_ROOT}
  • Download/Setup the Qualcomm SNPE SDK as follow:
./nemo
├── third_party
    ├── snpe
        ├── benchmarks
        ├── bin
        ├── include
        ├── lib
        ├── models
        ├── share
        ...

2. Prepare videos

  • Download a Youtube video
$NEMO_CODE_ROOT/nemo/tool/script/download_video.sh -c product_review
  • Encode the video
$NEMO_CODE_ROOT/nemo/tool/script/encode_video.sh -c product_review

Details are described in this file.

3. Prepare DNNs

  • Train a DNN
$NEMO_CODE_ROOT/nemo/dnn/script/train_video.sh -g 0 -c product_review -q high -i 240 -o 1080
  • (Optional) Convert the TF model to the Qualcomm SNPE dlc
$NEMO_CODE_ROOT/nemo/dnn/script/convert_tf_to_snpe.sh -g 0 -c product_review -q high -i 240 -o 1080
  • (Optional) Test the dlc on Qualcomm devices
$NEMO_CODE_ROOT/nemo/dnn/script/test_snpe.sh -g 0 -c product_review -q high -r 240 -s 4 -d [device id]

Details are described in this file.

4. Generate a cache profile

  • Setup: Build the SR-integrated codec (x86_64)
$NEMO_CODE_ROOT/nemo/cache_profile/script/setup.sh
  • Generate the cache profile using the codec
$NEMO_CODE_ROOT/nemo/cache_profile/script/select_anchor_points.sh -g 0 -c product_review -q high -i 240 -o 1080 -a nemo
  • (Optional) Analyze frame dependencies & frame types
$NEMO_CODE_ROOT/nemo/cache_profile/script/analyze_video.sh -g 0 -c product_review -q high -i 240 -o 1080 -a nemo

Details are described in this file.

5. Compare NEMO vs. baselines

  • Setup: Build the SR-integrated codec (arm64-v8)
$NEMO_CODE_ROOT/nemo/test/script/setup_local.sh 
  • Setup: Copy data to mobile devices
$NEMO_CODE_ROOT/nemo/test/script/setup_device.sh -c product_review -q high -r 240 -a nemo_0.5 -d [device id]
  • Measure the latency
$NEMO_CODE_ROOT/nemo/test/script/measure_latency.sh -c product_review -q high -r 240 -a nemo_0.5 -d [device id]
  • Measure the quality
$NEMO_CODE_ROOT/nemo/test/script/measure_quality.sh -g 0 -c product_review -q high -i 240 -o 1080 -a nemo_0.5 

Details are described in this file.

6. Play NEMO in Android smartphones

  • Setup: Copy data to mobile devices
$NEMO_CODE_ROOT/nemo/player/script/setup_device.sh -c product_review -q high -r 240 -a nemo_0.5 -d [device id] -a nemo_0.5_16
  • Run the NEMO player
  1. Enable USB debugging and USB install at mobiles.
  2. Open $NEMO_CODE_ROOT/nemo/player using Android Studio.
  3. Set ndk path in local.properties as ndk.dir=[ndk folder]/android-ndk-14b; I used android-ndk-14b.
  4. Build & run it!
App information
* product review: You can choose content.
* 1: it is deprecated.
* high: You can choose DNN quality level.
* Decode-Cache: You can choose Decode, Decode-SR (=per frame super-resolution), Decode-cache (=NEMO).
* None: You can choose total playback time.
* 0.5: You can choose the cache profile, 0.5 represents the quality threshold compared to per-frame super-resolutionl.

About

This is an official GitHub repository for the paper, "NEMO: Enabling Neural-enhanced Video Streaming on Commodity Mobile Devices".

Resources

Stars

Watchers

Forks

Packages

No packages published

Languages

  • Java 95.7%
  • Python 2.1%
  • Other 2.2%