Skip to content
/ kantv Public

workbench for learing&practising AI tech in real scenario on Android device, powered by GGML(Georgi Gerganov Machine Learning) and NCNN(Tencent NCNN) and FFmpeg

License

Apache-2.0 and 2 other licenses found

Licenses found

Apache-2.0
LICENSE
MIT
LICENSE-llamacpp
Unknown
LICENSE-zh
Notifications You must be signed in to change notification settings

zhouwg/kantv

 
 

Repository files navigation

KanTV

KanTV("Kan", aka Chinese PinYin "Kan" or Chinese HanZi "看" or English "watch/listen") , an open source project focus on Kan(aka "Watch/Listen" in English) online TV for Android-based device,derived from original ijkplayer , with many/much enhancements:

  • Watch online TV and local media (by customized FFmpeg and Exoplayer with updated version:FFmpeg 6.1, Exoplayer 2.15, source code of customized FFmpeg could be found here according to FFmpeg's license, source code of customized Exoplayer(based on my previous study in Exoplayer) could be found within source code of this project)

  • Record online TV to automatically generate videos (usable for short video creators to generate short video materials but pls respect IPR of original content creator/provider)

  • Set up a custom playlist and then use this software to watch the content of the custom playlist

  • Performance benchmark for Android-based mobile phone

  • Watch encrypted live stream(Google Widevine, Huawei WisePlay, ChinaDRM......)

  • Real-time English subtitle for online TV by the great & excellent & amazing whisper.cpp (PoC finished on Xiaomi 14. Xiaomi 14 or other powerful Android mobile phone IS HIGHLY required for this feature otherwise unexpected behaviour would happen)

  • UI refactor

  • Android turn-key project(customized/secondary software development activity could be done by Android Studio IDE)

Some goals of this project are:

  • Well-maintained "workbench" for AI R&D activites

    • Well-maintained "workbench" for ASR(Automatic Speech Recognition) researchers who was interested in practise state-of-the-art AI tech(like GGML's whisper.cpp) in real scenario
    • Well-maintained "workbench" for software programmer to learning AI technology in real scenario
    • ...
  • Android turn-key project for software programmers who was interested in device-side AI application on Android-based device

  • Give back to open source community after got/learnt a lot from open source community

How to build project

prerequisites

  • Host OS information:
uname -a

Linux 5.8.0-43-generic #49~20.04.1-Ubuntu SMP Fri Feb 5 09:57:56 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

cat /etc/issue

Ubuntu 20.04.2 LTS \n \l

  • tools & utilities
sudo apt-get update
sudo apt-get install build-essential -y
sudo apt-get install cmake -y
sudo apt-get install curl -y
sudo apt-get install python -y
sudo apt-get install tcl expect -y
sudo apt-get install nginx -y
sudo apt-get install git -y
sudo apt-get install vim -y
sudo apt-get install spawn-fcgi -y
sudo apt-get install u-boot-tools -y
sudo apt-get install ffmpeg -y
sudo apt-get install openssh-client -y

sudo dpkg --add-architecture i386
sudo apt-get install lib32z1 -y

sudo apt-get install -y android-tools-adb android-tools-fastboot autoconf \
        automake bc bison build-essential ccache cscope curl device-tree-compiler \
        expect flex ftp-upload gdisk acpica-tools libattr1-dev libcap-dev \
        libfdt-dev libftdi-dev libglib2.0-dev libhidapi-dev libncurses5-dev \
        libpixman-1-dev libssl-dev libtool make \
        mtools netcat python-crypto python3-crypto python-pyelftools \
        python3-pycryptodome python3-pyelftools python3-serial \
        rsync unzip uuid-dev xdg-utils xterm xz-utils zlib1g-dev

sudo apt-get install python3-pip -y
sudo apt-get install indent -y
pip3 install meson ninja

echo "export PATH=/home/`whoami`/.local/bin:\$PATH" >> ~/.bashrc

  • bazel

    download bazel-3.1.0 and install bazel manually

  wget https://github.com/bazelbuild/bazel/releases/download/3.1.0/bazel-3.1.0-linux-x86_64
  sudo ./bazel-3.1.0-installer-linux-x86_64.sh

bazel is NOT used currently but put it here for further usage in the future. Of course, this step could be skipped accordingly.

  • Android NDK & Android Studio

    download and install Android Studio and Android NDK manually

    Android Studio 4.2.1

    Android NDK-r26c

    then put Android NDK-r26c into /opt/kantv-toolchain accordingly

    ls /opt/kantv-toolchain/android-ndk-r26c
    
    
  • vim settings

borrow from http://ffmpeg.org/developer.html#Editor-configuration

set ai
set nu
set expandtab
set tabstop=4
set shiftwidth=4
set softtabstop=4
set noundofile
set nobackup
set fileformat=unix
set undodir=~/.undodir
set cindent
set cinoptions=(0
" Allow tabs in Makefiles.
autocmd FileType make,automake set noexpandtab shiftwidth=8 softtabstop=8
" Trailing whitespace and tabs are forbidden, so highlight them.
highlight ForbiddenWhitespace ctermbg=red guibg=red
match ForbiddenWhitespace /\s\+$\|\t/
" Do not highlight spaces at the end of line while typing on that line.
autocmd InsertEnter * match ForbiddenWhitespace /\t\|\s\+\%#\@<!$/

Fetch source codes

For Android APP/APK developers,

git clone https://github.com/cdeos/kantv.git
cd kantv
git checkout master

For C/C++ developers,

git clone --recurse-submodules https://github.com/cdeos/kantv.git
cd kantv
git checkout master

Build all dependent native codes

modify build/envsetup.sh accordingly before launch build

pay attention here and modify it accordingly if target Android device is NOT Xiaomi 14

. build/envsetup.sh

time ./build-all.sh

Build Android APK

  • Build APK from source code by Android Studio IDE manually

    Please attention some source codes in ASRResearchFragment.java which affect the running of the ASR demo and the size of the generated APK.

  • Latest prebuit APK could be found here Github(the size of the prebuilt APK is about 90M because it contains dependent model file of DeepSpeech for purpose of make ASR demo happy).

Run Android APK on real Android phone

This apk follows the principles of 'minimum permissions' and 'do not collect unnecessary user data' or EU's GDPR principle. When installing/using for the first time on an Android phone, only the following two permissions are required:

  • Access to storage is required to generate necessary temporary files
  • Access to device information is required to obtain current phone network status information, distinguishing whether the current network is Wi-Fi or mobile when playing online TV

The following is some English snapshots.

Screenshot_20240301_000503_com cdeos kantv Screenshot_20240301_000509_com cdeos kantv 1210108450

Screenshot_20240301_114116_com cdeos kantv Screenshot_2024_0304_131033

Screenshot_20240301_000602_com cdeos kantv Screenshot_20240301_000609_com cdeos kantv

Other English and Chinese snapshots could be found in release directory.

ChangeLog

Changelog could be found here.

Roadmap

  • real-time English subtitle for online English TV on Xiaomi 14(because it contains a very powerful mobile SoC) by great and excellent and amazing whisper.cpp

  • real-time Chinese subtitle for online English TV on Xiaomi 14(because it contains a very powerful mobile SoC) by great and excellent and amazing whisper.cpp

  • integrate gstreamer to project KanTV(VLC is also excellent and gstreamer is more complicated than VLC but gstreamer was supported by many semiconductor companies. anyway, they are both born in/come from EU)

  • bugfix in UI layer

  • UI refactor and “align to" UI in China's most popular and successful app WeChat(learn from WeChat)

  • ...

How to setup customized KanTV server in local development env

The computing power and network bandwidth of default kantvserver is very low due to insufficient fund, so setup a local End-2-End development env is strongly recommended.

  • setup a http server(by apache or nginx) in local development env

  • modify kant server address in app

1370107702

  • upload required files to local http server like this(dependent files for DeepSpeech could be found here)
   apk ->                              http(s)://local_http_server/kantv/apk/kantv-latest.apk
   apk version ->                      http(s)://local_http_server/kantv/apk/kantv-version.txt
   audio.wav  ->                       http(s)://local_http_server/kantv/deepspeech/audio.wav
   deepspeech-0.9.3-models.tflite ->   http(s)://local_http_server/kantv/deepspeech/deepspeech-0.9.3-models.tflite
   deepspeech-0.9.3-models.scorer ->   http(s)://local_http_server/kantv/deepspeech/deepspeech-0.9.3-models.scorer


   something for whisper.cpp      ->   http(s)://local_http_server/kantv/whisper/something

   something for paddlespeech     ->   http(s)://local_http_server/kantv/paddlespeech/something

How to create customized playlist for kantv apk

  • create test.m3u(recommend name and it's hardcoded in source code) like this:
  #EXTM3U
  #EXTINF:-1,hls
  http(s)://local_http_server/kantv/media/test.hls
  #EXTINF:-1,dash
  http(s)://local_http_server/kantv/media/test.dash
  #EXTINF:-1,rtmp
  http(s)://local_http_server/kantv/media/test.rtmp
  #EXTINF:-1,webrtc
  http(s)://local_http_server/kantv/media/test.rtc
  #EXTINF:-1,hevc(h265)
  http(s)://local_http_server/kantv/media/test.hevc
  #EXTINF:-1,h266
  http(s)://local_http_server/kantv/media/test.h266
  #EXTINF:-1,av1
  http(s)://local_http_server/kantv/media/test.av1
  #EXTINF:-1,testvideo-1 (pls attention following path is start with /)
  /test.mp4
  #EXTINF:-1,testvideo-2
  /video/test.ts

or just fetch favourite playlist from IPTV and rename it to test.m3u(pls attention that users/developers from Mainland China should review this issue)

  • upload test.m3u to local http server like this
 test.m3u                  ->   http(s)://local_http_server/kantv/epg/test.m3u

How to integrate proprietary codes with project KanTV for proprietary R&D activity

For AI expert who want to integrate proprietary codes(which contains IPR and consists of Java/JNI/native...) to customized/derived project of KanTV, Please refer to this opening issue How to integrate proprietary codes for proprietary R&D activity

Support

  • Please do not send e-mail to me for technical question. Public technical discussion on github is preferred.
  • feel free to submit issues or new features(focus on Android at the moment), volunteer support would be provided if time permits.

Contribution

Be sure to review the opening issues before contribute to project KanTV, We use GitHub issues for tracking requests and bugs, please see how to submit issue in this project .

Report issue in various Android-based phone or even submit PR to this project is great welcomed.

English is preferred in this project. thanks for cooperation and understanding.

Acknowledgement

Many/sincerely thanks to all contributors of the great open source community, especially all original authors and all contributors of the great Linux & Android & FFmpeg && whisper.cpp and other excellent projects.

Project KanTV has used/tried following open-source projects(list in here is not incomplete):

License

Copyright (c) 2017 Bilibili
Licensed under LGPLv2.1 or later
Copyright (c) 2021 - 2023 Project KanTV

Copyright (c) 2024 -  Authors of Project KanTV

Licensed under Apachev2.0 or later

Commercial Use

Project KanTV is licensed under Apachev2.0 or later, so itself is free/open for commercial use.

About

workbench for learing&practising AI tech in real scenario on Android device, powered by GGML(Georgi Gerganov Machine Learning) and NCNN(Tencent NCNN) and FFmpeg

Topics

Resources

License

Apache-2.0 and 2 other licenses found

Licenses found

Apache-2.0
LICENSE
MIT
LICENSE-llamacpp
Unknown
LICENSE-zh

Stars

Watchers

Forks

Packages

No packages published