docker:nvidia opencl support with instructions

.docker/nvidia.dockerfile

@@ -0,0 +1,32 @@
+FROM debian:bookworm
+
+ARG DEBIAN_FRONTEND=noninteractive
+RUN apt update -y
+RUN apt install -y curl \
+    apt-utils \
+    unzip \
+    tar \
+    curl \
+    xz-utils \
+    ocl-icd-libopencl1 \
+    opencl-headers \
+    clinfo \
+    build-essential \
+    gcc
+
+RUN mkdir -p /etc/OpenCL/vendors && \
+    echo "libnvidia-opencl.so.1" > /etc/OpenCL/vendors/nvidia.icd
+ENV NVIDIA_VISIBLE_DEVICES all
+ENV NVIDIA_DRIVER_CAPABILITIES compute,utility
+
+RUN curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs > /rustup.sh
+RUN chmod +x /rustup.sh
+RUN /rustup.sh -y
+
+RUN apt install -y opencl-dev
+
+RUN bash -c 'export PATH="$PATH:$HOME/.cargo/bin";rustup default nightly'
+
+COPY . /opt/rllama
+RUN bash -c 'export PATH="$PATH:$HOME/.cargo/bin";cd /opt/rllama;RUSTFLAGS="-C target-feature=+sse2,+avx,+fma,+avx2" cargo build --release --features server,opencl'
+RUN ln -s /opt/rllama/target/release/rllama /usr/bin

.docker/nvidia.md

@@ -0,0 +1,40 @@
+#rllama docker on nvidia
+
+## Getting OpenCL to work inside docker.
+Please note that this also requires some packages and modifications on your host system in order to allow the containers to use nvidia GPU features such as **compute**.
+
+
+For each of the described distro / distro-family you could follow the instructions at the given links below.
+
+**Note**: You also need an upto-date version of docker/docker-ce so be sure to follow the instructions to install docker for your distro from the [docker website](https://docs.docker.com/engine/install).
+
+**Note2**: I have only personally tested the instructions on fedora/nobara and hence, cannot guarantee the accuracy of the instructions for other distros.
+
+### Fedora / Fedora-based
+**[https://gist.github.com/JuanM04/fcbed16d0f4405a286adebee5fd31cb2](https://gist.github.com/JuanM04/fcbed16d0f4405a286adebee5fd31cb2)**
+
+
+### Debian / Debian-based / Ubuntu / Ubuntu-based
+**[https://www.howtogeek.com/devops/how-to-use-an-nvidia-gpu-with-docker-containers/](https://www.howtogeek.com/devops/how-to-use-an-nvidia-gpu-with-docker-containers/)**
+
+
+### Arch / Arch-based
+**[https://wiki.archlinux.org/title/Docker#Run_GPU_accelerated_Docker_containers_with_NVIDIA_GPUs](https://wiki.archlinux.org/title/Docker#Run_GPU_accelerated_Docker_containers_with_NVIDIA_GPUs)**
+
+Feel free to contribute/improve the instructions for existing and other distros.
+
+## Usage
+1. 
+```bash
+docker build -f ./.docker/nvidia.dockerfile -t rllama:nvidia .
+```
+2.
+```bash
+docker run --rm --gpus all --privileged -v /models/LLaMA:/models:z -it rllama:nvidia \
+    rllama --model-path /models/7B \
+           --param-path /models/7B/params.json \
+           --tokenizer-path /models/tokenizer.model \
+           --prompt "hi I like cheese"
+```
+
+Replace `/models/LLaMA` with the directory you've downloaded your models to. The `:z` in `-v` flag may or may not be needed depending on your distribution (I needed it on Fedora Linux)

README.md

@@ -59,6 +59,8 @@ features if you install manually from this Git repository instead.
 There is a Dockerfile you can use if you'd rather just get started quickly and
 you are familiar with `docker`. You still need to download the models yourself.
 
+
+### For CPU-only docker support:
 ```
 docker build -f ./.docker/cpu.dockerfile -t rllama .
 ```
@@ -75,6 +77,9 @@ Replace `/models/LLaMA` with the directory you've downloaded your models to.
 The `:z` in `-v` flag may or may not be needed depending on your distribution
 (I needed it on Fedora Linux)
 
+### For GPU-enabled docker support with nvidia:
+Follow the instructions [here](.docker/nvidia.md).
+
 ## LLaMA weights
 
 Refer to https://github.com/facebookresearch/llama/ As of now, you need to be