[FEATURE]: Migrate training into Celery

[andrewpeng02]

Feature Name

Migrate training into Celery

Your Name

Andrew Peng

Description

Currently, we do the training during the HTTP request. I plan on changing the train HTTP endpoints by scheduling a train job via Celery and returning the job id in the request. This offers numerous advantages

• Long training tasks (>2 min) shouldn't be done in an HTTP request. Scheduling the training job will allow the endpoint to return quickly
• Eventually, we can decouple the backend with the training, so that we can use cheaper EC2 instances for the Django backend, and GPU instances for the actual training

Notifying the user will be done in websockets in this issue (#920 (comment)), for now I'll create an HTTP endpoint to retrieve the training results that the user can ping.

[github-actions]

Hello @andrewpeng02! Thank you for submitting the Feature Request Form. We appreciate your contribution. 👋

We will look into it and provide a response as soon as possible.

To work on this feature request, you can follow these branch setup instructions:

1. Checkout the main branch:

```
 git checkout nextjs
```

2. Pull the latest changes from the remote main branch:

```
 git pull origin nextjs
```

3. Create a new branch specific to this feature request using the issue number:

```
 git checkout -b feature-1136
```

Feel free to make the necessary changes in this branch and submit a pull request when you're ready.

Best regards,
Deep Learning Playground (DLP) Team

[karkir0003]

@andrewpeng02 thanks for writing this issue. Is this docs on celery the updated ones: https://docs.celeryq.dev/en/stable/django/first-steps-with-django.html