Livestreams fail to progress to convert and move steps #457
Comments
|
Can you run the below to save the logs and upload them here? |
|
I would love to contribute to the issue if you both don't mind, I'm having the same problem |
|
@Majow I see the worker is panicing when trying to stop the chat. I've fixed the panic in #458 but I'm not 100% it's going to resolve your issue. When you're not archiving anything can you pull and use the If you want to try to recover the 'stuck' streams before restarting the container, follow these steps: #450 (comment) |
|
Here are my logs. They're 14MB of junk because I'm checking for livestreams every minute. Hope it helps though. |
|
@Dragonatorul Looks like something is failing to kill the chat download. Can you run If you don't have the |
|
Thanks! Makes sense on the chat, but what about the video convert and move? Now that I think about it, this was a problem before moving to the new queue system, but when it happened before I'd just stop the download task and start the convert task, which then triggered the move task. That way I'd have at least a partial download/vod available. Is there a way to do that now? The buttons to do that seem to be gone. This is the output. And this is the current queue. Only the last two items are actually live. The rest are stuck there for days now.
|
|
Also, this is my deployment docker compose if it helps. The temp folder is local, but the destination is over SMB, in another VM, but on the same hardware node. It's all being served through an Nginx Proxy Manager reverse proxy front-end. version: "3.3"
services:
ganymede-api:
container_name: ganymede-api
image: ghcr.io/zibbp/ganymede:latest
restart: unless-stopped
depends_on:
- ganymede-temporal
environment:
- TZ=$TZ
- DB_HOST=$DB_HOST
- DB_PORT=$DB_PORT
- DB_USER=$DB_USER
- DB_PASS=$DB_PASS
- DB_NAME=$DB_NAME
- DB_SSL=$DB_SSL
- JWT_SECRET=$JWT_SECRET
- JWT_REFRESH_SECRET=$JWT_REFRESH_SECRET
- TWITCH_CLIENT_ID=$TWITCH_CLIENT_ID
- TWITCH_CLIENT_SECRET=$TWITCH_CLIENT_SECRET
- FRONTEND_HOST=$FRONTEND_HOST
- COOKIE_DOMAIN=$ROOT_DOMAIN
# OPTIONAL
# - OAUTH_PROVIDER_URL=
# - OAUTH_CLIENT_ID=
# - OAUTH_CLIENT_SECRET=
# - OAUTH_REDIRECT_URL=http://IP:PORT/api/v1/auth/oauth/callback # Points to the API service
- TEMPORAL_URL=ganymede-temporal:7233
# WORKER
- MAX_CHAT_DOWNLOAD_EXECUTIONS=5
- MAX_CHAT_RENDER_EXECUTIONS=3
- MAX_VIDEO_DOWNLOAD_EXECUTIONS=5
- MAX_VIDEO_CONVERT_EXECUTIONS=3
volumes:
- ganymede_vods:/vods
- ganymede_logs:/logs
- ganymede_data:/data
- ganymede_tmp:/tmp
# ports:
# - 4800:4000
networks:
- npm_shared_public
dns:
- 192.168.1.50
- 1.1.1.1
- 8.8.8.8
ganymede-frontend:
container_name: ganymede-frontend
image: ghcr.io/zibbp/ganymede-frontend:latest
restart: unless-stopped
environment:
- API_URL=$API_URL
- CDN_URL=$CDN_URL
- SHOW_SSO_LOGIN_BUTTON=$SHOW_SSO_LOGIN_BUTTON
- FORCE_SSO_AUTH=$FORCE_SSO_AUTH
- REQUIRE_LOGIN=$REQUIRE_LOGIN
# ports:
# - 4801:3000
networks:
- npm_shared_public
dns:
- 192.168.1.50
- 1.1.1.1
- 8.8.8.8
ganymede-temporal:
image: temporalio/auto-setup:1
container_name: ganymede-temporal
depends_on:
- ganymede-db
environment:
- DB=postgresql # this tells temporal to use postgres (not the db name)
- DB_PORT=5432
- POSTGRES_USER=$DB_USER
- POSTGRES_PWD=$DB_PASS
- POSTGRES_SEEDS=ganymede-db # name of the db service
networks:
- npm_shared_public
ports:
- 7233:7233
ganymede-db:
container_name: ganymede-db
image: postgres:14
restart: unless-stopped
volumes:
- ganymede_db:/var/lib/postgresql/data
environment:
- POSTGRES_PASSWORD=$DB_PASS
- POSTGRES_USER=$DB_USER
- POSTGRES_DB=$DB_NAME
ports:
- 4803:5432
networks:
- npm_shared_public
ganymede-nginx:
container_name: ganymede-nginx
image: nginx
restart: unless-stopped
volumes:
- $GANYMEDE_GIT_PATH/nginx.conf:/etc/nginx/nginx.conf:ro
- ganymede_vods:/mnt/vods
# ports:
# - 4802:8080
networks:
- npm_shared_public
volumes:
git:
driver: local
driver_opts:
type: 'none'
o: 'bind'
device: ${GANYMEDE_GIT_PATH}
ganymede_data:
driver: local
driver_opts:
type: 'none'
o: 'bind'
device: ${GANYMEDE_DATA_PATH}
ganymede_logs:
driver: local
driver_opts:
type: 'none'
o: 'bind'
device: ${GANYMEDE_LOGS_PATH}
ganymede_db:
driver: local
driver_opts:
type: 'none'
o: 'bind'
device: ${GANYMEDE_DB_PATH}
ganymede_nginx:
driver: local
driver_opts:
type: 'none'
o: 'bind'
device: ${GANYMEDE_NGINX_PATH}
ganymede_tmp:
driver: local
driver_opts:
type: 'none'
o: 'bind'
device: ${GANYMEDE_TMP_PATH}
ganymede_vods:
driver: local
driver_opts:
type: "cifs"
o: "addr=${SHARE_HOST},username=${SHARE_USER},password=${SHARE_PASS},file_mode=0777,dir_mode=0777,iocharset=utf8"
device: "//${SHARE_HOST}/ganymede_vods"
networks:
npm_shared_public:
external: true
|
I believe it did fix my issues, no stream has been stuck since that. Will update you if anything changes. Thanks for solving this problem so fast! |
I think this is caused by a network connectivity issue. To be specific: A livestream was being recorded, when the power on the router cut out, thus cutting the network and internet connection. The server remained running as it was on a separate circuit. When the network came back on the stream recording appears to have continued on a new queue item.
However, all subsequent streams, even those that started normally 24h later, are stuck. The recording appears to have completed successfully, but it does not progress to the convert or move steps. I don't see any corresponding workflows in the workflow tab.
Additionally, the chat doesn't seem to work either. I have discord alerts configured and I keep receiving alerts such as this:
Edit: I just noticed the difference in the stream IDs (the numeric ID used by twitch). The one starting with 4 is the ID for the livestream itself. The second one starting with 2 is the ID of the corresponding VOD. What's weird is that they're both labeled as LIVE streams and are both stuck. I presume that's why the actual VOD wasn't archived either. I had the same problem with another stream, which I manually deleted from the VODs section, which was then re-downloaded as a VOD. But that was only AFTER deleting the "livestream" which had the VOD ID and was stuck. It's still downloading now, so I don't know if it will succeed yet.
The text was updated successfully, but these errors were encountered: