Frame Loss in video stream

[constantinfite]

🐛 Bug

I connect 3 cameras IP to my computer.
I run the command : python detect.py --weights ./weights/best.pt --conf 0.3 --iou 0.5 --source 'streams.txt' --save-txt --view-img.

stream.txt :
rtsp://192.168.1.20:554/1/h264major
rtsp://192.168.1.19:554/1/h264major
rtsp://192.168.1.18:554/1/h264major

But after some time it exits with this error :

File "C:\Users\const\Documents\PFE\Code\yolov5\utils\datasets.py", line 795, in letterbox
shape = img.shape[:2] # current shape [height, width]
AttributeError: 'NoneType' object has no attribute 'shape'
[h264 @ 000001c6faedcd00] error while decoding MB 68 0, bytestream -11

To Reproduce (REQUIRED)

It cannot be reproduced unless you have multiple camera IP

I think it links with a drop of fps.
My question is how can I fix this bug in the code and where can I modify to have an impact ?

Thanks

[glenn-jocher]

@constantinfite it appears cv2 retrieve() method fails to pull an image on one of the streams (network problems?), and then passes None instead of an image, which breaks the code in L795 there.

The solution would be to check for failed streams and replace them with empty images at the point of origin, either all black or all (114, 114, 114). Can you start a PR for this? The relevant region of the code is here in the streamloader:

yolov5/utils/datasets.py

Lines 295 to 306 in 17ac94b

	def update(self, index, cap):
	# Read next stream frame in a daemon thread
	n = 0
	while cap.isOpened():
	n += 1
	# _, self.imgs[index] = cap.read()
	cap.grab()
	if n == 4: # read every 4th frame
	_, self.imgs[index] = cap.retrieve()
	n = 0
	time.sleep(0.01) # wait time

[glenn-jocher]

@constantinfite can you try these two fixes and submit a PR with the one that works best (assuming one works)?

Solution 1: Black image on failed retrieval

 def update(self, index, cap): 
     # Read next stream frame in a daemon thread 
     n = 0 
     while cap.isOpened(): 
         n += 1 
         # _, self.imgs[index] = cap.read() 
         cap.grab() 
         if n == 4:  # read every 4th frame 
             success, im = cap.retrieve()
             self.imgs[index] = im if success else self.imgs[index] * 0
             n = 0 
         time.sleep(0.01)  # wait time 

Solution 2: Repeats last good image on failed retrieval

 def update(self, index, cap): 
     # Read next stream frame in a daemon thread 
     n = 0 
     while cap.isOpened(): 
         n += 1 
         # _, self.imgs[index] = cap.read() 
         cap.grab() 
         if n == 4:  # read every 4th frame 
             success, im = cap.retrieve()
             if success:
                 self.imgs[index] = im
             n = 0 
         time.sleep(0.01)  # wait time 

[constantinfite]

Ok thanks ! I will try on monday I don't have the camera during week end

[constantinfite]

@glenn-jocher thanks the first solution is working, I cannot do a PR because I have a older version of Yolov5.

[glenn-jocher]

@constantinfite ok great, I will implement the first solution in a PR then.

[glenn-jocher]

@constantinfite PR #2222 with first solution is merged. Thank you for your contributions and let us know if you run into any other issues.

[Okh2891996]

I think you can use multithreading to solve the problem