14

I capture and process an IP camera RTSP stream in a OpenCV 3.4.2 on Raspberry Pi. Unfortunately the processing takes quite a lot of time, roughly 0.2s per frame, and the stream quickly gets delayed.

I don't mind if I skip some frames so I'm looking for a way to seek to the end of the stream before capturing and processing the next frame.

vcap = cv2.VideoCapture("rtsp://{IPcam}/12")

while(1):
    ret, frame = vcap.read()
    time.sleep(0.2)              # <= Simulate processing time
    cv2.imshow('VIDEO', frame)
    if cv2.waitKey(1) == 27:
        break
    vcap.seek_to_end()           # <== How to do this?

How can I do that vcap.seek_to_end() to catch up with the stream, discard the missed frames, and start processing the most current one?

Thanks!

1
  • I'm also curious about this, have you tried using the .grab() method? Another option is to use gstreamer source in videocapture and then get gstreamer to drop frames somehow, but not sure. Commented Aug 13, 2018 at 15:00

6 Answers 6

18

Try this:

vcap = cv2.VideoCapture("rtspsrc location=rtsp://{IPcam}/12 ! decodebin ! videoconvert ! appsink max-buffers=1 drop=true")

This uses gstreamer to grab your camera feed, and will maintain a buffer of length 1 and drop the oldest as new incoming frames are received. Then, every time you call vcap.read() you should get the latest frame.

You can also try using the OMX decoder on the Raspberry Pi if you notice CPU usage is really high, as this will decode the video (assuming it's h264) on the GPU: ! rtph264depay ! h264parse ! omxh264dec ! appsink max-buffers=1 drop=true

You may need to recompile OpenCV as by default it's compiled with FFMPEG support, not gstreamer. This is fairly simple, just pass -D WITH_GSTREAMER=ON -D WITH_FFMPEG=OFF to the cmake command. Make sure you have the gstreamer development libs installed apt install libgstreamer1.0-dev libgstreamer-plugins-base1.0-dev.

1
  • As the time I'm writing, this is the only way I was able to read the last frame from an rtsp camera(without thread). The drawback is that it works well with a limited number of cameras. As the number grows, the cpu starts to feel the work (is always on 100%) Commented Nov 24, 2021 at 9:25
8

I rewrote threading implementation from Shubham into a class, to be able to stream many cameras:

import threading
from threading import Lock
import cv2

class Camera:
    last_frame = None
    last_ready = None
    lock = Lock()

    def __init__(self, rtsp_link):
        capture = cv2.VideoCapture(rtsp_link)
        thread = threading.Thread(target=self.rtsp_cam_buffer, args=(capture,), name="rtsp_read_thread")
        thread.daemon = True
        thread.start()

    def rtsp_cam_buffer(self, capture):
        while True:
            with self.lock:
                self.last_ready, self.last_frame = capture.read()


    def getFrame(self):
        if (self.last_ready is not None) and (self.last_frame is not None):
            return self.last_frame.copy()
        else:
            return None

Then, it can be used as:

capture = Camera('rtsp://...')

while True:
    frame = capture.getFrame()
1
  • That looks handy, I will give it a try next time I'm working on that project, cheers.
    – MLu
    Commented Jul 28, 2019 at 21:17
1

I managed a hack around it by creating a read thread which puts the frame in a variable and the application uses

import threading 
from threading import Lock
import cv2


rtsp_link = "rtsp://url"
vcap = cv2.VideoCapture(rtsp_link)

latest_frame = None
last_ret = None
lo = Lock()

def rtsp_cam_buffer(vcap):
    global latest_frame, lo, last_ret
    while True:
        with lo:
            last_ret, latest_frame = vcap.read()


t1 = threading.Thread(target=rtsp_cam_buffer,args=(vcap,),name="rtsp_read_thread")
t1.daemon=True
t1.start()


while True :
    if (last_ret is not None) and (latest_frame is not None):
        img = latest_frame.copy()
    else:
        print("unable to read the frame")
        time.sleep(0.2)
        continue

Its not the best way to do, but it solves the purpose.

1
  • That’s interesting idea, I’ll give it a try. Thanks!
    – MLu
    Commented Feb 26, 2019 at 11:01
1

Adding max-buffers=1 drop=true to the pipeline like so pipeline="rtspsrc location=rtsp://camera_ip_address latency=10 ! rtph264depay ! h264parse ! avdec_h264 ! videoconvert ! appsink max-buffers=1 drop=true" works for me

1
  • I am getting this error: [ WARN:0] global /tmp/opencv/modules/videoio/src/cap_gstreamer.cpp (1757) handleMessage OpenCV | GStreamer warning: Embedded video playback halted; module omxh264dec-omxh264dec0 reported: Could not initialize supporting library. [ WARN:0] global /tmp/opencv/modules/videoio/src/cap_gstreamer.cpp (886) open OpenCV | GStreamer warning: unable to start pipeline [ WARN:0] global /tmp/opencv/modules/videoio/src/cap_gstreamer.cpp (480) isPipelinePlaying OpenCV | GStreamer warning: GStreamer: pipeline have not been created Any idea? (of course, enabled gstreamer)
    – rpi_guru
    Commented Jun 21, 2020 at 23:48
1

The threading solution does work , but too processor hungry for embedded devices like raspberry pi and opencv is primarily used in embedded applications

The solution I used for the same is..

import time
FPS = *rtsp fps value*
cap = cv2.VideoCapture("RTSP URL"); 
CALIBRATION = 1.5

def skipFrames(timegap):
   global FPS,cap
   latest = None
   while True :  
      for i in range(timegap*FPS/CALIBRATION) :
        _,latest = cap.read()
        if(not _):
           time.sleep(0.5)#refreshing time
           break
      else:
        break
   return latest

gap = 0.1
while cap.isOpened(): 
   current = skipFrames(gap)
   s = time.time()
   """
   My time hungry task here , may be some object detection stuff
   """
   gap = time.time()-s

Increase or Decrease the CALIBRATION constant according to your needs , ofcource you wont be able to mitigate the streaming protocol latency here , yet this will help in minimalizing the lag close to that of the protocol latency

I know that my answer came after 2 years from the question , but might help people who might visit this question in future

1
  • A LOWER FPS value around 10 -15 always does a better job with opencv , yet other FPS too will do good Commented Oct 10, 2020 at 3:23
0

You can use the Onvif specification and use getSnapShotUri, by example with postman use the XML request:

<?xml version="1.0" encoding="UTF-8"?>
<soap:Envelope xmlns:soap="http://www.w3.org/2003/05/soap-envelope"
xmlns:trt="http://www.onvif.org/ver10/media/wsdl">
 <soap:Body>
    <trt:GetSnapShotUri>
    <trt:ProfileToken token="MainStreamProfileToken">MainStreamProfileToken</trt:ProfileToken>
 </trt:GetSnapShotUri>
 </soap:Body>
</soap:Envelope>

then you get the response:

<?xml version="1.0" encoding="UTF-8"?>
<s:Envelope xmlns:s="http://www.w3.org/2003/05/soap-envelope" xmlns:e="http://www.w3.org/2003/05/soap-encoding" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:wsa="http://www.w3.org/2005/08/addressing" xmlns:xmime="http://www.w3.org/2005/05/xmlmime" xmlns:tns1="http://www.onvif.org/ver10/topics" xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:xop="http://www.w3.org/2004/08/xop/include" xmlns:tt="http://www.onvif.org/ver10/schema" xmlns:wsnt="http://docs.oasis-open.org/wsn/b-2" xmlns:wstop="http://docs.oasis-open.org/wsn/t-1" xmlns:tds="http://www.onvif.org/ver10/device/wsdl" xmlns:tan="http://www.onvif.org/ver20/analytics/wsdl" xmlns:tr2="http://www.onvif.org/ver20/media/wsdl" xmlns:trt="http://www.onvif.org/ver10/media/wsdl" xmlns:tev="http://www.onvif.org/ver10/events/wsdl" xmlns:tptz="http://www.onvif.org/ver20/ptz/wsdl" xmlns:timg="http://www.onvif.org/ver20/imaging/wsdl" xmlns:ter="http://www.onvif.org/ver10/error"  xmlns:tmd="http://www.onvif.org/ver10/deviceIO/wsdl" xmlns:hikwsd="http://www.onvifext.com/onvif/ext/ver10/wsdl" xmlns:hikxsd="http://www.onvifext.com/onvif/ext/ver10/schema" >
    <s:Body>
        <trt:GetSnapshotUriResponse>
            <trt:MediaUri>
                <tt:Uri>http://192.168.1.88:80/tmpfs/auto.jpg</tt:Uri>
                <tt:InvalidAfterConnect>false</tt:InvalidAfterConnect>
                <tt:InvalidAfterReboot>false</tt:InvalidAfterReboot>
                <tt:Timeout>PT5S</tt:Timeout>
            </trt:MediaUri>
        </trt:GetSnapshotUriResponse>
    </s:Body>
</s:Envelope>

then you can use your address http://192.168.1.88:80/tmpfs/auto.jpg to get the frame, the problem is that it uses the second video stream at low res but it uses less CPU

If the camera suports onvif SetVideoEncoderConfiguration to increase resolution, else this method only use default second stream resolution

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.