1

In order to try an embedded AI, I want to stream an image dataset through a rtsp stream. What I tried to do is to read one image every X seconds and send it to the stream and infere my AI on it. I tried to use this github repo :https://gist.github.com/takidog/2c981c34d5d5b41c0d712f8ef4ac60d3#file-main-py

This is what I tried so far :

import cv2
import time

import subprocess as sp 

import glob, os

__PATH = "./DATASET"


os.chdir(__PATH)
IMG_LIST = glob.glob("*.jpg")
IMG_LIST_LEN = len(IMG_LIST)
IMG_INDEX = 0
IMG_DELAY = 2

IMG_WIDTH = 1280
IMG_HEIGHT = 720

IMG_SIZE = str(IMG_WIDTH)+"x"+str(IMG_HEIGHT)
FPS = 5

RTSP_SERVER = "rtsp://localhost:31415/stream"

COMMAND = ['ffmpeg',
           '-re',
            '-s', IMG_SIZE,
            '-r', str(FPS),
            '-i', '-',
            '-bufsize', '64M',
            '-maxrate', "4M",
            '-rtsp_transport', 'tcp',
            '-muxdelay','0.1',
            RTSP_SERVER]

process = sp.Popen(COMMAND,stdin=sp.PIPE)

while(True):

    CURRENT_IMG = cv2.imread(IMG_LIST[IMG_INDEX])
    IMG_INDEX = (IMG_INDEX+1)%IMG_LIST_LEN
    while(CURRENT_IMG.shape[0]!=720): #We dump images with a bad format
        CURRENT_IMG = cv2.imread(IMG_LIST[IMG_INDEX])
        IMG_INDEX = (IMG_INDEX+1)%IMG_LIST_LEN

        
    _,FRAME = cv2.imencode('.png', CURRENT_IMG)

    process.stdin.write(FRAME.tobytes())

    time.sleep(1/FPS)

Surprise surprise this does not work and gives me this error :

Input #0, png_pipe, from 'pipe:':
  Duration: N/A, bitrate: N/A
    Stream #0:0: Video: png, rgb24(pc), 1280x720, 25 fps, 25 tbr, 25 tbn, 25 tbc
[NULL @ 0x55ba3fe1b860] Unable to find a suitable output format for 'rtsp://localhost:31415/stream'
rtsp://localhost:31415/stream: Invalid argument
Traceback (most recent call last):
  File "main.py", line 47, in <module>
    process.stdin.write(FRAME.tobytes())
BrokenPipeError: [Errno 32] Broken pipe
5
  • Add the '-f', 'rtsp', arguments and try again.
    – Rotem
    Commented Jul 27, 2021 at 20:11
  • Hi, sorry for the late answer : when I add this argument it gives me : [tcp @ 0x5591ee1734c0] Connection to tcp://localhost:31415?timeout=0 failed: Connection refused Could not write header for output file #0 (incorrect codec parameters ?): Connection refused Error initializing output stream 0:0 -- Conversion failed! Traceback (most recent call last): File "main.py", line 47, in <module> process.stdin.write(FRAME.tobytes()) BrokenPipeError: [Errno 32] Broken pipe Commented Aug 2, 2021 at 7:02
  • I recommend you to try UDP first. The TCP protocol requires that the TCP server (receiving size) to be up before the streaming begins. It could be the reason for the error (I am not sure).
    – Rotem
    Commented Aug 2, 2021 at 7:18
  • I tried with -rtsp_transport udp but the error is still exactly the same Commented Aug 2, 2021 at 7:56
  • I posted a reproducible code sample. In case it still not working, try updating the version of FFmpeg. My answer tries to address the FFmpeg streaming issue (not addressing your AI inference case). You can probably replace FFplay (listener) with cap = cv2.VideoCapture, but I didn't test it.
    – Rotem
    Commented Aug 2, 2021 at 21:01

1 Answer 1

5

Here is a reproducible sample - hoping you can copy paste and execute, but nothing is promised...

The example applies the following stages:

  • Create 10 synthetic JPEG images in ./test_dataset folder, to be used as input.
  • Execute FFplay sub-process as RTSP listener.
    When using TCP protocol we should start the TCP server first (FFplay is used as a TCP server in out case).
    We also need the receiver process, because without it, FFmpeg streamer process halts after the first frame.
  • Execute FFmpeg sub-process for RTSP streaming.
    Cyclically read JPEG image to NumPy array (in BGR color format), and write the array as raw video frame to stdin pipe.
    Note: It is more efficient to write raw video frames, than encoding each frame to PNG (as used by your reference sample).

Here is the code:

import cv2
#import time
import subprocess as sp 
import glob
import os

img_width = 1280
img_height = 720


test_path = './test_dataset'  # Folder with synthetic sample images.

os.makedirs(test_path, exist_ok=True)  # Create folder for input images.

os.chdir(test_path)

ffmpeg_cmd = 'ffmpeg'  # May use full path like: 'c:\\FFmpeg\\bin\\ffmpeg.exe'
ffplay_cmd = 'ffplay'  # May use full path like: 'c:\\FFmpeg\\bin\\ffplay.exe'


# Create 10 synthetic JPEG images for testing (image0001.jpg, image0002.jpg, ..., image0010.jpg).
sp.run([ffmpeg_cmd, '-y', '-f', 'lavfi', '-i', f'testsrc=size={img_width}x{img_height}:rate=1:duration=10', 'image%04d.jpg'])


img_list = glob.glob("*.jpg")
img_list_len = len(img_list)
img_index = 0

fps = 5

rtsp_server = 'rtsp://localhost:31415/live.stream'

# You will need to start the server up first, before the sending client (when using TCP). See: https://trac.ffmpeg.org/wiki/StreamingGuide#Pointtopointstreaming
ffplay_process = sp.Popen([ffplay_cmd, '-rtsp_flags', 'listen', rtsp_server])  # Use FFplay sub-process for receiving the RTSP video.


command = [ffmpeg_cmd,
           '-re',
           '-f', 'rawvideo',  # Apply raw video as input - it's more efficient than encoding each frame to PNG
           '-s', f'{img_width}x{img_height}',
           '-pixel_format', 'bgr24',
           '-r', f'{fps}',
           '-i', '-',
           '-pix_fmt', 'yuv420p',
           '-c:v', 'libx264',
           '-bufsize', '64M',
           '-maxrate', '4M',
           '-rtsp_transport', 'tcp',
           '-f', 'rtsp',
           #'-muxdelay', '0.1',
           rtsp_server]

process = sp.Popen(command, stdin=sp.PIPE)  # Execute FFmpeg sub-process for RTSP streaming


while True:
    current_img = cv2.imread(img_list[img_index])  # Read a JPEG image to NumPy array (in BGR color format) - assume the resolution is correct.
    img_index = (img_index+1) % img_list_len  # Cyclically repeat images

    process.stdin.write(current_img.tobytes())  # Write raw frame to stdin pipe.

    cv2.imshow('current_img', current_img)  # Show image for testing

    # time.sleep(1/FPS)
    key = cv2.waitKey(int(round(1000/fps)))  # We need to call cv2.waitKey after cv2.imshow

    if key == 27:  # Press Esc for exit
        break


process.stdin.close()  # Close stdin pipe
process.wait()  # Wait for FFmpeg sub-process to finish
ffplay_process.kill()  # Forcefully close FFplay sub-process
cv2.destroyAllWindows()  # Close OpenCV window

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.