0

I have a android program that streams video via RTSP to a Wowza media server, my OpenCV program then gets this RTSP feed to process.

The problem is the video feed almost immediately gets corrupted and the following errors are output:

[h264 @ 0x1f05320] Invalid level prefix
[h264 @ 0x1f05320] error while decoding MB 5 5
[h264 @ 0x1dfb940] Invalid level prefix
[h264 @ 0x1dfb940] error while decoding MB 6 1
[h264 @ 0x1dfbde0] corrupted macroblock 7 5 (total_coeff=-1)
[h264 @ 0x1dfbde0] error while decoding MB 7 5
[h264 @ 0x1dfafe0] corrupted macroblock 7 6 (total_coeff=-1)
 ...

I have tried adding ?tcp to the end of the URL which removes the corrupt frames but leaves me with a large amount of lag that renders the feed useless (it's a real time application).

I have a different program that I use to test the stream and the RTSP feed works fine in that program (using OpenCV as well), it's only when I try to use the same feed in a separate program that computes ORB features that the feed gets corrupted beyond recognition. I'm getting the video like so:

  cv::Mat gray;
  CvCapture* img_scene = cvCaptureFromFile("rtsp://193.61.148.73:1935/serg/android_test"); //?tcp
  while(1) { //Create infinte loop for live streaming
    cv::Mat image = cvQueryFrame(img_scene);
    cvtColor(image, gray, CV_BGR2GRAY);

The program also streams fine in ffmeg using the tcpoption.

EDIT:

capture thread:

Mat captureThread() {
  if(captureOpen == false){
    img_scene = cvCaptureFromFile("rtsp://193.61.148.73:1935/serg/android_test?tcp");
  }
  while(1) {
    image = cvQueryFrame(img_scene);
    cvtColor(image, gray, CV_BGR2GRAY);
    return gray;
  }

}

Processing thread:

Mat processingThread(Mat gray, Mat img_object) {
  //calculate keypoint and descriptors
  //match keypoints and descriptors
  //draw good matches
  //draw homography
    return imgMatches;
}

main:

int main(int argc, char *argv[]) {
  img_object = imread( argv[1], CV_LOAD_IMAGE_GRAYSCALE );
  while(1) {
    thread t1(captureThread); 
    t1.join();
    captureOpen = true;
    thread t2(processingThread, gray, img_object);
    t2.join();
    imshow("Output", imgMatches);
    key = cvWaitKey(5);
    if (char(key) == 27) {
      break;
    }  
  }
  return 0;
}

2 Answers 2

1

try using separate threads for capturing and processing. I have also encountered this problem before and it largely went away once I have separated them into two threads. (I was using ffmpeg directly but it should be similar). You can see that this is really the case by putting a sleep() function in the opencv program you mentioned was working. You should immediately see the corruption starts to happen.

So whenever there is a delay in consuming the network feed, these kind of corruption seems to be much more frequent. Also if your cpu is too slow to process the stream, it will happen as well. If you are using ffmpeg directly, it's possible to discard those corrupted frames since they affect the results especially when doing motion detection.

Edit

Ok this is just pseudo code but generally it should look like this.

bool imgready=false;
Mat sharedmat;
mutex mtx;

Mat captureThread() {
  if(captureOpen == false){
    img_scene = cvCaptureFromFile("rtsp://193.61.148.73:1935/serg/android_test?tcp");
  }
  while(1) {

    mtx.lock();
    sharedmat = cvQueryFrame(img_scene);
    imgready=true;
    mtx.unlock();

  }

}

void processFunction(Mat im)
{
  //do whatever
}

void processingThread() {
  while(1)
  {
    if(imgready)
    {
      mtx.lock();
      Mat localmat=sharedmat.clone();
      processFunction(localmat);
      imgready=false;

      mtx.unlock();
    }else
      sleep(1000);//sleep 1 millisecond


  }
}


int main(int argc, char *argv[]) {

  thread t1(captureThread);
  thread t2(processingThread);

  t1.join();
  t2.join();

  return 0;
}
8
  • Thanks for the advice I'll try this out and let you know. (Sadly won't be until tomorrow as I've to demo my other working program to visitors today.)
    – Colin747
    Feb 3, 2015 at 9:40
  • It seemed to improve the feed, especially when using tcp however there is a large amount of lag (approx. 6 seconds) but I'm not sure if this lag is due to the video feed appearing to run in slow motion if that makes any sense? The wait time is set to 5 and it seems to run much slower than the it does in the other program using the same streaming method. Any advice?
    – Colin747
    Feb 5, 2015 at 11:45
  • I think lag caused by streaming server is pretty normal, but "appearing to run in slow motion" is not. I am not that familiar with server side streaming, I have only dealt with client side part before and have only interfaced with IP cameras. For cameras,there is almost no lag in the feed. However, when the camera feed goes through my company's other team video restreamer, it will add about 5-10 seconds lag.
    – Zaw Lin
    Feb 5, 2015 at 11:56
  • How are you doing the threading exactly? Maybe you can share some codes? I can share some samples written for linux, maybe it can give you some idea?
    – Zaw Lin
    Feb 5, 2015 at 11:57
  • I've added some additional code, let me know if you need more of the processing thread let me know as I didn't want to post lots of code.
    – Colin747
    Feb 5, 2015 at 12:35
0

It turns out the reason why the video feed was getting slowed down so badly was due to an expensive function call within findHomography:

Mat H = findHomography( obj, scene, CV_RANSAC, 3, mask );

I have changed the CV_RANSAC parameter to 0 so the line now looks like:

Mat H = findHomography( obj, scene, 0, 0, mask );

N.B. the forth parameter was also changed to 0 as it is a threshold value when using CV_RANSAC, more information can be found in the documentation here.

For anyone else finding this also pay attention to Zaw Lin's answer as it also helped clear up corruption problems in the video feed.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.