4

I am able to undistort RGB image successfully.

Now, I am working on directly undistort I420 data, instead of first converting it to RGB.

Below are the steps I followed after camera calibration.

K = cv::Matx33d(541.2152931632737, 0.0, 661.7479652584254,
                0.0, 541.0606969363056, 317.4524205037745,
                0.0,               0.0,               1.0);
D = cv::Vec4d(-0.042166406281296365, -0.001223961942208027, -0.0017036710622692108, 0.00023929900459453295);
newSize = cv::Size(3400, 1940);
cv::Matx33d new_K;
cv::fisheye::estimateNewCameraMatrixForUndistortRectify(K, D, cv::Size(W, H), cv::Mat::eye(3, 3, CV_64F), new_K, 1, newSize);    // W,H are the distorted image size
cv::fisheye::initUndistortRectifyMap(K, D, cv::Mat::eye(3, 3, CV_64F), new_K, newSize, CV_16SC2, mapx, mapy);

cv::remap(src, dst, mapx, mapy, cv::INTER_LINEAR);

Above code is giving me undistorted image successfully.

Now I want to undistort I420 data. So, now my src will be an I420/YV12 data. How can I undistort an I420 data, without converting it first to RGB?

By the way I420 is an image format with only 1 channel(unlike 3 channels in RGB). It has height = 1.5*image height. Its width is equal to image width.

Below code is to convert I420 to BGR

cvtColor(src, BGR, CV_YUV2BGR_I420, 3);

BGR - pixel arrangement BGR I420 - pixel arrangement I

2 Answers 2

6

The most efficient solution is resizing mapx and mapy and applying shrunk maps on down-sampled U and V channels:

  • Shrink mapx and mapy by a factor of x2 in each axis - create smaller maps matrices.
  • Divide all elements of shrank maps by 2 (applies mapping lower resolution image).
  • Apply mapx and mapy on Y color channel.
  • Apply shrunk_mapx and shrunk_mapy on down-sampled U and V color channels.

Here is a Python OpenCV sample code (please read the comments):

import cv2 as cv
import numpy as np

# For the example, read Y, U and V as separate images.
srcY = cv.imread('DistortedChessBoardY.png', cv.IMREAD_GRAYSCALE) #  Y color channel (1280x720)
srcU = cv.imread('DistortedChessBoardU.png', cv.IMREAD_GRAYSCALE) #  U color channel (640x360)
srcV = cv.imread('DistortedChessBoardV.png', cv.IMREAD_GRAYSCALE) #  V color channel (640x360)

H, W = srcY.shape[0], srcY.shape[1]

K = np.array([[541.2152931632737, 0.0, 661.7479652584254],      
              [0.0, 541.0606969363056, 317.4524205037745],
              [0.0,               0.0,               1.0]])

D = np.array([-0.042166406281296365, -0.001223961942208027, -0.0017036710622692108, 0.00023929900459453295])

# newSize = cv::Size(3400, 1940);
newSize = (850, 480)

# cv::Matx33d new_K;
new_K = np.eye(3)

# cv::fisheye::estimateNewCameraMatrixForUndistortRectify(K, D, cv::Size(W, H), cv::Mat::eye(3, 3, CV_64F), new_K, 1, newSize);    // W,H are the distorted image size
new_K = cv.fisheye.estimateNewCameraMatrixForUndistortRectify(K, D, (W, H), np.eye(3), new_K, 1, newSize)

# cv::fisheye::initUndistortRectifyMap(K, D, cv::Mat::eye(3, 3, CV_64F), new_K, newSize, CV_16SC2, mapx, mapy);
mapx, mapy = cv.fisheye.initUndistortRectifyMap(K, D, np.eye(3), new_K, newSize, cv.CV_16SC2);

# cv::remap(src, dst, mapx, mapy, cv::INTER_LINEAR);
dstY = cv.remap(srcY, mapx, mapy, cv.INTER_LINEAR)

# Resize mapx and mapy by a factor of x2 in each axis, and divide each element in the map by 2
shrank_mapSize = (mapx.shape[1]//2, mapx.shape[0]//2)
shrunk_mapx = cv.resize(mapx, shrank_mapSize, interpolation = cv.INTER_LINEAR) // 2
shrunk_mapy = cv.resize(mapy, shrank_mapSize, interpolation = cv.INTER_LINEAR) // 2

# Remap U and V using shunk maps
dstU = cv.remap(srcU, shrunk_mapx, shrunk_mapy, cv.INTER_LINEAR, borderValue=128)
dstV = cv.remap(srcV, shrunk_mapx, shrunk_mapy, cv.INTER_LINEAR, borderValue=128)

cv.imshow('dstY', dstY)
cv.imshow('dstU', dstU)
cv.imshow('dstV', dstV)

cv.waitKey(0)
cv.destroyAllWindows()

Result:

Y:
dstY

U:
dstU

V:
dstV

After converting to RGB:
RGB


C++ implementation considerations:

Since I420 format arranges Y, U and V as 3 continuous planes in memory, it's simple to set a pointer to each "plane", and treat it as a Grayscale image.
Same data ordering applies the output image - set 3 pointer to output "planes".

Illustration (assuming even width and height, and assume byte stride equals width):

srcY -> YYYYYYYY           dstY -> YYYYYYYYYYYY
        YYYYYYYY                   YYYYYYYYYYYY
        YYYYYYYY                   YYYYYYYYYYYY
        YYYYYYYY                   YYYYYYYYYYYY
        YYYYYYYY   remap           YYYYYYYYYYYY
        YYYYYYYY  ======>          YYYYYYYYYYYY
srcU -> UUUU                       YYYYYYYYYYYY
        UUUU               dstU -> YYYYYYYYYYYY
        UUUU                       UUUUUU
srcV -> VVVV                       UUUUUU
        VVVV                       UUUUUU
        VVVV                       UUUUUU
                           dstV -> VVVVVV
                                   VVVVVV
                                   VVVVVV
                                   VVVVVV

Implementing above illustration is C++

Under the assumption that width and height are even, and byte stride equals width, you can use the following C++ example for converting I420 to Y, U and V planes:

Assume: srcI420 is Wx(H*3/2) matrix in I420 format, like cv::Mat srcI420(cv::Size(W, H * 3 / 2), CV_8UC1);.

int W = 1280, H = 720;  //Assume resolution of Y plane is 1280x720

//Pointer to Y plane
unsigned char *pY = (unsigned char*)srcI420.data;

//Y plane as cv::Mat, resolution of srcY is 1280x720
cv::Mat srcY = cv::Mat(cv::Size(W, H), CV_8UC1, (void*)pY);

//U plane as cv::Mat, resolution of srcU is 640x360 (in memory buffer, U plane is placed after Y).
cv::Mat srcU = cv::Mat(cv::Size(W/2, H/2), CV_8UC1, (void*)(pY + W*H));

//V plane as cv::Mat, resolution of srcV is 640x360 (in memory buffer, V plane is placed after U).
cv::Mat srcV = cv::Mat(cv::Size(W / 2, H / 2), CV_8UC1, (void*)(pY + W*H + (W/2*H/2)));

//Display srcY, srcU, srcV for testing
cv::imshow("srcY", srcY);
cv::imshow("srcU", srcU);
cv::imshow("srcV", srcV);
cv::waitKey(0);

Above example uses pointer manipulations, without the need for copying the data.
You can use the same pointer manipulations for your destination I420 image.

Note: The solution is going to work in most cases, but not guaranteed to work in all cases.

4
  • Thanks for this wonderful answer. Regarding C++ implementation, it looks like 5 planes. 1 big and 4 small planes. Its easy to point header for the big one Mat(H, W, CV_8UC1, i420.data). How can a header be pointed for the small ones. Problem is they are horizontally concatenated.
    – Jai
    Jan 24, 2020 at 8:51
  • Take my word on it... there are 3 planes: Y plane 1280x720, then U plane 640x360 then V plane 640x360. When displaying U and V as if their width were 1280, you are getting something that looks like 5 planes.
    – Rotem
    Jan 24, 2020 at 9:13
  • I added a C++ code sample, for demonstrating how to implement the separation of I420 to Y, U, V planes.
    – Rotem
    Jan 24, 2020 at 11:12
  • I understood what you tried to say about existence of 3 planes not 5 planes. The alternate rows of U plane are separated and transformed into two U images. Same goes with V plane. Thanks for your well crafted response.
    – Jai
    Jan 24, 2020 at 11:31
0

EDIT: Components are not interleaved in the YV12 format, so the following will not work:

If the YV12 data is a one channel image, the interpolation of the remap operation is applied to the value represented by all three YUV data instead of individual Y, U and V components.

Therefore, roughly speaking, instead of doing a

c.YYYYYYYY, c.UU, c.VV

it will perform a

c.YYYYYYYYUUVV

during a linear interpolation.

You can perform a YV12 -> BGR color conversion after remap, but the colors of the interpolated pixels would be wrong.

Instead of doing a linear interpolation, try using a nearest-neighbor interpolation in remap. Then you should be able to get correct colors after YV12 -> BGR color conversion.

So, find mapx, mapy, then remap using INTER_NEAREST, and finally perform a YV12 -> BGR color conversion.

5
  • As far as I know in I420 data. First H number of rows are only Y data. and next H/2 number of rows are UV data.
    – Jai
    Jan 23, 2020 at 11:51
  • @Jai In that case, the suggested approach will not work. I was not aware of this data arrangement.
    – dhanushka
    Jan 23, 2020 at 12:00
  • Now I depicted the pixel arrangements in my post.
    – Jai
    Jan 23, 2020 at 13:23
  • If you are not interested in chroma, you can undistort the luminance channel just by creating a matrix header with that data. Does your camera support packed pixel formats? YUV - VideoLAN Wiki
    – dhanushka
    Jan 23, 2020 at 14:19
  • I understood. But I need the chroma also. I have to work with I420. Because later on, the I420 data will be fed to h264 video encoder,
    – Jai
    Jan 23, 2020 at 15:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.