1

I have a very curious problem with my Leopard Imaging M021 camera. The company doesn't really support linux, so the camera will only output raw data. I need it to run on a Beagleboard, so that's why I'm putting effort into getting it to work.

An employee of theirs has told me that it is in YUY2 format (usually 16 bits per pixel), but the high 4 bits are always 0, and the lower 12 bits contain information like so: YUYV description

Using the command:

fswebcam --device /dev/video0 --resolution 1280x720 --dumpframe test.raw

I get a file 1,843,200 bytes long, meaning a 1280x720 image with 2 bytes per pixel (16 bits per pixel).

HOWEVER, the only way I've been able to actually get the image to display correctly is if I use IrfanView's RAW image display and set it to 12 bits per pixel, not normalized, with a bayer pattern of GR, and a vertical flip. I have no idea why the vertical flip is required, because the other settings will display a skewed, weird image but not flipped. Then I use 12 BPP and it is flipped.

IrfanView Params

I suppose since the high 4 bits are always 0, that causes it to in actuality be 12 bits per pixel rather than 16?

I need to know what is really going on with the bytes in the file in order to write the conversion algorithm myself (unless anyone knows of an open source program that does the same thing as IrfanView).

Using Python, I've made a really quick script to just pull the Y component out and view it (expecting a greyscale version of the image), but I get a very distorted version. Is there something wrong with my code? Am I pulling out the data in the wrong order? What the heck does "not normalized" mean in IrfanView? Why is the GR Bayer pattern setting required to see the image in RGB?

with open('test.raw', 'r+b') as f:
    Y0 = []
    U = []
    Y1 = []
    V = []
    vals = [Y0, U, Y1, V]
    val = f.read(1)
    pixel = int.from_bytes(val, byteorder='big')
    i = 0
    vals[i].append(pixel)
    while val:
        i += 1
        val = f.read(1)
        if val != "":
            pixel = int.from_bytes(val, byteorder='big')
            vals[i % 4].append(pixel)

k = 0
with open("1.test", "w") as f:
    for i in range(720):
        for j in range(640):
            f.write(str(Y0[k]))
            f.write(" ")
            f.write(str(Y1[k]))
            f.write(" ")
            k += 1
        f.write("\n")

Resulting junk image: enter image description here

I would appreciate anyone's help or advice on this.

EDIT:

Further possibly helpful evidence. If I run this in matlab, just treating every 2 bytes as the exact pixel value:

fid = fopen('test.raw', 'r');
[I, count] = fread(fid , [1280, 720], 'uint16');
imagesc(I')
colormap(gray);

I get this image: sort of decent looking picture but no color

I'm still missing color information, because I just disregarded it. And it's still skewed a little bit. But it looks better. If you zoom in, the pattern of the image skew is good pixel, black pixel, good pixel, black pixel, etc. Does someone more knowledgeable of cameras and colors know what that is indicative of?

EDIT 2:

With the expert help of Mark Ransom, I wrote up a nice OpenCV script to read in the data, utilize CV_BayerGR2RGB to convert to RGB, and view the image. It works!

#include <vector>
#include <iostream>
#include <stdio.h>
#include <opencv2/opencv.hpp>
#include <opencv2/highgui/highgui.hpp>

int main() {
    // Each pixel is made up of 16 bits, with the high 4 bits always equal to 0
    unsigned char bytes[2];

    // Hold the data in a vector
    std::vector<unsigned short int> data;

    // Read the camera data
    FILE *fp = fopen("test.raw","rb");
    while(fread(bytes, 2, 1, fp) != 0) {
        // The data comes in little-endian, so shift the second byte right and concatenate the first byte
        data.push_back(bytes[0] | (bytes[1] << 8));
    }

    // Make a matrix 1280x720 with 16 bits of unsigned integers
    cv::Mat imBayer = cv::Mat(720, 1280, CV_16U);

    // Make a matrix to hold RGB data
    cv::Mat imRGB;

    // Copy the data in the vector into a nice matrix
    memmove(imBayer.data, data.data(), data.size()*2);

    // Convert the GR Bayer pattern into RGB, putting it into the RGB matrix!
    cv::cvtColor(imBayer, imRGB, CV_BayerGR2RGB);

    cv::namedWindow("Display window", cv::WINDOW_AUTOSIZE);
    // *15 because the image is dark
    cv::imshow("Display window", 15*imRGB);

    cv::waitKey(0);

    return 0;
}
7
  • 1
    It seems like the struct module would help.. docs.python.org/2/library/struct.html
    – tom10
    Sep 12, 2014 at 1:27
  • Check out the dcraw site; it has lots of good info about the arcane mysteries of RAW image decoding. And it's quite likely that you already have the dcraw program installed on your Linux system.
    – PM 2Ring
    Sep 12, 2014 at 1:37
  • @PM2Ring this isn't really RAW from the sensor, it's only raw in the sense that it's not encoded in a standard file format. Sep 12, 2014 at 1:44
  • @tom10, thanks for the tip, I didn't know about struct.
    – NSchrading
    Sep 12, 2014 at 2:32
  • @PM2Ring on second thought, I think it really is RAW and the advice about being YUY2 is incorrect. Sep 12, 2014 at 2:34

1 Answer 1

4

The evidence that you have, first that each byte has the upper 4 bits as 0 and the fact that you can see a proper color picture from Irfanview, tells me that the employee was incorrect and the format isn't actually YUY2 - it's GRGB.

This code should pull out the green pixels and double them up so you can get an idea of what the image looks like. Note that the even and odd lines will alternate between starting with G and (R or B).

with open('test.raw', 'r+b') as f:
    raw_g = []
    for y in range(720):
        for x in range(1280//4):
            if y % 2:
                r1, r0, g1, g0, b1, b0, g3, g2 = f.read(8)
            else:
                g1, g0, r1, r0, g3, g2, b1, b0 = f.read(8)
            g0 = g0 << 8 | g1
            g2 = g2 << 8 | g3
            raw_g.extend([g0, g0, g2, g2])

You'll probably want to right-shift these values >> 4 before you write them out to a gray-scale file.

Edit: The code had one remaining bug (an extra read), and I didn't realize the values were little-endian instead of big-endian. Those changes are already applied to the code sample above.

Since the values seemed a little dark, I also decided to apply a gamma correction - instead of n >> 4 I used int(255.999 * (n / 4095.0)**(1/2.2))

green channel with gamma correction

I'm afraid de-Bayering is a little beyond what I can pull off in a short StackOverflow answer.

6
  • In order to get your code to run, I had to make a few changes: for x in range(int(1280/4)): (convert float to int) if y % 2: (can't modulo by 0) And I guess it didn't work :( Resulting image: imgur.com/Qm4ZWVW
    – NSchrading
    Sep 12, 2014 at 3:06
  • @NSchrading sorry about being so sloppy, I've made changes to fix the errors. And I can't evaluate the output image without seeing it at full scale, I'm sure it's something very simple remaining. Sep 12, 2014 at 3:46
  • Imgur always compresses the image I think. If you want to play with the raw file, I've put it on my google drive and shared it. docs.google.com/file/d/0B-xKNHgy4GauZ2NkWkZreHVDa1E/edit Check out my new edit to the main post as well. I don't know if it gives you any helpful info, but it might. I appreciate you taking the time to help. I'm heading to bed and will work on this more tomorrow.
    – NSchrading
    Sep 12, 2014 at 4:07
  • @NSchrading thanks for providing the actual raw file, it helped with the final pieces of the puzzle. Sep 12, 2014 at 4:43
  • @MarkRansom: Nice work, Mark! I was just about to suggest that the earlier "solarized" image looked like it had Endian issues, but then I noticed you'd improved things rather drastically. :)
    – PM 2Ring
    Sep 12, 2014 at 5:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.