2

I am trying to simulate a line follower with Pioneer 3AT in Webots. This is the first step towards my application involving swarm robotics. I have placed camera. However, I am not able to display image processed with OpenCV in the simulation display (in realtime). For now, I am planning to just threshold the lane and display using the display node as mentioned in https://www.cyberbotics.com/doc/reference/display.

However, I am not able to get this up and running. The question is how do I display the processed numpy image array in the webots display?

Code that I currently have:

camera = Camera("camera")
camera.enable(TIME_STEP);
display = Display("display")

while (robot.step(timestep) != -1):
    cameraData = camera.getImage();
    image = np.frombuffer(cameraData, np.uint8).reshape((camera.getHeight(), camera.getWidth(), 4))

Now, how do I display the image in display? I tried imageNew, imagePaste etc. However, I am getting plenty of C/C++ errors via SWIG and I am not able to find any relevant examples too. (The Webots simulator comes with only one example that uses display city.wbt and that uses a static image. I am not sure if that could be used. But, the summarised code of that example which is in C is as below.)

// speedometer
WbDeviceTag display;
int display_width = 0;
int display_height = 0;
WbImageRef speedometer_image = NULL;

// initialize display (speedometer)
if (enable_display) {
    display = wb_robot_get_device("display");
    speedometer_image = wb_display_image_load(display, "speedometer.png");
}

// display background
wb_display_image_paste(display, speedometer_image, 0, 0, false);

P.S.: I have spent more than 4 hours on this and I am not able to find any solution for this. There appears to be a similar question on Qt (I am not using Qt). But, that question has gone unanswered: Ploting an image with 'imshow' of opencv in webots. Any help will be greatly appreciated as I am planning to opensource my entire project.

EDIT 1: I tried the following as per the following note for Java. But, I am getting a blank display screen. No luck after 10 hours of efforts.

Note [Java]: The Display.imageNew function can display the image returned by the Camera.getImage function directly if the pixel format argument is set to ARGB.

display.imageNew(cameraData, display.ARGB, camera.getHeight(), camera.getWidth())

2 Answers 2

2

I figured out an "official way" to get this up and running. However, I do not like this solution as it is not integrated directly into the simulator (and the simulations become too slow when used with this method). For now, I am accepting this answer. However, I will accept any other answer that integrates neatly into Webots.

cv2.startWindowThread()
cv2.namedWindow("preview")

while (robot.step(timestep) != -1):
    cv2.imshow("preview", image)
    cv2.waitKey(TIME_STEP)

Source: https://cyberbotics.com/forum?message=7950

1

You will find more example of Display device in Webots that might help you (they are in C/C++ but you should be able to apply similar mechanisms for Python):

In python you shoul dbe able to display an OpenCV image into the display with something similar to:

    imageRef = display.imageNew(camera.getImage().tolist(), Display.RGB)
    display.imagePaste(imageRef, 0, 0)
6
  • Could you please post a minimal working solution? I had through that before. But, I am getting SWIG errors when I try to access display memory. It looks like the C/C++ version uses memcpy which can't be replicated directly in Python (without digging into Webots source code). Oct 9, 2019 at 8:09
  • I added an example (not tested) in the answer. Oct 9, 2019 at 9:48
  • I am getting similar errors (that I used to get before) pastebin.com/erT8uLrT Oct 9, 2019 at 12:14
  • @PrasadRaghavendra here is a tested solution: imageRef = display.imageNew(frame.tolist(), Display.RGB). @DavidMansolino please update the code Feb 4, 2020 at 23:36
  • Hi @DarkoLukić thank you for the tested solution. I assume frame = camera.getImage() right? Feb 5, 2020 at 7:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.