43

For a real-time interactive Mandelbrot viewer I was making in R I am on the lookout for a performant way to display 1920x1080 raw hex color matrices as raster images in the hope of being able to achieve ca. 5-10 fps (calculating the Mandelbrot images themselves now achieves ca. 20-30 fps at moderate zooms, and certainly scrolling around should go fast) (of course there could be many applications of having access to fast 2D graphics in R). Using image() with option useRaster=TRUE, plot.raster or even grid.raster() doesn't cut it as displaying the raster image is way slower (in the best case ca. 1/4 of a second) than actually calculating it, so I am on the lookout for a more performant option, ideally using SDL or OpenGL acceleration.

I noticed that one should be able to call SDL, SDL_image and GL/OpenGL functions from R using the rdyncall package, which should have much better performance.

Although this package is archived on CRAN, it is still fully functional. See paper here and Mercurial repository here.

To install the archived version:

library(devtools)
install_version("rdyncall",
                version="0.7.5",
                repos="http://cran.us.r-project.org")

The SDL, SDL_image and SDL_mixer DLLs (version 1.2) (on Windows) have to be installed first from https://libsdl.org/release/, https://www.libsdl.org/projects/SDL_image/release/ and https://www.libsdl.org/projects/SDL_mixer/release/)(the 64 bit DLLs are to be put underR-4.2.1/bin/x64`). On Ubuntu they can be installed using

sudo apt-get install libsdl1.2-dev libsdl-image1.2-dev libsdl-mixer1.2

Some demos of how to make SDL & OpenGL calls are available at https://dyncall.org/demos/soulsalicious/index.html (1980s computer-game style starfield, with music included).

Am I correct that with this package one should be able to display a 2D image raster using SDL & opengl acceleration? If so, has anyone any thoughts how to do this (I'm asking because I have no experience in using either SDL or OpenGL)?

To open a 1920 x 1080 SDL window I think I have to use (gathered from some OpenGL examples and windowed.R script in https://dyncall.org/demos/soulsalicious/soulsalicious.tar.gz, fullscreen also possible, see fullscreen.R)

init <- function()
{
  require(rdyncall)
  dynport(SDL)
  SDL_Init(SDL_INIT_VIDEO)
  dynport(GL)
  dynport(GLU)
  dynport(SDL_image)
  
  SDL_GL_SetAttribute(SDL_GL_RED_SIZE,8)
  SDL_GL_SetAttribute(SDL_GL_GREEN_SIZE,8)
  SDL_GL_SetAttribute(SDL_GL_BLUE_SIZE,8)
  SDL_GL_SetAttribute(SDL_GL_DOUBLEBUFFER,1)
  x_res <- 1920
  y_res <- 1080
  win <- SDL_SetVideoMode(x_res, y_res, 32, 
                          SDL_HWSURFACE + SDL_OPENGL + SDL_DOUBLEBUF)
  SDL_WM_SetCaption("SDL surface",NULL)
  glEnable(GL_TEXTURE_2D)
  # Set the projection matrix for the image
  glMatrixMode(GL_PROJECTION)
  glLoadIdentity()
  x_min=1
  x_max=x_res
  y_min=1
  y_max=y_res
  glOrtho(x_min, x_max, y_min, y_max, -1, 1)
  # Set the modelview matrix for the image
  glMatrixMode(GL_MODELVIEW)
  glLoadIdentity()
}
init()

I gather I should then set up some pixel transfer map using something like

glPixelMapfv(GL_PIXEL_MAP_I_TO_R, nb_colors, map_colors)
glPixelMapfv(GL_PIXEL_MAP_I_TO_G, nb_colors, map_colors)
glPixelMapfv(GL_PIXEL_MAP_I_TO_B, nb_colors, map_colors)

then create a buffer for the pixel data using another pixels <- glPixelMapfv call & draw the pixel data to the screen using glDrawPixels and swap the back and front buffers to display the image using SDL_GL_SwapBuffers(win) and then wait for the user to close the window & then clean up using SDL_Quit() etc. Trouble is I have no OpenGL or SDL experience, so would anybody know how to carry out these last few steps? (I am using SDL version 1.2 here)

Some timings of non-OpenGL options which are too slow for my application:

# some example data & desired colour mapping of [0-1] ranged data matrix
library(RColorBrewer)
ncol=1080
cols=colorRampPalette(RColorBrewer::brewer.pal(11, "RdYlBu"))(ncol)
colfun=colorRamp(RColorBrewer::brewer.pal(11, "RdYlBu"))
col = rgb(colfun(seq(0,1, length.out = ncol)), max = 255)
mat=matrix(seq(1:1080)/1080,nrow=1920,ncol=1080,byrow=TRUE)
mat2rast = function(mat, col) {
  idx = findInterval(mat, seq(0, 1, length.out = length(col)))
  colors = col[idx]
  rastmat = t(matrix(colors, ncol = ncol(mat), nrow = nrow(mat), byrow = TRUE))
  class(rastmat) = "raster"
  return(rastmat)
}
system.time(mat2rast(mat, col)) # 0.24s

# plot.raster method - one of the best?
par(mar=c(0, 0, 0, 0))
system.time(plot(mat2rast(mat, col), asp=NA)) # 0.26s

# grid graphics - tie with plot.raster?
library(grid)
system.time(grid.raster(mat2rast(mat, col),interpolate=FALSE)) # 0.28s

# base R image()
par(mar=c(0, 0, 0, 0))
system.time(image(mat,axes=FALSE,useRaster=TRUE,col=cols)) # 0.74s # note Y is flipped to compared to 2 options above - but not so important as I can fill matrix the way I want

# ggplot2 - just for the record...
df=expand.grid(y=1:1080,x=1:1920)
df$z=seq(1,1080)/1080
library(ggplot2)
system.time({q <- qplot(data=df,x=x,y=y,fill=z,geom="raster") + 
                scale_x_continuous(expand = c(0,0)) + 
                scale_y_continuous(expand = c(0,0)) +
                scale_fill_gradientn(colours = cols) + 
                theme_void() + theme(legend.position="none"); print(q)}) # 11s 
11
  • Try quadmesh package for conversion of raster to efficient rgl form.
    – mdsumner
    Jan 6, 2018 at 3:57
  • Well that doesn't solve my problem I reckon - I don't want to get a 3D height map of my matrix, but just a 2D colour-mapped levelplot/heatmap/raster. Jan 6, 2018 at 4:06
  • 1
    You can set z = 0. I'm not suggesting it solves your problem! I don't know the solution and am keen to find one too. Using raster or GDAL to provide level-of-detail specific to the current window is about the best I've seen, but it's hard to string the pieces together
    – mdsumner
    Jan 6, 2018 at 4:34
  • btw, I don't think you are driving magick here, there's no conversion to that format (surely plot doesn't do that implicitly??)
    – mdsumner
    Jan 6, 2018 at 4:38
  • 1
    No I had been kind of hoping for an answer here, but unfortunately I didn't get one so far... Nov 19, 2018 at 8:45

2 Answers 2

1

According to the RGL package introduction, it is :

a visualization device system for R, using OpenGL as the rendering backend. An rgl device at its core is a real-time 3D engine written in C++. It provides an interactive viewpoint navigation facility (mouse + wheel support) and an R programming interface.

As RGL is a real time 3D engine, I expect that using RGL for 2D will give you a fast display.

Please note that this is an old project so I am not sure that it fit your requirement.

You can take a look on this paper and see some result images in this gallery

6
  • I tried the RGL library, but it's only for 3D OpenGL, not 2D, which I would need here... So I'm afraid this will not work (I tried that option before)... Apr 17, 2018 at 12:26
  • That rdyncall package I think would be the way to go, but I'm afraid I would need some help to use it, as I'm not an OpenGL expert... Apr 17, 2018 at 12:27
  • Do you try to plot data in 3D on a (XY) plan/grid and fix the camera (position and direction) on the Z axe toward the XY grid ?
    – A. STEFANI
    Apr 17, 2018 at 12:38
  • 1
    Yes that's what I tried, but my data is not 3D so this is not efficient - there are very efficient OpenGL methods for pure 2D graphics which should be used here... Apr 17, 2018 at 12:54
  • 1
    @derhass Well at least the small subset of OpenGL that's used in the RGL package wasn't sufficient for what I wanted to do, but feel free to prove me wrong... Using SDL and OpenGL directly using rdyncall on the other hand should work (looking at the demo of the random field linked above), but I haven't figured out how to get it to work for my specific problem... Apr 18, 2018 at 8:05
1

Not an SDL or OpenGL based solution (which should be faster still), but the nativeRaster format implemented in https://github.com/coolbutuseless/nara seems to be ca. 10x faster to display 640x480 image rasters than grid.raster().

To display the raster just call

library(remotes)
remotes::install_github("coolbutuseless/nara")
library(nara)

# Setup a fast graphics device that can render quickly
x11(type = 'dbcairo', antialias = 'none', width = 8, height = 6)
grid.raster(nara::raster_to_nr(ras), interpolate = FALSE)

where ras is a regular grid raster.

There is a couple of nice demos including an R version of pacman, https://github.com/coolbutuseless/pacman and of the game Anotherworld, https://github.com/coolbutuseless/anotherworld.

And also a demo where I used nara to do real-time Mandelbrot fractal zooms, cf. function zoom() in https://github.com/tomwenseleers/mandelExplorer.

See video here.

enter image description here

1
  • 1
    {nara} is great, but I will try an OpenGL solution. In the past, I have been porting GLFW just for this purpose. But I am not there yet. github.com/ramiromagno/glfw. Dec 20, 2022 at 22:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.