For a real-time interactive Mandelbrot viewer I was making in R I am on the lookout for a performant way to display 1920x1080 raw hex color matrices as raster images in the hope of being able to achieve ca. 5-10 fps (calculating the Mandelbrot images themselves now achieves ca. 20-30 fps at moderate zooms, and certainly scrolling around should go fast) (of course there could be many applications of having access to fast 2D graphics in R). Using image()
with option useRaster=TRUE
, plot.raster
or even grid.raster()
doesn't cut it as displaying the raster image is way slower (in the best case ca. 1/4 of a second) than actually calculating it, so I am on the lookout for a more performant option, ideally using SDL or OpenGL acceleration.
I noticed that one should be able to call SDL, SDL_image and GL/OpenGL functions from R using the rdyncall
package, which should have much better performance.
Although this package is archived on CRAN, it is still fully functional. See paper here and Mercurial repository here.
To install the archived version:
library(devtools)
install_version("rdyncall",
version="0.7.5",
repos="http://cran.us.r-project.org")
The SDL
, SDL_image
and SDL_mixer
DLLs (version 1.2) (on Windows) have to be installed first from https://libsdl.org/release/, https://www.libsdl.org/projects/SDL_image/release/ and https://www.libsdl.org/projects/SDL_mixer/release/)(the 64 bit DLLs are to be put under
R-4.2.1/bin/x64`).
On Ubuntu they can be installed using
sudo apt-get install libsdl1.2-dev libsdl-image1.2-dev libsdl-mixer1.2
Some demos of how to make SDL & OpenGL calls are available at https://dyncall.org/demos/soulsalicious/index.html (1980s computer-game style starfield, with music included).
Am I correct that with this package one should be able to display a 2D image raster using SDL
& opengl
acceleration? If so, has anyone any thoughts how to do this (I'm asking because I have no experience in using either SDL or OpenGL)?
To open a 1920 x 1080 SDL window I think I have to use
(gathered from some OpenGL examples and windowed.R
script in https://dyncall.org/demos/soulsalicious/soulsalicious.tar.gz, fullscreen also possible, see fullscreen.R
)
init <- function()
{
require(rdyncall)
dynport(SDL)
SDL_Init(SDL_INIT_VIDEO)
dynport(GL)
dynport(GLU)
dynport(SDL_image)
SDL_GL_SetAttribute(SDL_GL_RED_SIZE,8)
SDL_GL_SetAttribute(SDL_GL_GREEN_SIZE,8)
SDL_GL_SetAttribute(SDL_GL_BLUE_SIZE,8)
SDL_GL_SetAttribute(SDL_GL_DOUBLEBUFFER,1)
x_res <- 1920
y_res <- 1080
win <- SDL_SetVideoMode(x_res, y_res, 32,
SDL_HWSURFACE + SDL_OPENGL + SDL_DOUBLEBUF)
SDL_WM_SetCaption("SDL surface",NULL)
glEnable(GL_TEXTURE_2D)
# Set the projection matrix for the image
glMatrixMode(GL_PROJECTION)
glLoadIdentity()
x_min=1
x_max=x_res
y_min=1
y_max=y_res
glOrtho(x_min, x_max, y_min, y_max, -1, 1)
# Set the modelview matrix for the image
glMatrixMode(GL_MODELVIEW)
glLoadIdentity()
}
init()
I gather I should then set up some pixel transfer map using something like
glPixelMapfv(GL_PIXEL_MAP_I_TO_R, nb_colors, map_colors)
glPixelMapfv(GL_PIXEL_MAP_I_TO_G, nb_colors, map_colors)
glPixelMapfv(GL_PIXEL_MAP_I_TO_B, nb_colors, map_colors)
then create a buffer for the pixel data using another pixels <- glPixelMapfv
call & draw the pixel data to the screen using glDrawPixels
and swap the back and front buffers to display the image
using SDL_GL_SwapBuffers(win)
and then wait for the user to close the window & then clean up using SDL_Quit()
etc. Trouble is I have no OpenGL or SDL experience, so would anybody know how to carry out these last few steps? (I am using SDL version 1.2 here)
Some timings of non-OpenGL options which are too slow for my application:
# some example data & desired colour mapping of [0-1] ranged data matrix
library(RColorBrewer)
ncol=1080
cols=colorRampPalette(RColorBrewer::brewer.pal(11, "RdYlBu"))(ncol)
colfun=colorRamp(RColorBrewer::brewer.pal(11, "RdYlBu"))
col = rgb(colfun(seq(0,1, length.out = ncol)), max = 255)
mat=matrix(seq(1:1080)/1080,nrow=1920,ncol=1080,byrow=TRUE)
mat2rast = function(mat, col) {
idx = findInterval(mat, seq(0, 1, length.out = length(col)))
colors = col[idx]
rastmat = t(matrix(colors, ncol = ncol(mat), nrow = nrow(mat), byrow = TRUE))
class(rastmat) = "raster"
return(rastmat)
}
system.time(mat2rast(mat, col)) # 0.24s
# plot.raster method - one of the best?
par(mar=c(0, 0, 0, 0))
system.time(plot(mat2rast(mat, col), asp=NA)) # 0.26s
# grid graphics - tie with plot.raster?
library(grid)
system.time(grid.raster(mat2rast(mat, col),interpolate=FALSE)) # 0.28s
# base R image()
par(mar=c(0, 0, 0, 0))
system.time(image(mat,axes=FALSE,useRaster=TRUE,col=cols)) # 0.74s # note Y is flipped to compared to 2 options above - but not so important as I can fill matrix the way I want
# ggplot2 - just for the record...
df=expand.grid(y=1:1080,x=1:1920)
df$z=seq(1,1080)/1080
library(ggplot2)
system.time({q <- qplot(data=df,x=x,y=y,fill=z,geom="raster") +
scale_x_continuous(expand = c(0,0)) +
scale_y_continuous(expand = c(0,0)) +
scale_fill_gradientn(colours = cols) +
theme_void() + theme(legend.position="none"); print(q)}) # 11s