7

I have a small augmented reality app that I'm developing and would like to know how to save a screenshot of what the user sees with a tap of a button or a timer.

The app works by overlaying live camera feed above another UIView. I can save screenshots by using power button +home button, these are saved to camera roll. However, Apple will not render the AVCaptureVideoPreviewLayer, even if I ask the window to save itself. It will create a transparent piece of canvas where the preview layer is.

What's the proper way for an augmented reality app to save screenshots, including transparency and subviews?

//displaying a live preview on one of the views
    -(void)startCapture
{
        captureSession = [[AVCaptureSession alloc] init];
        AVCaptureDevice *audioCaptureDevice = nil;

        //           AVCaptureDevice *audioCaptureDevice = [AVCaptureDevice defaultDeviceWithMediaType:AVMediaTypeVideo];

        NSArray *videoDevices = [AVCaptureDevice devicesWithMediaType:AVMediaTypeVideo];

        for (AVCaptureDevice *device in videoDevices) {

            if(useFrontCamera){
                if (device.position == AVCaptureDevicePositionFront) {
                    //FRONT-FACING CAMERA EXISTS
                    audioCaptureDevice = device;
                    break;
                }
            }else{
                if (device.position == AVCaptureDevicePositionBack) {
                    //Rear-FACING CAMERA EXISTS
                    audioCaptureDevice = device;
                    break;
                }
            }

        }


        NSError *error = nil;
        AVCaptureDeviceInput *audioInput = [AVCaptureDeviceInput deviceInputWithDevice:audioCaptureDevice error:&error];
        if (audioInput) {
            [captureSession addInput:audioInput];
        }
        else {
            // Handle the failure.
        }


if([captureSession canAddOutput:captureOutput]){
 captureOutput = [[AVCaptureVideoDataOutput alloc] init];
    [captureOutput setAlwaysDiscardsLateVideoFrames:YES];
    [captureOutput setSampleBufferDelegate:self queue:queue];
    [captureOutput setVideoSettings:videoSettings];

    dispatch_release(queue);
}else{
//handle failure
}

        previewLayer = [AVCaptureVideoPreviewLayer layerWithSession:captureSession];
        UIView *aView = arOverlayView;
        previewLayer.frame =CGRectMake(0,0, arOverlayView.frame.size.width,arOverlayView.frame.size.height); // Assume you want the preview layer to fill the view.

        [aView.layer addSublayer:previewLayer];
        [captureSession startRunning];


}

//ask the entire window to draw itself in a graphics context. This call will not render

//the AVCaptureVideoPreviewLayer . It has to be replaced with a UIImageView or GL based view. //see following code for creating a dynamically updating UIImageView -(void)saveScreenshot {

   UIGraphicsBeginImageContext(appDelegate.window.bounds.size);

    [appDelegate.window.layer renderInContext:UIGraphicsGetCurrentContext()];

    UIImage *screenshot = UIGraphicsGetImageFromCurrentImageContext();
    UIGraphicsEndImageContext();

    UIImageWriteToSavedPhotosAlbum(screenshot, self, 
                                   @selector(image:didFinishSavingWithError:contextInfo:), nil);


}


//image saved to camera roll callback
- (void)image:(UIImage *)image didFinishSavingWithError:(NSError *)error 
  contextInfo:(void *)contextInfo
{
    // Was there an error?
    if (error != NULL)
    {
        // Show error message...
         NSLog(@"save failed");

    }
    else  // No errors
    {
         NSLog(@"save successful");
        // Show message image successfully saved
    }
}

Here's the code for creating the image:

//you need to add your view controller as a delegate to the camera output to be notified of buffereed data

-(void)activateCameraFeed
{
//this is the code responsible for capturing feed for still image processing
 dispatch_queue_t queue = dispatch_queue_create("com.AugmentedRealityGlamour.ImageCaptureQueue", NULL);

    captureOutput = [[AVCaptureVideoDataOutput alloc] init];
    [captureOutput setAlwaysDiscardsLateVideoFrames:YES];
    [captureOutput setSampleBufferDelegate:self queue:queue];
    [captureOutput setVideoSettings:videoSettings];

    dispatch_release(queue);

//......configure audio feed, add inputs and outputs

}

//buffer delegate callback
- (void)captureOutput:(AVCaptureOutput *)captureOutput didOutputSampleBuffer:(CMSampleBufferRef)sampleBuffer fromConnection:(AVCaptureConnection *)connection
{
    if ( ignoreImageStream )
        return;
       [self performImageCaptureFrom:sampleBuffer];

} 

Create a UIImage:

- (void) performImageCaptureFrom:(CMSampleBufferRef)sampleBuffer
{
    CVImageBufferRef imageBuffer;

    if ( CMSampleBufferGetNumSamples(sampleBuffer) != 1 )
        return;
    if ( !CMSampleBufferIsValid(sampleBuffer) )
        return;
    if ( !CMSampleBufferDataIsReady(sampleBuffer) )
        return;

    imageBuffer = CMSampleBufferGetImageBuffer(sampleBuffer); 

    if ( CVPixelBufferGetPixelFormatType(imageBuffer) != kCVPixelFormatType_32BGRA )
        return;

    CVPixelBufferLockBaseAddress(imageBuffer,0); 

    uint8_t *baseAddress = (uint8_t *)CVPixelBufferGetBaseAddress(imageBuffer);
    size_t bytesPerRow = CVPixelBufferGetBytesPerRow(imageBuffer);
    size_t width = CVPixelBufferGetWidth(imageBuffer);
    size_t height = CVPixelBufferGetHeight(imageBuffer); 

    CGImageRef newImage = nil;

    if ( cameraDeviceSetting == CameraDeviceSetting640x480 )
    {
        CGColorSpaceRef colorSpace = CGColorSpaceCreateDeviceRGB();
        CGContextRef newContext = CGBitmapContextCreate(baseAddress, width, height, 8, bytesPerRow, colorSpace, kCGBitmapByteOrder32Little | kCGImageAlphaPremultipliedFirst);
        newImage = CGBitmapContextCreateImage(newContext);
        CGColorSpaceRelease( colorSpace );
        CGContextRelease(newContext);
    }
    else
    {
        uint8_t *tempAddress = malloc( 640 * 4 * 480 );
        memcpy( tempAddress, baseAddress, bytesPerRow * height );
        baseAddress = tempAddress;
        CGColorSpaceRef colorSpace = CGColorSpaceCreateDeviceRGB();
        CGContextRef newContext = CGBitmapContextCreate(baseAddress, width, height, 8, bytesPerRow, colorSpace,  kCGBitmapByteOrder32Little | kCGImageAlphaNoneSkipFirst);
        newImage = CGBitmapContextCreateImage(newContext);
        CGContextRelease(newContext);
        newContext = CGBitmapContextCreate(baseAddress, 640, 480, 8, 640*4, colorSpace, kCGBitmapByteOrder32Little | kCGImageAlphaPremultipliedFirst);
        CGContextScaleCTM( newContext, (CGFloat)640/(CGFloat)width, (CGFloat)480/(CGFloat)height );
        CGContextDrawImage(newContext, CGRectMake(0,0,640,480), newImage);
        CGImageRelease(newImage);
        newImage = CGBitmapContextCreateImage(newContext);
        CGColorSpaceRelease( colorSpace );
        CGContextRelease(newContext);
        free( tempAddress );
    }

    if ( newImage != nil )
    {

//modified for iOS5.0 with ARC    
    tempImage =  [[UIImage alloc] initWithCGImage:newImage scale:(CGFloat)1.0 orientation:cameraImageOrientation];
    CGImageRelease(newImage);

//this call creates the illusion of a preview layer, while we are actively switching images created with this method
        [self performSelectorOnMainThread:@selector(newCameraImageNotification:) withObject:tempImage waitUntilDone:YES];
    }

    CVPixelBufferUnlockBaseAddress(imageBuffer,0);
}

update the interface with a UIView that can actually be rendered in a graphics context:

- (void) newCameraImageNotification:(UIImage*)newImage
{
    if ( newImage == nil )
        return;

        [arOverlayView setImage:newImage];
//or do more advanced processing of the image
}
2

1 Answer 1

5

If you are wanting a snapshot of what's on screen, this is what I'm doing in one of my camera apps. I haven't touched this code in a long time so there might be a better 5.0 way now but this is solid with over 1 million downloads. There is a function for grabbing a UIView based screen and one for grabbing an Open/GLES1 screen:

//
//  ScreenCapture.m
//  LiveEffectsCam
//
//  Created by John Carter on 10/8/10.
//

#import "ScreenCapture.h"

#import <QuartzCore/CABase.h>
#import <QuartzCore/CATransform3D.h>
#import <QuartzCore/CALayer.h>
#import <QuartzCore/CAScrollLayer.h>

#import <OpenGLES/EAGL.h>
#import <OpenGLES/ES1/gl.h>
#import <OpenGLES/ES1/glext.h>
#import <QuartzCore/QuartzCore.h>
#import <OpenGLES/EAGLDrawable.h>


@implementation ScreenCapture

+ (UIImage *) GLViewToImage:(GLView *)glView
{
    UIImage *glImage = [GLView snapshot:glView]; // returns an autoreleased image
    return glImage;
}

+ (UIImage *) GLViewToImage:(GLView *)glView withOverlayImage:(UIImage *)overlayImage
{
    UIImage *glImage = [GLView snapshot:glView]; // returns an autoreleased image

    // Merge Image and Overlay
    //
    CGRect imageRect = CGRectMake((CGFloat)0.0, (CGFloat)0.0, glImage.size.width*glImage.scale, glImage.size.height*glImage.scale);
    CGImageRef overlayCopy = CGImageCreateCopy( overlayImage.CGImage );
    CGColorSpaceRef colorSpace = CGColorSpaceCreateDeviceRGB();
    CGContextRef context = CGBitmapContextCreate(NULL, (int)glImage.size.width*glImage.scale, (int)glImage.size.height*glImage.scale, 8, (int)glImage.size.width*4*glImage.scale, colorSpace, kCGImageAlphaPremultipliedLast);
    CGContextDrawImage(context, imageRect, glImage.CGImage);
    CGContextDrawImage(context, imageRect, overlayCopy);
    CGImageRef newImage = CGBitmapContextCreateImage(context);
    UIImage *combinedViewImage = [[[UIImage alloc] initWithCGImage:newImage] autorelease];
    CGImageRelease(newImage);
    CGImageRelease(overlayCopy);
    CGContextRelease(context);
    CGColorSpaceRelease(colorSpace);

    return combinedViewImage;
}

+ (UIImage *) UIViewToImage:(UIView *)view withOverlayImage:(UIImage *)overlayImage
    {
    UIImage *viewImage = [ScreenCapture UIViewToImage:view]; // returns an autoreleased image

    // Merge Image and Overlay
    //
    CGRect imageRect = CGRectMake((CGFloat)0.0, (CGFloat)0.0, viewImage.size.width*viewImage.scale, viewImage.size.height*viewImage.scale);
    CGImageRef overlayCopy = CGImageCreateCopy( overlayImage.CGImage );
    CGColorSpaceRef colorSpace = CGColorSpaceCreateDeviceRGB();
    CGContextRef context = CGBitmapContextCreate(NULL, (int)viewImage.size.width*viewImage.scale, (int)viewImage.size.height*viewImage.scale, 8, (int)viewImage.size.width*4*viewImage.scale, colorSpace, kCGImageAlphaPremultipliedLast);
    CGContextDrawImage(context, imageRect, viewImage.CGImage);
    CGContextDrawImage(context, imageRect, overlayCopy);
    CGImageRef newImage = CGBitmapContextCreateImage(context);
    UIImage *combinedViewImage = [[[UIImage alloc] initWithCGImage:newImage] autorelease];
    CGImageRelease(newImage);
    CGImageRelease(overlayCopy);
    CGContextRelease(context);
    CGColorSpaceRelease(colorSpace);

    return combinedViewImage;
}



+ (UIImage *) UIViewToImage:(UIView *)view
{
    // Create a graphics context with the target size
    // On iOS 4 and later, use UIGraphicsBeginImageContextWithOptions to take the scale into consideration
    // On iOS prior to 4, fall back to use UIGraphicsBeginImageContext
    //
    // CGSize imageSize = [[UIScreen mainScreen] bounds].size;
    CGSize imageSize = CGSizeMake( (CGFloat)480.0, (CGFloat)640.0 );        // camera image size

    if (NULL != UIGraphicsBeginImageContextWithOptions)
        UIGraphicsBeginImageContextWithOptions(imageSize, NO, 0);
    else
        UIGraphicsBeginImageContext(imageSize);

    CGContextRef context = UIGraphicsGetCurrentContext();

    // Start with the view...
    //
    CGContextSaveGState(context);
    CGContextTranslateCTM(context, [view center].x, [view center].y);
    CGContextConcatCTM(context, [view transform]);
    CGContextTranslateCTM(context,-[view bounds].size.width * [[view layer] anchorPoint].x,-[view bounds].size.height * [[view layer] anchorPoint].y);
    [[view layer] renderInContext:context];
    CGContextRestoreGState(context);

    // ...then repeat for every subview from back to front
    //
    for (UIView *subView in [view subviews]) 
    {
        if ( [subView respondsToSelector:@selector(screen)] )
            if ( [(UIWindow *)subView screen] == [UIScreen mainScreen] )
                continue;

        CGContextSaveGState(context);
        CGContextTranslateCTM(context, [subView center].x, [subView center].y);
        CGContextConcatCTM(context, [subView transform]);
        CGContextTranslateCTM(context,-[subView bounds].size.width * [[subView layer] anchorPoint].x,-[subView bounds].size.height * [[subView layer] anchorPoint].y);
        [[subView layer] renderInContext:context];
        CGContextRestoreGState(context);
    }

    UIImage *image = UIGraphicsGetImageFromCurrentImageContext();   // autoreleased image

    UIGraphicsEndImageContext();

    return image;
}

+ (UIImage *) snapshot:(GLView *)eaglview
{
    NSInteger x = 0;
    NSInteger y = 0;
    NSInteger width = [eaglview backingWidth];
    NSInteger height = [eaglview backingHeight];
    NSInteger dataLength = width * height * 4;

    NSUInteger i;
    for ( i=0; i<100; i++ )
    {
        glFlush();
        CFRunLoopRunInMode(kCFRunLoopDefaultMode, (float)1.0/(float)60.0, FALSE);
    }

    GLubyte *data = (GLubyte*)malloc(dataLength * sizeof(GLubyte));

    // Read pixel data from the framebuffer
    //
    glPixelStorei(GL_PACK_ALIGNMENT, 4);
    glReadPixels(x, y, width, height, GL_RGBA, GL_UNSIGNED_BYTE, data);

    // Create a CGImage with the pixel data
    // If your OpenGL ES content is opaque, use kCGImageAlphaNoneSkipLast to ignore the alpha channel
    // otherwise, use kCGImageAlphaPremultipliedLast
    //
    CGDataProviderRef ref = CGDataProviderCreateWithData(NULL, data, dataLength, NULL);
    CGColorSpaceRef colorspace = CGColorSpaceCreateDeviceRGB();
    CGImageRef iref = CGImageCreate(width, height, 8, 32, width * 4, colorspace, kCGBitmapByteOrder32Big | kCGImageAlphaPremultipliedLast, ref, NULL, true, kCGRenderingIntentDefault);

    // OpenGL ES measures data in PIXELS
    // Create a graphics context with the target size measured in POINTS
    //
    NSInteger widthInPoints;
    NSInteger heightInPoints;

    if (NULL != UIGraphicsBeginImageContextWithOptions)
    {
        // On iOS 4 and later, use UIGraphicsBeginImageContextWithOptions to take the scale into consideration
        // Set the scale parameter to your OpenGL ES view's contentScaleFactor
        // so that you get a high-resolution snapshot when its value is greater than 1.0
        //
        CGFloat scale = eaglview.contentScaleFactor;
        widthInPoints = width / scale;
        heightInPoints = height / scale;
        UIGraphicsBeginImageContextWithOptions(CGSizeMake(widthInPoints, heightInPoints), NO, scale);
    }
    else
    {
        // On iOS prior to 4, fall back to use UIGraphicsBeginImageContext
        //
        widthInPoints = width;
        heightInPoints = height;
        UIGraphicsBeginImageContext(CGSizeMake(widthInPoints, heightInPoints));
    }

    CGContextRef cgcontext = UIGraphicsGetCurrentContext();

    // UIKit coordinate system is upside down to GL/Quartz coordinate system
    // Flip the CGImage by rendering it to the flipped bitmap context
    // The size of the destination area is measured in POINTS
    //
    CGContextSetBlendMode(cgcontext, kCGBlendModeCopy);
    CGContextDrawImage(cgcontext, CGRectMake(0.0, 0.0, widthInPoints, heightInPoints), iref);

    // Retrieve the UIImage from the current context
    UIImage *image = UIGraphicsGetImageFromCurrentImageContext();   // autoreleased image

    UIGraphicsEndImageContext();

    // Clean up
    free(data);
    CFRelease(ref);
    CFRelease(colorspace);
    CGImageRelease(iref);

    return image;
}

@end
11
  • I checked your app, it's amazing! Trying the + (UIImage *) UIViewToImage:(UIView *)view method now
    – Alex Stone
    Jan 25, 2012 at 20:27
  • 1
    This did not work, because I'm using an AVCaptureVideoPreviewLayer , while it seems that your app is using a GLView for live feed rendering. When calling the save view code, the preview layer is not rendered in the image
    – Alex Stone
    Jan 25, 2012 at 20:34
  • There are methods there for both UIView based and OpenGL based screen captures. Not sure what you tried because my app uses the AVCaptureVideoPreviewLayer too. Jan 25, 2012 at 21:20
  • 2
    OK, it was too much to post here so I placed all the code at jellyfilledstudios.com/2011/11/23/live-effects-cam What I've done is written a foundation for an augmented reality app that lets you swap between UIView as your primary camera image layer or an Open/GLES1 view as your primary image layer with the camera being a texture. For this example I've also thrown in a slightly pointless backinglayer view and overlaylayer view just to demonstrate that layer views (including Open/GL) will be captured. Hope this helps. Jan 26, 2012 at 21:01
  • 1
    @JohnCarter: I place the camera preview in UIView in my app. Now I m using your UIView screen capture, but it shows black screen in the camera preview. What do now
    – TamilKing
    Nov 8, 2013 at 5:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.