Blog

Zoom In With OpenGL ES

Gestures, Graphics, iOS Development, OpenGL

Zooming in on an image with a pinch gesture and then panning around that image has been a feature we have all became familiar with since the introduction of the very first iPhone. It still remains an important experience for every mobile user and for developers it is still a feature that is implemented in a lot of apps. To create the correct solution to zoomable images we must think about memory management as this could terminate our app, and we must think about CPU performance which could cost the users battery life and also create a very glitchy UI.

The best way to deal with memory management is to only display the image size that is needed. i.e. a 320x568 screen with a 2x retina display does not need to be displaying an image any bigger than 640x1136 if the image is at a zoom scale of 1. So the first thing to do is only draw the image (with the correct aspect ratio) with width and height at most (screen dimension x screen scale factor x zoom scale).

This way of managing the memory now needs an efficient method for the CPU to deal with the drawing of the images. Since users will be zooming, doing any drawing the on the main thread could be detrimental the performance of the UI and cause stutters as a new image gets drawn to be displayed. So the best method here is to make sure any images drawn are done on a background thread, and then when the drawing is completed we can send this to the main thread to be displayed. In addition we can increase CPU performance further by using OpenGL ES to do all of the on screen rendering since animations happen quite a lot when it comes with zooming - just think about the double tap gesture we all use. Open GL also gives us the extra benefit of being able to write custom shaders and therefore we can write image filters with no cost on the CPU.

Drawing images in a background thread is helpful because we don't block the main thread but it too comes with problems, namely we may be at a point where we have requested the image drawn at a zoom scale of 2 and zoom scale of 3 as the user has continued zooming in (or more likely getting the 3x image after the 2x image as the user is zooming out) but we could get a delay and have the 3x image come before the 2x image in which case is great providing we know not to use the 2x image. Therefore we must unregister ourselves from hearing about a new image being available if it is not the one we want.

At the moment of creating the image we want to display on screen we need to create the new texture used for OpenGL ES. This is a fairly long operation (noticeable if the user is still zooming) so we need to be able to create the texture in an asynchronous way. We do this by using an asynchronous texture loader and once the texture loader has finished we go back to the main thread and update the texture that is to be displayed. This method has absolutely no impact on the main thread and is almost unnoticeable to the user aside from the image appears sharper.

Now we have our approach we just have to think about how the user will zoom in to an image. Very often on iPhone the aspect ratio of an image (usually 4:3) is not the same as the screen (usually 16:9) so we fit the image for the screen and we get sides or tops that have no content in them. These parts of the image are very dull and no user wants to zoom in on these, however if you take a minute to look at the Photos app on your iPhone or iPad you will find that you can zoom in on them then at the moment you let go the image snaps to its' bounds and everything is fine. I don't know if this is intended behaviour but even if it is I still feel that it looks a little bit odd and much prefer a zooming solution where the content is highlighted. The solution to this is simple, before the view updates and shows the new boundaries of where the user has zoomed in we check whether the image could be closer to the bounds of the screen. If the image has a height of 300 and the screen has a height of 320 then we should keep a space of 10 at the top and a space of 10 at the bottom independent of the location of the users intended zoom position. Once the images dimensions are bigger than that of the screens then we can zoom into any point without having to worry about the position of the image. This is the same intention for a double tap gesture.

One of my favourite UX in iOS is the elastic effect when panning out of the content bounds that you just don't seem to get on other devices. If we used a UIScrollView we would happily get this for free but we are sadly not. So to get this we calculate how much the user is panning past the bounds and use the hyperbolic tan function to reduce the amount the user pans outside. The hyperbolic tan function is perfect for this scenario since at 0 the function has a gradient of exactly 1 (which is just at the point we will ask the function to kick in) and it has an asymptote at 1 so we know it can't pan for infinity and there is some limit (which we can set) as to how far the user can go.

At this point we have combined optimisation in memory, CPU and UI. This is now an image view we can use whenever we want the user to be able to zoom in on an image but in some cases we would like to use these to swipe through images in some sort of album. The panning of the image can get in the way of its' superview (scrollview) ability to scroll and so I have added a 'panningDelegate' which will ask the delegate whether image view should pan Left, Right, Up or Down, when the pan will move out of the bounds of the image. You could also use this panningDelegate just to remove the elastic effect if you don't like it.

To use this image view please get the code from GitHub.

The files you will need in your project for the image view to work are 'AGLKBaseEffect', 'AGLKImageEffect', 'AGLKImageView', 'AGLKViewListener', 'AGLKVertextAttributeBuffer', 'AGLKContext', 'AGLKMath', 'AGLKImageRenderer', 'AGLKImageShader.fsh' and 'AGLKImageShader.vsh'. 'AGLKViewListener' is a simple class that I use to delete drawables on 'AGLKImageView' when the app goes into the background state.

Get your project started

Talk to us today about web and app projects