Mahmoud Afifi, a recent graduate from the Department of Electrical Engineering & Computer Science at the Lassonde School of Engineering, has won two awards for his doctoral dissertation.
Afifi was awarded the CS-Can/Info-Can 2021 Canadian Computer Science Distinguished Dissertation Award, which recognizes excellence in research and writing by a computer science doctoral student completing their degree at a Canadian institution.
He also received the 2021 Canadian Image Processing and Pattern Recognition Society (CIPPRS) John Barron Doctoral Dissertation Award, given annually to the top PhD thesis in areas covered by the Conference on Computer and Robot Vision (CRV) including computer vision, robot vision, robotics, medical imaging, image processing or pattern recognition.
Department of Electrical Engineering & Computer Science professor at Lassonde and the Canada Research Chair Computer Vision, Michael S. Brown, supervised Afifi’s thesis “Image Colour Correction, Enhancement and Editing.”
“I’m happy to see Dr. Afifi’s hard work, creativity and scientific efforts have been acknowledged by these two awards,” said Brown. “It was my honour to be his supervisor.”
Afifi’s work focused on understanding and implementing colour correction processes to produce colour-accurate images from digital cameras. One part of his thesis was to correct the colour of images with inaccurate white balance, that is, how warm or cool the overall colours of a photograph look, especially the parts that are, or should be, white. (If you’ve ever tried to take a photo of a scene lit simultaneously by office fluorescent lights and sunset streaming in from a window, only to discover that your image has an unpleasant bluish or reddish cast, then you have encountered a white balance problem.)
“Whether it’s because of the white balance tints or the effect of lighting on the image, our goal was to try to make white objects look white,” said Afifi, noting that it was a task that was more complicated than it may seem. He proposed several methods for white balance correction and editing. He worked with Adobe Research to correct images post-capture after the photograph has already been taken. Another area he worked on with Adobe was auto recolouring a photo based on the objects of the scene, a significant evolution from typical recolouring filters, like those found in Instagram, which do not understand the contents of the image.
The second aspect of his research was a colour correction for exposure errors. A photographer usually adjusts several camera settings to make sure that the right amount of light is let into the sensor (the exposure) to accurately capture the colours of an image. Underexpose the image, and navy blue or dark brown objects start to become indistinguishable from dark, black shadows. Overexpose the image and pastels get washed out and appear white. Even when you try to correct the exposure using post-production photo editing tools, the software may not be able to restore the original colour of the object as that information has been lost.
Afifi tackled this problem of colour degradation arising from exposure errors, carrying out an internship with Samsung to work on the post-capture correction.
Afifi also worked on illuminant estimation during an internship with Google, where he, along with researchers from Google, designed a neural network to perform illumination estimation, and moreover, developed a sensor-independent approach so that the model does not need to be rebuilt or retrained for different sensors or camera hardware.
The consequence of accurate colour and white balance corrections not only helps with producing better-looking images but also provides better inputs for downstream processes such as object identification within photographs. Although deep learning approaches that are used in such high-level image processing are already fairly accurate, they are very sensitive to the reference data used to train the neural network model. “If the training data used to identify cats are all photos taken with good white balancing, the deep learning model may not be able to accurately identify cats in images with inaccurate white balance,” explained Afifi. He addressed the problem by developing a colour augmentation technique to improve network robustness against images with white balance mistakes.
Afifi’s research resulted in 13 papers, five patent applications, a collaboration with Adobe, and internships at Samsung and Google. He credited his success to his experience in Brown’s lab, where he was able to learn to treat problems from different angles in a supportive research environment. “If you have a deep understanding of the problem that you like to solve, you will find it easy to get innovative and think outside of the box,” said Afifi.
While he found research to be fun, at times it was also challenging. “You have to keep the passion up, deadline to deadline, idea to idea and with short timeframes,” said Afifi, “and it was also challenging to find that balance between the research and my kids, wife and family.”
When he applied for these awards, Afifi did not think he would get them but was ecstatic to learn the news from Brown. “Even my daughter told her friends,” he said. Afifi, who graduated in 2021, now works at Apple in colour processing as a camera algorithms engineer.