For every curve on our face, there exists a cosmetic product that can project it beautifully. With rapid increase in incomes due to growing economics, increasing interests in cosmetics for skin care, salon and spa treatments due to changing lifestyles and climate changes have drastically increased the demand for skin care products. This can be illustrated by the fact that the global cosmetics market was valued in 2017 at USD 532 billion and this number is expected to reach USD 863 billion by 2024 according to Zion Market Research (2018)

In the beauty industry, it is important to remember that not all products work for all people. We’re all unique with different skin type, skin tone and have different goals that we wish to achieve using beauty products. Most brands currently sell one-size-fits-all products that is believed to magically work on all skin types and conditions. Customers still blindly believe in the hype that they will look exactly like the advertised model with glowing skin without considering their personal skin condition and end up spending money on products that doesn’t help them. In fact, a study done by SkinStore in 2017 mentions that an average American woman could spend up to $300,000 on face products during her lifetime. With the huge variety of products being sold in the market, consumers often find themselves confused about which products to buy and rely on the salesperson's expertise. Men are often reluctant to ask specific questions about beauty products publicly. In an industry that requires a high level of personalization of products, does not seem to offer customized products to its target customers and this presents an opportunity for computer science to close the gap.

With the advent of artificial intelligence (AI), there has been a rise of the personalisation trend whereby consumers are able to have their specific needs fulfilled using a tailor-made formula instead of generic cosmetics items. Computer vision is a sub-branch of AI which deals with the technology for creating artificial systems based on information from images and media. Using computer vision one can recognize facial features, analyze the data and then make predictions about the appearance. Furthermore, consumers can look at the effects of products and new looks on their face without physically testing them using computer vision and augmented reality. With companies have huge data of previous purchases, data analysis on consumer data will lead to better cosmetics. With data analytics, products can be tailored accordingly to varying consumers. Consumers can be recommended different brands and their products focussing on exact ratios of ingredients and optimized formulas. This personalised approach would help alleviate the problem of consumers not being able to figure out what to buy and wasting their money on generic products.

The objective of this project is to develop a mobile application that a user can use to assist them in their skincare needs. The mobile application is intended to break the cycle of trying and testing new skincare products for the users, and streamline the process of facial self care for the user.

We want to address the issue of a user picking the wrong products for themselves, and picking “broad-brush” products which promise too much and deliver too little. With our application, a user will be able to get a specialized and personalized list of products which will cater to the user’s needs and requirements. And instead of getting bombarded with aisles upon aisles of products - most of which are irrelevant to the user - we want to deliver a list that has as low redundancy and as high relevance to the user as possible.

In achieving the aforementioned state we have 2 major goals:
- Correct classification of the user’s skin type, problems and severity of problems using the selfie and questionnaire (see 3. Methodology for details), is a primary goal
- Building a reliable recommendation system that takes in the users information and then gives the a treatment plan or selection of products that will help to alleviate the users problems.

To sum up, we want a mobile application that servers as handy and convenient dermatologist, readily available to any person with facial skin related problems. And just like a dermatologist, our application will cater to the individual needs of the person in matters of preference such as allergies, likes/dislikes, etc; and more importantly in matters of nailing down the problem and providing the right treatment.

For acne detection, we will be utilizing a research paper by Zhao, Zhang & Spoelstra (2019), by using a pretrained CNN model that was trained using selfie image labeled from 1 to 5 according to the person’s acne severity.

For wrinkles detection, a lot of sources suggests the use of Gabor filter, one of the papers is titled Fast Detection of Facial Wrinkles Based on Gabor Features Using Image Morphology and Geometric Constraints by Batool and Chellappa (2014). Fortunately, Gabor filter is supported in OpenCV library. Another alternative is to use Hessian filter as suggested in a research paper titled Wrinkle Detection Using Hessian Line Tracking by Ng, Yap, Costen, and Li (2015).

To predict the skin age from selfie image, we can either use Active Appearance Model (AAM) or Convolutional Neural Network (CNN) approach. There exists some existing implementation of Fast-AAM. However, a research paper by Padme and Desai (2015) suggests that AAM and some other older literature are relying on a controlled facial images which might not be suitable for mobile selfie. If that’s the case, then a pretrained CNN model trained using IMDB database will be used.

Currently we are planning to use the difference of actual age and predicted skin age to determine the sallowness. Further development could consider yellowish skin.

By utilizing the facial landmark information and wrinkle detection algorithm, we are able to detect one’s Crow’s feet if exists.

For skin tone/colour detection, there are several research papers that provide techniques for implementation. The first method is the use of HSV and YCrCb colour spaces to determine the user skin tone. Another method is to use k-means clustering algorithm with the support of OpenCV library to detect skin tone, as suggested by Jalangan (2018) in one of his articles. The skin tone information can be useful to measure the severity of dark-eye circle and eyebags.

To detect dark-eye circle and eyebags, we will utilize the facial landmark information (below the eyes) and the skin tone to determine the severity of dark-eye circle and eyebags.