Intelligent skin analysis with Herbsom
This project was carried out as part of the TechLabs “Digital Shaper Program” in Münster (summer term 2021).
Abstract
To suggest the most suitable active ingredient composition for a skin cream, we developed, at the request of the start-up Herbsom, an algorithm that enables skin analysis based on facial photos. After a potential customer has uploaded a photo of the face, the skin type is categorised. This categorisation then allows a conclusion on which ingredients should be added to the base cream for addressing the individual skin characteristics and problems.
Introduction
Our project idea comes from the Münster-based start-up Herbsom, which launches customisable natural cosmetics on the market. The founders noticed that many people find it difficult to classify their own skin type.
We have developed an algorithm that enables skin analysis based on photos of faces, so that suggestions can be made for the most individually suitable ingredient compositions. After a potential customer uploads a photo of their face, it will be categorised into, for example, wrinkled vs. smooth and pimply vs. clear skin. This categorisation then allows a conclusion on which ingredients should be added to the base cream to address the individual skin characteristics and problems. Our goal was for the algorithm to predict as accurately as possible the applicable skin category.
Method
Overall, we used Python (Pytorch/Pandas/Fastai etc.) via Jupyter Notebooks (Colab) as well as Slack and Notion to organize our work progress.
Slack was our main communication tool during the whole project. Here meetings were organised, updates given, problems posted and solutions sought together. For our virtual meetings we used Zoom. With the help of Notion we made use of the Kanban-Board in order to organize our workflows.
The most important tool to develop our algorithm was Python. In Google’s Colaboratory web-bases IDE, we specifically used the Pytorch, Pandas, and Fastai librarys. Our Learner code is mainly based on fast.ai’s classes, which were provided to us in videos by fast.ai on Tech Labs’ learning platform edyoucated.
First of all, we cleaned and organised all the data we got from Herbsom. Since the data was acquried by multiple ways, we converted everything into equivalent files to set the basis for our later algorithm. The related pictures were transformed, normalised and afterwards croped to the to actual face by an extern Face Recognition API .
In a next step, we made the date more “accessible”, i.e. we automated the import and manipulation of the datasets, which ultimately created a datasheet accessible for Machine Learning. On this basis, we created three different learners: wrinkle classification, skin type and pimples classification.
These learners are all based on Convolutional Neural Networks. Due to the limited data quantity we experimented with different pretrained Models, finally receiving the best results with Pytorchs Resnet18 Architecture. The Learner differ mainly in the classification approach used. While we used multiclassification for the skin type to get a specific skin type suggested (i.e. oily skin, dry skin, etc), a regression model was suitable for the pimple and wrinkle learner. Thus, the Learner suggests a number between 0 (no pimples/wrinkles) and 1 (very many pimples/wrinkles) and the active ingredients can be adjusted even more precisely.
In a final step, we converted the Learner into a web-based application to implement on Herbsom’s website. Users can quickly and easily upload their own photo and have it classified.
Results
Herbsom wanted a program that would make it easier for customers to assess their skin type and choose the right products. Our end result makes this possible primarily for wrinkle and pimple classification. In this cases an accuracy of more than 90% is attained. Extending our results to categories that can also be represented on a scale from to (e.g. how shiny is the skin) seems equally feasible. Due to the limited data and time given, the categorization of skin types was not achievable with sufficent accuracy.
Although, we stay in contact with Herbsom to address this problem as the start-up’s data acquisation is not terminated yet.
(However, the categorization of skin type into “normal skin”, “combination skin”, “dry skin” will probably have to remain in the hands of the customer in the near future.)
