Renzoe Box

Computer Vision • Skin-Tone Matching • Color Correction • MediaPipe

Built the "Match Me" algorithm — a computer vision system that uses a phone camera and color correction cards to accurately map a user's skin tone to their ideal foundation shade. Powered by OpenCV for image processing and MediaPipe's facial tessellation mesh for precise skin region sampling.

Problem

Finding the right foundation shade is one of the biggest pain points in beauty shopping. Skin tones vary dramatically across lighting conditions — what looks like a match in-store fluorescent light looks completely wrong in natural daylight. Online shopping makes this even harder: you can't try the product on, and product photos are unreliable.

The challenge: build a system that can accurately capture a user's true skin tone using just a phone camera — compensating for ambient lighting, camera white balance, and screen color profiles — and map it to the correct foundation shade from the product catalog.

The Match Me Algorithm

Color Correction Cards

Users hold a physical color correction card next to their face while taking a photo. The card contains known color reference patches — similar to a Pantone or X-Rite ColorChecker. By comparing the captured colors of these patches to their known true values, the system calculates a color correction matrix that compensates for the specific lighting conditions and camera characteristics in that moment.

Facial Tessellation (MediaPipe)

MediaPipe's Face Mesh provides a 468-point 3D facial landmark tessellation — a dense mesh of triangles covering the entire face. The algorithm uses specific regions of this mesh to sample skin tone from areas that best represent the user's true color:

Forehead

Large, relatively flat area with consistent skin exposure

Cheeks

Primary matching zone — minimal shadow, good surface area

Jawline

Secondary zone for undertone validation

Average Skin Tone Extraction

Pixels within each tessellation region are sampled and averaged in multiple color spaces (RGB, LAB, HSV) to produce a robust skin tone representation. The LAB color space is particularly important because it's perceptually uniform — meaning numerical distance in LAB space corresponds to how different two colors actually look to the human eye. The corrected LAB values are then matched against the foundation catalog using nearest-neighbor distance in LAB space.

Technical Stack

Computer Vision

OpenCV for image processing, color space conversion, histogram analysis, and color correction matrix computation

Face Detection

MediaPipe Face Mesh — 468-point facial tessellation for precise skin region identification

Color Science

LAB, HSV, and RGB color space analysis for perceptually accurate skin tone matching

Color Correction

Physical calibration cards with known reference patches to normalize lighting and camera differences

Language

Python with NumPy for matrix operations and efficient pixel-level processing

Matching Engine

Nearest-neighbor search in LAB color space against the foundation shade database

Processing Pipeline

1

Capture

User takes a photo holding the color correction card next to their face

2

Card Detection

OpenCV detects and extracts the color reference patches from the card

3

Color Correction

Compute correction matrix by comparing captured patches to known reference values

4

Face Mesh

MediaPipe generates 468-point facial tessellation to identify skin regions

5

Skin Sampling

Extract and average pixel values from forehead, cheeks, and jawline regions

6

Color Space Transform

Apply correction matrix and convert to LAB color space for perceptual accuracy

7

Shade Matching

Nearest-neighbor search against foundation catalog in LAB space → recommended shade

Technical Details

468

Face Landmarks

3

Color Spaces

7

Pipeline Steps

LAB

Perceptual Space

Impact

  • Created a camera-based skin tone matching system that works across different lighting conditions using physical color correction cards
  • Used MediaPipe facial tessellation (468 landmarks) for precise, anatomically-aware skin region sampling — no manual face cropping needed
  • Implemented matching in LAB color space for perceptually accurate results — the same color science used by professional cosmetics labs
  • Built the full pipeline from capture to recommendation — card detection, color correction, face mesh, skin sampling, and shade matching