Intro

CLIP (Contrastive Language-Image Pretraining), Predict the most relevant text snippet given an image

it is ML (machine learning) model used in Immich to do text search on media (photo and video) files

Immich support a lot of model, than can be found at https://huggingface.co/immich-app and than configured in settings

Different models a takes different time to index media. Some model are Multilanguage, some are not.

Time of media storage processing

Time to process 320 000 photos and 17 000 videos with size of 5.4 TB

nllb-clip-large-siglip__mrl takes even more time than ViT-B-16-SigLIP-512__webli too process media

Quality of search by some random queries

Summary

Word/Phrase	XLM-Roberta-Large-Vit-B-16Plus	ViT-B-16-SigLIP-512__webli	nllb-clip-large-siglip__mrl
anxiety
apple
apple tree	+
arguing people
car			- (infra red 50%)
car on a parking lot
city fog			-(some screenshots)
exchange rate	+	(screenshots)	(screenshots)
feel sad	30%
fog		-(some screenshots)	-(some screenshots)
green fence			60%+ (IR)
honda civic
peach
person feel anxiety
Range Rover	+UAZ		+Toyota
river fog			-(50% screenshots)
serenity		(lot of screenshots)
TV
відчувати радість	0%		40%
відчувати сум	20%		(bags/packing)
заправка wog	100%/40%		100%/70%
заправка кло	100%/38%		100%/10%
зелений забор	100%		80%+ (IR)
курс обміну валют	(8 photo / screenshots )	20%	(screenshots)
люди в лісі		(50% w/o people)	(people+ forest-)
люди які сперечаються		(shops)
людина що відчувають розчарування		(ad boards)
мати сумніви		(mothers)
машина на стоянці			(machinery
машина		(city streets)	(machinery
персик
почуття суму	20%
радість
Рейндж Ровер	+UAZ
Розчарування
спокій
сум
сумніватись
сумувати
тв
туман на річці		/ (fog+ river-)
туман у місті		/ (fog+ city?)
туман
Хонда Сівік	(only logo)
Хонда Цивік	(mostly logo)
Хонда Цівік	(mostly logo)
ціна на бензин	(30% exchange rates)
яблуко
яблуня			/ (more like aplle)

Image found Examples

Car (English)

Green Fence (English)

туман на річці ("River fog" Ukrainian)