In the previous articles, we established how to build a photo gallery with its own search engine. But where do we find the images we like? We need to manually find sources of “good” images and then manually check if an image is “good”. Can we automate both of these tasks? And the answer is yes.
“Many applications require being able to decide whether a new observation belongs to the same distribution as existing observations (it is an inlier), or should be considered as different (it is an outlier). Often, this ability is used to clean real data sets. Two important distinctions must be made:”
| outlier detection | The training data contains outliers which are defined as observations that are far from the others. Outlier detection estimators thus try to fit the regions where the training data is the most concentrated, ignoring the deviant observations. |
| novelty detection | The training data is not polluted by outliers and we are interested in detecting whether a new observation is an outlier. In this context an outlier is also called a novelty. |
(https://scikit-learn.org/stable/modules/outlier_detection.html)
Novelty detection seems promising – we can gather a dataset of images we like, train a novelty detection algorithm, and for each new image test if it’s “good” for us.
There are a lot of anomaly detection algorithms, but I decided to go with Gaussian Mixture Model, because it’s quite fast and doesn’t require holding training dataset in memory, like k-nn based algorithms (for example, LOF).
btw, I found this wonderful library, PyOD, which implements a lot of anomaly detection algorithms.
Image features are extracted with CLIP, because it’s awesome and I use it everywhere.
n_components was found through trial and error.
from sklearn.mixture import GaussianMixture
gmm = GaussianMixture(n_components = 16, covariance_type = 'full')
gmm.fit(features)
After that we can score samples, where score is log-likelihood of each sample.
gmm.score_samples(features)
This is the histogram of scores of training(clean) dataset (x is gmm score)
and this is the histogram of scores of unfiltered dataset (/r/EarthPorn/). Scores are clipped at 0 and -3000 for better visibility.
Now we can choose a threshold. Lower threshold ⇒ more images, more outliers and vice versa.
Unfortunately, the presence of watermarks doesn’t have much effect on GMM score. So, I’ve trained a binary classifier (no_watermark/watermark). I’ve annotated 22k images and uploaded the dataset to kaggle.
I’ve found that downscaling image to 224×224 erases subtle watermarks, so I’ve decided to resize images to 448×448, get features of each 224×224 quadrant and concatenate them. Accuracy is about 97-98%, but there are still false-negatives. Probably need a bigger and more diverse dataset.
Pic – plot of losses, blue – train split, orange – test split.
[Github]
anti_sus is a zeromq server for filtering outlier images. It receives a batch of rgb images (numpy array) and returns indexes of good images.
It has 2 step filtering:
- gmm score threshold
- watermark detection
In the future, I’d like to add models that can evaluate the image quality (IQA) and detect if an image is synthetic aka generated with GANs or Diffusion models.
[Github]
nomad is as super hacky reddit parser that uses Pushshift API to get new posts from reddit. Supports flickr and imgur image download.
154 images in ~14 hours, with threshold of 700.
Top 15 subreddits:
[('u_Sovornia', 15),
('itookapicture', 5),
('EarthPorn', 5),
('Outdoors', 3),
('fujifilm', 3),
('flyfishing', 2),
('Washington', 2),
('sunset', 2),
('travelpictures', 2),
('RedDeadOnline', 2),
('SonyAlpha', 2),
('iPhoneography', 2),
('SkyPorn', 1),
('MaldivesHoliday', 1),
('natureisbeautiful', 1)]
We can see that we get a pretty diverse list of subreddits. If we let it run for a while, we’ll get a list of subreddits that are similar to our interests, and we can parse them individually.
We can use this combination of nomad+anti_sus in two different ways: we can use it as a standalone tool and just save new images to the file system, or we can integrate it with scenery. This way, new images will be added to our photo gallery automatically, and we can use ambience to check if an image is a duplicate. At the time of writing, it’s preferred to use phash, I am currently researching the possibility of making use of local features from local_features_web, but it’s too memory/computationally expensive. Why not just use CLIP features? Too unreliable, lots of errors
btw I cleaned /r/Earthporn/ and it’s on scenery.cx now.
Article on github: https://github.com/qwertyforce/anti_sus/blob/main/automatic_image_mining.md
If you found any inaccuracies or have something to add, feel free to submit PR or raise an issue.
