> ## Documentation Index > Fetch the complete documentation index at: https://docs.obz.ai/llms.txt > Use this file to discover all available pages before exploring further. # Data Inspection and Outlier Detection > Deep dive into Data Inspector functionality on natural images. We extract features from images that will be used to assess their data quality and to identify outliers. Firstly, let's start with importing necessary PyTorch modules. ```python theme={null} from torchvision.datasets import Imagenette from torch.utils.data import DataLoader from torchvision.transforms import v2 import torch import numpy as np import matplotlib.pyplot as plt ``` Next, let's prepare the datasets. In general, you will need two separate sets of images: **Reference Dataset**: This dataset will be used to extract reference image features and to fit the outlier detection models. In ML, this may be a training dataset. **Inference Dataset**: This dataset will be treated as incoming new data on which you want to perform outlier detection. In ML, this may be testing or validation dataset or any new samples in production. For this tutorial, we will use a subset of the ImageNet dataset called **Imagenette**. The Reference data is from be the Imagenette training data, whereas the Inference data will be from the validation dataset. ```python theme={null} # Transforms TRANSFORMS = v2.Compose([v2.ToImage(), v2.ToDtype(torch.float32, scale=True), v2.CenterCrop(size=(160,160)), v2.Resize(size=(224,224))]) NORMALIZE = v2.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]) # Datasets train_ds = Imagenette(root="./example_data", split='train', size='160px', transform=TRANSFORMS, download=True) ref_set = torch.utils.data.Subset(train_ds, indices=[i for i in range(0, len(train_ds), 10)]) val_ds = Imagenette(root="./example_data", split='val', size='160px', transform=TRANSFORMS) # DataLoaders ref_loader = DataLoader(ref_set, batch_size=32, shuffle=False) inf_loader = DataLoader(val_ds, batch_size=6, shuffle=True) # Labels mapping CLASS_NAMES = ["tench", "English springer", "cassette player", "chain saw", "church", "French horn", "garbage truck", "gas pump", "golf ball", "parachute"] LOGIT2NAME = { 0: "tench", 1: "English springer", 2: "cassette player", 3: "chain saw", 4: "church", 5: "French horn", 6: "garbage truck", 7: "gas pump", 8: "golf ball", 9: "parachute" } ``` ## Data Inspector Module Setup You are now ready to create an Outlier Detector -- using the Reference dataset to obtain the typical distribution of visual features and applying that fitted model to the Inference dataset. The Obz AI package is designed to be highly modular and customizable. As a first step, we’ll import the specific feature extractors and the outlier detection algorithm you want to use. ```python theme={null} from obzai.data_inspector.extractor import FirstOrderExtractor, CLIPExtractor from obzai.data_inspector.detector import GMMDetector, PCAReconstructionLossDetector ``` ### First Order Extractor and GMM Detector `FirstOrderExtractor` is a straightforward and fast tool designed to extract first-order statistical features from images. These features summarize general properties of the pixel intensity values, such as mean, variance, skewness, etc. For example, they are useful for identifying images that are overly bright/dark or excessively variable in their intensities compared to the Reference dataset. **Note:** First-order statistical features are invariant to the arrangement of pixels; in other words, they do not capture spatial relationships within the image. At any point, you can view the list of features that `FirstOrderExtractor` computes by accessing its `.feature_names` attribute. This helps you to understand exactly which statistics are being extracted from your images. ```python theme={null} first_order_extrc = FirstOrderExtractor() first_order_extrc.feature_names ``` \['entropy', 'min', 'max', '10th\_percentile', '90th\_percentile', 'mean', 'median', 'interquartile\_range', 'range', 'mean\_absolute\_deviation', 'robust\_mean\_absolute\_deviation', 'root\_mean\_square', 'skewness', 'kurtosis', 'variance', 'uniformity'] `GMMDetector` is an outlier detection method that utilizes a Gaussian Mixture Model (GMM). To configure and use the `GMMDetector`, please follow the steps below: * `extractors` - Sequence of Extractor objects which process your data. Currently, only the `FirstOrderExtractor` is accepted. * `n_components` - A number of Gaussian components for the mixture model. This controls the complexity of the model and how finely it can separate data clusters. * `outlier_quantile` - Set the quantile threshold to determine what is considered an outlier. Data points falling below this quantile are classified as outliers. * `show_progress` - If set to `True`, a progress bar will be displayed during feature extraction to visualize operation progress. After initialization, you can fit the model to your reference data by calling `.fit` method with a reference data. Ensure that the data you want to model comes in the form of a PyTorch `DataLoader` object. ```python theme={null} gmm_detector = GMMDetector(extractors=[first_order_extrc], n_components=3, outlier_quantile=0.01, show_progress=True) gmm_detector.fit(ref_loader) ``` **That's all!** Your GMM based Outlier Detector is ready to use. You can pass batches of images in the inference data into `.detect()` method. This method returns a named tuple with: * `img_features` - extracted features for each image in the batch. * `outliers` - boolean vector indicating if samples in the batch are outliers. We can run this outlier detection on a single batch of images, and display the outputs. ```python theme={null} # Example code to run inference on a single batch image_batch, _ = next(iter(inf_loader)) detection_results = gmm_detector.detect(image_batch) print("Outlier detection results for a batch:") print(detection_results.outliers) print("First Order Features for a batch:") print(detection_results.img_features) print("Detection scores:") print(detection_results.scores) ``` We can now run this outlier detection on \~1000 samples from the inference data ```python theme={null} ## Example code to run inference on multiple batches (upto 1000 samples) # Process multiple batches from the inference loader firstorder_total_samples = 0 firstorder_total_outliers = 0 firstorder_outliers = [] firstorder_features = [] firstorder_scores = [] for batch_images, _ in inf_loader: # Process batch batch_results = gmm_detector.detect(batch_images) firstorder_total_outliers += batch_results.outliers.sum() firstorder_total_samples += len(batch_results.outliers) firstorder_outliers.append(batch_results.outliers.flatten()) firstorder_features.append(batch_results.img_features['FirstOrderExtractor'].T) firstorder_scores.append(batch_results.scores.flatten()) # Break after 1000 samples if firstorder_total_samples >= 1000: break print(f"Processed {firstorder_total_samples} samples") print(f"Found {firstorder_total_outliers} first-order outliers ({(firstorder_total_outliers/firstorder_total_samples)*100:.2f}%)") ``` We can visualize the results by creating a histogram of outlier status and a distribution of scores based on GMM. In the second figure, we are plotting the distributrion of scores with y-axis in a log scale. ```python theme={null} # Convert lists to numpy arrays for easier manipulation firstorder_outliers_array = np.concatenate(firstorder_outliers) firstorder_scores_array = np.concatenate(firstorder_scores) # Calculate counts for normal and outlier samples firstorder_outlier_counts = np.bincount(firstorder_outliers_array.astype(int)) # Create the same plots with a minimal look fig, (ax1, ax2) = plt.subplots(2, 1, figsize=(15, 10)) # Plot 1: Outlier Status Histogram with minimal style ax1.bar(['Normal', 'Outlier'], firstorder_outlier_counts, color='black') ax1.set_title('Distribution of Outliers', pad=20) ax1.set_ylabel('Count') percentage = (firstorder_outlier_counts[1] / len(firstorder_outliers_array)) * 100 ax1.text(1, firstorder_outlier_counts[1]/2, f'{percentage:.1f}%', ha='center', va='center', color='white') # Plot 2: Detection Scores Distribution with minimal style ax2.hist(firstorder_scores_array, bins=500, color='black', edgecolor='none') ax2.set_yscale('log') # Set y-axis to logarithmic scale ax2.set_title('Distribution of Detection Scores', pad=20) ax2.set_xlabel('Score') ax2.set_ylabel('Count (log scale)') ax2.legend() plt.tight_layout() plt.show() ``` png For completeness, it is good to visualize some images within the normal range and outside of the range. ```python theme={null} # Get first batch of outliers from scores and images normal_images = [] outlier_images = [] batch_idx = 0 for batch_images, _ in inf_loader: detection_results = gmm_detector.detect(batch_images) # Add normal images normal_idx = (~detection_results.outliers).nonzero()[0] if len(normal_idx) > 0: for idx in normal_idx[:3-len(normal_images)]: if len(normal_images) < 3: normal_images.append(batch_images[idx].cpu().numpy()) # Add outlier images outlier_idx = detection_results.outliers.nonzero()[0] if len(outlier_idx) > 0: for idx in outlier_idx[:3-len(outlier_images)]: if len(outlier_images) < 3: outlier_images.append(batch_images[idx].cpu().numpy()) # Break if we have enough images if len(normal_images) == 3 and len(outlier_images) == 3: break batch_idx += 1 # Create figure with subplots fig, axes = plt.subplots(2, 3, figsize=(10, 5)) # Plot normal images for i, img in enumerate(normal_images): axes[0, i].imshow(img.transpose(1, 2, 0)) axes[0, i].axis('off') axes[0, i].set_title('Within Range', fontsize=10) # Plot outlier images for i, img in enumerate(outlier_images): axes[1, i].imshow(img.transpose(1, 2, 0)) axes[1, i].axis('off') axes[1, i].set_title('Outside of Range (Outliers)', fontsize=10) plt.suptitle('Examples of Outliers based on First Order Features', fontsize=10) plt.tight_layout() plt.show() ``` png ### CLIP Extractor and PCA Reconstruction Loss Detector `GMMDetector` applied on First Order Features provides a straightforward quantification that are invariant to the spatial arrangement of objects. We provide more advanced outlier detection based on principal component analysis (PCA), which are suitable for high dimensional embedding features. Here, we look at how to use a deep learning model (CLIP; Contrastive Language-Image Pre-training) to extract embedding features that take account for spatial relationships. To handle this type of high dimensional features, we use `PCAReconstructionLossDetector` as an outlier detection algorithm. *There will an image schematic presentation of Reconstruction Loss idea.* To set up the `PCAReconstructionLossDetector`, you need to provide the following arguments: * **extractor**: This is an already-instantiated extractor object that will be used to turn images into feature vectors. It is best to use an extractor that outputs features with high dimensions, such as `CLIPExtractor`. * **n\_components**: This is the number of principal components (PCs). In other words, it controls how many dimensions will be in the reduced latent space. * **outlier\_quantile**: This value is used to decide if an image should be considered an outlier. If the reconstruction loss (the difference between the original and reconstructed image in feature space) for an image is higher than this quantile, then the detector will classify that image as an outlier. * **show\_progress**: Set this to `True` if you want to see a progress bar while features are being extracted. If you don’t want to see a progress bar, set it to `False`. Note that unlike `GMMDetector`, `PCAReconstructionLossDetector` considers an unusually high value (e.g., reconstruction loss) to indicate evidence for an outlier. ```python theme={null} clip_extractor = CLIPExtractor() pca_detector = PCAReconstructionLossDetector(extractor=clip_extractor, n_components=128, outlier_quantile=0.01, show_progress=True) ``` ```python theme={null} pca_detector.fit(ref_loader) ``` After having fitted this outlier detection algorithm on our reference data, we can apply it on samples from the inference data. ```python theme={null} ## Example code to run inference on a single batch # image_batch, _ = next(iter(inf_loader)) # detection_results = pca_detector.detect(image_batch) # print("CLIP features:") # print(detection_results.img_features) # print("Outlier status:") # print(detection_results.outliers) # print("PCA Reconstruction Loss") # print(detection_results.scores) # print("2D PCA Components") # print(detection_results.projector_results.pca_coords) # print("2D UMAP Components") # print(detection_results.projector_results.umap_coords) ## Example code to run inference on multiple batches (upto 1000 samples) # Process multiple batches from the inference loader clip_total_samples = 0 clip_total_outliers = 0 clip_outliers = [] clip_features = [] clip_scores = [] for batch_images, _ in inf_loader: # Process batch batch_results = pca_detector.detect(batch_images) clip_total_outliers += batch_results.outliers.sum() clip_total_samples += len(batch_results.outliers) clip_outliers.append(batch_results.outliers.flatten()) clip_features.append(batch_results.img_features['CLIPExtractor'].T) clip_scores.append(batch_results.scores.flatten()) # Break after 1000 samples if clip_total_samples >= 1000: break print(f"Processed {clip_total_samples} samples") print(f"Found {clip_total_outliers} outliers ({(clip_total_outliers/clip_total_samples)*100:.2f}%)") ``` ```python theme={null} # Convert lists to numpy arrays for easier manipulation clip_outliers_array = np.concatenate(clip_outliers) clip_scores_array = np.concatenate(clip_scores) # Calculate counts for normal and outlier samples clip_outlier_counts = np.bincount(clip_outliers_array.astype(int)) # Create the same plots with a minimal look fig, (ax1, ax2) = plt.subplots(2, 1, figsize=(15, 10)) # Plot 1: Outlier Status Histogram with minimal style ax1.bar(['Normal', 'Outlier'], clip_outlier_counts, color='black') ax1.set_title('Distribution of Outliers', pad=20) ax1.set_ylabel('Count') percentage = (clip_outlier_counts[1] / len(clip_outliers_array)) * 100 ax1.text(1, clip_outlier_counts[1]/2, f'{percentage:.1f}%', ha='center', va='center', color='white') # Plot 2: Detection Scores Distribution with minimal style ax2.hist(clip_scores_array, bins=500, color='black', edgecolor='none') ax2.set_yscale('log') # Set y-axis to logarithmic scale ax2.set_title('Distribution of Detection Scores', pad=20) ax2.set_xlabel('Score') ax2.set_ylabel('Count (log scale)') ax2.legend() plt.tight_layout() plt.show() ``` png ```python theme={null} clip_outlier_counts[1]/2 ``` We can now visualize some of samples that are considered outliers by the CLIP-based detection method. ```python theme={null} # Get first batch of outliers from scores and images normal_images = [] outlier_images = [] batch_idx = 0 for batch_images, _ in inf_loader: detection_results = pca_detector.detect(batch_images) # Add normal images normal_idx = (~detection_results.outliers).nonzero()[0] if len(normal_idx) > 0: for idx in normal_idx[:3-len(normal_images)]: if len(normal_images) < 3: normal_images.append(batch_images[idx].cpu().numpy()) # Add outlier images outlier_idx = detection_results.outliers.nonzero()[0] if len(outlier_idx) > 0: for idx in outlier_idx[:3-len(outlier_images)]: if len(outlier_images) < 3: outlier_images.append(batch_images[idx].cpu().numpy()) # Break if we have enough images if len(normal_images) == 3 and len(outlier_images) == 3: break batch_idx += 1 # Create figure with subplots fig, axes = plt.subplots(2, 3, figsize=(10, 5)) # Plot normal images for i, img in enumerate(normal_images): axes[0, i].imshow(img.transpose(1, 2, 0)) axes[0, i].axis('off') axes[0, i].set_title('Within Range', fontsize=10) # Plot outlier images for i, img in enumerate(outlier_images): axes[1, i].imshow(img.transpose(1, 2, 0)) axes[1, i].axis('off') axes[1, i].set_title('Outside of Range (Outliers)', fontsize=10) plt.suptitle('Examples of Outliers based on CLIP', fontsize=10) plt.tight_layout() plt.show() ``` png ### Inspecting reference features When you fit an Outlier Detector, it internally extracts and stores a set of reference features from your data. If you wish to access these reference features for inspection or further analysis, you can do so easily. Simply use the `.return_reference_features()` method on your detector instance. This method will return the reference features associated with the given detector. Let’s see how to retrieve the reference features from a `GMMDetector` object: ```python theme={null} refs_gmm = gmm_detector.return_reference_features() ``` Similary you can retrieve reference features from you PCA Reconstruction Loss Detector. ```python theme={null} refs_pca = pca_detector.return_reference_features() ``` This particular type of a detector produces also 2 dimensional represenations of data points prepared with PCA and UMAP algorithms. It is particularly usefull for visualization on scatter plot. You can return those embedded features by calling `.return_reference_2D_components()` ```python theme={null} embedding_coords = pca_detector.return_reference_2D_components() print("## PCA Coordinates of Reference Set ##") print(embedding_coords.pca_coords) print("## UMAP Coordinates of Reference Set ##") print(embedding_coords.umap_coords) ``` ## PCA Coordinates of Reference Set x\_coor y\_coor 0 9.501494 7.013512 1 17.202290 12.576481 2 15.603800 10.027860 3 14.825594 10.396540 4 11.999342 10.963834 .. ... ... 942 5.190470 -10.450700 943 6.155877 -10.435268 944 4.931119 -11.205107 945 7.275045 -11.389244 946 1.419314 -6.772340 \[947 rows x 2 columns] ## UMAP Coordinates of Reference Set x\_coor y\_coor 0 9.501342 7.013336 1 17.202326 12.576518 2 15.603857 10.027987 3 14.825516 10.396494 4 11.999347 10.963912 .. ... ... 942 5.190484 -10.450880 943 6.155859 -10.435368 944 4.931139 -11.205077 945 7.275099 -11.389211 946 1.419346 -6.772443 \[947 rows x 2 columns]