Efficient Way to Find Indices of Unique Rows in PyTorch Tensor

To efficiently find the indices of unique rows in a PyTorch tensor, you can leverage torch.unique with the return_inverse or return_counts flags. By sorting the tensor along the first dimension and using these flags, you can map each row to its unique identifier and then filter for rows that appear exactly once. This approach avoids explicit Python loops, ensuring high performance on GPU devices while maintaining clarity in your data processing pipeline.

Understanding the Problem Context

In machine learning workflows, particularly when handling datasets with duplicates, identifying unique samples is crucial for model training integrity. You might encounter scenarios where a PyTorch tensor contains repeated rows representing identical data points. The goal is not just to remove duplicates but to know which original indices correspond to these unique entries. This information helps in debugging data pipelines or ensuring balanced sampling during training. Standard Python approaches using sets or dictionaries often fail because tensors are not hashable. Furthermore, converting tensors to NumPy arrays for processing can introduce significant overhead due to data transfer between CPU and GPU memory. Therefore, a pure PyTorch solution is preferred for its efficiency and compatibility with automatic differentiation graphs if needed.

Method 1: Using torch.unique with return_inverse

The most straightforward method involves using the torch.unique function. This function returns three values: the unique elements, their inverse indices, and optionally the counts of each element. To find unique rows, we treat each row as an element. First, ensure your tensor is contiguous. Non-contiguous tensors can cause issues with some operations. Then, apply torch.unique along dimension 0. If you need the indices of the first occurrence of each unique row, use the return_inverse=True flag. However, this gives you the mapping back to the unique set, not necessarily the count. To isolate strictly unique rows (those appearing only once), you should combine this with return_counts=True. Here is how you can implement this:

import torch

def get_unique_row_indices(tensor):
    # Ensure the tensor is contiguous for efficient processing
    tensor = tensor.contiguous()
    
    # Get unique rows, their inverse indices, and counts
    unique_rows, inverse_indices, counts = torch.unique(
        tensor, 
        dim=0, 
        sorted=True, 
        return_inverse=True, 
        return_counts=True
    )
    
    # Identify which unique rows appear exactly once
    is_unique = counts == 1
    
    # Map back to original indices
    # We need to find which original indices correspond to the unique rows
    # inverse_indices maps each original row to its position in unique_rows
    # We filter original indices where the corresponding unique row has count 1
    
    unique_mask = is_unique[inverse_indices]
    original_indices = torch.arange(tensor.size(0))
    
    return original_indices[unique_mask]

This code creates a boolean mask indicating which original rows belong to unique groups. It then filters the range of indices to return only those belonging to single-occurrence rows.

Method 2: Sorting and Grouping for Large Tensors

For very large tensors, torch.unique can be memory-intensive. An alternative approach involves sorting the tensor and then comparing adjacent rows. This method is more algorithmic but can be more memory-efficient. Sort the tensor based on its rows. Then, compare each row with its neighbor. If a row is different from both its predecessor and successor (handling edge cases), it is unique. This requires careful implementation to handle duplicates correctly.

def get_unique_indices_sort_method(tensor):
    # Sort the tensor and keep track of original indices
    sorted_tensor, sort_indices = torch.sort(tensor, dim=0)
    
    # Compare adjacent rows
    # Shift rows up and down
    prev_rows = torch.cat([sorted_tensor[:1], sorted_tensor[:-1]])
    next_rows = torch.cat([sorted_tensor[1:], sorted_tensor[-1:]])
    
    # Check if current row is different from previous and next
    diff_prev = (sorted_tensor != prev_rows).any(dim=1)
    diff_next = (sorted_tensor != next_rows).any(dim=1)
    
    # A row is unique if it differs from both neighbors
    is_unique = diff_prev & diff_next
    
    # Map back to original indices
    unique_sorted_indices = torch.where(is_unique)[0]
    original_indices = sort_indices[unique_sorted_indices]
    
    return original_indices

While this method avoids the overhead of computing full counts, it still requires sorting, which is O(N log N). For most practical purposes, Method 1 is more concise and leverages optimized C++ kernels within PyTorch.

Performance Considerations

When choosing between methods, consider the size of your tensor and available memory. torch.unique is highly optimized but may consume extra memory for intermediate tensors. If you are working on GPU, ensure that all operations remain on the same device to avoid costly transfers. Additionally, if your tensor contains floating-point numbers, be aware of precision issues. Two rows that are mathematically identical might differ by epsilon due to numerical errors. In such cases, consider rounding or using a tolerance-based comparison before applying uniqueness checks. For related insights on handling data shapes in PyTorch, you might find Understanding the Returned Shape of Targets in PyTorch Dataloader useful, as proper shape management often precedes advanced operations like this.

Frequently Asked Questions

What happens if my tensor has NaN values?

NaN values are treated as distinct from other values and from themselves in standard comparisons. torch.unique will typically group NaNs together. If you want to exclude NaN rows entirely, filter them out before applying the uniqueness check using torch.isnan() and logical indexing.

Can I use this method for 2D images instead of rows?

Yes, as long as you define "row" as a slice along a specific dimension. For images, you might flatten the spatial dimensions and treat each pixel vector as a row, though this is computationally expensive. It is better suited for tabular data or flattened feature vectors.

Is there a way to get unique columns instead of rows?

Simply transpose the tensor before applying the function. Use tensor.T or tensor.permute(1, 0) to swap axes, then apply the same logic. Remember to adjust the returned indices accordingly if you need them mapped back to the original column positions.

How does this handle duplicate unique rows?

If multiple rows are identical and appear more than once, they are not considered "unique" in the sense of having a count of one. The methods above specifically target rows that appear exactly once. If you want to keep one instance of every duplicate group, use return_inverse=True without the count filter.

Does this work with mixed data types?

No, PyTorch tensors require homogeneous data types. If your data includes strings or mixed types, you must convert them to a numeric representation first, such as encoding categories into integers. Once converted to a numeric tensor, the uniqueness check works seamlessly.