Script to Find Common Strings Between Two PDF Files

To find common strings between two PDF files on Linux, use the pdfplumber library in Python to extract text from each document. By converting the extracted content into sets of words or lines, you can perform a set intersection operation to identify shared vocabulary efficiently without manual parsing.

Prerequisites and Installation

Before writing the script, ensure your Linux environment has Python 3 installed. The most robust method for extracting text from PDFs in Python is using pdfplumber, which handles complex layouts better than older tools like PyPDF2. You may also need system-level dependencies for compiling certain libraries if they are not pre-built wheels available for your distribution. First, create a virtual environment to isolate your project dependencies. This prevents conflicts with system-wide packages.

python3 -m venv pdf_compare_env
source pdf_compare_env/bin/activate

Next, install the required Python package.

pip install pdfplumber

If you encounter compilation errors during installation, you might need to install development headers for Python and potentially libjpeg depending on your OS. For example, on Ubuntu:

sudo apt-get install python3-dev libjpeg-dev zlib1g-dev

This setup ensures that the extraction engine functions correctly across various PDF structures.

Extracting Text from PDFs

The core challenge in comparing PDFs is that they are binary formats containing compressed streams, not plain text. We must parse these streams to retrieve human-readable content. The following function uses pdfplumber to open a PDF file and iterate through every page to collect all text.

import pdfplumber

def extract_text_from_pdf(pdf_path):
    """
    Extracts all text from a given PDF file.
    
    Args:
        pdf_path (str): Path to the PDF file.
        
    Returns:
        str: Combined text from all pages.
    """
    full_text = ""
    try:
        with pdfplumber.open(pdf_path) as pdf:
            for page in pdf.pages:
                # Extract text from the current page
                page_text = page.extract_text()
                if page_text:
                    full_text += page_text + "\n"
    except Exception as e:
        print(f"Error reading {pdf_path}: {e}")
    return full_text

This approach concatenates text from all pages with newlines. It handles cases where a page might have no extractable text by checking if page_text is not None. This raw string contains all the characters we will later tokenize.

Tokenizing and Comparing Strings

Once we have the raw text, we need to normalize it to find meaningful commonalities. Simply comparing sentences often fails due to minor formatting differences. A more effective strategy is to compare individual words or n-grams. For this solution, we will compare unique words to find common vocabulary. We convert the text to lowercase to ignore case sensitivity and remove punctuation. Then, we split the text into a list of words and convert it into a Python set. Sets allow for efficient intersection operations.

import re

def get_word_set(text):
    """
    Converts text into a set of unique, lowercase words.
    
    Args:
        text (str): Raw text extracted from PDF.
        
    Returns:
        set: Set of unique words.
    """
    # Convert to lowercase
    text = text.lower()
    # Remove punctuation and split into words
    words = re.findall(r'\b\w+\b', text)
    return set(words)

def find_common_strings(pdf1_path, pdf2_path):
    """
    Finds common strings (words) between two PDF files.
    
    Args:
        pdf1_path (str): Path to the first PDF.
        pdf2_path (str): Path to the second PDF.
        
    Returns:
        set: Common words found in both PDFs.
    """
    text1 = extract_text_from_pdf(pdf1_path)
    text2 = extract_text_from_pdf(pdf2_path)
    
    set1 = get_word_set(text1)
    set2 = get_word_set(text2)
    
    # Find intersection
    common_words = set1.intersection(set2)
    return common_words

The regular expression r'\b\w+\b' captures word boundaries, ensuring that hyphenated words or underscores are handled appropriately based on your specific needs. If you need to preserve phrases instead of single words, you would modify the tokenization step to split by lines or sentences instead.

Running the Script

To make this utility executable, wrap the logic in a main block that accepts command-line arguments. This allows you to run the script directly from the terminal without modifying the code for different files.

import sys
import argparse

def main():
    parser = argparse.ArgumentParser(description="Find common strings between two PDFs.")
    parser.add_argument("pdf1", help="Path to the first PDF file")
    parser.add_argument("pdf2", help="Path to the second PDF file")
    args = parser.parse_args()
    
    common = find_common_strings(args.pdf1, args.pdf2)
    
    if common:
        print(f"Found {len(common)} common words:")
        for word in sorted(common):
            print(word)
    else:
        print("No common strings found.")

if __name__ == "__main__":
    main()

Save this as compare_pdfs.py. You can then execute it in your terminal:

python compare_pdfs.py document_a.pdf document_b.pdf

This output provides a clean list of shared terminology. If you suspect issues with filenames containing spaces or special characters when passing them via bash, remember to quote the arguments properly, similar to troubleshooting Bash Script with Suspected Issues of Spaces, Dashes, Filenames and Quoting.

Optimizations and Considerations

For very large PDFs, loading the entire text into memory might be inefficient. In such cases, consider processing pages incrementally or using a database to store word frequencies. Additionally, stop-word removal (filtering out common words like "the", "and", "is") can help highlight significant commonalities rather than trivial ones. You can achieve this by maintaining a list of English stop words and filtering them out before creating the sets. Furthermore, if the PDFs contain scanned images rather than digital text, pdfplumber will return empty strings. In those scenarios, you would need to integrate Optical Character Recognition (OCR) libraries like pytesseract before performing the comparison. This adds complexity but is necessary for physical document digitization projects.

Frequently Asked Questions

Why does my script return no results even though the PDFs look identical?

The PDFs might contain hidden metadata, different font encodings, or scanned images that prevent text extraction. Ensure you are comparing the visible text content and not just the file structure. Check the extracted text variable to verify data is being read correctly.

Can I compare whole sentences instead of individual words?

Yes, modify the get_word_set function to split text by periods or newlines instead of using regex word boundaries. Store these sentences in a set and perform the intersection operation on them. This helps identify shared paragraphs or specific phrasing.

What if one PDF is password protected?

pdfplumber generally cannot open encrypted PDFs without the password. You would need to pass the password argument to the open method if the library supports it, or decrypt the file beforehand using tools like qpdf. Attempting to open an encrypted file without credentials will raise an exception.

How do I handle case sensitivity in the comparison?

The provided solution converts all text to lowercase before tokenizing. This ensures that "Hello" and "hello" are treated as the same string. If you need case-sensitive comparison, remove the .lower() call, but be aware that this increases the likelihood of missing matches due to capitalization differences.

Is there a way to visualize the common strings?

You can export the common words to a CSV file or generate a word cloud using libraries like wordcloud. This provides a visual representation of shared vocabulary, making it easier to spot dominant themes in both documents compared to a simple text list.