Efficiently Count Instances of Letter-Digraphs/-Bigrams in a Corpus - Results Shown in Spreadsheet

You can efficiently count letter-digraphs and bigrams in a large text corpus by using Python’s collections.Counter combined with list comprehensions. This approach processes text in memory without loading entire files into complex data structures, ensuring high performance even for gigabyte-scale datasets. The resulting counts are then exported directly to a CSV file, which opens seamlessly in Excel or Google Sheets for immediate analysis and visualization.

Understanding Bigrams and Digraphs

A bigram (or digraph) is a sequence of two adjacent characters from a string. In natural language processing and cryptanalysis, analyzing the frequency of these pairs helps identify patterns, detect anomalies, or study linguistic properties. For example, in English, "th" and "he" are among the most common bigrams. When working with a corpus—a large collection of texts—counting these occurrences manually is impossible due to the volume of data. Instead, an algorithmic approach is required to iterate through the text and tally each unique pair. The core challenge lies in efficiency. A naive solution might involve nested loops or repeated string slicing that creates thousands of intermediate string objects, leading to high memory usage and slow execution times. By leveraging Python’s built-in libraries, specifically itertools and collections, we can create a streamlined pipeline that minimizes overhead. This method is particularly useful when dealing with Linux environments where command-line tools and Python scripts are commonly used for data analysis tasks.

Setting Up the Environment

Before writing the counting script, ensure you have Python 3 installed on your system. Most modern Linux distributions include Python by default, but you may need to install development headers if you plan to use additional packages later. You do not need any external libraries like pandas or nltk for this basic task, as standard library modules are sufficient and faster for simple counting operations. If you encounter permission errors while trying to install packages globally, refer to our guide on handling Externally managed Environment PIP errors. This often occurs in newer Debian-based systems where package managers restrict global Python installations to prevent conflicts with system tools. Using a virtual environment is recommended for isolated project dependencies, though for this specific script, the standard library alone suffices.

Implementing the Counter Logic

The most efficient way to generate bigrams is to use zip() with iterator shifting. By creating two iterators over the same string—one offset by one character—you can pair every character with its successor. This technique avoids explicit indexing and reduces the number of object creations. Here is the core logic:

from collections import Counter
import csv

def count_bigrams(text):
    # Convert to lowercase and remove non-alphabetic characters if needed
    cleaned_text = ''.join(char.lower() for char in text if char.isalpha())
    
    # Create bigrams using zip
    bigrams = zip(cleaned_text, cleaned_text[1:])
    
    # Join pairs into strings and count them
    bigram_strings = [''.join(pair) for pair in bigrams]
    
    return Counter(bigram_strings)

This function returns a Counter object, which is a dictionary subclass designed for counting hashable objects. It automatically handles the tallying process in C-speed, making it significantly faster than manual dictionary updates. The cleaned_text step ensures that spaces, punctuation, and numbers are ignored, focusing only on alphabetic bigrams. If your corpus includes mixed case, converting to lowercase prevents "Th" and "th" from being counted separately.

Processing Large Files Efficiently

For very large corpora, reading the entire file into memory at once can cause MemoryError exceptions. To avoid this, process the file in chunks or line by line. However, be cautious because bigrams can span across line breaks. A robust solution involves maintaining a buffer of the last character from the previous chunk.

def process_file_large(filepath):
    counter = Counter()
    last_char = None
    
    with open(filepath, 'r', encoding='utf-8') as f:
        for line in f:
            # Handle bigram crossing line boundaries
            if last_char is not None:
                counter[last_char + line[0]] += 1
            
            # Process internal bigrams
            cleaned_line = ''.join(c.lower() for c in line if c.isalpha())
            bigrams = zip(cleaned_line, cleaned_line[1:])
            for pair in bigrams:
                counter[''.join(pair)] += 1
                
            # Update buffer for next iteration
            if cleaned_line:
                last_char = cleaned_line[-1]
            else:
                last_char = None
                
    return counter

This generator-like approach keeps memory usage constant regardless of file size. It correctly identifies bigrams that might appear at the end of one line and the beginning of the next, assuming the original text was continuous. Note that if your corpus consists of separate documents rather than a single continuous stream, you may want to reset last_char between documents to avoid cross-document artifacts.

Exporting Results to Spreadsheet

Once the counts are computed, exporting them to a spreadsheet format allows for easy sorting and filtering. The csv module in Python’s standard library makes this straightforward. We sort the results by frequency in descending order to highlight the most significant bigrams.

def export_to_csv(counter, filename):
    # Sort by count descending
    sorted_items = counter.most_common()
    
    with open(filename, 'w', newline='', encoding='utf-8') as f:
        writer = csv.writer(f)
        writer.writerow(['Bigram', 'Count'])
        for bigram, count in sorted_items:
            writer.writerow([bigram, count])

The most_common() method returns a list of tuples sorted by count. Writing this to a CSV file produces a structure that Excel interprets correctly. You can then apply filters, create pivot tables, or generate bar charts directly within the spreadsheet application. This workflow bridges the gap between raw data processing and human-readable insights. For users interested in automating other terminal-based data transformations, you might also find our article on calling ifcconvert from the terminal helpful for understanding how to pipe data between command-line tools and scripts.

Frequently Asked Questions

How do I handle punctuation in the bigram count?

The provided code filters out non-alphabetic characters using isalpha(), effectively ignoring punctuation. If you need to include punctuation as part of the bigrams, remove the filter condition and simply join all characters. Be aware that this will increase the vocabulary size significantly.

Can I count trigrams or n-grams instead?

Yes, you can generalize the zip approach by adding more offsets. For trigrams, use zip(text, text[1:], text[2:]). For arbitrary n-grams, consider using sklearn.feature_extraction.text.CountVectorizer or implementing a sliding window loop, though the zip method remains highly efficient for small n values.

What if my text contains Unicode characters?

The code uses UTF-8 encoding when opening files, which supports Unicode. Ensure your input text is properly decoded. If you encounter encoding errors, verify that your source files are saved in UTF-8 format. The isalpha() method works correctly with Unicode letters in Python 3.

Is this method faster than using Pandas?

For simple counting tasks, collections.Counter is generally faster and uses less memory than Pandas because it avoids the overhead of DataFrame creation and alignment. Pandas is beneficial when you need to perform complex joins, merges, or statistical aggregations alongside the counting process.

How can I visualize the top 10 bigrams?

You can print the top 10 items directly from the Counter object using counter.most_common(10). For visualization, export the CSV and use Excel’s charting tools, or use Python libraries like Matplotlib to plot a horizontal bar chart of the top frequencies.