Skip to main content

Generic API Key Detection: Information Theory vs. Unknown Unknowns

Traditional secret scanning utilities (such as git-secrets or baseline configurations of container scanners) rely predominantly on Regular Expressions (Regex). This methodology is highly efficient for deterministic tokens possessing rigid structures: AWS IAM keys universally begin with AKIA, OpenAI tokens with sk-, and classic GitHub tokens with ghp_.

However, the MLOps ecosystem is deeply fragmented. Vector database providers (Pinecone, Weaviate), experiment tracking platforms (Weights & Biases, MLflow), and proprietary internal microservices generate authentication keys that consist of 32 to 64 purely random characters (typically UUIDs, Hexadecimal, or Base64 encoded strings). These keys lack a deterministic prefix.

To a Regex engine, these are "Unknown Unknowns." The scanner will silently bypass the string, misclassifying it as a standard string literal or a Git commit hash.

The Mathematical Solution: Shannon Entropy​

To detect prefix-less cryptographic secrets, security architecture must shift from pattern matching to information theory. We calculate Shannon Entropy, a mathematical measure of the "chaos" or unpredictability in the distribution of characters within a given string.

The entropy formula is defined as: $H(X) = - \sum p(x) \log_2 p(x)$

Where $p(x)$ represents the probability of occurrence for each unique character in the string.

  • Low Entropy: A string like "password123" exhibits highly predictable, repeating character patterns. Its entropy score will be mathematically low (approximately 2.5 to 3.0 bits per character).
  • High Entropy: A cryptographically secure API key like "7f8a9d12-b3c4-9e8f" consists of a uniform distribution of randomized characters, pushing its entropy score toward the mathematical limit for its specific character set (typically > 4.5 for Hex strings and > 5.5 for Base64).
import math
from collections import Counter

# Calculating the Shannon Entropy of a string to detect dense cryptographic material
def calculate_shannon_entropy(data_string: str) -> float:
    if not data_string:
        return 0.0
        
    entropy = 0.0
    length = len(data_string)
    frequencies = Counter(data_string)
    
    for freq in frequencies.values():
        probability = freq / length
        entropy -= probability * math.log2(probability)
        
    return entropy

# If the calculated entropy > 4.5 and length > 20, the string is highly suspicious

Context-Aware AST Parsing​

Relying exclusively on high entropy generates unacceptable false-positive rates (as compressed data, file checksums, and serialized object IDs also exhibit high entropy). The critical component for accurate detection is Execution Context.

Veritensor resolves this architectural challenge by constructing the Abstract Syntax Tree (AST) of the target source code (Python, TypeScript, Go) and fusing mathematical entropy with syntactic heuristics. The engine does not merely evaluate isolated strings; it evaluates the variable identifiers to which those strings are assigned.

If a variable identifier matches authorization heuristics (e.g., pinecone_api_key, auth_token, w_b_secret, db_password), AND the assigned literal value exhibits high Shannon entropy, Veritensor deterministically flags the sequence as a compromised secret. This approach mathematically eliminates the blind spots inherent in legacy Regex-based scanners.