Cryptanalysis

Text statistics

Exact definitions for Decipher’s text statistics, with the assumptions that change their meaning.

Also found as index of coincidence · entropy · chi-square · coincidence at lag

Choose the sample first

A statistic describes the units supplied to it, not the displayed ciphertext in the abstract. Decipher can use the whole input, periodic phases, or fixed blocks. It can count Latin letters, whitespace-separated tokens, or visible characters. Results from different choices are not directly comparable.

Length is the number of selected units. Distinct units is the number of different values among them. Those two numbers set the scale for everything that follows.

Index of coincidence

Decipher's index of coincidence is the probability that two different positions, selected without replacement, contain the same unit. Let N be the sample length and fₓ the count of unit x.Friedman, Index of Coincidence Campaigne, Index of Coincidence

Raw finite-sample index

IC = Σₓ fₓ(fₓ − 1) / [N(N − 1)]

IC for AABB

Numerator: 2(1) + 2(1) = 4
Denominator: 4(3) = 12
IC: 4 / 12 = 1 / 3

Shannon entropy

Let p̂ₓ be the observed proportion of unit x. Entropy measures how spread out that observed distribution is. With a base-2 logarithm, its unit is bits per selected unit.Shannon, Mathematical Theory of Communication

H = −Σₓ p̂ₓ log₂(p̂ₓ)

Order does not enter entropy

AABB: H = 1 bit
ABAB: H = 1 bit

The two samples have the same counts, so they have the same entropy despite their different order.

Chi-square against uniform

Decipher compares observed counts Oₓ with equal expected counts Eₓ across the configured inventory. Unobserved inventory members still contribute. The displayed number is the raw statistic; Decipher does not attach a p-value or compare against English frequencies.NIST, Chi-square goodness of fit

Uniform inventory of size m

Eₓ = N / mχ² = Σₓ (Oₓ − Eₓ)² / Eₓ

AAAA over a two-unit inventory

Observed: 4, 0
Expected: 2, 2
χ²: (4−2)²/2 + (0−2)²/2 = 4

Coincidence at lag

At lag k, Decipher compares each unit with the unit k positions later and reports the proportion of exact matches. This categorical equality measure is not the numerical autocorrelation built from deviations around a mean.NIST, Autocorrelation

C(k) = #{i : uᵢ = uᵢ₊ₖ} / (N − k)

ABAB

Lag 1: 0 matches out of 3 = 0
Lag 2: 2 matches out of 2 = 1

A peak can support a candidate period, but repeated symbols, harmonics, and a short denominator can produce peaks too. Read it alongside the actual repetitions and phases.

Zero can mean no comparison

For an empty sample, for IC with fewer than two units, or for a lag with no remaining pair, the current calculation returns 0. That is a practical result convention. It is not evidence that the text is uniform, random, or free of repetition.