Noisy channel model

See Chapter B of.[1]

Given an alphabet ${\displaystyle \Sigma }$, let ${\displaystyle \Sigma ^{*}}$ be the set of all finite strings over ${\displaystyle \Sigma }$. Let the dictionary ${\displaystyle D}$ of valid words be some subset of ${\displaystyle \Sigma ^{*}}$, i.e., ${\displaystyle D\subseteq \Sigma ^{*}}$.

The noisy channel is the matrix

${\displaystyle \Gamma _{ws}=\Pr(s|w)}$,

where ${\displaystyle w\in D}$ is the intended word and ${\displaystyle s\in \Sigma ^{*}}$ is the scrambled word that was actually received.

The goal of the noisy channel model is to find the intended word given the scrambled word that was received. The decision function ${\displaystyle \sigma$ :\Sigma ^{*}\to D} is a function that, given a scrambled word, returns the intended word.

Methods of constructing a decision function include the maximum likelihood rule, the maximum a posteriori rule, and the minimum distance rule.

In some cases, it may be better to accept the scrambled word as the intended word rather than attempt to find an intended word in the dictionary. For example, the word schönfinkeling may not be in the dictionary, but might in fact be the intended word.

One naturally wonders if the problem of translation could conceivably be treated as a problem in cryptography. When I look at an article in Russian, I say: 'This is really written in English, but it has been coded in some strange symbols. I will now proceed to decode.

— Warren Weaver, Letter to Norbert Wiener, March 4, 1947

See chapter 1, and chapter 25 of.[2]

Suppose we want to translate a foreign language to English, we could model ${\displaystyle P(E|F)}$ directly: the probability that we have English sentence E given foreign sentence F, then we pick the most likely one ${\displaystyle {\hat {E}}=\arg \max _{E}P(E|F)}$. However, by Bayes law, we have the equivalent equation:

$${\displaystyle {\hat {E}}={\underset {E\in {\text{ English }}}{\operatorname {argmax} }}\overbrace {P(F\mid E)} ^{\text{translation model }}\overbrace {P(E)} ^{\text{language model}}}$$

Consequently, if we have a good model of English, and a good English-to-foreign translator, we immediately get a good foreign-to-English translator.

Speech recognition can be thought of as translating from a sound-language to a text-language. Consequently, we have

$${\displaystyle {\hat {T}}={\underset {T\in {\text{ Text }}}{\operatorname {argmax} }}\overbrace {P(S\mid T)} ^{\text{speech model }}\overbrace {P(T)} ^{\text{language model}}}$$

where ${\displaystyle P(S|T)}$ is the probability that a speech sound S is produced if the speaker is intending to say text T. Intuitively, this equation states that the most likely text is a text that's both a likely text in the language, and produces the speech sound with high probability.

The utility of the noisy-channel model is not in capacity. Theoretically, any noisy-channel model can be replicated by a direct ${\displaystyle P(T|S)}$ model. However, the noisy-channel model factors the model into two parts which are appropriate for the situation, and consequently it is generally more well-behaved.

When a human speaks, it does not produce the sound directly, but first produces the text it wants to speak in the language centers of the brain, then the text is translated into sound by the motor cortex, vocal cords, and other parts of the body. The noisy-channel model matches this model of the human, and so it is appropriate. This is justified in the practical success of noisy-channel model in speech recognition.

• Coding theory