MaxEnt principle and closure under AWGN

Wiki card — concept node for exp-families-stability.

(i) Formal statement

MaxEnt principle (Jaynes 1957). Among all densities \(p\) on \(\mathbb{R}^d\) satisfying the moment constraints \(\mathbb{E}_p[\varphi_i] = \eta_i\) (\(i = 1, \ldots, r\)), the entropy-maximising density (subject to existence) is

\[ p^\star(x) \;=\; Z^{-1}\, e^{-\theta^\top \varphi(x)}, \qquad Z = \int e^{-\theta^\top \varphi}, \]

where \(\theta\) is the Lagrange multiplier dual to the moment constraint. The maximised entropy is \(h(p^\star) = \theta^\top \eta + \log Z\).

The MaxEnt density is precisely the exponential-family density (1) of the problem statement — Jaynes' theorem furnishes the inverse direction: every exponential family is a MaxEnt family under its own sufficient-statistic constraints.

AWGN (additive white Gaussian noise) channel. The map \(X \mapsto X + Z\) with \(Z \sim \mathcal{N}(0, \sigma^2 I)\) independent; equivalently, convolution \(p \mapsto p * \gamma_\sigma\).

Stability question restated. Q1 asks: for which \(\varphi\) does the MaxEnt class \(\{p_\theta\}\) remain closed under AWGN?

(ii) Role in Q1 / Q2 / Q3

• Q1 (MaxEnt ∩ AWGN framing, Shannon persona). The intersection of "MaxEnt-closed under moment constraints \(\mathbb{E}[\varphi]\)" and "AWGN-closed under convolution \(*\gamma_\sigma\)" is the target class of Q1. MaxEnt fixes the family form (1); AWGN imposes the closure under \(* \gamma\); the intersection is the problem.
• Q2 (Gaussian saturation). When \(\varphi_i \in \{\mathrm{vec}(xx^\top), x\}\), MaxEnt gives Gaussian; AWGN preserves Gaussian. The intersection is precisely the Gaussian family.
• Q3 (Hamilton–Jacobi dual). The HJ equation (6) is the dual (Legendre-transform) form of the MaxEnt duality between \(\theta\) and \(\eta\) evolved along AWGN time. Computing \(\partial_t h(X_t) = \tfrac12 J(X_t)\) (de Bruijn) on the MaxEnt family gives a linear ODE on \(\theta_t\) if and only if \(J(X_t)\) is affine in \(\eta_t\) — which is (Q2 saturation) iff \(\varphi\) is quadratic-linear.

(iii) References

• Jaynes (1957) — Information Theory and Statistical Mechanics. doi:10.1103/PhysRev.106.620.
• Shannon (1948) — A Mathematical Theory of Communication; EPI, AWGN capacity. doi:10.1002/j.1538-7305.1948.tb01338.x.
• Cover, Thomas (2006) — Elements of Information Theory, ch. 12 (MaxEnt), ch. 17 (EPI, AWGN). doi:10.1002/047174882X.

(iv) Worked miniature — Gaussian is MaxEnt and AWGN-closed

MaxEnt under \(\mathbb{E}[x] = \mu\), \(\mathbb{E}[xx^\top] = \Sigma + \mu\mu^\top\). Lagrangian \(\mathcal{L}(p) = -\int p\log p + \lambda_0(\int p - 1) + \lambda^\top \int x p + \tfrac12\mathrm{tr}(\Lambda \int xx^\top p)\). Setting \(\delta\mathcal{L} = 0\): \(-\log p - 1 + \lambda_0 + \lambda^\top x + \tfrac12 x^\top\Lambda x = 0\), hence \(p(x) \propto \exp(\tfrac12 x^\top\Lambda x + \lambda^\top x)\). For integrability, \(\Lambda \prec 0\), so \(p\) is Gaussian with precision \(-\Lambda\). Matching moments gives \(-\Lambda = \Sigma^{-1}\), \(\lambda = -\Sigma^{-1}\mu\): \[ p^\star \;=\; \mathcal{N}(\mu, \Sigma). \]

AWGN closure. \(X \sim \mathcal{N}(\mu, \Sigma)\), \(Z \sim \mathcal{N}(0, \sigma^2 I)\), \(X + Z \sim \mathcal{N}(\mu, \Sigma + \sigma^2 I)\). Gaussian goes to Gaussian. ✓

MaxEnt-AWGN commutative diagram. \[ \begin{array}{ccc} \{p : \mathbb{E}\varphi = \eta\} & \xrightarrow{\mathrm{MaxEnt}} & p_\eta \\ \big\downarrow\, *\gamma_\sigma & & \big\downarrow\, *\gamma_\sigma \\ \{p : \mathbb{E}\varphi = \eta + \delta\eta\} & \xrightarrow{\mathrm{MaxEnt}} & p_{\eta + \delta\eta} \end{array} \] The diagram commutes iff the MaxEnt family is AWGN-closed — i.e., iff Q1 holds for \(\varphi\).

Shannon obstruction. For \(\varphi = x^4\) (one-dim, \(d = 1\)), MaxEnt gives \(p \propto e^{-\theta x^4}\) (Laplace-type). AWGN does not preserve this: \(p * \gamma_\sigma\) has different tail structure. The MaxEnt ∩ AWGN intersection is empty in this case — the Q1-obstruction for \(\deg \varphi \ge 3\).