论文标题
使用熵中的相位过渡分析训练---第一部分：一般理论
Analyzing Training Using Phase Transitions in Entropy---Part I: General Theory
论文作者
论文摘要
我们分析了由条件变量变化引起的序列的条件熵的相变。例如，在训练学习系统的参数时，发生这种过渡，因为从训练阶段到数据阶段的过渡会导致测量系统响应的条件熵导致不连续的跳跃。对于大规模系统，我们提出了一种计算通过单次训练获得的互信息结合的方法，并证明可以使用条件熵的两个衍生物之间的差异来计算这种结合。系统模型不需要参数中的高斯性或线性性，也不需要最差的噪声近似值或任何未知参数的明确估计。该模型适用于在通信，信号处理和机器学习中采用培训作为其操作的一部分的广泛算法和方法。
We analyze phase transitions in the conditional entropy of a sequence caused by a change in the conditional variables. Such transitions happen, for example, when training to learn the parameters of a system, since the transition from the training phase to the data phase causes a discontinuous jump in the conditional entropy of the measured system response. For large-scale systems, we present a method of computing a bound on the mutual information obtained with one-shot training, and show that this bound can be calculated using the difference between two derivatives of a conditional entropy. The system model does not require Gaussianity or linearity in the parameters, and does not require worst-case noise approximations or explicit estimation of any unknown parameters. The model applies to a broad range of algorithms and methods in communication, signal processing, and machine learning that employ training as part of their operation.
