Information Theory
arXiv:cs.IT
Covers theoretical and experimental aspects of information theory and coding. Includes source coding, channel coding, data compression, and cryptographic protocols.
Trending in Information Theory
A Dual Approach for Hierarchical Information-Theoretic Tree Abstractions
In this paper, we consider establishing a formal connection between two distinct tree-abstraction problems inspired by the information-bottleneck (IB) method. Specifically, we consider the hard- and soft-constrained formulations that have recently appeared in the literature to determine the conditions for which the two approaches are equivalent. Our analysis leverages concepts from Lagrangian relaxation and duality theory to relate the dual function of the hard-constrained problem to the Q-function employed in Q-tree search and shows the connection between tree phase transitions and solutions to the dual problem obtained by exploiting the problem structure. An algorithm is proposed that employs knowledge of the tree phase transitions to find a setting of the dual variable that solves the dual problem. Furthermore, we present an alternative approach to select the dual variable that leverages the integer programming formulation of the hard-constrained problem and the strong duality of linear programming. To obtain a linear program, we establish that a relaxation of the integer programming formulation of the hard-constrained tree-search problem has the integrality property by showing that the program constraint matrix is totally unimodular. Empirical results that corroborate the theoretical developments are presented and discussed throughout.
2512.01985
Dec 2025Information Theory
Forget BIT, It is All about TOKEN: Towards Semantic Information Theory for LLMs
Large language models (LLMs) have demonstrated remarkable capabilities in numerous real-world applications. While the vast majority of research conducted from an experimental perspective is progressing rapidly, it demands substantial computational power, data, and other resources. Therefore, how to open the black-box of LLMs from a theoretical standpoint has become a critical challenge. This paper takes the theory of rate-distortion function, directed information, and Granger causality as its starting point to investigate the information-theoretic principles behind LLMs, leading to the development of semantic information theory for LLMs, where the fundamental unit is token, rather than bits that lacks any semantic meaning. By defining the probabilistic model of LLMs, we discuss structure-agnostic information-theoretic measures, such as the directed rate-distortion function in pre-training, the directed rate-reward function in post-training, and the semantic information flow in inference phase. This paper also delves deeply into the theory of token-level semantic embedding and the information-theoretically optimal vectorization method. Thereafter, we propose a general definition of autoregression LLM, where the Transformer architecture and its performance such as ELBO, generalization error bound, memory capacity, and semantic information measures can be derived theoretically. Other architectures, such as Mamba/Mamba2 and LLaDA, are also discussed in our framework. Consequently, this paper provides a theoretical framework for understanding LLMs from the perspective of semantic information theory, which also offers the necessary theoretical tools for further in-depth research.
2511.012021
Nov 2025Information Theory
