Keynote Speakers

A recent line of work has emerged that uses machine learning / AI to learn the encoders and/or decoders of error-correcting codes. This is often done for AWGN channels in both the forward error correction and feedback settings. Do these codes learn dramatically new non-linear codes? When or why might they outperform model-based or algorithmically designed codes? This talk will survey recent trends in learned error-correcting encoders, present open problems, and outline some of my work on “interpreting’’ a few such codes.  Our interpretations can sometimes approximate the encoders and/or decoder black boxes, offer insights into training, lead to orders-of-magnitude reductions in the number of involved parameters, and provide an understanding of how error correction is performed.  However, can we actually learn something new from these learned codes?

The problem of designing codes for deletion- and permutation-error correction has received renewed interest due to applications in DNA-based data storage systems that use nanopore sequencers as readout  platforms. In almost all instances, these errors are assumed to be imposed independently of each other and of the sequence context. Such assumptions are not valid in practice because nanopore errors tend to occur within specific contexts and are caused by system implementation issues. We introduce two new problems to address these issues, pertaining to contextual nanopore deletion-error correction and intra-substring permutation error-correction. In the former case, we assume that single deterministic deletions follow (complete) runlengths of length exceeding some predefined threshold. Our results in this case include both bounds on the rate of the code and code constructions, based on forbidden-substring encoding, string correlations and generalizations of VT codes. In the latter case, we assume that symbols in one or more nonoverlapping substrings of a string can undergo arbitrary or cyclic permutations. The results highlight the redundancy reductions of permutation-correction codes compared to well-known burst correction codes such as Fire codes, the proofs of which rely on probability concentration arguments.

Diffusion models have become a dominant paradigm in generative modeling. Their connection to information theory has been noted with increasing frequency since their introduction in 2015 and has recently come into sharper focus. They are typically described in terms of a forward-backward paradigm, where a forward process dissipates information by passing a sample through a family of noisy channels, and a backward process reverses this evolution in distribution via Bayes’ rule. In this talk, I will connect this perspective to classical information-theoretic conservation laws such as EXIT, GEXIT, and I-MMSE, which were developed by our community in the 2000s. In the ideal setting, with known distributions and optimal estimators, these conservation laws hold exactly and lead to equivalent information measures across a broad class of degraded channel paths. In the data-driven setting, the relevant tools are conservation laws for mismatched estimation, and performance is limited by estimation error stemming from limited training data and restricted model capacity. The interaction between the channel family, approximation error, and overall performance then becomes more subtle. Our viewpoint unifies masked and Gaussian diffusion within a common framework and clarifies what happens in practical regimes, where learned denoisers are not matched to the true posteriors. The resulting picture helps explain why some channel paths are better than others in practice, even when they are equivalent in the ideal setting.

Modern AI is built on a single computational primitive — matrix multiplication — which consumes the majority of its time and energy. One way to reduce this cost is to lower the precision of the operands, decreasing both communication and arithmetic overhead. In this talk I will describe the information-theoretic tradeoffs that arise in quantized matrix multiplication, and show how information-theoretic insights enable the design of more efficient algorithms for quantizing real-world LLMs down to 2-4 bits per parameter without significant loss of performance.