A comprehensive, hands-on tutorial series for understanding and applying multivariate information theory to neuroscience and complex systems. This repository provides progressive notebooks that build from foundational concepts to complete analysis pipelines using state-of-the-art Python packages.
This tutorial series takes you from information theory basics to sophisticated analyses of higher-order interactions in neural data:
- Information theory fundamentals: Entropy, mutual information, and their multivariate extensions
- Higher-order interactions: Synergy, redundancy, and emergent phenomena like the XOR problem
- Practical tools: HOI, Frites, and XGI packages for real-world analysis
- Time-resolved analysis: Dynamic information flow in neural time series
- Network representations: Hypergraphs for higher-order structures
- Complete pipelines: End-to-end analysis from raw data to scientific insights
The tutorial consists of six progressive Jupyter notebooks, each building on the previous ones:
Building the Mathematical Groundwork
Start from scratch with the fundamental concepts of information theory. Learn what entropy really measures, how to compute mutual information, and why these measures matter for understanding dependencies.
Key Topics:
- Entropy for discrete and continuous variables
- Joint and conditional entropy
- Mutual information as a measure of dependence
- Conditional mutual information
- Hands-on implementations from first principles
Prerequisites: Basic Python, NumPy, and probability theory
Discovering Synergy and Higher-Order Effects
Experience the mind-bending realization that two variables can each carry zero information individually yet contain perfect information together. This is the gateway to understanding why pairwise analysis is insufficient.
Key Topics:
- The XOR function and its information-theoretic properties
- Why Venn diagrams fail for 3+ variables
- Interaction information and its sign
- Synergy vs. redundancy: the fundamental distinction
- "Explaining away" and conditional independence
Prerequisites: Notebook 1
Scaling to Professional Analysis Tools
Transition from manual calculations to the HOI package, designed specifically for computing higher-order information measures on real datasets. Learn to detect synergistic ensembles and decompose information systematically.
Key Topics:
- O-information: detecting synergy vs. redundancy
- S-information: measuring total complexity
- Partial Information Decomposition (PID)
- Multiple estimators: Gaussian copula, KNN, binning
- Computational scaling and optimization
- Application to neural population codes
Prerequisites: Notebooks 1-2
From Static Snapshots to Dynamic Information Flow
Move beyond static analysis to track how information emerges and flows through neural systems over time. Learn rigorous statistical testing to distinguish real effects from noise.
Key Topics:
- Time-resolved mutual information
- Permutation testing for significance
- Cluster-based correction for multiple comparisons
- Dynamic functional connectivity
- Fixed-effect vs. random-effect analysis
- Gaussian Copula MI (GCMI)
- Integration with MNE-Python for M/EEG data
Prerequisites: Notebooks 1-3
From Information Measures to Network Topology
Represent higher-order interactions as network structures using hypergraphs. Learn why traditional pairwise networks are insufficient and how hypergraphs naturally capture the structures discovered by information theory.
Key Topics:
- Hypergraphs: edges connecting multiple nodes simultaneously
- Creating and manipulating hypergraphs with XGI
- Mapping HOI results to hyperedges
- Hypergraph statistics: degree, clustering, connectivity
- Visualizing higher-order structures
- Temporal hypergraph sequences
- Network topology in neuroscience context
Prerequisites: Notebooks 1-4
From Raw Data to Scientific Insight
The synthesis: a complete analysis pipeline integrating HOI, Frites, and XGI to answer a realistic neuroscience question about visual decision-making. This represents a publication-ready workflow you can adapt for your own research.
Research Question: How do hierarchical brain regions (V1 → MT → LIP → PFC) interact synergistically to transform sensory evidence into decisions?
Analysis Components:
- When does information flow through the hierarchy? (Frites)
- Which ensembles show synergistic coding? (HOI)
- How is the network organized topologically? (XGI)
- Does synergy emerge early (sensory) or late (decision)?
Prerequisites: Notebooks 1-5
- Clone the repository:
git clone https://github.com/lordgrilo/cnww-hoi.git
cd cnww-hoi- Create a virtual environment (recommended):
python -m venv venv
source venv/bin/activate # On Windows: venv\Scripts\activate- Install dependencies:
pip install -r requirements.txtThe main packages you'll be using:
numpy,scipy,matplotlib,seaborn: Scientific computing and visualizationhoi: Higher-Order Interactions packagefrites: Framework for Information Theoretical analysisxgi: Comple(X) Group Interactions (hypergraphs)jax,jaxlib: Fast numerical computation (backend for HOI)mne: Neurophysiological data analysis (used by Frites)
- Launch Jupyter:
jupyter notebookStart with 01_information_theory_foundations.ipynb and progress sequentially.
If you're already familiar with information theory, you can jump to:
- Notebook 3 for HOI package usage
- Notebook 4 for time-resolved analysis with Frites
- Notebook 5 for hypergraph network representations
- Notebook 6 for the complete integration pipeline
Recommended time: 10-15 hours total
- Week 1: Notebooks 1-2 (foundations and intuition)
- Week 2: Notebook 3 (HOI package)
- Week 3: Notebooks 4-5 (Frites and XGI)
- Week 4: Notebook 6 (integration)
Recommended time: 3-5 hours
- Skim Notebooks 1-2 for notation
- Focus on Notebooks 3-6 for practical tools
- Adapt Notebook 6 pipeline to your data
These techniques are applicable to:
Neuroscience:
- Neural population coding and ensemble analysis
- Brain network connectivity (fMRI, M/EEG, electrophysiology)
- Information routing and cognitive processing
- Consciousness and integrated information
Complex Systems:
- Gene regulatory networks
- Ecological community interactions
- Social network dynamics
- Climate system dependencies
Machine Learning:
- Feature interaction analysis
- Model interpretability
- Emergent computational properties
- Deep network representations
- Python: 3.8 or higher
- RAM: 8GB minimum (16GB recommended for large datasets)
- GPU: Optional but recommended for HOI on large datasets (JAX supports CUDA)
- OS: Linux, macOS, or Windows
See requirements.txt for specific versions. Key compatibility notes:
- JAX installation varies by platform (CPU vs GPU)
- Frites requires MNE-Python >= 0.23
- XGI works with NetworkX but extends it
All notebooks use simulated data generated with fixed random seeds for reproducibility. The simulation parameters are based on realistic neuroscience scenarios, allowing you to:
- Understand exact ground truth
- Validate methods systematically
- Explore parameter spaces
- Adapt to your own data
Contributions are welcome! Areas for contribution:
- Additional examples from other domains
- Improved visualizations
- Performance optimizations
- Bug fixes and clarifications
- Translations
Please open an issue first to discuss major changes.
Information Theory:
- Shannon, C. E. (1948). A Mathematical Theory of Communication. Bell System Technical Journal.
- Cover, T. M., & Thomas, J. A. (2006). Elements of Information Theory. Wiley.
Higher-Order Interactions:
- Rosas, F. E., et al. (2019). Quantifying high-order interdependencies via multivariate extensions of the mutual information. Physical Review E.
- Williams, P. L., & Beer, R. D. (2010). Nonnegative decomposition of multivariate information. arXiv:1004.2515.
- Timme, N., et al. (2014). Synergy, redundancy, and multivariate information measures: an experimentalist's perspective. Journal of Computational Neuroscience.
Neuroscience Applications:
- Schneidman, E., et al. (2006). Weak pairwise correlations imply strongly correlated network states. Nature.
- Panzeri, S., et al. (2017). Cracking the neural code for sensory perception by combining statistics and interventions. Neuron.
- Stringer, C., et al. (2019). High-dimensional geometry of population responses in visual cortex. Nature.
- HOI: https://hoi.readthedocs.io/
- Frites: https://brainets.github.io/frites/
- XGI: https://xgi.readthedocs.io/
- JAX: https://jax.readthedocs.io/
- MNE-Python: https://mne.tools/
- Issues: Please use GitHub Issues for bugs and questions
- Discussions: GitHub Discussions for general questions and sharing results
- Email: [Your contact email]
This project is licensed under the MIT License - see the LICENSE file for details.
This tutorial integrates three excellent open-source packages:
- HOI by Etienne Combrisson and team
- Frites by Etienne Combrisson and team
- XGI by the Comple(X) Group Interactions team
Thanks to the broader neuroscience and complex systems communities for developing the theoretical foundations these tools implement.
Happy Learning! 🧠💡
Questions? Found a bug? Have suggestions? Open an issue or start a discussion!