Infotheory, written in C++, and usable in Python as well, is a software package to perform information theoretic analysis on multivariate data. This package implements traditional as well as more recent measures that arise from multivariate extensions to information theory, specifically

1. Entropy [1]
2. Mutual Information [2]
3. Partial Information Decomposition [3]
• Unique Information
• Redundant Information
• Synergistic Information

The main highlights of this package include:

• written in C++ for efficiency
• ease of use via python bindings and compatibility with numpy
• an API that allows adding the data once to then be able to perform various analyses across different sub-spaces of the dataset very quickly
• use of sparse data structures that work well with high-dimensional data
• user-controllable estimation of data distribution using averaged shifted histograms [4]
• flexibilty to specify binning allows proper estimation of information measures between continuous and discrete variables
• perform PI-decomposition over 3 (two sources and 1 target) and 4 (three variables and 1 target) variables

The package is available open-source on GitHub. While the C++ headers should function on any platform, in its current release the python package has been tested on Linux and MacOS.

#### Citation

If you use this package, please cite preprint available at: https://arxiv.org/abs/1907.02339

Candadai, M., & Izquierdo, E. J. (2019). infotheory: A C++/Python package for multivariate information theoretic analysis. arXiv preprint arXiv:1907.02339.

@article{candadai2019infotheory,
title={infotheory: A C++/Python package for multivariate information theoretic analysis},
journal={arXiv preprint arXiv:1907.02339},
year={2019}
}


### Installation⇧

Python
pip install --upgrade infotheory

In MacOS, you might have to add two environment variables using
export CXXFLAGS="-mmacosx-version-min=10.9"
export LDFLAGS="-mmacosx-version-min=10.9"

C++
#include "Infotools.h"
Since Infotools accepts TVector arrays as arguments, you will also need to include VectorMatrix.h, which contains the TVector class and use objects of TVector to add data and invoke metrics.
#include "VectorMatrix.h"

### Usage⇧

Examples and benchmarks are available here.

Infotheory is a package that has been designed to make it easy and intuitive to use. The steps involved in using the package in C++ as well as Python are as follows:

1. Create the object
The following are required to create and object and setup the analyses - $dims$, the total dimensionality of all variables combined; $nreps$, the number of shifted binnings that should be averaged across;. $nreps=0$ does no shifted binning and is ideal for discrete valued data and any non-negative value of nreps produces that many shifts on each side to average over. Once this data is available, objects in python can be created as follows

import infotheory
dims = 3
nreps = 0
it = infotheory.InfoTools(dims, nreps)

2. Specify how the data should be binned
Since the package uses a sparse representation, it is required that the bin boundaries are pre-specified before adding the data. This could either be done manually or the user could ask the package to create equal width bins by providing appropriate data.

a. To specify equal width bins, the following arguments are required: $nbins$, a list specifying the number of bins for data along each dimension; $maxs$, a list specifying the maximum value of the data along each dimension; and finally $mins$, a list specifying the minimum value of the data along each dimension

nbins = [2, 3, 3] # for each dim
mins = [0, 0, 0] # for each dim
maxs = [1, 1, 1] # for each dim
it.set_equal_interval_binning(nbins, mins, maxs)

b. Alternatively one can explicitly specify the bin boundaries by providing the left boundary for each bin along each dimension of the data.

boundaries = [[0,0.5], [0, 0.5, 1], [0, 0.5, 1]]
it.set_bin_boundaries(boundaries)

Data is added either one at a time, or all at a time, such that each datapoint is a vector of length $dims$ i.e. a single concatenated vector of all random variables. When adding multiple data points at once, a list of lists will need to be used.

for _ in range(1000): # adding 1000 random points
# Alternatively


4. Invoking information theoretic tool
Once all data has been added, the different information theoretic metrics that are available as part of the package can be invoked. Since each data point was added as a singe concatenated vector, the different dimensions need to be identified while invoking different metrics. This is done by passing an argument along with the call which will be a list of length $dims$, and contain an ID to match dimensions belonging to the same variable. For instance, with $dims=3$, if the first two dimensions represent the first variable and the next represents the second variable, this list, $var\_ids=[0,0,1]$. This allows for the flexibility of analyzing sub-spaces of the data by using $-1$ to ignore dimensions for a particular analysis. For the same $dims$ setup mentioned here,

mi = it.mutual_info([0, 0, 1])

measures mutual information between two 2D random variables that are available in the first two and the third dimension of the dataset respectively. With the same dataset, one can then go on to invoke

mi = it.mutual_info([0, -1, 1])

to measure the mutual information between just the first dimension of the first random variable and the second. As a rule, dimensions with $-1$ are ignored. As one might have guessed, for entropy only the $0$ values are paid attention to and for PID the list requires $0, 1$ and $2$ to identify the three variables in the dataset, where $0$ is always the variable about which information is estimated.

This final step can be repeated for all analyses and this whole process can be started over for a new dataset. Below, you can find sample code in Python on how to use this package to estimate mutual information between two 2D random variables. See this page for a detailed account of the different measures available and sample code for all of them.

API:

A demo in C++ for the same program is available here.

### Contact⇧

Having trouble with Infotheory? Want to contribute? Contact Madhavun at madvncv [at] gmail.com