0

I have a set of points that lie on a n-dimensional sphere (hence, they have norm one). I need to compute the entropy of this set. Is there a tool which allows me to do something like that? Otherwise, how can I do it?

My guess is that, since the entropy has to be computed from a probability distribution, I need to divide this problem in 2:

1) Some function which takes the set of points and outputs a probability distribution which approximates this set.

2) A function which takes the probability distribution (or its density) and gives me its entropy.

All of this while knowing that the points lie on the n-sphere. Thanks!

2 Answers 2

1

I am not sure how much data you exactly have. But based on your explanation, I have the following answer. I will suggest to use Kernel Density Estimation to estimate the pdf. This module is available in scipy and scikit-learn.

In scikit-learn, you have more options to choose for kernel to be used in KDE.

import numpy as np
from sklearn.neighbors.kde import KernelDensity

### create data ###
sample_count = 1000
n = 6
data = np.random.randn(sample_count, n)
data_norm = np.sqrt(np.sum(data*data, axis=1))
data = data/data_norm[:, None]   # Normalized data to be on unit sphere


## estimate pdf using KDE with gaussian kernel
kde = KernelDensity(kernel='gaussian', bandwidth=0.2).fit(data)

log_p = kde.score_samples(data)  # returns log(p) of data sample
p = np.exp(lop_p)                # estimate p of data sample
entropy = -np.sum(p*lop_p)       # evaluate entropy
0

I'm not sure but I guess a more efficient way would be to replace np.sum(p*np.log(p)) by -np.mean(log(p)). You should take advantage of your data and perform a monte carlo integration, my implementation would be :

def get_entropy(X):
   if len(X.shape)==1:
       X=X.reshape(-1,1)
   params = {'bandwidth': np.logspace(-10, 10, 20)}
   gs = GridSearchCV(KernelDensity(), params)
   gs.fit(X)
   kde=gs.best_estimator_
   log_probs=kde.score_samples(X)
   return -np.mean(log_probs)

I tested it on several common distributions and I'm always quite close to the analytical derivation of the entropy if enough samples are provided.

I'd say the previous answer is wrong, let's say we consider our dataset times 2 (each sample appear twice). Then the estimated entropy should not move but the previous formula would say the entropy has been doubled.

Averaging log(p) over the dataset is a MonteCarlo integration method, summing p*log(p) over the dataset isn't. Correct me if I'm wrong.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.