oneminusp.com Computational Finance, Markets, Programming & co

Calculating Entropy the Functional Way

Previously, I wrote a short article on how to implement fold left in R. It was fairly obvious that there must be a builtin function for it in R. At the time, I just assumed it would be "reduce" or it would not exist, however the proper function name is called "Reduce" with a capital R -- as a side note, I do not really understand the naming scheme of functions in the R base library.

So here's the fairly obvious way on how to calculate Shannon's entropy in R using Reduce:

> fentropy <- function(x,y) { x + (-y * log2(y)) }
> Reduce(fentropy, c(0.5,0.5), 0)
[1] 1
> Reduce(fentropy, c(0.25,0.25,0.25,0.25), 0)
[1] 2

First for the binary case with answer 1, and then for four values uniformly distributed.

Last but not least, we could also write an entropy function the "R way" which uses its nice functions which work over vectors:

entropy <- function(ps) {
     H = -sum(ifelse(ps>0, ps * log2(ps), 0))
     return(H)
}