Softmax

Here is code to approximate log(1+exp(-x)) and log(exp(a)+exp(b)) pretty accurately and fast with C or C++ (only a 6X speedup over direct evaluation, but worth having if you do this a lot).

I wonder if I use another type of expansion whether things would be better than the Taylor expansions I’ve used. A Pade expansion (see Numerical recipes for example) does perform worse, something to do with the positions of the poles (at $\pm i\pi$). I also wonder about using fixed point maths. A key point is that if you are doing lots of these operations you’ll want to think about the order in which you do them. If you choose the order correctly you can ignore a lot of the computations. I may return to these issues some day.

Contact me if you want a ‘more liberal’ license and I’ll think about it. I guess this is a fairly simple piece of code.

Copyright: Written by and Copyright (C) 2003 Iain Murray
http://www.gatsby.ucl.ac.uk/~iam23/

This program is free software; you can redistribute it 
and/or modify it under the terms of the GNU General
Public License as published by the Free Software
Foundation; either version 2 of the License, or (at
your option) any later version.

This program is distributed in the hope that it will
be useful, but WITHOUT ANY WARRANTY; without even the
implied warranty of MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.  See the GNU General Public
License for more details.

The GNU General Public License should be included with
this file.  If not, you can view it at
http://www.gnu.org/copyleft/gpl.html
or write to the Free Software Foundation, Inc., 59
Temple Place - Suite 330, Boston, MA  02111-1307, USA.

softmax_2003_04_03.tar.gz