Artificial Life Course

P(NOT a) = 1 - P(a)

P(a OR b) = P(a) + P(b) - P(a AND b)

P(a AND b) = P(a,b)

P(a|b) = P(a AND b) / P(b)

P(a AND b) = P(a|b) * P(b) = P(b|a) * P(a)

P(a|b) = P(a), P(b|a) = P(b)

S = k log W: Entropy = Boltzmann\\u2019s-constant * log( function of # of possible micro-states )
I = K log \\u03a9: Entropy = constant (usually dropped) * log( function of # of possible micro-states )

        n
H = - K \\u2211 pi log pi
       i=1

K is a positive constant

e.g. for two possibilities:

H = - (p log p + q log q)

H(x,y) = - \\u2211 p(xi,yj) log p(xi,yj)
          i,j

H(x,y) \\u2264 H(x) + H(y)

H(y|x) = - \\u2211 p(xi) p(yj|xi) log p(yj|xi)
H(y|x) = - \\u2211 p(xi,yj) log p(yj|xi)

H(y) \\u2265 H(y|x)

HMax = log n

Ho(x) = H(x) / HMax

I(x,y) = \\u2211 p(xi,yj) log [ p(xi,yj) / p(xi)p(yj) ]

I(x,y) = H(x) - H(x|y) = H(y) - H(y|x)
I(x,y) >= 0
I(x,y) = I(y,x)

Probability and Uncertainty:
marginal: H(x) = - \\u2211 p(xi) log p(xi)
joint: H(x,y) = - \\u2211 p(xi,yj) log p(xi,yj)
conditional: H(y|x) = - \\u2211 p(xi,yj) log p(yj|xi)
mutual: I(x,y) = \\u2211 p(xi,yj) log [ p(xi,yj) / p(xi)p(yj) ]

von der Malsburg (1973) neuron activation rule:
dHk(t)/dt = \\u2212\\u03b1kHk(t) + \\u2211 plkHl(t) + \\u2211 sikAi(t)
           activation    lateral    connections from
              decay    connections  "retinal" layer

The Shannon information rate (the average information transmitted from L to M per message):
R = \\u2211\\u2211 P(L,M) log[ P(L,M) / P(L)P(M) ] = H(L) - H(L|M) = H(M) - H(M|L)

 - P(L) is the probability of the input message being L
 - P(M) is the probability of the output message being M
 - P(L,M) is the joint probability of the input being L and the output being M

For the case in which P(L) is uniform over L:
R = - \\u2211M [A(M)/A] log[A(M)/A] - (\\u03b3\\u03bd/A) \\u2211M s(M)

 - P(M) = A(M) / A
 - P(M|L) log P(M|L) is zero except for borderline L
 - \\u03b3 is a positive number whose value depends on the details of the noise process, and which determines P(M|L) for borderline L as a function of distance from the border

The linear response rule:
M = a1 + \\u2211Lc

Potential range of Hebb-type rules:
\\u2206ci = a2LiM + a3Li + a4M + a5

a2-5 are arbitrary constants (a2 > 0)

1. (Leaky) Integrate & Fire (I&F):
v\\u2019 = I + a - b v
if v \\u2265 vthresh then fire and reset v = c

Four floating-point operations, plus one compare, per ms.
3 of the 20 identified behaviors.

2. Integrate & Fire with Adaptation (I&FA)
v\\u2019 = I + a - b v + g (d - v)
if v \\u2265 vthresh then fire and reset v = c
g\\u2019 = (e \\u03b4(t) - g) / \\u03c4

10 floating-point operations, plus compare, per ms
5 of 20 identified behaviors.

3. Integrate & Fire or Burst (I&FB)
v\\u2019 = I + a - b v + g H(v - vh) h (vT - v)
if v \\u2265 vthresh then fire and reset v = c
h\\u2019 = { -h / \\u03c4- , if v > vh
(1 - h) / \\u03c4+ , if v < vh

Between 9 and 13 operations per ms (depending on v)
9 to 11 behaviors still unmodeled.

4. Resonate & Fire (R&F)
z\\u2019 = I + (b + i\\u03c9) z
if Im(z) \\u2265 athresh then fire and reset z = z0(z)
10 operations, plus compare, per ms
9 behaviors unmodeled

5. Quadratic Integrate & Fire (QI&F)
v\\u2019 = I + a (v - vrest) (v - vthresh)
if v \\u2265 vpeak then fire and reset v = vreset
7 operations, plus compare, per ms
15 of the 20 behaviors

6. Izhikevich spiking model (ISM)
v\\u2019 = 0.04 v2 + 5 v + 140 - u + I
u\\u2019 = a (bv - u)
if v \\u2265 30 mv then has fired, so reset:
v = c
u = u + d

13 floating point operations, plus compare, per ms in
most regimes
all 20 behaviors with different parameters

7. FitzHugh-Nagumo
v\\u2019 = a + b v + c v2 + d v3 - u
u\\u2019 = \\u03b5 (ev - u)

18 floating point operations, per step
10 of the defined 20

8. Hindmarsh-Rose
v\\u2019 = u - F(v) + I - w
u\\u2019 = G(v) - u
w\\u2019 = (H(v) - w) / \\u03c4

30 floating point operations, per step
17 of the 20 behaviors

9. Morris-Lecar
60 floating-point ops per step
6 to 8 behaviors unmodeled

10. Wilson polynomial neurons

45 floating-point ops per step, but requires
Can theoretically produce all 20 behaviors

11. Hodgkin-Huxley
120 floating-point ops per step
all 20 behaviors

I(X) = \\u03a3H{xi} \\u2212 H(X) >= 0
MI(x1,x2) = H(x1) + H(x2) \\u2013 H(x1x2)
H{xi} is the entropy of the ith individual element xi