Step04-Numerical Difference

The purpose of implementing Variable and Function class was to compute differential automatically. So, what is differential?

Differential is simply a gradient, which is a change in an extremely short time interval. We can describe defferential as a following formula:

$f^{'}=\displaystyle \lim_{h \to 0}\frac{f(x+h)-f(x)}{h}$

Then how could we implement numerical difference as a code? We can replace the variable h(which is an extremely small number) with 1e-4. Also instead of forward and backward difference, it's better to use central difference because of relatively low error. (증명 정리하기) Next code is a function that computes central difference.

def numerical_diff(f, x, eps=1e-4):#eps=h

    x0=Variable(x.data-eps)#(x-h)

    x1=Variable(x.data+eps)#(x+h)

    y0=f(x0)#f(x-h)

    y1=f(x1)#f(x+h)

    return (y1.data-y0.data)/(2*eps)#central difference formula (f(x+h)-f(x-h))/2h

Let's caclulate the central difference of Square function.

f=Square()#f(x)=x^2

x=Variable(np.array(2.0))

dy=numerical_diff(f,x)

print(dy)

The result is 4.000000000004, which includes a very small amount of error, 0.000000000004.

Also, this function can calculate a central difference of composite function:

def f(x):

    A=Square()

    B=Exp()

    C=Square()

    return C(B(A(x)))

x=Variable(np.array(0.5))

dy=numerical_diff(f,x)

print(dy)

#result: 3.2974426293330694

Although the error is tiny here, numerical difference still has some limitations. Error might be increase depending on which calculation it is executing, and the amount of computation can also increase dramatically.