User:Jimmy Novik/Cost Function

The cost function is a function that describes how well a machine learning algorithm approximates some function $f^{*}$, or, in other words, how close our algorithm is at achieving its main purpose of predicting the output based on a given input.

Most of the time the cost function being used is the cross-entropy between the training data and the model’s predictions.

The cost function is given by:

$J(\theta )=-\mathbb {E} _{x,y~p_{data}}\log {p}_{model}(y|x)$,

Which gets expanded to:

$J(\theta )={\frac {1}{2}}\mathbb {E} _{x,y~p_{data}}\|y-f(x;\theta )\|^{2}+const$

Extra

Mean square distance function

${\rm {MSD}}\equiv \langle |\mathbf {x} (t)-\mathbf {x_{0}} |^{2}\rangle ={\frac {1}{N}}\sum _{i=1}^{N}|\mathbf {x^{(i)}} (t)-\mathbf {x^{(i)}} (0)|^{2}$