basic neural network ,problem in training

62eb91de06000acaf116dc4bdfba0157
0
simshon 101 Sep 25, 2011 at 22:42

after reading some articles about neural network(back-propagation) i try to write a simple neural network by myself.

ive decided XOR neural-network, my problem is when i am trying to train the network, if i use only one example to train the network,lets say 1,1,0(as input1,input2,targetOutput). after 500 trains +- the network answer 0.05. but if im trying more then one example (lets say 2 different or all the 4 possibilities) the network aims to 0.5 as output :( i searched in google for my mistakes with no results :S ill try to give as much details as i can to help find what wrong:

-ive tried networks with 2,2,1 and 2,4,1 (inputlayer,hiddenlayer,outputlayer).

-the output for every neural defined by:

double input = 0.0;
for (int n = 0; n < layers.Count; n++)
input += layers[n].Output * weights[n];
while ‘i’ is the current layer and weight are all the weights from the previous layer.

-the last layer(output layer) error is defined by:

value*(1-value)*(targetvalue-value);
while ‘value’ is the neural output and ‘targetvalue’ is the target output for the current neural.

-the error for the others neurals define by:

foreach neural in the nextlayer
sum+=neural.value*currentneural.weights[neural];

myerror=myoutput*(1-myoutput)*sum;

-all the weights in the network are adapt by this formula(the weight from neural -> neural 2)

weight+=LearnRate*neural.myvalue*neural2.error;
while LearnRate is the nework learning rate(defined 0.25 at my network). -the biasweight for each neural is defined by:

bias+=LearnRate*neural.myerror*neural.Bias;
bias is const value=1.

that pretty much all i can detail, as i said the output aim to be 0.5 with different training examples :(

ive upload my project here:

http://www.multiupload.com/G68E57N4BM .

im realy stuck here,dunno where my mistake after checking the code again and again:(

thank you very very much for your help:)

12 Replies

Please log in or register to post a reply.

6eaf0e08fe36b2c23ca096562dd7a8b7
0
__________Smile_ 101 Sep 26, 2011 at 14:46

This is definitely bad:

//Calc OutputFor InputLayer
for (int i = 0; i < layers[0].Count; i++)
    layers[0][i].Output = sigmoid(layers[0][i].Input);

Never transform input values before feeding it into first neuron. While it may works for small inputs around 0, your whole network lose great deal of adaptivity. Also there are unnecessary redundancy here:

//Add Bias
layers[i][j].Input += layers[i][j].Bias * weights[i][j, 0];
//layers[i][j].Input += layers[i][j].Bias;

Remove layers[j].Bias altogether and use trainable weights[j, 0] only. Also “+=” instead of “=” scares me. Do you have nonzero values from previous run and use it? More clear code will be
**

layers[i][j].Input = weights[i][j, 0];

As for learning algorithm, backpropagation is not the best method for setting neuron weights. There are more powerful algorithms, but there are no best approach for all cases. Try to play with learning constants, start from different set of weights and then choose the best.

62eb91de06000acaf116dc4bdfba0157
0
simshon 101 Sep 26, 2011 at 16:52

1)
for every neuron there is input,output and bias(which is 1).
input is the sum of all the previous layers neurons(their output*weight with this neuron).
then the output of the neuron is the value of activation(input),
where is the problem here?

2)about the bias, as i understand the network need bias in case all the inputs are 0,then the network wont train without non-zero value(which is the bias).

3)about the ‘+=’,before each train im set all the input&output of all the neurons to 0.0.

tyvm for your help:)

6eaf0e08fe36b2c23ca096562dd7a8b7
0
__________Smile_ 101 Sep 26, 2011 at 17:29

For every neuron must hold

O = F(sum(I*w)+b),

but for first layer you have

O = F(I+b).

Effectively you preprocess input signal with sigmoid and lessen dynamic range as a result. You cannot compensate lack of sensitivity with learning.

62eb91de06000acaf116dc4bdfba0157
0
simshon 101 Sep 26, 2011 at 20:02

@’

‘]For every neuron must hold

O = F(sum(I*w)+b),

but for first layer you have

O = F(I+b).

Effectively you preprocess input signal with sigmoid and lessen dynamic range as a result. You cannot compensate lack of sensitivity with learning.

i dont understand your last sentence:S

but i output dont need to be like that :
o=sigmoid(sum(I*w)+b*bw)

while bw is the bias weight?

6eaf0e08fe36b2c23ca096562dd7a8b7
0
__________Smile_ 101 Sep 26, 2011 at 21:14

@simshon

i dont understand your last sentence:S

Well, simply replace

SetInputs(input);
//Calc OutputFor InputLayer
for (int i = 0; i < layers[0].Count; i++)
    layers[0][i].Output = sigmoid(layers[0][i].Input);

by

for (int i = 0; i < tempinput.Count; i++)
    layers[0][i].Output = layers[0][i].Input = tempinput[i];

That can improve learning ability in certain cases.

62eb91de06000acaf116dc4bdfba0157
0
simshon 101 Sep 26, 2011 at 21:25

ive already try to not use the activation function on the input class,but the training proccess is still wrong somehow:S

6eaf0e08fe36b2c23ca096562dd7a8b7
0
__________Smile_ 101 Sep 26, 2011 at 21:42

Hmm…why do you use O*(1-O)*E for error correction? This may be wrong, try E/(O*(1-O)) or simply E.

62eb91de06000acaf116dc4bdfba0157
0
simshon 101 Sep 26, 2011 at 23:55

i just followed few tutorials formulas,such as:

http://www.codeproject.com/KB/recipes/BP.aspx.

what is E in your:E/(o*(1-o)?

6eaf0e08fe36b2c23ca096562dd7a8b7
0
__________Smile_ 101 Sep 27, 2011 at 13:12

By E I mean (target - output) or sum(weight*error) or delta in tutorial.

O*(1-O)*E doesn’t make sense to me: it slows learning on tails of sigmoid where sensitivity already low. Mathematically justified will be E/(O*(1-O)) cause [sigmoid(x)]’ = -sigmoid(x)*(1-sigmoid(x)). I think tutorial author simply made mistake.

62eb91de06000acaf116dc4bdfba0157
0
simshon 101 Sep 27, 2011 at 21:33

stil the output aims to be 0.5:S

6eaf0e08fe36b2c23ca096562dd7a8b7
0
__________Smile_ 101 Sep 27, 2011 at 22:38

Maybe you don’t have enough neurons?

62eb91de06000acaf116dc4bdfba0157
0
simshon 101 Sep 29, 2011 at 22:14

ive try with more layers and more then 2 and even 4 in each layer(hiddens)