matlab - Backpropagation error : conceptual or programing? -

i wrote following backpropagation algorithm model 2 input identity function

clc % clear  nh = 3;              % neurons in hidden layer ni = 2;              % neurons in input layer  eta = .001;          % learning rate traningsize =100;  x=1:traningsize      input   = rand(traningsize,ni);     test    = input; end      nk = size(test,2);    % neurons in output layer  b1 = rand(1,nh);%+ .5; b2 = rand(1,nk);%- .5; w1 = rand(nh,ni) + .5; w2 = rand(nk,nh) - .5;  figure hold on;  iter = 1 :5000     errsq = 0;     x = 1: traningsize          a0      = input(x,:);         ex      = test(x,:);          [a1, a2]= feedforward(a0,w1,w2,b1,b2);          del2    = (a2-ex) .* (1-a2) .* (a2);         del1    = (del2 * w2) .* (1-a1) .* (a1);          delb2   = del2;         delb1   = del1;          delw2   = zeros(nk,nh);         = 1:nh             j = 1:nk                 delw2   = a1(i) * del2(j);             end         end         = 1:ni             j = 1:nh                 delw1   = a0(i) * del1(j);             end         end          b2 = b2 - eta * delb2;         b1 = b1 - eta * delb1;          w2 = w2 - eta * delw2;         w1 = w1 - eta * delw1;          errsq = errsq + sum(a2-ex) .* sum(a2-ex);      end  cost = errsq /(2 * traningsize); plot(iter,cost,'o');  if cost < 0.005     cost     break end  end     cost 

the feedforward function :

function [a1, a2]  = feedforward(a0,w1,w2,b1,b2)      z1      = a0 * w1' + b1;     a1      = sig(z1);     z2      = a1 * w2' + b2;     a2      = sig(z2);   end 

the cost function plot

now messing ?

is programmatic error have escaped notice ? or implementing algorithm wrongly?

when test resulting weights calculated cost trained results wrong

blue > expected output ; red > output of neural network

also why cost value rises before decreasing (like in figure 1)

as turn out needed correct programing bug;

    delw2   = zeros(nk,nh);     = 1:nh         j = 1:nk             delw2(j,i)   = a1(i) * del2(j);   % forgot index delw2         end     end     delw1   = zeros(nh,ni);                   % initilize delwi1 (although optional)     = 1:ni         j = 1:nh             delw1(j,i)   = a0(i) * del1(j);   % forgot index delw1         end     end 

and eliminate sigmoid function output layer. ie make linear output layer acceptable results.

i'm not sure why linear output layer strictly required though , comments on that.