-
Notifications
You must be signed in to change notification settings - Fork 9
Layers.c
In the current version of learning lab (30/1/2019) there are 4 types of different layers you can build:
- Fully-Connected layers
- Convolutional layers
- Residual layers
- Batch normalized layers
In this type of layer you have some input neurons, some output neurons and output and input neurons are connected by weights. Each output neuron is connected to each input neuron, each connection is a weight saved as a float number. Furthermore each output neuron has a bias always as float number.
fcl* fully_connected(int input, int output, int layer, int dropout_flag, int activation_flag, float dropout_threshold)
int input:= is the number of input neurons of this layer
int output:= is the number of output neurons of this layer
int layer:= is a index => [0,N] that indicates which position is occupied by this layer
For example:
layer = 0 this layer is the first layer of the network, layer = n this layer is in the position n +1 in the network.
int dropout_flag:= This parameter can be set as NO_DROPOUT, DROPOUT, DROPOUT_TEST.
If it is set to DROPOUT, then after this layer there will be not a dropout, if is set to DROPOUT then during the training there is a dropout_threshold% (last parameter of fully-connected) that a neuron will not be considered, if it is set to DROPOUT_TEST then during you test phase the outputs coming from this layer will be shifted by dropout_threshold.
For example:
I used a threshold of 0.3 during the training. I will set the dropout flag to DROPOUT. Then during the test i will create a new threshold parameter 1-0.3 = 0.7. So, during the test phase i will set the dropout flag to DROPOUT_TEST and dropout threshold to 0.7.
int activation_flag:= indicates what kind of activation you want to apply. These are the activation that can be applied for now:
#define NO_ACTIVATION 0
#define SIGMOID 1
#define RELU 2
#define SOFTMAX 3
#define TANH 4
#define LEAKY_RELU 5
float dropout_threshold:= is the threshold used during the dropout.
typedef struct fcl { //fully-connected-layers
int input,output,layer,dropout_flag;//dropout flag = 1 if dropout must be applied
int activation_flag; // activation flag = 0 -> no activation, flag = 1 -> sigmoid, = 2 -> relu, = 3 -> softmax, 4->tanhh
float* weights;// output*input
float* d_weights;// output*input
float* d1_weights;// output*input
float* d2_weights;// output*input
float* biases; //output
float* d_biases; //output
float* d1_biases; //output
float* d2_biases; //output
float* pre_activation; //output
float* post_activation; //output
float* dropout_mask;//output
float* dropout_temp;//output
float* temp;//output
float* temp3;//output
float* temp2;//input
float* error2;//input
float dropout_threshold;
} fcl;
You can access to each parameter with fcl* your_name_fcl_structure->param, the comments next to each vector parameter indicate the dimension of that vector.
The weights are the effective weights of the layer, same for the biases, each other vector is used for the feed forward, backpropagation and gradient descent.
If you want to access to the result after the activation function it is stored in post_activation vector, otherwise if you want to access to the result before the activation function it is stored in pre_activation vector
If you want to deallocate the space occupied by the fully connected layer you can do that with
void free_fully_connected(fcl* f)
If you want to save your fully connected layer in a .bin file, you can do that with:
void save_fcl(fcl* f, int n)
The f layer will be saved on a file named as n.bin where n is a number that will be converted in string format. To Load you fcl* layer:
fcl* load_fcl(FILE* fr)
This kind of layers are very adaptable layers. According to your flags it can compute the convolution or not, the local response normalization or not, the pooling or not. after the convolution can be applied padding or not, after the pooling can be applied padding or not.
cl* convolutional(int channels, int input_rows, int input_cols, int kernel_rows, int kernel_cols, int n_kernels, int stride1_rows, int stride1_cols, int padding1_rows, int padding1_cols, int stride2_rows, int stride2_cols, int padding2_rows, int padding2_cols, int pooling_rows, int pooling_cols, int normalization_flag, int activation_flag, int pooling_flag, int layer, int convolutional_flag)
int channels:= it indicates the channels of the input layer respect to this layer
int input_rows:= indicates the rows of the input layer respect to this layer
int input_cols:= indicates the columns of the input layer respect to this layer
int kernel_rows:= is the number of rows of each kernel of this layer, if you use this layer only for pooling then kernel_rows can be set to each number you wish
int kernel_cols:= same of kernel_rows but for the columns of the kernels
int n_kernels:= is the number of kernels indicates the depth of the current layer. If you use only pooling then n_kernels must be set = to channels
int stride1_rows:= indicates the stride used for the rows during the convolution, it must be = to stride1_cols
int padding1_rows:= is the padding used for the rows after the convolution it must be set = to padding1_cols
int stride2_rows:= is the stride for the rows used during the pooling it must be set = to stride2_cols
int padding2_rows:= is the padding for the rows used after the pooling it must be set = to padding2_cols
int pooling_rows:= is the pooling dimension used for the rows
int pooling_cols:= is the pooling dimension used for the cols
int normalization_flag:= can be set as NO_NORMALIZATION or LOCAL_RESPONSE_NORMALIZATION
int activation_flag:= can be set with all the activation we saw before except for softmax
int pooling_flag:= can be set as NO_POOLING, MAX_POOLING, AVARAGE_POOLING
int layer:= is the position of this layer inside the network
int convolutional_flag:= can be set as NO_CONVOLUTION, CONVOLUTION
For the padding, the padding parameters indicate extra dimension per row and column of the output matrix. this extra dimension will be filled with 0s. So for example if i have an output matrix 2x2 with padding = 1 it will be 4x4 with padding = 2 will be 6x6 and so on...
typedef struct cl { //convolutional-layers
int channels, input_rows, input_cols,layer, convolutional_flag;
int kernel_rows, kernel_cols, n_kernels;
int stride1_rows, stride1_cols, padding1_rows, padding1_cols;
int stride2_rows, stride2_cols, padding2_rows, padding2_cols;
int pooling_rows, pooling_cols;
int normalization_flag, activation_flag, pooling_flag; // activation flag = 0, no activation, = 1 sigmoid, = 2 relu, pooling flag = 1 max-pooling, = 2 avarage-pooling
int rows1, cols1, rows2,cols2;
float** kernels; //n_kernels - channels*kernel_rows*kernel_cols
float** d_kernels; //n_kernels - channels*kernel_rows*kernel_cols
float** d1_kernels; //n_kernels - channels*kernel_rows*kernel_cols
float** d2_kernels; //n_kernels - channels*kernel_rows*kernel_cols
float* biases; //n_kernels
float* d_biases; //n_kernels
float* d1_biases; //n_kernels
float* d2_biases; //n_kernels
float* pre_activation;//n_kernels*((input_rows-kernel_rows)/stride1_rows +1 + 2*padding1_rows)*((input_cols-kernel_cols)/stride1_cols +1 + 2*padding1_cols)
float* post_activation;//n_kernels*((input_rows-kernel_rows)/stride1_rows +1 + 2*padding1_rows)*((input_cols-kernel_cols)/stride1_cols +1 + 2*padding1_cols)
float* post_normalization;//n_kernels*((input_rows-kernel_rows)/stride1_rows +1 + 2*padding1_rows)*((input_cols-kernel_cols)/stride1_cols +1 + 2*padding1_cols)
float* post_pooling;//n_kernels*((((input_rows-kernel_rows)/stride1_rows +1 + 2*padding1_rows) - pooling_rows)/stride2_rows + 1 + 2*padding2_rows)*((((input_cols-kernel_cols)/stride1_cols +1 + 2*padding1_cols) - pooling_cols)/stride2_cols + 1 + 2*padding2_cols)
float* temp;//n_kernels*rows1*cols1
float* temp2;//n_kernels*rows1*cols1
float* temp3;//n_kernels*rows1*cols1
float* error2;//channels*input_rows*input_cols
} cl;
Your weights can be accessed from kernels and same for biases, like with fully-connected layers all the other vectors are used for feed forward, back propagation and gradient descent. The comments next to the vectors are the dimensions There are 4 parameters very important:
- rows1
- cols1
- rows2
- cols2
rows1 = ((input_rows-kernel_rows)/stride1_rows +1 + 2*padding1_rows)
cols1 = ((input_cols-kernel_cols)/stride1_cols +1 + 2*padding1_cols)
rows2 = ((((input_rows-kernel_rows)/stride1_rows +1 + 2padding1_rows) - pooling_rows)/stride2_rows + 1 + 2padding2_rows)
cols2 = ((((input_cols-kernel_cols)/stride1_cols +1 + 2padding1_cols) - pooling_cols)/stride2_cols + 1 + 2padding2_cols)
If there is no convolution (convolutional_flag = NO_CONVOLUTION) then:
rows2 = ((input_rows-pooling_rows)/stride2_rows +1 + 2*padding2_rows)
cols2 = ((input_cols-pooling_cols)/stride2_cols +1 + 2*padding2_cols)
If you want to deallocate the space occupied by the Convolutional layer you can do that with
void free_convolutional(cl* c)
If you want to save your convolutional layer in a .bin file, you can do that with:
void save_cl(cl* c, int n)
The c layer will be saved on a file named as n.bin where n is a number that will be converted in string format. To Load you cl* layer:
cl* load_cl(FILE* fr)
Residual layers are composed by 1 or more convolutional layers and at the end of the feed forward of the residual layer, if the feed forward has been computed through the "model_tensor_input_feed_forward" function, then the input state of the residual layer will be added to the last convolutional layer of this residual. After that there will be an activation function (RELU as default) to this new output.
rl* residual(int channels, int input_rows, int input_cols, int n_cl, cl** cls);
int channels:= is the channel input of the first convolutional layer inside the residual layer
int input_rows:= is the input rows of the first convolutional layer inside the residual layer
int input cols:= is the input columns of the first convolutional layer inside the residual layer
int n_cl:= is the number of convolutional layer inside the residual layer
cl** cls:= is the convolutional layers
typedef struct rl { //residual-layers
int channels, input_rows, input_cols, n_cl;
float* input;
cl* cl_output;
cl** cls;
} rl;
the float* input vector is the input given to the first convolutional layer. it will be saved to be added in a second time during the feed forward. the output given from the addition between the output of the last convolutional layer and the input stored in float* input is saved in cl_output->post_activation, if there is any activation in cl_output->activation_flag, otherwise is stored in cl_output->pre_activation. the default activation_flag is RELU, but you can change this activation setting
your_residual_layer->cl_output->activation_flag = SIGMOID // RELU // TANH // LEAKY_RELU // NO_ACTIVATION
If you want to deallocate the space occupied by the Residual layer you can do that with
void free_residual(rl* r)
If you want to save your residual layer in a .bin file, you can do that with:
void save_rl(rl* r, int n)
The r layer will be saved on a file named as n.bin where n is a number that will be converted in string format. To Load you rl* layer:
rl* load_rl(FILE* fr)