Computes a 2D convolution over input images. Intended to be used in 2d convolution forward pass. This applies a 2D cross-correlation, not to be confused with the mathematical convolution.

Arguments

• input : Tensor - 4D Tensor batch of images of the size [N,C_in,H_in,W_in]
• weight : Tensor - 4D Tensor convolving kernel weights of the size [C_out,C_in,kH,kW]
• bias : Tensor - 3D Tensor bias of the size [C_out,1,1]
• padding : Tuple - Tuple with height and width of the padding
• stride : Tuple - Tuple with height and width of the stride

Computes gradients of a 2D convolution. Intended to be used after #conv2d to calculate gradients in backward pass.

Arguments

• input : Tensor - 4D Tensor batch of images of the size [N,C_in,H_in,W_in]
• weight : Tensor - 4D Tensor convolving kernel weights of the size [C_out,C_in,kH,kW]
• bias : Tensor - 3D Tensor bias of the size [C_out,1,1]
• grad_output : Tensor - 4D Tensor gradient of size [N, C_out, H_out, W_out]
• padding : Tuple - Tuple with height and width of the padding
• stride : Tuple - Tuple with height and width of the stride

Computes a forward dropout activation

Arguments

• input : Tensor - Tensor to activate
• mask : Tensor - Mask to dropout
• probability : Float - Probability of dropout

Computes a forward dropout activation

Arguments

• input : Tensor - Tensor to activate
• mask : Tensor - Mask to dropout
• probability : Float - Probability of dropout

Computes a backwards dropout derivative

Arguments

• gradient : Tensor - Tensor used to compute backwards pass
• mask : Tensor - Mask to apply to the gradient
• probability : Float - Probability of dropout

Exponential linear unit activation

Arguments

• x : Tensor - Tensor to activate

Exponential linear unit activation

Arguments

• x : Tensor - Tensor to activate

ELU derivative

Arguments

• gradient : Tensor - Tensor to derive
• cached : Tensor - Cached Tensor from activation

Computes a 2D convolution over input images. Intended to be used in 2d convolution forward pass. This applies a 2D cross-correlation, not to be confused with the mathematical convolution.

Arguments

• input : Tensor - 4D Tensor batch of images of the size [N,C_in,H_in,W_in]
• weight : Tensor - 4D Tensor convolving kernel weights of the size [C_out,C_in,kH,kW]
• bias : Tensor - 3D Tensor bias of the size [C_out,1,1]
• padding : Tuple - Tuple with height and width of the padding
• stride : Tuple - Tuple with height and width of the stride

Computes gradients of a 2D convolution. Intended to be used after #conv2d to calculate gradients in backward pass.

Arguments

• input : Tensor - 4D Tensor batch of images of the size [N,C_in,H_in,W_in]
• weight : Tensor - 4D Tensor convolving kernel weights of the size [C_out,C_in,kH,kW]
• bias : Tensor - 3D Tensor bias of the size [C_out,1,1]
• grad_output : Tensor - 4D Tensor gradient of size [N, C_out, H_out, W_out]
• padding : Tuple - Tuple with height and width of the padding
• stride : Tuple - Tuple with height and width of the stride

Leaky ReLU activation function

Arguments

• x : Tensor - Argument to activate

Leaky ReLU activation function

Arguments

• x : Tensor - Argument to activate

Leaky ReLU activation function

Arguments

• x : Tensor - Argument to activate

Leaky ReLU activation function

Arguments

• x : Tensor - Argument to activate

Leaky ReLU derivative

Arguments

• gradient : Tensor - Tensor to derive
• cached : Tensor - Cached Tensor from activation

Leaky ReLU derivative

Arguments

• gradient : Tensor - Tensor to derive
• cached : Tensor - Cached Tensor from activation

Returns labels, as well as X and Y training inputs for the IRIS dataset.

Returns a struct containing features, labels, as well as test_features and test_labels for the MNIST dataset

Computes the maxpooling of a Tensor

Arguments

• input : Tensor - Tensor to pool
• kernel : Tuple - Kernel height and width
• target : Tensor - Tensor truth values
• padding : Tuple - Tuple with height and width of the padding
• stride : Tuple - Tuple with height and width of the stride

Computes the maxpooling gradient

Arguments

• shape : Array - Shape of gradient output
• max_indices : Tensor - Pooled max indices
• grad_output : Tensor - Output from forward pass

Mean relative error for Tensor, mean of the element-wise |y_true - y|/max(|y_true|, |y|) Normally the relative error is defined as |y_true - y| / |y_true|, but here max is used to make it symmetric and to prevent dividing by zero, guaranteed to return zero in the case when both values are zero.

Mean squared error loss

Arguments

• input : Tensor - Predicted values
• target : Tensor - Truth values

Computes gradients of mean squared error loss

Arguments

• gradient : Tensor - Tensor gradient computed from MSE forwards
• cache : Tensor 4D - Cached Tensor from activation
• target : Tensor - Tensor truth values

Compute numerical gradient for any function w.r.t. to an input Tensor, useful for gradient checking, recommend using float64 types to assure numerical precision. The gradient is calculated as: (f(x + h) - f(x - h)) / (2*h) where h is a small number, typically 1e-5 f(x) will be called for each input elements with +h and -h pertubation. Iterate over all elements calculating each partial derivative

Compute numerical gradient for any function w.r.t. to an input value, useful for gradient checking, recommend using float64 types to assure numerical precision. The gradient is calculated as: (f(x + h) - f(x - h)) / (2*h) where h is a small number, typically 1e-5.

ReLU activation function

Arguments

• x : Tensor - Argument to activate

ReLU activation function

Arguments

• x : Tensor - Argument to activate

ReLU activation function

Arguments

• x : Tensor - Argument to activate

ReLU activation function

Arguments

• x : Tensor - Argument to activate

Derivative of the ReLU activation function

Arguments

• gradient : Tensor - Tensor to derive
• cached : Tensor Cached Tensor from activation

Derivative of the ReLU activation function

Arguments

• gradient : Tensor - Tensor to derive
• cached : Tensor Cached Tensor from activation

Sigmoid takes a real value as input and outputs another value between 0 and 1. It’s easy to work with and has all the nice properties of activation functions: it’s non-linear, continuously differentiable, monotonic, and has a fixed output range.

Arguments

• x : Tensor - Tensor to activate

Examples

a = [0.1, 0.34, 0.65].to_tensor
puts Num::NN.sigmoid(a) # => [0.524979, 0.584191, 0.65701 ]

Sigmoid takes a real value as input and outputs another value between 0 and 1. It’s easy to work with and has all the nice properties of activation functions: it’s non-linear, continuously differentiable, monotonic, and has a fixed output range.

Arguments

• x : Tensor - Tensor to activate

Examples

a = [0.1, 0.34, 0.65].to_tensor
puts Num::NN.sigmoid(a) # => [0.524979, 0.584191, 0.65701 ]

Sigmoid takes a real value as input and outputs another value between 0 and 1. It’s easy to work with and has all the nice properties of activation functions: it’s non-linear, continuously differentiable, monotonic, and has a fixed output range.

Arguments

• x : Tensor - Tensor to activate

Examples

a = [0.1, 0.34, 0.65].to_tensor
puts Num::NN.sigmoid(a) # => [0.524979, 0.584191, 0.65701 ]

Sigmoid takes a real value as input and outputs another value between 0 and 1. It’s easy to work with and has all the nice properties of activation functions: it’s non-linear, continuously differentiable, monotonic, and has a fixed output range.

Arguments

• x : Tensor - Tensor to activate

Examples

a = [0.1, 0.34, 0.65].to_tensor
puts Num::NN.sigmoid(a) # => [0.524979, 0.584191, 0.65701 ]

Sigmoid cross entropy loss

Arguments

• input : Tensor - Predicted values
• target : Tensor - Truth values

Sigmoid cross entropy loss

Arguments

• input : Tensor - Predicted values
• target : Tensor - Truth values

Computes gradients of sigmoid cross entropy loss

Arguments

• gradient : Tensor - Tensor gradient computed from SCE forwards
• cache : Tensor 4D - Cached Tensor from activation
• target : Tensor - Tensor truth values

Computes gradients of sigmoid cross entropy loss

Arguments

• gradient : Tensor - Tensor gradient computed from SCE forwards
• cache : Tensor 4D - Cached Tensor from activation
• target : Tensor - Tensor truth values

Derivative of the Sigmoid function

Arguments

• gradient : Tensor - Tensor to derive
• cached : Tensor - Cached Tensor from activation

Examples

a = [0.1, 0.34, 0.65].to_tensor
puts Num::NN.d_sigmoid(a) # => [0.249376, 0.242912, 0.225348]

Derivative of the Sigmoid function

Arguments

• gradient : Tensor - Tensor to derive
• cached : Tensor - Cached Tensor from activation

Examples

a = [0.1, 0.34, 0.65].to_tensor
puts Num::NN.d_sigmoid(a) # => [0.249376, 0.242912, 0.225348]

Computes softmax cross entropy loss

Arguments

• input : Tensor - Predicted values
• target : Tensor - Truth values

Computes gradients of SmCE loss

Arguments

• gradient : Tensor - Tensor gradient computed from SmCE forwards
• cache : Tensor 4D - Cached Tensor from activation
• target : Tensor - Tensor truth values

Tanh squashes a real-valued number to the range [-1, 1]. It’s non-linear. But unlike Sigmoid, its output is zero-centered. Therefore, in practice the tanh non-linearity is always preferred to the sigmoid nonlinearity.

Arguments

• x : Tensor - Tensor to activate

Examples

a = [0.1, 0.34, 0.65].to_tensor
Num::NN.tanh(a) # => [0.099668, 0.327477, 0.57167 ]

Tanh squashes a real-valued number to the range [-1, 1]. It’s non-linear. But unlike Sigmoid, its output is zero-centered. Therefore, in practice the tanh non-linearity is always preferred to the sigmoid nonlinearity.

Arguments

• x : Tensor - Tensor to activate

Examples

a = [0.1, 0.34, 0.65].to_tensor
Num::NN.tanh(a) # => [0.099668, 0.327477, 0.57167 ]

Tanh squashes a real-valued number to the range [-1, 1]. It’s non-linear. But unlike Sigmoid, its output is zero-centered. Therefore, in practice the tanh non-linearity is always preferred to the sigmoid nonlinearity.

Arguments

• x : Tensor - Tensor to activate

Examples

a = [0.1, 0.34, 0.65].to_tensor
Num::NN.tanh(a) # => [0.099668, 0.327477, 0.57167 ]

Tanh squashes a real-valued number to the range [-1, 1]. It’s non-linear. But unlike Sigmoid, its output is zero-centered. Therefore, in practice the tanh non-linearity is always preferred to the sigmoid nonlinearity.

Arguments

• x : Tensor - Tensor to activate

Examples

a = [0.1, 0.34, 0.65].to_tensor
Num::NN.tanh(a) # => [0.099668, 0.327477, 0.57167 ]

Derivative of the Tanh function

Arguments

• x : Tensor - Tensor to derive
• cached : Tensor - Cached Tensor from activation

Examples

a = [0.1, 0.34, 0.65].to_tensor
Num::NN.d_tanh(a) # => [0.990066, 0.892759, 0.673193]

Derivative of the Tanh function

Arguments

• x : Tensor - Tensor to derive
• cached : Tensor - Cached Tensor from activation

Examples

a = [0.1, 0.34, 0.65].to_tensor
Num::NN.d_tanh(a) # => [0.990066, 0.892759, 0.673193]