MagmaSharp – .NET High Level API for MAGMA


Few weeks ago, I was doing research and I needed a fast program for Singular Value Decomposition. I have SVD implementation in my open source project called Daany which is using the SVD implementation of Accord.NET great Machine Learning Framework. However, the decomposition is working fine and smooth for small matrices with few hundreds rows/cols but for matrices with more than 500 rows and columns it is pretty slow. So I was forced to think about of using different library in order to speed up the SVD calculation. I could use some of python libraries eg. TensorFlow, PyTorch or SciPy or similar libraries from R and c++. I have used such libraries and I know how they are fast. But I still wanted to have approximately same speed on .NET as well.

Then I decided to look how can I use some of available c++ based libraries. Once I switch to c++ based project I would not be able to use .NET framework where other parts of my research are implemented. So only solution was to implement a wrapper around a c++ library and use pInvoke in order to expose required methods in C# code.

The first idea was to use LAPACK/BLAS numerical library to calculate not only SVD but whole set of Linear Algebra routines. LAPACK/BLAS libraries have long history back to 70s of the 20th century. They are proved to be very fast and reliable. However they are not supported for GPU.

Then I came to MAGMA which is nothing but LAPACK for GPU. MAGMA is very complex and fast library which requires CUDA. However if the machine has no CUDA, the library cannot be used.

The I decided to use hybrid approach and use MAGMA whenever the machine has CUDA, otherwise use LAPACK as computation engine. This approach is the most complex and required advance skills in C++ and C#. So after a more than a month of the implementation the MagmaSharp is published as GitHub open source project with the fist public release MagmaSharp 0.02.01 at

MagmaSharp v0.02.01

The first release of MagmaSharp supports MAGMA Device routines: Currently the library supports MAGMA driver routines for general rectangular matrix:

  1. gesv – solve linear system, AX = B, A is general non-symetric matrix,
  2. gels – least square solve, AX = B, A is rectangular,
  3. geev – eigen value solver for non-symetric matrix, AX = X \lambda
  4. gesvd– singular value decomposition (SVD), A = U \sigma V^T .

The library supports float and double value types.

Software requirements

The project is build on .NET Core 3.1 and .NET Standard 2.1. It is built and tested on Windows 10 1909 only.

Software (Native Libraries) requirements

In order to compile, build and use the library the following native libraries are needed to be installed.

However, if you install the MagmaSharp as Nuget package, both libraries are included, so you don’t have to install it.

How to use MagmaSharp

MagmaSharp is packed as Nuget and can be added to your .NET project as ordinary .NET component. You don’t have to worry about native libraries and dependencies. Everything is included in the package. The package can be installed from this link, or just search for MagmaSharp.

How to Build MagmaSharp from the source

  1. Download the MagmaSharp source code from the GitHub page.

  2. Reference Magma static library and put it to folder MagmaLib. Magma static library can be downloaded and built from the Official site.

  3. Open ‘MagmaSharp.sln’ with Visual Studio 2019.

  4. Make sure the building architecture is x64.

  5. Restore Nuget packages.

  6. Build and run the Solution.

How to start with MagmaSharp

The best way to start with MahmaSharp is to take a look at the MagmaSharp.XUnit project, there is a small example how to use each of the implemented method with or without CUDA device.

C# Jupyter Notebook Part 2/n

What is .NET Jupyter Notebook

In this blog post, we are going to explore the main features in the new C# Juypter Notebook. For those who used Notebook from other programming languages like Python or R, this would be an easy task. First of all, the Notebook concept provides a quick, simple and straightforward way to present a mix of text and $ \Latex $, source code implementation and its output. This means you have a full-featured platform to write a paper or blog post, presentation slides, lecture notes, and other educated materials.

The notebook consists of cells, where a user can write code or markdown text. Once he completes the cell content confirmation for cell editing can be achieved by Ctrl+Enter or by press run button from the notebook toolbar. The image below shows the notebook toolbar, with a run button. The popup combo box shows the type of cell the user can define. In the case of text, Markdown should be selected, for writing source code the cell should be Code.

run button

To start writing code to C# Notebook, the first thing we should do is install NuGet packages or add assembly references and define using statements. In order to do that, the following code installs several nuget packages, and declare several using statements. But before writing code, we should add a new cell by pressing + toolbar button.

The first few NuGet packages are packages for ML.NET. Then we install the XPlot package for data visualization in .NET Notebook, and then we install a set of Daany packages for data analytics. First, we install Daany.DataFrame for data exploration and analysis, and then Daany.DataFrame.Ext set of extensions for data manipulation used with ML.NET.

//ML.NET Packages
#r "nuget:Microsoft.ML.LightGBM"
#r "nuget:Microsoft.ML"
#r "nuget:Microsoft.ML.DataView"

//Install XPlot package
#r "nuget:XPlot.Plotly"

//Install Daany.DataFrame 
#r "nuget:Daany.DataFrame"
#r "nuget:Daany.DataFrame.Ext"
using System;
using System.Linq;

//Daany data frame
using Daany;
using Daany.Ext;

//Plotting functionalities
using XPlot.Plotly;

//ML.NET using
using Microsoft.ML;
using Microsoft.ML.Data;
using Microsoft.ML.Trainers.LightGbm;

The output for the above code:

run button

Once the NuGet packages are installed successfully, we can start with data exploration. But before this declare few using statements:

We can define classes or methods globally. The following code implements the formatter method for displaying Daany.DataFrame in the output cell.

// Temporal DataFrame formatter for this early preview
using Microsoft.AspNetCore.Html;
Formatter<DataFrame>.Register((df, writer) =>
    var headers = new List<IHtmlContent>();
    headers.AddRange(df.Columns.Select(c => (IHtmlContent) th(c)));
    //renders the rows
    var rows = new List<List<IHtmlContent>>();
    var take = 20;
    for (var i = 0; i < Math.Min(take, df.RowCount()); i++)
        var cells = new List<IHtmlContent>();
        foreach (var obj in df[i])
    var t = table(
                r => tr(r))));
}, "text/html");

For this demo we will used famous Iris data set. We will download the file from the internet, load it by using Daany.DataFrame, a display few first rows. In order to do that we run the folloing code:

var url = "";
var cols = new string[] {"sepal_length","sepal_width", "petal_length", "petal_width", "flower_type"};
var df = DataFrame.FromWeb(url, sep:',',names:cols);

The output looks like this: run button

As can be seen, the last line from the previous code has no semicolon, which means the line should be displayed in the output cell. Let’s move on, and implement two new columns. The new columns will be sepal and petal area for the flower. The expression we are going to use is:

$$ PetalArea = petal_width \cdot petal_length;\ SepalArea = sepal_width \cdot sepal_length; $$

As can be seen, the $\LaTeX$ is fully supported in the notebook.

The above formulea is implemented in the following code:

//calculate two new columns into dataset
df.AddCalculatedColumn("SepalArea", (r, i) => Convert.ToSingle(r["sepal_width"]) * Convert.ToSingle(r["sepal_length"]));
df.AddCalculatedColumn("PetalArea", (r, i) => Convert.ToSingle(r["petal_width"]) * Convert.ToSingle(r["petal_length"]));

run button

The data frame has two new columns. They indicate the area for the flower. In order to see basic statistics parameters for each of the defined columns, we call Describe method.

//see descriptive stats of the final ds

run button

From the table above, we can see the flower column has only 3 values. The most frequent value has a frequency equal to 50, which is an indicator of a balanced dataset.

Data visualization

The most powerful feature in Notebook is a data visualization. In this section, we are going to plot some interesting charts.

In order to see how sepal and petal areas are spread in 2D plane, the following plot is implemented:

//plot the data in order to see how areas are spread in the 2d plane
//XPlot Histogram reference:

var faresHistogram = Chart.Plot(new Graph.Histogram(){x = df["flower_type"], autobinx = false, nbinsx = 20});
var layout = new Layout.Layout(){title="Distribution of iris flower"};

run button

The chart is also an indication of a balanced dataset.

Now lets plot areas depending on the flower type:

// Plot Sepal vs. Petal area with flower type

var chart = Chart.Plot(
    new Graph.Scatter()
        x = df["SepalArea"],
        y = df["PetalArea"],
        mode = "markers",
        marker = new Graph.Marker()
            color = df["flower_type"].Select(x=>x.ToString()=="Iris-virginica"?1:(x.ToString()=="Iris-versicolor"?2:3)),
            colorscale = "Jet"

var layout = new Layout.Layout(){title="Plot Sepal vs. Petal Area & color scale on flower type"};
chart.WithLabels(new string[3]{"Iris-virginica","Iris-versicolor", "Iris-setosa"});
chart.WithXTitle("Sepal Area");
chart.WithYTitle("Petal Area");
chart.Width = 800;
chart.Height = 400;


run button

As can be seen from the chart above, flower types are separated almost linearly, since we used petal and sepal areas instead of width and length. With this transformation, we can get a 100% accurate ml model.

Machine Learning

Once we finished with data transformation and visualization we can define the final data frame before machine learning application. To end this we are going to select only two columns for features and one label column which will be flower type.

//create new data-frame by selecting only three columns
var derivedDF = df["SepalArea","PetalArea","flower_type"];

run button

Since we are going to use ML.NET, we need to declare Iris in order to load the data into ML.NET.

//Define an Iris class for machine learning.
class Iris
    public float PetalArea { get; set; }
    public float SepalArea { get; set; }
    public string Species { get; set; }
//Create ML COntext
MLContext mlContext = new MLContext(seed:2019);

Then load the data from Daany data frame into ML.NET:

//Load Data Frame into Ml.NET data pipeline
IDataView dataView = mlContext.Data.LoadFromEnumerable<Iris>(derivedDF.GetEnumerator<Iris>((oRow) =>
    //convert row object array into Iris row

    var prRow = new Iris();
    prRow.SepalArea = Convert.ToSingle(oRow["SepalArea"]);
    prRow.PetalArea = Convert.ToSingle(oRow["PetalArea"]);
    prRow.Species = Convert.ToString(oRow["flower_type"]);
    return prRow;

Once we have data, we can split it into train and test sets:

//Split dataset in two parts: TrainingDataset (80%) and TestDataset (20%)
var trainTestData = mlContext.Data.TrainTestSplit(dataView, testFraction: 0.2);
var trainData = trainTestData.TrainSet;
var testData = trainTestData.TestSet;

The next step in prepare the data for training is define pipeline for dtaa transformation and feature engineering:

//one encoding output category column by defining KeyValues for each category
IEstimator<ITransformer> dataPipeline =
mlContext.Transforms.Conversion.MapValueToKey(outputColumnName: "Label", inputColumnName: nameof(Iris.Species))

//define features columns
.Append(mlContext.Transforms.Concatenate("Features",nameof(Iris.SepalArea), nameof(Iris.PetalArea)));

Once we completes the preparation paert, we can perform the training part. The training start by calling Fit to the pipeline:

 // Define LightGbm algorithm estimator
IEstimator<ITransformer> lightGbm = mlContext.MulticlassClassification.Trainers.LightGbm();
//train the ML model
TransformerChain<ITransformer> model = dataPipeline.Append(lightGbm).Fit(trainData);

Once the training is completes, we have trained model which can be evaluated. In order to print the evaluation result with formatting, we are going to install Daany DataFrame extension which has implementation of printing results

//evaluate train set
var predictions = model.Transform(trainData);
var metricsTrain = mlContext.MulticlassClassification.Evaluate(predictions);
ConsoleHelper.PrintMultiClassClassificationMetrics("TRAIN Iris DataSet", metricsTrain);
ConsoleHelper.ConsoleWriteHeader("Train Iris DataSet Confusion Matrix ");

run button

//evaluate test set
var testPrediction = model.Transform(testData);
var metricsTest = mlContext.MulticlassClassification.Evaluate(testPrediction);
ConsoleHelper.PrintMultiClassClassificationMetrics("TEST Iris Dataset", metricsTest);
ConsoleHelper.ConsoleWriteHeader("Test Iris DataSet Confusion Matrix ");

run button

As can be seen, we have a 100% accurate model for Iris flower recognition. Now, let’s add a new column into the data frame called Prediction to have a model prediction in the data frame.
In order to do that, we are evaluating the model on the train and the test data set. Once we have a prediction for both sets, we can join them and add as a separate column in Daany data frame. The following code does exactly what we described previously.

var flowerLabels = DataFrameExt.GetLabels(predictions.Schema).ToList();
var p1 = predictions.GetColumn<uint>("PredictedLabel").Select(x=>(int)x).ToList();
var p2 = testPrediction.GetColumn<uint>("PredictedLabel").Select(x => (int)x).ToList();
//join train and test
var p = p1.Select(x => (object)flowerLabels[x-1]).ToList();
//add new column into df
var dic = new Dictionary<string, List<object>> { { "PredictedLabel", p } };
var dff = derivedDF.AddColumns(dic);

run button The output above shows the first few rows in the data frame. To see the few last rows from the data frame we call a Tail method.


run button

In this blog post, we saw how can we be more productive when using .NET Jupyter Notebook with Machine Learning and Data Exploration and transformation, by using ML.NET and Daany – DAtaANalYtics library. Complete source code for this notebook can be found at GitHub repo:

In depth LSTM Implementation using CNTK on .NET platform

In this blog post the implementation of the LSTM recurrent neural network in CNTK will be shown in detail. The implementation will cover LSTM implementation based on Hochreiter & Schmidhuber (1997) paper which can be found here here. The great blog post about LSTM can also be found at colah’s blog, that explains in details the structure of the LSTM cell, as well as some of the most used LSTM variants. In this blog post the LSTM recurrent network will be implemented using CNTK, a deep learning tool using C# programming language and .NET Core platform. Also in case you want to see how to use pure C# without any additional library for LSTM implementation, you can see great MSDN article: Test Run – Understanding LSTM Cells Using C# By James McCaffrey.

Whole implementation of LSTM RNN is part of ANNdotNET – deep learning tool on .NET platform. More information about the project can be found at GitHub Project page:

Introduction to LSTM Recurrent Network

Classic neural networks are built on the fact that data don’t have any order when entering into the network, and the output depend only on the input features. In case when the output depends on features and previous outputs, the classic feed forward neural network cannot help. The solution for such problem may be neural network which can be recursively provides the previous outputs. This kind of network is called recurrent neural network RNN, and it was introduced by the Hopfields in the 1980s, and later popularized when the back-propagation algorithm was improved in the beginning of 1990s. Simple concept of the recurrent neural network can be shown on the following image.

The current output of the recurrent network is defined by the current input Xt, and also on states related on the previous network outputs ht-1, ht-2,

The concept of the recurrent neural network is simple and easy to implement, but the problem raises during the training phase due to unpredictable gradient behavior. During training phase, gradient problem of neural network can be summarized in two categories: the vanishing and the exploding gradient.

The recurrent neural network is based on back-propagation algorithm, specially developed for the recurrent ANN, which is called back-propagation through time, BPTT.  In vanishing gradient problem parameters updates are proportional to the gradient of the error, which in most cases negligibly small, and results that the corresponding weights are constant and stop the network from further training.

On the other hand, exploding gradient problem refers to the opposite behavior, where the updates of weights (gradient of the cost function) became large in each back-propagation step. This problem is caused by the explosion of the long-term components in the recurrent neural network.

The solution to the above problems is specific design of there current network called Long Short-Term Memory, LSTM. One of the main advantages of the LSTM is that it can provide a constant error flow. In order to provide a constant error flow, the LSTM cell contains set of memory blocks,which have the ability to store the temporal state of the network. The LSTM also has special multiplicative units called gates that control the information flow.

The LSTM cell consists of:

  • input gate – which controls the flow of the input activations into the memory cell,
  • output gate which controls the output flow of the cell activation.
  • forget gate, which filters the information from the input and previous output and decides which one should be remembered or forgot and dropped out.

Besides three gates the LSTM cell contains cell update which is usually tanh layer to be part of the cell state.

In each LSTM cell the three variables are coming into the cell:

  • the current input xt,
  • previous output ht-1 and
  • previous cell state ct-1.

On the other hand, from each LSTM cell two variables are getting out:

  • the current output ht and
  • the current cell state ct.

Graphical representation of the LSTM cell is shown on the following image.

In order to implement LSTM recurrent network, first the LSTM cell should be implemented. The LSTM cell has three gates, and two internal states, which should be determined in order to calculate the current output and current cell state.

The LSTM cell can be define as neural network where the input vector x=\left(x_1,x_2,x_3,\ldots x_t\right)  in time t, maps to the output vector y=\left(y_1,\ y_2,\ \ldots,y_m\right), through the calculation of the following layers:

  • the forget gate sigmoid layer for the time t, ft  is calculated by the previous output ht-1 the input vector xt, and the matrix of weights from the forget layer Wf with addition of corresponded bias bi:


  • the input gate sigmoid layer for the time t, it  is calculated by the previous output ht-1 the input vector xt, and the matrix of weights from the input layer Wi with addition of corresponded bias bi:


  • the cell state in time t, Ct  is calculated from the forget gate 
    ft  and the previous cell state Ct-1. The result is summed wth the input gate it and the cell update state {\widetilde{c}}_t, that is tanh layer calculated by the previous output ht-1 the input vector xt, and the weight matrix for the cell with addition of corresponded bias bi:

C_t=f_t\ \otimes C_{t-1}+\ i_t\otimes\tanh{\left(\ W_C\bullet\left[h_{t-1},x_t\right]+b_C\right).}

  • the output gate sigmoid layer for the time t, ot is calculated by the previous output ht-1, the input vector xt, and the matrix of weights from the output layer Wo with addition of corresponded bias bi:

o_t=\sigma\left(\ W_0\bullet\left[h_{t-1},x_t\right]+b_0\right).

The final stage of the LSTM cell is current output ht calculation. The current output  is calculated with the multiplication operation \otimes between output gate layer and tanh layer of the current cell state Ct .


The current output ht, has passed through the network as the previous state for the next LSTM cell, or as the input for neural network output layer.

LSTM with Peephole connection

One of the LSTM variant which is implemented in python based CNTK is LSTM with peephole connection which is first introduced by Gers & Schmidhuber (2000). LSTM with peephole connection let each gate (forget, input and output) look at the cell state.

Now the gates with peephole connection can be expressed sothat the started terms of each gates are extended with additional matrix of Ct.So, the forget gate with peephole can be expressed:

f_t=\sigma\left(W_f\bullet\left[{C_{t-1},\ h}_{t-1},x_t\right]+b_f\right).

Similarly, the input gate and the output gate with peephole connection are expressed as:

i_t=\sigma\left(W_i\bullet\left[C_{t-1},\ h_{t-1},x_t\right]+b_i\right),

o_t=\sigma\left(\ W_0\bullet\left[{C_{t-1},h}_{t-1},x_t\right]+b_0\right).

With peephole connection LSTM cell get additional matrix for each gate and the number of LSTM parameters are increased by additional 3mXm parameters, where m – is output dimension.

Implementation of LSTM Recurrent network

The CNTK is Microsoft open source library for deep learning written in C++, but it can be run from various programming languages: Python,C#, R, Java. In order to use the library in C#, the CNTK related Nugget package has to be installed, and the project must be built for 64bit architecture.

  1. So open Visual Studio 2017 and create simple .NET Core Console application.
  2. Then install CNTK GPU Nugget package to your recently created console application.

Once the startup project is created the LSTM CNTK implementation can be started.

Implementation of the LSTM Cell

As stated previously the implementation presented in this blog post is  originally implemented in ANNdotNET – open source project for deep learning on .NET platform. It can be found at official GitHub project page

The LSTM recurrent network starts by implementation of the LSTMCell class. The LSTMCell class is derived from the NetworkFoundation class which implements basic neural network operations. The Basic operations are implemented through the implementation of the following methods:

  • Bias – bias parameters implementation
  • Weights – implementation of the weights parameters
  • Layer – implementation of the classic fully connected linear layer
  • AFunction – applying activation function on the layer.

NetworkFoundation class is shown in the next code snippet

// ANNdotNET - Deep Learning Tool on .NET Platform                                                      
// Copyright 2017-2018 Bahrudin Hrnjica                                                                                                                                       //
// This code is free software under the MIT License                                     //
// See license section of  //
// Bahrudin Hrnjica                                                                     
// Bihac, Bosnia and Herzegovina                                                         //
using CNTK;
using NNetwork.Core.Common;

namespace NNetwork.Core.Network
public class NetworkFoundation

public Variable Layer(Variable x, int outDim, DataType dataType, DeviceDescriptor device, uint seed = 1 , string name="")
    var b = Bias(outDim, dataType, device);         
    var W = Weights(outDim, dataType, device, seed, name);

    var Wx = CNTKLib.Times(W, x, name+"_wx");
    var l = CNTKLib.Plus(b,Wx, name);

    return l;

public Parameter Bias(int nDimension, DataType dataType, DeviceDescriptor device)
    //initial value
    var initValue = 0.01;
    NDShape shape = new int[] { nDimension };
    var b = new Parameter(shape, dataType, initValue, device, "_b");
    return b;

public Parameter Weights(int nDimension, DataType dataType, DeviceDescriptor device, uint seed = 1, string name = "")
    //initializer of parameter
    var glorotI = CNTKLib.GlorotUniformInitializer(1.0, 1, 0, seed);
    //create shape the dimension is partially known
    NDShape shape = new int[] { nDimension, NDShape.InferredDimension };
    var w = new Parameter(shape, dataType, glorotI, device, name=="" ? "_w" : name);
    return w;

public Function AFunction(Variable x, Activation activation, string outputName="")
    switch (activation)
        case Activation.None:
            return x;
        case Activation.ReLU:
            return CNTKLib.ReLU(x, outputName);
        case Activation.Softmax:
            return CNTKLib.Sigmoid(x, outputName);
        case Activation.Tanh:
            return CNTKLib.Tanh(x, outputName);

As can be seen, methods implement basic neural buildingblocks, which can be apply to any network type. Once the NetworkFoundation baseclass is implemented, the LSTM cell class implementation starts by definingthree properties and custom constructor, that is shown in the following code snippet:

// ANNdotNET - Deep Learning Tool on .NET Platform                                                       
// Copyright 2017-2018 Bahrudin Hrnjica                                                                                                                                       //
// This code is free software under the MIT License                                     //
// See license section of  //
// Bahrudin Hrnjica                                                                     
// Bihac, Bosnia and Herzegovina                                                         //
using CNTK;
using NNetwork.Core.Common;

namespace NNetwork.Core.Network.Modules
public class LSTM : NetworkFoundation
    public Variable X { get; set; } //LSTM Cell Input
    public Function H { get; set; } //LSTM Cell Output
    public Function C { get; set; } //LSTM Cell State

public LSTM(Variable input, Variable dh, Variable dc, DataType dataType, Activation actFun, bool usePeephole, bool useStabilizer, uint seed, DeviceDescriptor device)
    //create cell state
    var c = CellState(input, dh, dc, dataType, actFun, usePeephole, useStabilizer, device, ref seed);

    //create output from input and cell state
    var h = CellOutput(input, dh, c, dataType, device, useStabilizer, usePeephole, actFun, ref seed);

    //initialize properties
    X = input;
    H = h;
    C = c;

Properties X, H and C, hold current values of the LSTM cell,once the LSTM object is created. The LSTM constructor takes several arguments:

  • the first three are variables for the input, previous output and previous cell state;
  • the activation function of the cell update layer. 

The constructor also contains two arguments for creation a different LSTM variant: peepholes, and self-stabilization, and few other self-explained arguments. The LSTM constructor creates cell state and output by calling CellState and CellOutput methods respectively. Thei mplementation of those methods is shown on the next code snippet:

public Function CellState(Variable x, Variable ht_1, Variable ct_1, DataType dataType, 
    Activation activationFun, bool usePeephole, bool useStabilizer, DeviceDescriptor device, ref uint seed)
    var ft = AGate(x, ht_1, ct_1, dataType, usePeephole, useStabilizer, device, ref seed, "ForgetGate");
    var it = AGate(x, ht_1, ct_1, dataType, usePeephole, useStabilizer, device, ref seed, "InputGate");
    var tan = Gate(x, ht_1, ct_1.Shape[0], dataType, device, ref seed);

    //apply Tanh (or other) to gate
    var tanH = AFunction(tan, activationFun, "TanHCt_1" );

    //calculate cell state
    var bft = CNTKLib.ElementTimes(ft, ct_1,"ftct_1");
    var bit = CNTKLib.ElementTimes(it, tanH, "ittanH");

    //cell state
    var ct = CNTKLib.Plus(bft, bit, "CellState");
    return ct;

public Function CellOutput(Variable input, Variable ht_1, Variable ct, DataType dataType, DeviceDescriptor device, 
    bool useStabilizer, bool usePeephole, Activation actFun ,ref uint seed)
    var ot = AGate(input, ht_1, ct, dataType, usePeephole, useStabilizer, device, ref seed, "OutputGate");

    //apply activation function to cell state
    var tanHCt = AFunction(ct, actFun, "TanHCt");

    //calculate output
    var ht = CNTKLib.ElementTimes(ot, tanHCt,"Output");

    //create output layer in case different dimensions between cell and output
    var c = ct;
    Function h = null;
    if (ht.Shape[0] != ct.Shape[0])
        //rectified dimensions by adding linear layer
        var so = !useStabilizer? ct : Stabilizer(ct, device);
        var wx_b = Weights(ht_1.Shape[0], dataType, device, seed++);
        h = wx_b * so;
        h = ht;

    return h;

Above methods have been implemented by using previously defined gates and blocks. The method AGate creates LSTM gate. The method is called two times in order to create forget and input gates. Then the Gate method is called in order to create linear layer for the update cell state. The activation function is provided as the constructor argument. Implementation of AGate and Gate functions is shown in the following code snippet:

public Variable AGate(Variable x, Variable ht_1, Variable ct_1, DataType dataType, bool usePeephole,
    bool useStabilizer, DeviceDescriptor device, ref uint seed, string name)
    //cell dimension
    int cellDim = ct_1.Shape[0];
    //define previous output with stabilization of if defined
    var h_prev = !useStabilizer ? ht_1 : Stabilizer(ht_1, device);

    //create linear gate
    var gate = Gate(x, h_prev, cellDim, dataType, device, ref seed);
    if (usePeephole)
        var c_prev = !useStabilizer ? ct_1 : Stabilizer(ct_1, device);
        gate = gate + Peep(c_prev, dataType, device, ref seed);
    //create forget gate
    var sgate = CNTKLib.Sigmoid(gate, name);
    return sgate;

private Variable Gate(Variable x, Variable hPrev, int cellDim,
                            DataType dataType, DeviceDescriptor device, ref uint seed)
    //create linear layer
    var xw_b = Layer(x, cellDim, dataType, device, seed++);
    var u = Weights(cellDim, dataType, device, seed++,"_u");
    var gate = xw_b + (u * hPrev);
    return gate;

As can be seen AGate calls the Gate method in order to create linear layer, and then apply the activation function.

In order to create LSTM variant with peephole connection, as well as LSTM with self-stabilization, two additional methods are implemented. The peephole connection is explained previously. The implementation of Stabilizer methods is based on the implementation found at C# examples on the CNTK github page, with minor modification and re-factorization.

internal Variable Stabilizer(Variable x, DeviceDescriptor device)
    //define floating number
    var f = Constant.Scalar(4.0f, device);

    //make inversion of prev. value
    var fInv = Constant.Scalar(f.DataType, 1.0 / 4.0f);

    //create value of 1/f*ln (e^f-1)
    double initValue = 0.99537863;

    //create param with initial value
    var param = new Parameter(new NDShape(), f.DataType, initValue, device, "_stabilize");

    //make exp of product scalar and parameter
    var expValue = CNTKLib.Exp(CNTKLib.ElementTimes(f, param));

    var cost = Constant.Scalar(f.DataType, 1.0) + expValue;

    var log = CNTKLib.Log(cost);

    var beta = CNTKLib.ElementTimes(fInv, log);

    //multiplication of the variable layer with constant scalar beta
    var finalValue = CNTKLib.ElementTimes(beta, x);

    return finalValue;

internal Function Peep(Variable cstate, DataType dataType, DeviceDescriptor device, ref uint seed)
    //initial value
    var initValue = CNTKLib.GlorotUniformInitializer(1.0, 1, 0, seed);

    //create shape which for bias should be 1xn
    NDShape shape = new int[] { cstate.Shape[0] };

    var bf = new Parameter(shape, dataType, initValue, device, "_peep");

    var peep = CNTKLib.ElementTimes(bf, cstate);
    return peep;

The Peep method is based on previous description in the blog post, that simply adds the additional set of parameters which includes the previous cell state into Gates.

Implementation of the LSTM Recurrent Network

Once we have the LSTM cell implementation it is easy toimplement recurrent network based on LSTM. Previously the LSTM is defined withthree input variables: input and two previous state variables. Those previous states should be defined not as real variables but as placeholders, and should be changed dynamically for each iteration. So, the recurrent network starts by defining placeholders of previous output and previous cell state. Then the LSTMcell object is created. Once the the LSTM is created, the actual values is replaced by the previous values by calling the CNTK method PastValue. Then the placeholders are replaced with the past values of the variables. At the end the method return the CNTK Function object, which can be one of two cases, which is controlled by the returnSequence argument:

  • first case where the method returns the full sequence,
  • second case where the methods return the last element of the sequence.
// ANNdotNET - Deep Learning Tool on .NET Platform                                      
// Copyright 2017-2018 Bahrudin Hrnjica                                                 
// This code is free software under the MIT License                                     
// See license section of  
// Bahrudin Hrnjica                                                                     
// Bihac, Bosnia and Herzegovina                                                         
//                                                                  //
using CNTK;
using NNetwork.Core.Common;
using NNetwork.Core.Network.Modules;
using System;
using System.Collections.Generic;

namespace NNetwork.Core.Network
public class RNN
public static Function RecurrenceLSTM(Variable input, int outputDim, int cellDim, DataType dataType, DeviceDescriptor device, bool returnSequence=false,
    Activation actFun = Activation.TanH, bool usePeephole = true, bool useStabilizer = true, uint seed = 1)
    if (outputDim &lt;= 0 || cellDim &lt;= 0)
        throw new Exception("Dimension of LSTM cell cannot be zero.");
    //prepare output and cell dimensions 
    NDShape hShape = new int[] { outputDim };
    NDShape cShape = new int[] { cellDim };

    //create placeholders
    //Define previous output and previous cell state as placeholder which will be replace with past values later
    var dh = Variable.PlaceholderVariable(hShape, input.DynamicAxes);
    var dc = Variable.PlaceholderVariable(cShape, input.DynamicAxes);

    //create lstm cell
    var lstmCell = new LSTM(input, dh, dc, dataType, actFun, usePeephole, useStabilizer, seed, device);

    //get actual values of output and cell state
    var actualDh = CNTKLib.PastValue(lstmCell.H);
    var actualDc = CNTKLib.PastValue(lstmCell.C);

    // Form the recurrence loop by replacing the dh and dc placeholders with the actualDh and actualDc
    lstmCell.H.ReplacePlaceholders(new Dictionary&lt;Variable, Variable&gt; { { dh, actualDh }, { dc, actualDc } });

    //return value depending of type of LSTM layer
    if (returnSequence)
        return lstmCell.H;
        return CNTKLib.SequenceLast(lstmCell.H); 


As can be seen, the RNN class contains only one static method, which return the CNTK Function object which contains the recurrent network with LSTM cell. The method takes several arguments: input variable, dimension of the output of the recurrent network, dimension of the LSTM cell, and the additional arguments for creation different variants of the LSTM cell.

Implementation of Test Application

Now that the full LSTM based recurrent network is implemented, we are going to provide the test application that can test basic LSTM functionality. The application contains two test methods in order to check:

  • number of LSTM parameters, and
  • output and cell states of the LSTM cell for two iterations.

Testing the correct number of the parameters

The first method implements validation of the correct numberof LSTM parameters. The LSTM cell has three kinds of matrices: U and W and bfor each LSTM component: forget, input and output gate, and cell update.
Let assume the number of input dimension is n,and the number of output is m. Also let assume that dimension number of the cell is equal to output dimension. We can defined the following matrices:

  • U matrix with dimensions of mxn
  • W matrix with dimensions of mxm
  • B matrix (vector) with dimensions 1xm

In total the LSTM has P_{\left(LSTM\right)}=4\bullet\left(m^2+m\bullet n+m\right).

In case the LSTM has peephole connection the the number of parameters is increased with additional C matrix with 1xm parameters.

In total the LSTM with peephole connection has P_{\left(LSTM\right)}=4\bullet\left(m^2+m\bullet n+m\right)+3\bullet1\bullet m. The test method is implemented for n=3, and m=4, so the total number of parameters for default LSTM cell is P(n)=4(9+6+3)=4*18=72. With peephole connection the LSTM cell has P(n)= 4(9+6+3)+3*1*4 = 4*18+3*16 = 72+12=84.

In case the LSTM cell is defined with self-stabilization parameter, the additional 4xm parameters are defined.

Now that we defined parameter number for pure LSTM, with peephole and self-stabilization, we can implement test methods based on n=3 and m=4:

public void LSTM_Test_Params_Count()
    //define values, and variables
    Variable x = Variable.InputVariable(new int[] { 3 }, DataType.Float, "input");
    Variable y = Variable.InputVariable(new int[] { 4 }, DataType.Float, "output");

    //Number of LSTM parameters
    var lstm1 = RNN.RecurrenceLSTM(x,4,4, DataType.Float,device, Activation.Tanh,true,true,1);

    var ft = lstm1.Inputs.Where(l=&gt;l.Uid.StartsWith("Parameter")).ToList();
    var consts = lstm1.Inputs.Where(l =&gt; l.Uid.StartsWith("Constant")).ToList();
    var inp = lstm1.Inputs.Where(l =&gt; l.Uid.StartsWith("Input")).ToList();

    //bias params
    var bs = ft.Where(p=&gt;p.Name.Contains("_b")).ToList();
    var totalBs = bs.Sum(v =&gt; v.Shape.TotalSize);
    var ws = ft.Where(p =&gt; p.Name.Contains("_w")).ToList();
    var totalWs = ws.Sum(v =&gt; v.Shape.TotalSize);
    Assert.AreEqual(totalWs, 24);
    var us = ft.Where(p =&gt; p.Name.Contains("_u")).ToList();
    var totalUs = us.Sum(v =&gt; v.Shape.TotalSize);
    Assert.AreEqual(totalUs, 36);

    var totalOnly = totalBs + totalWs + totalUs;
    var totalWithSTabilize = totalOnly + totalst;
    var totalWithPeep = totalOnly + totalPh;

    var totalP = totalOnly + totalst + totalPh;
    var totalParams = ft.Sum(v=&gt;v.Shape.TotalSize);

Testing the output and cell state values

In this test the network parameters input, previous output and cell states are setup. The result of this test is weather the LSTM cell returns correct output and cell state values for first and second iteration. The implementation of this test is shows on the following code snippet:

public void LSTM_Test_WeightsValues()

    //define values, and variables
    Variable x = Variable.InputVariable(new int[] { 2 }, DataType.Float, "input");
    Variable y = Variable.InputVariable(new int[] { 3 }, DataType.Float, "output");

    //data 01
    var x1Values = Value.CreateBatch&lt;float&gt;(new NDShape(1, 2), new float[] { 1f, 2f }, device);
    var ct_1Values = Value.CreateBatch&lt;float&gt;(new NDShape(1, 3), new float[] { 0f, 0f, 0f }, device);
    var ht_1Values = Value.CreateBatch&lt;float&gt;(new NDShape(1, 3), new float[] { 0f, 0f, 0f }, device);

    var y1Values = Value.CreateBatch&lt;float&gt;(new NDShape(1, 3), new float[] { 0.0629f, 0.0878f, 0.1143f }, device);

    //data 02
    var x2Values = Value.CreateBatch&lt;float&gt;(new NDShape(1, 2), new float[] { 3f, 4f }, device);
    var y2Values = Value.CreateBatch&lt;float&gt;(new NDShape(1, 3), new float[] { 0.1282f, 0.2066f, 0.2883f }, device);

    //Create LSTM Cell with predefined previous output and prev cell state
    Variable ht_1 = Variable.InputVariable(new int[] { 3 }, DataType.Float, "prevOutput");
    Variable ct_1 = Variable.InputVariable(new int[] { 3 }, DataType.Float, "prevCellState");
    var lstmCell = new LSTM(x, ht_1, ct_1, DataType.Float, Activation.Tanh, false, false, 1, device);

    var ft = lstmCell.H.Inputs.Where(l =&gt; l.Uid.StartsWith("Parameter")).ToList();
    var pCount = ft.Sum(p =&gt; p.Shape.TotalSize);
    var consts = lstmCell.H.Inputs.Where(l =&gt; l.Uid.StartsWith("Constant")).ToList();
    var inp = lstmCell.H.Inputs.Where(l =&gt; l.Uid.StartsWith("Input")).ToList();

    //bias params
    var bs = ft.Where(p =&gt; p.Name.Contains("_b")).ToList();
    var pa = new Parameter(bs[0]);
    pa.SetValue(new NDArrayView(pa.Shape, new float[] { 0.16f, 0.17f, 0.18f }, device));
    var pa1 = new Parameter(bs[1]);
    pa1.SetValue(new NDArrayView(pa1.Shape, new float[] { 0.16f, 0.17f, 0.18f }, device));
    var pa2 = new Parameter(bs[2]);
    pa2.SetValue(new NDArrayView(pa2.Shape, new float[] { 0.16f, 0.17f, 0.18f }, device));
    var pa3 = new Parameter(bs[3]);
    pa3.SetValue(new NDArrayView(pa3.Shape, new float[] { 0.16f, 0.17f, 0.18f }, device));
    //set value to weights parameters
    var ws = ft.Where(p =&gt; p.Name.Contains("_w")).ToList();
    var ws0 = new Parameter(ws[0]);
    var ws1 = new Parameter(ws[1]);
    var ws2 = new Parameter(ws[2]);
    var ws3 = new Parameter(ws[3]);
    (ws0).SetValue(new NDArrayView(ws0.Shape, new float[] { 0.01f, 0.03f, 0.05f, 0.02f, 0.04f, 0.06f }, device));
    (ws1).SetValue(new NDArrayView(ws1.Shape, new float[] { 0.01f, 0.03f, 0.05f, 0.02f, 0.04f, 0.06f }, device));
    (ws2).SetValue(new NDArrayView(ws2.Shape, new float[] { 0.01f, 0.03f, 0.05f, 0.02f, 0.04f, 0.06f }, device));
    (ws3).SetValue(new NDArrayView(ws3.Shape, new float[] { 0.01f, 0.03f, 0.05f, 0.02f, 0.04f, 0.06f }, device));
    //set value to update parameters
    var us = ft.Where(p =&gt; p.Name.Contains("_u")).ToList();
    var us0 = new Parameter(us[0]);
    var us1 = new Parameter(us[1]);
    var us2 = new Parameter(us[2]);
    var us3 = new Parameter(us[3]);
    (us0).SetValue(new NDArrayView(us0.Shape, new float[] {  0.07f, 0.10f, 0.13f, 0.08f, 0.11f, 0.14f, 0.09f, 0.12f, 0.15f }, device));
    (us1).SetValue(new NDArrayView(us1.Shape, new float[] {  0.07f, 0.10f, 0.13f, 0.08f, 0.11f, 0.14f, 0.09f, 0.12f, 0.15f }, device));
    (us2).SetValue(new NDArrayView(us2.Shape, new float[] {  0.07f, 0.10f, 0.13f, 0.08f, 0.11f, 0.14f, 0.09f, 0.12f, 0.15f }, device));
    (us3).SetValue(new NDArrayView(us3.Shape, new float[] {  0.07f, 0.10f, 0.13f, 0.08f, 0.11f, 0.14f, 0.09f, 0.12f, 0.15f }, device));

    //Evaluate model after weights are setup
    var inV = new Dictionary&lt;Variable, Value&gt;();
    inV.Add(x, x1Values);
    inV.Add(ht_1, ht_1Values);
    inV.Add(ct_1, ct_1Values);

    //evaluate output when previous values are zero
    var outV11 = new Dictionary&lt;Variable, Value&gt;();
    outV11.Add(lstmCell.H, null);
    lstmCell.H.Evaluate(inV, outV11, device);
    //test  result values
    var result = outV11[lstmCell.H].GetDenseData&lt;float&gt;(lstmCell.H);
    Assert.AreEqual(result[0][0], 0.06286034f);//
    Assert.AreEqual(result[0][1], 0.0878196657f);//
    Assert.AreEqual(result[0][2], 0.114274308f);//

    //evaluate cell state
    var outV = new Dictionary&lt;Variable, Value&gt;();
    outV.Add(lstmCell.C, null);
    lstmCell.C.Evaluate(inV, outV, device);

    var resultc = outV[lstmCell.C].GetDenseData&lt;float&gt;(lstmCell.C);
    Assert.AreEqual(resultc[0][0], 0.114309229f);//
    Assert.AreEqual(resultc[0][1], 0.15543206f);//
    Assert.AreEqual(resultc[0][2], 0.197323829f);//

    //evaluate second value, with previous values as previous state
    //setup previous state and output
    ct_1Values = Value.CreateBatch&lt;float&gt;(new NDShape(1, 3), new float[] { resultc[0][0], resultc[0][1], resultc[0][2] }, device);
    ht_1Values = Value.CreateBatch&lt;float&gt;(new NDShape(1, 3), new float[] { result[0][0], result[0][1], result[0][2] }, device);

    //Prepare for the evaluation
    inV = new Dictionary&lt;Variable, Value&gt;();
    inV.Add(x, x2Values);
    inV.Add(ht_1, ht_1Values);
    inV.Add(ct_1, ct_1Values);

    outV11 = new Dictionary&lt;Variable, Value&gt;();
    outV11.Add(lstmCell.H, null);
    lstmCell.H.Evaluate(inV, outV11, device);

    //test  result values
    result = outV11[lstmCell.H].GetDenseData&lt;float&gt;(lstmCell.H);
    Assert.AreEqual(result[0][0], 0.128203377f);//
    Assert.AreEqual(result[0][1], 0.206633776f);//
    Assert.AreEqual(result[0][2], 0.288335562f);//

    //evaluate cell state
    outV = new Dictionary&lt;Variable, Value&gt;();
    outV.Add(lstmCell.C, null);
    lstmCell.C.Evaluate(inV, outV, device);

    //evaluate cell state with previous value
    resultc = outV[lstmCell.C].GetDenseData&lt;float&gt;(lstmCell.C);
    Assert.AreEqual(resultc[0][0], 0.227831185f);//
    Assert.AreEqual(resultc[0][1], 0.3523231f);//
    Assert.AreEqual(resultc[0][2], 0.4789199f);//

In this article the implementation of the LSTM cell is presented in details from the theory and implementation. Also the article contains two test methods in order to prove the correctness of the implementation. The result values of the output and the cell states are compared with manually calculated values.

Create CIFAR-10 Deep Learning Model With ANNdotNET GUI Tool

With ANNdotNET 1.2 the user is able to create and train deep learning models for image classification. Image classification module provides minimum of GUI actions in order to fully prepare data set. In this post, we are going to create and train deep learning model for CIFAR-10 data set, and see how it easy to do that with ANNdotNET v1.2.

In order to prepare data we have to download CIFAR-10 data set from official web site . The CIFAR-10 data set is provided in 6 binary batch files that should be extracted and persisted on your local machine. Number 10 in the name means that data set is created for 10 labels.The following image shows 10 labels of CIFAR-10 data set each label with few sample images.

CIFAR-10 data set (Learning Multiple Layers of Features from Tiny Images, Alex Krizhevsky, 2009.)

The data set contains 60 000 (50 000 for training and validation, and 10 000 for test) tinny colored images dimensions of 32×32. There is also bigger version of the data set CIFAR-100 with 100 labels. Our task is to create deep learning model capable of recognizing only one of 10 predefined labels from each image.

Data preparation

In order to prepare images, we need to do the following:

The following image shows extracted data set persisted in 10 label folders. The bird folder is opened and shows all images labeled for bird. The test folder contains all images created for testing the model once the model is trained.

In order to properly save all images, we need to create simple C# Console application which should extract and save all 60 000 images. Complete C# program can be downloaded from here.

In order to successfully extract the images, we have to see how those images are stored in binary files. From the official site we can see that there are 5 for training and 1 for test binary files: data_batch_1.bin, data_batch_2.bin, …, data_batch_5.bin, as well as test_batch.bin.

Each of these files is formatted as follows so that the first byte of the array is label index, and the next 3072 bytes represent the image. Each batch contains 10 000 images.

Important to know is that images are stored in CHW format which means that 1d image array is created so that the first 1024 bytes are the red channel values, the next 1024 the green, and the final 1024 the blue. The values are stored in row-major order, so the first 32 bytes are the red channel values of the first row of the image. To end this, all those information have been carried out when implementing the Extractor application. The most important methods are reshaping the 1D byte array into [3, height, width] image tensor, and creating the image from the byte tensor. The following implementation shows how 1D byte array is transformed into 3channel bitmap tensor.

static int[][][] reshape(int channel, int height, int width,  byte[] img)
    var data = new int[channel][][];
    int counter = 0;
    for(int c = 0; c < channel; c++)
        data[c] = new int[height][];
        for (int y = 0; y < height; y++)
            data[c][y] = new int[width];
            for (int x = 0; x < width; x++)
                data[c][y][x] = img[counter];
    return data;

Once the 1D byte array is transformed into tensor, the image can be created and persisted on disk. The following method iterates through all 10000 images in one batch file, extract them and persist on disk.

public static void extractandSave(byte[] batch, string destImgFolder, ref int imgCounter)
    var nStep = 3073;//1 for label and 3072 for image
    for (int i = 0; i < batch.Length; i += nStep)
        var l = (int)batch[i];
        var img = new ArraySegment<byte>(batch, i + 1, nStep - 1).ToArray();
// data in CIFAR-10 dataset is in CHW format, which means CHW: RR...R, GG..G, BB..B;

        // while HWC: RGB, RGB, ... RGB
        var reshaped = reshape(3, 32, 32, img);
        var image = ArrayToImg(reshaped);
        //check if folder exist
        var currentFolder = destImgFolder + classNames[l];

        if (!Directory.Exists(currentFolder))

        //save image to specified folder
        image.Save(currentFolder + "\\" + imgCounter.ToString() + ".png");


Run Cifar-Extractor console application and the process of downloading, extracting and saving images will be finished in few minutes. The most important is that CIFAR-10 data set will be stored in c://sc/datasets/cifar-10 path. This is important later, when we create image classifier.

Now that we have 60000 tiny images on disk arranged by labels we can start creating deep learning model.

Create new image classification project file in ANNdotNET

Open the latest ANNdotNET v1.2 and select New-> Image Classification project. Enter CIFAR project name and press save button. The following image shows CIFAR new ann-project:

Once we have new project, we can start defining image labels by pressing Add button. For each 10 labels we need to add new label item in the list. In each item the following fields should be defined:

  • Image label
  • Path to images with the label.
  • Query – in case we need to get all images within the specified path with certain part of the name. In case all images withing the specified path are images that indicate one label, query should be empty string.

Beside Label item, image transformation should be defined in order to define the size of the images, as well as how many images create validation/test data set.

Assuming the CIFAR-10 data set is extracted at c:/sc/datasets/cifar-10 folder, the following image shows how label items should be defined:

In case label item should be removed from the list, this is done by selecting the item, and then pressing Remove button. Beside image properties, we should defined how many images belong to validation data set. As can be seen 20% of all extracted images will be created validation data set. Notice that images from the test folder are not part of those two data set. they will be used for testing phase once the model is trained. Now that we done with data preparation we can move to the next step: creating mlconifg file.

Create mlconfig in ANNdotNET

By selecting New MLConfig command the new mlconfig file is created within the project explorer. Moreover by pressing F2 key on selected mlconfig tree item, we can easily change the name into “CIRAF-10-ConvNet”. The reason why we gave such name is because we are going to use convolution neural networks.

In order to define mlconfig file we need to define the following:

  • Network configuration using Visual Network Designer
  • Define Learning parameters
  • Define training parameters

Create Network configuration

By using Visual Network Designer (VND) we can quickly create network model. For this CIFAR-10 data set we are going to create 11 layers model with 4 Constitutional, 2 Pooling, 1 DropOut and 3 Dense layer, all followed by Scale layer:

Scale (1/255)->Conv2D(32,[3,3])->Conv2D(32,[3,3])->Pooling2d([2,2],2)->Conv2D(64,[3,3])->Conv2D(64,[3,3])->Pooling2d([2,2],2)->DropOut(0.5)->Dense(64, TanH)->Dense(32, TanH)->Dense(10,Softmax)

This network can be created so that we select appropriate layer from the VND combo box and click on Add button. The first layer is Scale layer, since we need to normalize the input values to be in interval (0,1). Then we created two sequence of Convolution, Pooling layers. Once we done with that, we can add two Dense layers with 64 and 32 neurons with TanH activation function. The last layer is output layer that must follow the output dimension, and Softmax activation function.

Once network model is defined, we can move to the next step: Setting learning and training parameters.

Learning parameters can be defined through the Learning parameters interface: For this model we can select:

  • AdamLearner with 0.005 rate and 0.9 momentum value. Loss function is Classification Error, and the evaluation function is Classification Accuracy

In order to define the training parameters we switch to Training tab page and setup:

  • Number of epoch
  • Minibatch size
  • Progress frequency
  • Randomize minibatch during training

Now we have enough information to start model training. The training process is started by selecting Run command from the application ribbon. In order to get good model we need to train the model at least few thousands epoch. The following image shows trained model with training history charts.

The model is trained with exactly of 4071 epochs, with network parameters mentioned above. As can be seen from the upper chart, mini-batch loss function was CrossEntropyWithSoftmax, while the evaluation function was classification accuracy.  The bottom chart shows performance of the training and validation data sets for each 4071 epoch. We can also recognize that validation data set has roughly the same accuracy as training data set which indicates the model is trained well.  More details about model performance can be seen on the next image:

Upper charts of the image above show actual and predicted values for training (left) and validation (right). Most of the point values are blue and overlap the orange which indicates that most of value are correctly predicted. The charts can be zoomed and view details of each value.The bottom part of the evaluation show performance parameters of the model for corresponded data set. As can be seen the trained model has 0.91 overall accuracy for training data set and 0.826 overall accuracy for validation data set, which indicate pretty good accuracy of the model. Moreover, the next two images shows confusion matrix for the both data sets, which in details shows how model predict all 10 labels.

The last part of the post is testing model for test data set. For that purpose we selected 10 random images from each label of the test set, and evaluate the model. The following images shows the model correctly predicted all 10 images.


ANNdotNET v1.2 image classification module offers complete data preparation and model development for image classification. The user can prepare data for training, create network model with Neural Network Designer, and perform set of statistical tools against trained model in order to validate and evaluate model. The important note is that the data set of images must be stored on specific location in order to use this trained model shown in the blog post. The trained model, as well as mlcofig files, can be load directly into ANNdotNET project explorer by doublick on feed example.

ANNdotNET as open source project provides outstanding way in complete development of deep learning model.