# Descriptive statistics and data normalization with CNTK and C#

As you probably know CNTK is Microsoft Cognitive Toolkit for deep learning. It is open source library which is used by various Microsoft products. Also the CNTK is powerful library for developing custom ML solutions from various fields with different platforms and languages. What is also so powerful in the CNTK is the way of the implementation. In fact the library is implemented as series of computation graphs, which  is fully elaborated into the sequence of steps performed in a deep neural network training.

Each CNTK compute graph is created with set of nodes where each node represents numerical (mathematical) operation. The edges between nodes in the graph represent data flow between operations. Such a representation allows CNTK to schedule computation on the underlying hardware GPU or CPU. The CNTK can dynamically analyze the graphs in order to to optimize both latency and efficient use of resources. The most powerful part of this is the fact thet the CNTK can calculate derivation of any constructed set of operations, which can be used for efficient learning  process of the network parameters. The flowing image shows the core architecture of the CNTK.

On the other hand, any operation can be executed on CPU or GPU with minimal code changes. In fact we can implement method which can automatically takes GPU computation if available. The CNTK is the first .NET library which provide .NET developers to develop GPU aware .NET applications.

What this exactly mean is that with this powerful library you can develop complex math computation directly to GPU in .NET using C#, which currently is not possible when using standard .NET library.

For this blog post I will show how to calculate some of basic statistics operations on data set.

Say we have data set with 4 columns (features) and 20 rows (samples). The C# implementation of this 2D array is show on the following code snippet:

static float[][] mData = new float[][] {
new float[] { 5.1f, 3.5f, 1.4f, 0.2f},
new float[] { 4.9f, 3.0f, 1.4f, 0.2f},
new float[] { 4.7f, 3.2f, 1.3f, 0.2f},
new float[] { 4.6f, 3.1f, 1.5f, 0.2f},
new float[] { 6.9f, 3.1f, 4.9f, 1.5f},
new float[] { 5.5f, 2.3f, 4.0f, 1.3f},
new float[] { 6.5f, 2.8f, 4.6f, 1.5f},
new float[] { 5.0f, 3.4f, 1.5f, 0.2f},
new float[] { 4.4f, 2.9f, 1.4f, 0.2f},
new float[] { 4.9f, 3.1f, 1.5f, 0.1f},
new float[] { 5.4f, 3.7f, 1.5f, 0.2f},
new float[] { 4.8f, 3.4f, 1.6f, 0.2f},
new float[] { 4.8f, 3.0f, 1.4f, 0.1f},
new float[] { 4.3f, 3.0f, 1.1f, 0.1f},
new float[] { 6.5f, 3.0f, 5.8f, 2.2f},
new float[] { 7.6f, 3.0f, 6.6f, 2.1f},
new float[] { 4.9f, 2.5f, 4.5f, 1.7f},
new float[] { 7.3f, 2.9f, 6.3f, 1.8f},
new float[] { 5.7f, 3.8f, 1.7f, 0.3f},
new float[] { 5.1f, 3.8f, 1.5f, 0.3f},};


If you want to play with CNTK and math calculation you need some knowledge from Calculus, as well as vectors, matrix and tensors. Also in CNTK any operation is performed as matrix operation, which may simplify the calculation process for you. In standard way, you have to deal with multidimensional arrays during calculations. As my knowledge currently there is no .NET library which can perform math operation on GPU, which constrains the .NET platform for implementation of high performance applications.

If we want to compute average value, and standard deviation for each column, we can do that with CNTK very easy way. Once we compute those values we can used them for normalizing the data set by computing standard score (Gauss Standardization).

The Gauss standardization is calculated by the flowing term:

$nValue= \frac{X-\nu}{\sigma}$,
where X- is column values, $\nu$ – column mean, and $\sigma$– standard deviation of the column.

For this example we are going to perform three statistic operations,and the CNTK automatically provides us with ability to compute those values on GPU. This is very important in case you have data set with millions of rows, and computation can be performed in few milliseconds.

Any computation process in CNTK can be achieved in several steps:

1. Read data from external source or in-memory data,
2. Define Value and Variable objects.
3. Define Function for the calculation
4. Perform Evaluation of the function by passing the Variable and Value objects
5. Retrieve the result of the calculation and show the result.

All above steps are implemented in the following implementation:

using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.Linq;
using System.Text;
using CNTK;
namespace DataNormalizationWithCNTK
{
class Program
{
static float[][] mData = new float[][] {
new float[] { 5.1f, 3.5f, 1.4f, 0.2f},
new float[] { 4.9f, 3.0f, 1.4f, 0.2f},
new float[] { 4.7f, 3.2f, 1.3f, 0.2f},
new float[] { 4.6f, 3.1f, 1.5f, 0.2f},
new float[] { 6.9f, 3.1f, 4.9f, 1.5f},
new float[] { 5.5f, 2.3f, 4.0f, 1.3f},
new float[] { 6.5f, 2.8f, 4.6f, 1.5f},
new float[] { 5.0f, 3.4f, 1.5f, 0.2f},
new float[] { 4.4f, 2.9f, 1.4f, 0.2f},
new float[] { 4.9f, 3.1f, 1.5f, 0.1f},
new float[] { 5.4f, 3.7f, 1.5f, 0.2f},
new float[] { 4.8f, 3.4f, 1.6f, 0.2f},
new float[] { 4.8f, 3.0f, 1.4f, 0.1f},
new float[] { 4.3f, 3.0f, 1.1f, 0.1f},
new float[] { 6.5f, 3.0f, 5.8f, 2.2f},
new float[] { 7.6f, 3.0f, 6.6f, 2.1f},
new float[] { 4.9f, 2.5f, 4.5f, 1.7f},
new float[] { 7.3f, 2.9f, 6.3f, 1.8f},
new float[] { 5.7f, 3.8f, 1.7f, 0.3f},
new float[] { 5.1f, 3.8f, 1.5f, 0.3f},};
static void Main(string[] args)
{
//define device where the calculation will executes
var device = DeviceDescriptor.UseDefaultDevice();

//print data to console
Console.WriteLine($"X1,\tX2,\tX3,\tX4"); Console.WriteLine($"-----,\t-----,\t-----,\t-----");
foreach (var row in mData)
{
Console.WriteLine($"{row[0]},\t{row[1]},\t{row[2]},\t{row[3]}"); } Console.WriteLine($"-----,\t-----,\t-----,\t-----");

//convert data into enumerable list
var data = mData.ToEnumerable<IEnumerable<float>>();

//assign the values
var vData = Value.CreateBatchOfSequences<float>(new int[] {4},data, device);
//create variable to describe the data
var features = Variable.InputVariable(vData.Shape, DataType.Float);

//define mean function for the variable
var mean =  CNTKLib.ReduceMean(features, new Axis(2));//Axis(2)- means calculate mean along the third axes which represent 4 features

//map variables and data
var inputDataMap = new Dictionary<Variable, Value>() { { features, vData } };
var meanDataMap = new Dictionary<Variable, Value>() { { mean, null } };

//mean calculation
mean.Evaluate(inputDataMap,meanDataMap,device);
//get result
var meanValues = meanDataMap[mean].GetDenseData<float>(mean);

Console.WriteLine($""); Console.WriteLine($"Average values for each features x1={meanValues[0][0]},x2={meanValues[0][1]},x3={meanValues[0][2]},x4={meanValues[0][3]}");

//Calculation of standard deviation
var std = calculateStd(features);
var stdDataMap = new Dictionary<Variable, Value>() { { std, null } };
//mean calculation
std.Evaluate(inputDataMap, stdDataMap, device);
//get result
var stdValues = stdDataMap[std].GetDenseData<float>(std);

Console.WriteLine($""); Console.WriteLine($"STD of features x1={stdValues[0][0]},x2={stdValues[0][1]},x3={stdValues[0][2]},x4={stdValues[0][3]}");

//Once we have mean and std we can calculate Standardized values for the data
var gaussNormalization = CNTKLib.ElementDivide(CNTKLib.Minus(features, mean), std);
var gaussDataMap = new Dictionary<Variable, Value>() { { gaussNormalization, null } };
//mean calculation
gaussNormalization.Evaluate(inputDataMap, gaussDataMap, device);

//get result
var normValues = gaussDataMap[gaussNormalization].GetDenseData<float>(gaussNormalization);
//print data to console
Console.WriteLine($"-------------------------------------------"); Console.WriteLine($"Normalized values for the above data set");
Console.WriteLine($""); Console.WriteLine($"X1,\tX2,\tX3,\tX4");
Console.WriteLine($"-----,\t-----,\t-----,\t-----"); var row2 = normValues[0]; for (int j = 0; j < 80; j += 4) { Console.WriteLine($"{row2[j]},\t{row2[j + 1]},\t{row2[j + 2]},\t{row2[j + 3]}");
}
Console.WriteLine(\$"-----,\t-----,\t-----,\t-----");
}

private static Function calculateStd(Variable features)
{
var mean = CNTKLib.ReduceMean(features,new Axis(2));
var remainder = CNTKLib.Minus(features, mean);
var squared = CNTKLib.Square(remainder);
//the last dimension indicate the number of samples
var n = new Constant(new NDShape(0), DataType.Float, features.Shape.Dimensions.Last()-1);
var elm = CNTKLib.ElementDivide(squared, n);
var sum = CNTKLib.ReduceSum(elm, new Axis(2));
var stdVal = CNTKLib.Sqrt(sum);
return stdVal;
}
}

public static class ArrayExtensions
{
public static IEnumerable<T> ToEnumerable<T>(this Array target)
{
foreach (var item in target)
yield return (T)item;
}
}
}


The output for the source code above should look like:

# Using CNTK with Visual Studio 2017 and Python

In the next few steps will show how to install CNTK and python environment in Visual Studio 2017.

1. First download the latest CNTK version from the official GitHub page, or just click on the following link: https://github.com/Microsoft/CNTK/releases

The release page will show the latest bits. Click on the CPU only package, accept the license and download the zip file.

1. Once that you have zip file on your PC, create the folder C:/local on disk and unzip the package in to it.
2. The next step performs the installation of the library as well as installation of the Python related distribution anaconda 4.1.1.
3. Open C:\local\cntk\Scripts\install\windows path and run install.bat file. You will need administrative rights in order to successfully install all required components.
4. The following image shows the installation process:

1. As can be seen first you have to run batch file (step 2), then press 1 and ENTER in order to continue with the installation process and press ‘y‘, to perform downloading required components.
2. The installation process takes several minutes to complete. The first component to be installed is Anaconda 4.1.1 which is needed in order to setup  CNTK.

1. Once the anaconda is installed, the process of CNTK installation starts and passes very quickly since we already download all CNTK bits.

1. Now that we have CNTK installed, the last installation step is installation of the Visual Studio Tool for Python.
2. Run the Visual Studio 2017 Installer and after the installed is show, just select the python components similar picture shows below:

1. Once the installation is completed run Visual Studio 2017.
2. From the Visual Studio 2017 Tool menu select Python and then select Python Environment:

1. From the Python Environment window select Anaconda 4.1.1 and update symbols DB, by pressing the button pointed on the image below:

1. Once we have environment updated, Press “Make this the default environment for the new projects” option in order to apply the environment for the future Python CNTK based projects.
2. Also the path for Python and Python scripts should be registered in Global Environment OS.

1. Once the previous steps are performed successfully, we can start writing CNTK aware python code in Visual Studio 2017.
2. OPen VS 2017 and Anaconda 4.1.1 environment and type.

import cntk

print(“CNTK verion:”, cntk__version__)

1. Similar output should be appear
2. print(“CNTK version:”, cntk.__version__)

# MicrosoftML package

Microsoft has released Microsoft R Server 9.0 (MRS9.0) with very interesting package called MicrosoftML. “Micrsooft ML” stands for Microsoft Machine Learning R package which you can use on R Server. R Server is commercial version of popular R Client distribution, which solves mayor problems when working with R. R Server contains set of cutting-edge technology to work with big data, as well as set of enhanced packages for parallelization and distributing computing.
MRS 9.0 is coming with “MicrosoftML” package which contains set of several Machine Learning algorithms developed in various Microsoft products in the last 10 years. You can combine the algorithms delivered in this package with pre-existing parallel external memory algorithms such as the RevoScaleR package as well as open source innovations such as CRAN R packages to deliver the best predictive analytic.
MicrosoftML package includes the following algorithms:

• Fast linear learner, with support for L1 and L2 regularization,
• Fast boosted decision tree,
• Fast random forest,
• Logistic regression, with support for L1 and L2 regularization,
• GPU-accelerated Deep Neural Networks (DNNs) with convolutions,
• Binary classification using a One-Class Support Vector Machine.

In order to fully use the power of MicrosoftML, and RevoScaleR you need to download MRS 9.0 from the MSDN or Visual Studio Dev Essentials subscription. Once the zip file is downloaded, unzip it, and run setup file.

The following required components  were missing when my installation is started.  Seems the MRS contains the latest .NET Core components, which is pretty cool:

After the prerequested components installed, the MRS installation process can start.

By clicking the Next button the Installation process starts:

Select the path where you want to install MRS, and press the Next button:

If everything went ok, the installation process is finished after less than minute, and the final dialog window appears:

By clicking the Finish button MRS is installed on you PC.

# Run MRS 9.0 by using R Tool fo Visual Studio, RTVS

Now it is time to run some R code. YOu have two posibilities to run R code. The first option is that you use the R Studio proffesion tool for running R code. It is free and open source which you can download from rstudio.com. If you are MS Developer you usualy write the code in the Visual Studio. So you can download RTVS from this link and run R code from Visual Studio.

Now that you have right tool to run R code, we can start with setting the MRS environment.

First thing you should do is to point RTVS to use MRS 9.0 instead of curently using some other distribution. So open the Visual Studio, select R Tools->Edit Options

The Option dialog appears. Set the R Engine to point installation folder of the MRS. Since my installation location was on Program Files folder, the picture below show my installation path.

After you set the right installation folder , restart the Visual Studio:

When the Visual Studio is running, open R Open R Interactive window. You should have similar text if you set up MRS path correctly:

Select New Project from the File->New menu option.

Name it FirstRServerDemo and click Ok. Now you are ready to write first MRS R code:

In the next post we will continue exploration the MicrosoftML library package and new set of Machine Learning algorithms added in this latest version.

# Details of my session at ATD12

Today, I gave session at Advanced Technology Day conference in Zagreb. It was very excited to see full room of people at the presentation, mostly developers from .NET world interesting in R and Data Science. This is good sign that the Data Science and the R are becoming more and more popular at daily basis. Most popularity for the R will bring R Tool for Visual Studio, which means the R language became member of the family of the Visual Studio.

For those who were asking about my slides and demo sample here is the information: