Integrating torchsharp functionality into Bonsai

[ncguilbeault]

Overview

This pull request introduces a new package, Bonsai.ML.Torch, which adds functionality from the TorchSharp package into Bonsai. It adds functions to perform basic tensor manipulations, linear algebra, inference with neural networks, and more. The Bonsai.ML.Torch package is still under heavy development and there are still many features in the TorchSharp package which have not been included. For the first release, this PR focuses on operators that support creating, manipulating, and transforming Tensor objects.

Summary of Changes

Addition of New Bonsai.ML.Torch Package:

• The Bonsai.ML.Torch project was added to the solution file Bonsai.ML.sln.
• Bonsai.ML.Torch adds a reference to the TorchSharp package.

Tensor Operations:

• Many common .NET data types (double, float, arrays, etc.) can be converted to Tensor objects using the ToTensor class.
• ToTensor has overloads for key OpenCV.Net data types (IplImage, Mat). IplImage and Mat data types can be converted to Tensor objects using zero-copy, but current conversions from Tensor to Mat or IplImage objects require a complete data copy. One possible solution to this: Support zero-copy OpenCV <-> TorchSharp conversions #55.
• Tensor objects can be initialized in various ways, including initializing tensors with Ones, Zeros, ARange, LinSpace, Empty, etc. Custom tensors can be defined using the CreateTensor class and specifying the values using Python-like syntax(e.g. "[0]" for 1D tensor with a single value, "[[0, 1]]" for a 1x2 tensor, "1.4" for single values, or "[]" for a 1D empty tensor).
• Tensor objects can be converted back to .NET arrays using the ConvertToArray node (for flattening data) and the ConvertToNDArray node (for specifying ranked multidimensional arrays).
• Tensor objects can be manipulated in various ways. For example, Reshape can be used to change the dimensions of a Tensor, Concat can be used to concatenate tensors along a specified dimension, ConvertScalarType can convert the data type of a Tensor, etc.
• GPU processing is supported by using the InitializeDeviceType with CUDA-compatible GPUs on startup. From there, the CUDA object can be passed to the Device parameter of many Tensor creation methods. Alternatively computed Tensor objects can be transferred to/from CUDA devices using the ToDevice node and setting the Device parameter using the desired device object.
• The Tensor object contains many static extension methods which can be accessed using the ExpressionTransform node. For example, it is possible to call it.sum() to sum all elements of a tensor, or it.T to transpose a tensor.
• Many existing reactive operators will also work with overloaded Tensor operators. For example, two Tensor objects can be passed to Zip -> Multiply.
• Specific elements of Tensor objects can be indexed using the Index class. The Index node allows specifying the indices using Python-like syntax. It is also possible to define tensor indices explicitly (e.g., BooleanIndex, ColonIndex, etc.), and use standard Bonsai reactive operators for indexing (e.g., Zip -> Index).

Neural Networks:

• This PR adds some initial support for the nn.Module class, the base class for instantiating neural network objects.
• A Module object can be loaded from predefined architectures (e.g., AlexNet, MNIST, and MobileNet) and can optionally load a set of weights from disk. The weights must come from a model with an identical architecture. Model weights can be saved using SaveModel. For more information about this and how to convert PyTorch modules to the correct format, see this wiki article.
• Running inference with a Module object can be done using the Forward node. Currently, the Forward method will always work by transforming a Tensor object as input into a Tensor output.
• Training a Module can be done using the Backward class. At this time, only a very small subset of optimizers (Adam), loss functions (NegativeLogLikelihood), and learning rate schedulers are supported. in the future, we plan to bring additional support for customizing training protocols in Bonsai workflows.
• The package provides an ITorchModule interface and an adapter class, TorchModuleAdapter, to allow both nn.Module objects and jit.ScriptModule objects to be loaded and used in a similar way.

Linear Algebra:

• Several functions are added to allow complex algebraic computations, such as calculating the Determinant, EigenvalueDecomposition, Inverse, etc.

Vision:

• Adds a single node, Normalize, for transforming image tensors using means and standard deviations for each channel.

[glopesdev]

@ncguilbeault This all looks great, only very few things I noticed for this round, it's in great shape and almost there I think!

[glopesdev]

This is a really nice overview of available operations, I wonder if we should include docfx xref links to each operator so that people would be able to navigate to them directly from here, but not strictly necessary at this point.

[glopesdev]

Example syntax:

[`ToTensor`](xref:Bonsai.ML.Torch.ToTensor)