C++ transfer learning APIs

Backpropagation

namespace coral
class SoftmaxRegressionModel
#include <softmax_regression_model.h>

An implementation of the softmax regression function (multinominal logistic regression) that operates as the last layer of your classification model, and allows for on-device training with backpropagation (for this layer only).

Input for this layer must be an image embedding, which should be the output of your embedding extractor (the backbone of your model). Once given here, the input is fed to a fully-connected layer where weights and bias are applied, and then passed to the softmax function to receive the final probability for each class.

Public Functions

SoftmaxRegressionModel(int feature_dim = 0, int num_classes = 0, float weight_scale = 0.01, float reg = 0.0)

Inputs: feature_dim (int): The dimension of the input feature (length of the feature vector).

num_classes (int): The number of output classes. weight_scale (float): A weight factor for computing new weights. The backpropagated weights are drawn from standard normal distribution, then multiplied by this number to keep the scale small. reg (float): The regularization strength.

float GetAccuracy(const Eigen::MatrixXf &mat_x, const std::vector<int> &correct_labels)

Computes how many of the labels are classified correctly for a given input.

Inputs: tensor mat_x of size NXD of input data, Vector labels of size N. Output: a decimal float between 0 and 1 of the accuracy. Accuracy = number classified correctly / total inputs

std::vector<int> RunInference(const Eigen::MatrixXf &mat_x)

Runs an inference using the current weights.

Inputs: tensor mat_x of size NxD of input data (image embeddings) Outputs: Vector of inferred label index for each input embedding

void Train(const TrainingData &data, const TrainConfig &train_config, float learning_rate)

Finds the optimal weights and biases for the last layer according to the specifications of weight_updater (sgd_updater for this model) and train_config.

void AppendLayersToEmbeddingExtractor(const tflite::Model &embedding_extractor_model, flatbuffers::FlatBufferBuilder *fbb) const

Appends the learned weight and bias values to embedding extractor tflite model and saves to file.

Private Functions

float GetLoss(const Eigen::MatrixXf &mat_x, const Eigen::MatrixXf &labels)

Calculates the loss of the current model for the given data, using a cross-entropy loss function.

Inputs: tensor mat_x of size NxD of input data (image embeddings) to test wehere N is number of inputs and D is the dimension of each input. Tensor labels of size NxC of labels in one-hot vector format where C is the number of classes possible in the model Output: value of the averaged cross entropy loss for each input embedding.

std::vector<Eigen::MatrixXf> GetGrads(const Eigen::MatrixXf &mat_x, const Eigen::MatrixXf &labels)

Calculates the gradients of the current weights using cross entropy loss as the loss function and backpropogation using CrossEntropyGradient, SoftmaxGradient, and FullyConnectedGradient.

Uses same inputs as get_loss(). Outputs: vector of weights of the model: mat_w_ and vec_b_

Private Members

Eigen::MatrixXf mat_w_

Weights and biases of the model that we want to learn.

Eigen::MatrixXf vec_b_
float logit_min_

Minimum / maximum values of the last fully connected layer’s output.

These values will be learned from datga and used to quantize the output of the learned fully-connected layer.

float logit_max_
int feature_dim_ = 0

Parameters set when initializing model.

int num_classes_ = 0
float weight_scale_ = 0.01
float reg_ = 0.0
std::vector<Eigen::MatrixXf> caches_

Caches to improve backpropogation performance by storing intermediate results of forward operators, in order of logits, probs, loss.

struct TrainConfig
#include <softmax_regression_model.h>

A struct to define the training configuration to be used for last-layer training.

Public Functions

TrainConfig(int num_iter, int batch_size, int print_every)

Public Members

int num_iter
int batch_size
int print_every
struct TrainingData
#include <softmax_regression_model.h>

A struct to define the training and validation input to a model.

Public Members

Eigen::MatrixXf training_data
Eigen::MatrixXf validation_data
std::vector<int> training_labels
std::vector<int> validation_labels
namespace coral
class MultiVariateNormalDistribution
#include <multi_variate_normal_distribution.h>

Multi variate normal distribution implemented with Eigen library.

Public Functions

MultiVariateNormalDistribution(const Eigen::VectorXf &mean, const Eigen::MatrixXf &cov)
Eigen::MatrixXf Sample(int num)

Samples ‘num’ samples from distribution.

Returns a [dim, num] shape matrix.

Private Members

Eigen::VectorXf mean_

Mean of the distribution.

Eigen::MatrixXf cov_

Covariance matrix of the distribution.

Eigen::MatrixXf p_

Multiplies this matrix with a random variable X which is drawn from N(0, I) will produce a sample drawn from N(0, cov_).

int dim_

The dimension of the covariance matrix.

Eigen::EigenSolver<Eigen::MatrixXf> solver_

Eigen solver which is used to compute eigen value and eigen vectors.

std::normal_distribution<float> rand_gaussian_

Gaussian random number generator.

namespace coral

Functions

Eigen::MatrixXf CrossEntropyLoss(const Eigen::MatrixXf &c, const Eigen::MatrixXf &p)

Computes the cross entropy between two probability distributions using CE(c,p) = - sum(c*log(p)) and returns the average loss of the batch.

Inputs: vector of size 2 of [c, p] Both c and p tensors are size NxC where N is the number of distributions and C is the length of each distribution. When used with softmax, p is the probability output from softmax and c is a batch of one-hot vectors for class labels. Output: vector of size 1; tensor is 1x1 containing the average loss value.

Eigen::MatrixXf CrossEntropyGradient(const Eigen::MatrixXf &c, const Eigen::MatrixXf &p)

This class computes the gradient of the Cross Entropy Loss with respect to each of the elements in probability distribution p A good reference for this is: https://deepnotes.io/softmax-crossentropy Inputs: vector of size 2 of [c, p] c and p described in CrossEntropyLoss class; Loss is output of the Compute method in CrossEntropyLoss class.

Output: vector of size 1; tensor is NxC gradient with respect to p

Eigen::MatrixXf FullyConnected(const Eigen::MatrixXf &mat_x, const Eigen::MatrixXf &mat_w, const Eigen::MatrixXf &mat_b)

Forward pass operator for the fully connected layer that computes Y = X*W + b A good reference for this is: http://cs231n.github.io/linear-classify/#score Input: vector of tensors in order of data mat_x, weights mat_w, and bias vec_b.

mat_x is size NxD where N is number of inputs and D is number of dimensions. mat_w is size DxC. vec_b is size 1xC. Output: vector of size 1 that is layer output Y

std::vector<Eigen::MatrixXf> FullyConnectedGradient(const Eigen::MatrixXf &mat_x, const Eigen::MatrixXf &mat_w, const Eigen::MatrixXf &b, const Eigen::MatrixXf &dmat_y)

Backward pass operator that computes gradients of the inputs to the fully connected layer A good reference for this is: http://cs231n.stanford.edu/2017/handouts/linear-backprop.pdf Input: vector of tensors in order of data mat_x, weights mat_w, bias b, grad dmat_y.

The tensors mat_x, mat_w, vec_b are as described in FullyConnected class, dmat_y is size NxC. Output: vector of tensors of gradients in order of dmat_x, dmat_w, dvec_b and correspond in size to mat_x, mat_w, vec_b respectively

Eigen::MatrixXf Softmax(const Eigen::MatrixXf &logits)

Forward pass operator for the softmax classifier layer that computes the probibilities of each sample in the tensor being in each class.

A good reference for this is: http://cs231n.github.io/linear-classify/#softmax Input: vector of size 1 of unnormalized logits; tensor is NxC array where N is number of inputs and C is number of classes. Output: vector of size 1 of normalized probabilities; tensor is NxC array.

Eigen::MatrixXf SoftmaxGradient(const Eigen::MatrixXf &logits, const Eigen::MatrixXf &dprobs)

This class computes the gradient of the Softmax operator with respect to each of the elements in the vector of unnormalized logits.

A good reference for this is: https://deepnotes.io/softmax-crossentropy Input: vector of size 2 of tensors [logits, dprobs]. logits is NxC array where N is number of inputs and C is number of classes. dprobs is NXC array of gradients of Loss with respect to softmax output. Output: vector of size 1; tensor is NxC gradient of Loss with respect to logits.

Eigen::MatrixXf SoftmaxLocalGradient(Eigen::MatrixXf::RowXpr prob)

Helper function to compute local gradient of softmax.

void SgdUpdate(const std::vector<Eigen::MatrixXf> &grads, float learning_rate, std::vector<Eigen::MatrixXf*> weights)

Updates the value of weights based on grads.

Inputs: grads is a vector of tensors of gradients to be used to update the weights in a particular layer of a neural net, and weights is a vector of tensors of the weights that we want to update. Each element grads[i] is the same size as its corresponding element weights[i]. When used to update a fully connected layer, the grads are dW and db from the output of FullyConnectedGradient.Compute and the weights are W and b. The learning rate is how fast the model learns; this value determines how much the weights are changed based on their gradient.

Weight imprinting

namespace coral
struct ImprintingClass

Public Members

std::vector<float> weights

Sum of the weights generated by imprinting extractor.

bool trainable

Indicates whether the weights can be updated or not.

class ImprintingEngine
#include <imprinting_engine.h>

Class that implements Imprinted Weights transfer learning method proposed in paper https://arxiv.org/pdf/1712.07136.pdf.

Public Functions

ImprintingEngine(const ImprintingEngine&) = delete
ImprintingEngine &operator=(const ImprintingEngine&) = delete
absl::Span<const char> ExtractorModelBuffer() const

Returns extractor model buffer.

absl::Status SerializeModel(flatbuffers::FlatBufferBuilder *fbb)

Returns serialzied bytes of re-trained model.

The model will contain: [embedding_extractors] -> L2Norm -> Conv2d -> Mul -> Reshape -> Softmax

absl::Status Train(absl::Span<const float> embedding, int class_id)

Trains the model with the given embedding vector of a certain category.

Inputs: embedding vector

If training a new category, the class id should be exactly of the next class.

If training existing category with more images, the imprinting engine must be under keep_classes mode.

Call this function multiple times to train multiple different categories.

const std::vector<ImprintingClass> &GetClasses() const

Returns currently trained classes.

int embedding_dim() const

Returns number of embedding dimenssions for each class.

Public Static Functions

std::unique_ptr<ImprintingEngine> Create(std::unique_ptr<ImprintingModel> model, bool keep_classes)

Validates input model and initializes embedding_extractor.

The input model comes with L2Norm layer. It is supposed to be a full classification model.

Users can choose whether to keep previous classes by setting keep_classes to true or false.

Returns pair of EdgeTpuApiStatus and error message. Error message is only relevant when status is not kEdgeTpuApiOk.

Private Functions

ImprintingEngine(std::unique_ptr<ImprintingModel> model)

Private Members

std::unique_ptr<ImprintingModel> model_
std::vector<ImprintingClass> classes_
class ImprintingModel

Public Functions

absl::Span<const char> ExtractorModelBuffer() const

Returns extractor model buffer by truncating the prototype model.

std::vector<ImprintingClass> LoadExistingClasses() const

Returns classes from the prototype model.

absl::Status SerializeModel(const std::vector<ImprintingClass> &classes, flatbuffers::FlatBufferBuilder *fbb)

Serializes prototype model with the given array of classes.

int embedding_dim() const

Returns embedding vector dimension.

Public Static Functions

absl::Status Create(const tflite::Model &prototype, std::unique_ptr<ImprintingModel> *out_model)

Creates ImprintingModel instance from tflite::Model.

std::unique_ptr<ImprintingModel> CreateOrDie(const tflite::Model &prototype)

Creates ImprintingModel instance from tflite::Model or crashes on error.

Private Functions

ImprintingModel() = default

Private Members

std::unique_ptr<tflite::ModelT> model_t_
flatbuffers::FlatBufferBuilder extractor_fbb_
float fc_quant_scale_
int64_t fc_quant_zero_point_
int embedding_dim_