Importing Fashions from TensorFlow, PyTorch, and ONNX » Deep Studying

How do you import a mannequin created in TensorFlow™ or PyTorch™ and convert it into MATLAB Code?

First, have in mind there are completely different choices for working with deep studying fashions in MATLAB.

1. Utilizing fashions created in MATLAB with Deep Studying Toolbox
2. Changing from different frameworks into MATLAB
3. Co-executing fashions from different frameworks with MATLAB

This weblog put up focuses on possibility 2, however you may see the advantages of selecting between these 3 choices utilizing this comparability desk beneath, and consider all obtainable fashions to instantly import into MATLAB utilizing the MATLAB Deep Studying Mannequin Hub.

How sophisticated is it to transform a mannequin? In lots of circumstances it is fairly easy. In different circumstances, there could also be a couple of further steps earlier than efficiently changing a third get together mannequin into MATLAB. The aim of this weblog is to provide steering on methods to navigate these steps.

Here’s what we’ll cowl at this time:

Comply with alongside step-by-step on this put up, and obtain the code right here: https://github.com/matlab-deep-learning/Picture-Classification-in-MATLAB-Utilizing-Transformed-TensorFlow-Mannequin

Importing TensorFlow Fashions

Importing TensorFlow Fashions utilizing SavedModel Format

When TensorFlow 2.0 grew to become obtainable (Sep 2019), the SavedModel format was launched and is now the popular technique for saving pretrained fashions. Here is the code to import this mannequin format.

Use tf.keras.functions.resnet_v2.ResNet50V2 to instantiate the ResNet50V2 mannequin.

The ResNet50V2 mannequin is skilled with pictures from the ImageNet database. Get the category names from SqueezeNet, which can also be skilled with ImageNet pictures.

squeezeNet = squeezenet;
ClassNames = squeezeNet.Layers(finish).Courses;

Import ResNet50V2 within the SavedModel format through the use of the importTensorFlowNetwork perform.

internet = importTensorFlowNetwork("ResNet50V2",...
   OutputLayerType="classification",...
   Courses=ClassNames)

Importing TensorFlow Fashions utilizing the HDF5 (Keras) Format

Import ResNet50V2 within the HDF5 format through the use of the importKerasNetwork perform.

internet = importKerasNetwork("ResNet50V2.h5",...
  OutputLayerType="classification",...
  Courses=ClassNames)

The software program throws a warning concerning the Keras model of the imported mannequin. Within the documentation web page of every import and export perform, there’s a Limitations part the place the supported TensorFlow or ONNX™ variations are described. For instance, see Limitations of importTensorFlowNetwork.

Tip: Despite the fact that you may efficiently import ResNet50V2 with each importKerasNetwork and importTensorFlowNetwork, I like to recommend utilizing importTensorFlowNetwork. Causes for this turn out to be obvious when attempting to import a extra complicated mannequin with layers not supported for conversion into built-in MATLAB layers, as proven within the subsequent part.

Importing Extra Complicated TensorFlow Fashions

This part reveals methods to import the TensorFlow mannequin PNASNetLargeV4 which, in contrast to ResNet50V2, incorporates a fancy feature-vector layer. This layer can’t be transformed instantly right into a built-in MATLAB layer, so you may want a customized layer. The excellent news is that MATLAB will autogenerate customized layers for you.

To get the PNASNetLargeV4 mannequin, run this code in Python®. You could find the code at https://tfhub.dev/google/imagenet/pnasnet_large/feature_vector/4.

import tensorflow as tf
import tensorflow_hub as hub
mannequin = tf.keras.Sequential([
  hub.KerasLayer("https://tfhub.dev/google/imagenet/pnasnet_large/feature_vector/4",
  trainable=True),
  tf.keras.layers.Dense(1000, activation='softmax')
  ])
mannequin.construct([None, 331, 331, 3])

The PNASNetLargeV4 mannequin is skilled with pictures from the ImageNet database.

Importing utilizing the HDF5 (Keras) Format (not really useful)

If you import PNASNetLargeV4, the software program can’t convert a number of the TensorFlow layers into equal built-in MATLAB layers.

First, attempt to import the mannequin in HDF5 format through the use of the importKerasNetwork perform.

internet = importKerasNetwork("PNASNetLargeV4.h5",...
  OutputLayerType="classification",...
  Courses=ClassNames)

The error message suggests utilizing importKerasLayers to import the community structure. importKerasLayers will import the mannequin as a layer graph with placeholder layers. You have to manually change the placeholder layers to make use of the community for prediction or to retrain the community. You’ll be able to change a placeholder layer with a handwritten layer (documentation instance: Assemble Community from Pretrained Keras Layers) or functionLayer (documentation instance: Exchange Unsupported Keras Layer with Operate Layer).

Importing utilizing the SavedModel Format (really useful)

Alternatively, you should utilize the importTensorFlowNetwork perform. It differs from its Keras equal as a result of it has a singular function; it might autogenerate customized layers. Throughout importing, importTensorFlowNetwork generates customized layers instead of the TensorFlow layers that aren’t supported for conversion into built-in MATLAB layers.

internet = importTensorFlowNetwork("PNASNetLargeV4",...
OutputLayerType="classification",...
Courses=ClassNames)

Tip: A fast method to test if customized layers had been generated for you is to examine the package deal that was created throughout the import course of.  E.g., for PNASNetLargeV4, the import perform generates one massive customized layer (kKerasLayer2Layer29666.m):

What if there isn’t a SavedModel model of a mannequin, simply HDF5?

On this state of affairs, I might suggest to attempt to convert the TensorFlow mannequin from HDF5 to saved mannequin format. Generally, that is the way in which to carry out the conversion from inside TensorFlow:

Nonetheless, you would possibly encounter a use case that requires “mannequin surgical procedure”. For instance, if an unsupported layer is written within the older model, and also you try to reserve it within the newer SavedModel model, TensorFlow might throw an error stating that it would not know methods to convert the mannequin. On this state of affairs you should do one of many following:

1. Manually re-code the layer within the newer TensorFlow SavedModel format previous to importing the mannequin into MATLAB.
2. Import the older mannequin into MATLAB utilizing importKerasLayers. It should create a placeholder layer instead of the customized layer, which the software program couldn’t import. Then, from inside MATLAB you manually re-code the layer.

Each choices require you to jot down layers in both TensorFlow (possibility 1) or MATLAB (possibility 2).

Importing PyTorch Fashions

First, you’ll want to convert the PyTorch mannequin to the ONNX mannequin format by following the directions in Exporting a Mannequin from PyTorch to ONNX. Then, you may import the ONNX mannequin into MATLAB through the use of importONNXNetwork. Right here, we present you methods to import the squeezenet.onnx mannequin, which you can even discover within the PyTorch Mannequin Zoo.

internet = importONNXNetwork("squeezeNet.onnx");

The importONNXNetwork perform may also generate customized layers when the software program can’t convert ONNX operators into equal built-in MATLAB layers. Import the shufflenet-9.onnx mannequin with autogenerated customized layers. By default, importONNXNetwork returns a DAGNetwork object that is able to use for prediction.

internet = importONNXNetwork("shufflenet-9.onnx",PackageName="shuffleNet");

importONNXNetwork saves the customized layers within the package deal +shuffleNet, within the present folder, equally to importTensorFlowNetwork.

You may also export a skilled Deep Studying Toolbox community to the ONNX mannequin format through the use of the exportONNXNetwork perform.

exportONNXNetwork(internet,"myNet.onnx")

This instance reveals methods to import a pretrained TensorFlow mannequin within the SavedModel format, and use the imported community to categorise a picture. The TensorFlow mannequin incorporates layers that aren’t supported for conversion into built-in MATLAB layers. The software program robotically generates customized layers whenever you import these layers.

Import Pretrained TensorFlow Mannequin

Use tf.keras.functions.efficientnet_v2.EfficientNetV2L to instantiate the EfficientV2L mannequin.

The EfficientNetV2L mannequin is skilled with pictures from the ImageNet database. Get the category names from SqueezeNet, which can also be skilled with ImageNet pictures.

squeezeNet = squeezenet;
ClassNames = squeezeNet.Layers(finish).Courses;

Import the TensorFlow mannequin EfficientNetV2L within the SavedModel format through the use of the importTensorFlowNetwork perform. Specify the output layer kind for a picture classification drawback.

internet = importTensorFlowNetwork("EfficientNetV2L",...
  OutputLayerType="classification",...
  Courses=ClassNames);

Discover the autogenerated customized layers.

PackageName="+EfficientNetV2L";
s = what(['.' PackageName]); 
ind = zeros(1,size(s.m));

for i = 1:size(internet.Layers)
  for j = 1:size(s.m)
    if strcmpi(class(internet.Layers(i)),[PackageName(2:end) '.' s.m{j(1:end-2)])
      ind(j) = i;
    finish
  finish
finish

show(ind)

ind = 2

show(internet.Layers(ind))

Analyze the imported community.

analyzeNetwork(internet)

As anticipated, the imported community, which incorporates autogenerated customized layers, reveals no warnings or errors within the community analyzer. This implies the imported community is able to use for prediction.

Learn and Preprocess Enter Picture

TensorFlow offers the tf.keras.functions.efficientnet_v2.preprocess_input technique to preprocess picture enter information for the EfficientNetV2L mannequin. Right here, we replicate the enter preprocessing by resizing, rescaling, and normalizing the enter picture.

Learn the picture you need to classify and show the scale of the picture. The picture is 792-by-1056 pixels and has three colour channels (RGB).

Im = imread("peacock.jpg");
dimension(Im)

Resize the picture to the enter dimension of the community.

InputSize = internet.Layers(1).InputSize;
Im = imresize(Im,InputSize(1:2));

The inputs to  EfficientNetV2L require additional preprocessing. Rescale the picture. Normalize the picture by subtracting the coaching pictures imply and dividing by the coaching pictures customary deviation. To seek out the values that you must use, see https://github.com/keras-team/keras-applications/blob/grasp/keras_applications/imagenet_utils.py.

ImProcessed = rescale(Im,0,1);
meanIm = [0.485 0.456 0.406];
stdIm = [0.229 0.224 0.225];
ImProcessed = (ImProcessed-reshape(meanIm,[1 1 3]))./reshape(stdIm,[1 1 3]);

Classify Picture Utilizing the Imported Community

Classify the picture utilizing the imported community. Present the picture with the classification label.

label = classify(internet,Im);
imshow(Im)
title(strcat("Predicted label: ",string(label)))

You may also use the imported community with the Predict block of the Deep Studying Toolbox, to categorise a picture in Simulink. For an instance, see the earlier put up: Bringing TensorFlow Fashions into MATLAB. The truth that the imported community incorporates autogenerated customized layers doesn’t hinder modeling in Simulink.

Interoperability with deep studying frameworks, akin to TensorFlow and PyTorch, allows you to do extra in MATLAB and Simulink. It offers the pliability to take full benefit of the MATLAB ecosystem and combine it with assets developed by the open-source neighborhood. It additionally allows you to mix workflows that embody data-centric preprocessing, mannequin tuning, mannequin compression, mannequin integration, and computerized code era for fashions developed exterior MATLAB.

Comparability of capabilities for working with deep studying fashions in MATLAB

Functionality	Fashions created utilizing Deep Studying Toolbox	Fashions Transformed from different FrameWorks	Co-Execution
Integrates with pre and put up processing in MATLAB
Requires set up of merchandise solely
Helps debugging from MATLAB
Inference efficiency in MATLAB and Simulink
Accessible MATLAB utility examples
Requires no datatype conversion and information reformatting
Protection for embedded code era with MATLAB Coder, GPU Coder & Deep Studying HDL Toolbox
Requires no further libraries for standalone deployment from MATLAB Compiler
Entry fashionable fashions in a single line of code
Entry to open-source fashions from TensorFlow and PyTorch

	Most assist
	Restricted assist
	No assist

Extra About Help Packages

You have to have assist packages to run the import and export capabilities in Deep Studying Toolbox. If the assist package deal will not be put in, every perform offers a obtain hyperlink to the corresponding assist package deal within the Add-On Explorer. A really useful apply is to obtain the assist package deal to the default location for the model of MATLAB you might be working. You may also instantly obtain the assist packages from File Trade.

Desk: Help packages required to run import and export capabilities.

Desk: Evaluating high-level options of the three capabilities that may import deep studying networks.

	importTensorFlowNetwork	importKerasNetwork	importONNXNetwork
Help for TensorFlow	✅*	✅**
Help for PyTorch			✅
Help for autogeneration of customized layers	✅		✅

* At the moment helps TensorFlow variations 2.0-2.6 (completely examined). Generally, fashions saved in newer TensorFlow variations are additionally importable.
Requires Deep Studying Toolbox Converter for TensorFlow Fashions

** Helps TensorFlow-Keras variations as much as 2.2.4, with restricted assist for variations 2.2.5 to 2.4.0.
Requires Deep Studying Toolbox Converter for TensorFlow Fashions

The software program within the assist packages is up to date month-to-month. These month-to-month updates can embody new layer assist for import and export, up to date TensorFlow and ONNX model assist, and bug fixes. Main options are launched with the final launch.

 

 

Get the MATLAB code

Revealed with MATLAB® R2020b