Equivalence Testing » Developer Zone

October 26, 2023

212

That is half 4 in my collection on MATLAB Take a look at through which we’ll have a look at the brand new equivalence testing performance it gives for MATLAB code. Beforehand, I’ve coated:

What’s equivalence testing and why do I want it?

In MATLAB, we are able to rework MATLAB code to C or C++ utilizing MATLAB Coder, or to .NET assemblies, Java courses, or Python packages utilizing MATLAB Compiler SDK. In such circumstances, we are going to usually need to confirm that the reworked model of our code behaves in precisely the identical manner as the unique MATLAB code. That is notably essential in security vital environments comparable to medical gadgets or ADAS the place it’s the reworked code that’s used within the remaining product.

Equivalence (or back-to-back) testing is a type of dynamic testing that checks that these two software program techniques produce the identical output when given the identical inputs. MATLAB Take a look at gives a brand new framework that makes it simple to write down such equivalence assessments.

Primary syntax

C/C++ code

The syntax for writing equivalence assessments utilizing the framework supplied by MATLAB Take a look at might be very acquainted for those who’re used to writing class-based unit assessments. Given a operate myAdd outlined as:

operate y = myAdd(a,b)

y = a + b;

finish

The equivalence take a look at for C code is:

classdef tEquivalence < matlabtest.coder.TestCase

strategies(Take a look at)

operate tMyAdd(testCase)

buildResults = construct(testCase,“myAdd”,Inputs={1,2});

executionResults = execute(testCase,buildResults);

verifyExecutionMatchesMATLAB(testCase,executionResults)

finish

There are some things to notice:

Our take a look at class inherits from matlabtest.coder.TestCase quite than the standard matlab.unittest.TestCase.
The brand new construct methodology builds the C binary that we are going to execute as a part of the take a look at. We specify the identify of the entry level operate we need to construct (myAdd) and a few default inputs. These “compile-time” inputs are required for code technology and are just like codegen’s -args flag. It’s an elective argument because it’s potential to write down features with no enter arguments.
The brand new execute methodology executes the C binary utilizing the default inputs. You can too specify completely different “run-time” inputs right here for those who so want.
The verifyExecutionMatchesMATLAB methodology does what it says – it produces a verification failure if the output from the C code doesn’t match that of MATLAB.

.NET, Java, Python

For MATLAB Compiler SDK workflows, the syntax may be very comparable:

classdef tDeployment < matlabtest.compiler.TestCase

strategies (Take a look at)

operate pythonEquivalence(testCase)

buildResults = construct(testCase,“myAdd.m”,“pythonPackage”);

executionResults = execute(testCase,buildResults,{1,2});

verifyExecutionMatchesMATLAB(testCase,executionResults);

finish

Once more, the documentation offers full particulars of the syntax comparable to methods to specify further MATLAB Compiler SDK choices.

Reusing unit assessments for equivalence testing

On this part I’m going to indicate how assessments which have be written to confirm a MATLAB algorithm may be repurposed for equivalence testing. The implementation shortestPath and corresponding 14 take a look at factors are based mostly on the instance venture that comes with MATLAB Take a look at.

Overview

The fundamental construction of the assessments is:

classdef tMATLABTests < matlab.unittest.TestCase

strategies (Take a look at)

operate aTestPoint(testCase)

% Generate inputs

adjacency = …

startIdx = …

endIdx = …

expectedResult = …

debugTxt = …

verifyPathLength(testCase, adjacency,startIdx,endIdx, expectedResult,debugTxt);

finish

the place the tactic verifyPathLength is of the shape:

operate verifyPathLength(testCase,adjacency,startIdx,endIdx, expectedResult,debugTxt)

% Execute the design

actualResult = shortestPath(adjacency, startIdx, endIdx);

% Affirm the anticipated

msgTxt = sprintf(‘Failure context: %s’, debugTxt);

testCase.verifyEqual(actualResult, expectedResult, msgTxt);

finish

I can use the Take a look at Browser so as to add tMATLABTests to my listing of assessments and run them:

Output of tMATLABTests within the Take a look at Browser.

verifyPathLength accommodates all of the code that we might want to modify to modify the assessments from being commonplace MATLAB unit assessments to equivalence assessments. Nevertheless, it could be good to retain the unique assessments while including the flexibility to run equivalence assessments with out duplicating code. We will do that by subclassing tMATLABTests and overloading the verifyPathLength methodology.

Equivalence take a look at for C code

To assist the equivalence testing of C code, I’m going to create a subclass of tMATLABTests known as tEquivalenceTestsForC. This code must do 4 issues:

Inherit from matlabtest.coder.TestCase along with tMATLABTests in order that we’ve entry to the equivalence testing performance that we’d like.
Construct the code – it’s best if we do that as soon as for the take a look at class quite than for every take a look at level individually. We will use a TestClassSetup block to do that and retailer the consequence as a property. There’s an extra catch that the adjacency enter isn’t of mounted measurement. We have to inform MATLAB Coder about this utilizing the coder.typeof operate.
Execute the code for the given inputs.
Examine the outcomes to MATLAB.

Right here’s the code:

classdef tEquivalenceTestsForC < tMATLABTests & matlabtest.coder.TestCase

properties (Entry = personal)

BuildResults

finish

strategies (TestClassSetup)

operate generateCode(testCase)

% Construct the operate as soon as and retailer the outcomes for reuse –

% way more environment friendly that constructing for each take a look at level.

% Outline enter arguments that can enable shortestPath to construct.

% Since adjacency may be of variable measurement, we use coder.typeof

% to declare that it is a double matrix that may be as much as 20×20

% and that each rows and columns can change measurement.

adjacency = coder.typeof(0,[20 20],[true true]);

inputs = { adjacency -1 2};

% Construct the operate

testCase.BuildResults = testCase.construct(“shortestPath”,Inputs=inputs);

finish

strategies

operate verifyPathLength(testCase,adjacency,startIdx,endIdx,~,~)

% Execute the operate in each MATLAB and C utilizing the inputs

% supplied by every take a look at level.

executionResults = testCase.execute(testCase.BuildResults,Inputs={adjacency,startIdx,endIdx});

% Confirm C matches MATLAB.

testCase.verifyExecutionMatchesMATLAB(executionResults)

finish

As earlier than, we are able to add tEquivalenceTestsForC to the Take a look at Browser and run the assessments:

Output of tEquivalenceTestsForC within the Take a look at Browser.

Equivalence take a look at for Python

To assist the equivalence testing of Python code, I’m going to create a second subclass of MATLABTests known as tEquivalenceTestsForPython. While this instance is for Python, you possibly can comply with the identical process for Java or .NET. Our code must do 4 issues:

Inherit from matlabtest.compiler.TestCase (observe compiler, not coder!) along with tMATLABTests in order that we’ve entry to the equivalence testing performance that we’d like.
Construct the code – it’s best if we do that as soon as for the take a look at class quite than for every take a look at level individually. We will use a TestClassSetup block to do that and retailer the consequence as a property.
Execute the code for the given inputs.
Examine the outcomes to MATLAB.

Be aware that since Python is a dynamically typed language, we don’t want to make use of the coder.typeof assemble that we had for C.

Right here’s the code:

classdef tEquivalenceTestsForPython < tMATLABTests & matlabtest.compiler.TestCase

properties (Entry = personal)

BuildResults

finish

strategies (TestClassSetup)

operate generateCode(testCase)

% Construct the operate as soon as and retailer the outcomes for reuse –

% way more environment friendly that constructing for each take a look at level.

testCase.BuildResults = testCase.construct(“shortestPath.m”,“pythonPackage”);

finish

strategies

operate verifyPathLength(testCase,adjacency,startIdx,endIdx,~,~)

% Execute the operate in each MATLAB and Python utilizing the

% inputs supplied by every take a look at level.

executionResults = testCase.execute(testCase.BuildResults,{adjacency,startIdx,endIdx});

% Confirm Python matches MATLAB.

testCase.verifyExecutionMatchesMATLAB(executionResults)

finish

We will then run the assessments within the Take a look at Browser. Be aware that you will need to have a appropriate model of Python put in – see pyenv for extra particulars.

Output of tEquivalenceTestsForPython within the Take a look at Browser.

On this case we’ve a take a look at failure which we might want to examine additional.

Conclusion

On this weblog put up we’ve appeared on the new performance out there in MATLAB Take a look at to assist equivalence testing when remodeling MATLAB code to C, C++, Python, Java, or .NET. Equivalence testing is especially essential in security vital environments comparable to medical gadgets or ADAS. We additionally checked out how inheritance may be leveraged to reuse current MATLAB assessments for equivalence testing.

Within the subsequent and remaining put up within the collection, I’ll discover dependency-based take a look at choice.

Code itemizing

shortestPath

operate pathLength = shortestPath(adjMatrix, startIdx, endIdx) %#codegen

% SHORTEST_PATH – Finds size of shortest path between nodes in a graph

%

% OUT = SHORTEST_PATH(ADJMTX, STARTIDX, ENDIDX) Takes a graph represented by

% its adjacency matrix ADJMTX together with two node STARTIDX, ENDIDX as

% inputs and returns a integer containing the size of the shortest path

% from STARTIDX to ENDIDX within the graph.

%% Validy testing on the inputs

% This code ought to by no means throw an error and as an alternative ought to return

% error codes for invlid inputs.

ErrorCode = 0;

pathLength = -1;

% Examine the validity of the adjacency matrix

if (~isAdjMatrixValid(adjMatrix))

ErrorCode = -9;

finish

% Examine the validity of the startIdx

if ~isNodeValid(startIdx)

ErrorCode = -19;

finish

% Examine the validity of the endIdx

if ~isNodeValid(endIdx)

ErrorCode = -29;

finish

[nodeCnt, n] = measurement(adjMatrix);

% Begin or finish node is just too giant

if startIdx > nodeCnt || endIdx > nodeCnt

ErrorCode = -99;

finish

% Begin or finish node is just too small

if startIdx < 1 || endIdx < 1

ErrorCode = -199;

finish

if (ErrorCode<0)

pathLength = ErrorCode;

return;

finish

%% Self-loop path is at all times 0

if startIdx == endIdx

pathLength = 0;

return;

finish

%% Dijkstra’s Algorithm

% Dijkstra’s Algorithm is used to iteratively discover the graph breadth

% first and replace the shortest path till we attain the top node.

% Initialization

max = realmax;

visited = false(1, nodeCnt);

% The space vector maintains the present recognized shortest path from

% the beginning node to each node. As nodes are processed one after the other

% the space vestor is up to date

distance = repmat(max, 1, nodeCnt);

distance(startIdx) = 0;

for iterStep = 1:nodeCnt

% At every iteration establish the present node to course of that

% isn’t but visited and has the smallest distance from the beginning.

% This breadth first search ensures that we are going to at all times attain nodes

% by the shortest potential path.

min = max;

nodeIdx = -1;

for v = 1:n

if ~visited(v) && distance(v) <= min

min = distance(v);

nodeIdx = v;

finish

% Cease iterating when the present distance is most as a result of

% this means no remaining nodes are reachable

if (min==max)

return;

finish

% Mark the present node visited and examine if that is finish index

visited(nodeIdx) = true;

if nodeIdx == endIdx

pathLength = distance(nodeIdx);

if (pathLength==realmax)

% No path exists so set distance to -1;

pathLength = -1;

finish

return;

finish

% Replace distances of unvisited nodes adjoining to the present node

for v = 1:nodeCnt

if(~visited(v) && adjMatrix(nodeIdx, v) ~= 0 && distance(nodeIdx) ~= max)

distVal = distance(nodeIdx) + adjMatrix(nodeIdx, v);

if distVal < distance(v)

distance(v) = distVal;

finish

operate out = isNodeValid(node)

% For full protection we have to create detrimental assessments that make every

% successively make every validity situation false

if(isscalar(node) && isnumeric(node) && ~isinf(node) && ground(node) == node)

out = true;

else

out = false;

finish

operate out = isAdjMatrixValid(adjMatrix)

% have to be a sq. matrix with solely 0, 1, or realmax entries.

[m, n] = measurement(adjMatrix);

% For full protection we have to create detrimental assessments that make every

% successively make every validity situation false

if (m==n) && isempty(discover((adjMatrix ~= 0) & (adjMatrix ~= 1), 1))

out = true;

else

out = false;

finish

tMATLABTests

classdef tMATLABTests < tCommon

strategies (Take a look at,TestTags=“InputTests”)

operate check_invalid_start_1(testCase)

adjMatrix = testCase.graph_straight_seq();

startIdx = -1;

endIdx = 2;

expOut = -199;

verifyPathLength(testCase, adjMatrix, startIdx, endIdx, expOut, ‘Invalid begin index, idx<1’);

finish

operate check_invalid_start_2(testCase)

adjMatrix = testCase.graph_straight_seq();

startIdx = 12;

endIdx = 2;

expOut = -99;

verifyPathLength(testCase, adjMatrix, startIdx, endIdx, expOut, ‘Invalid begin index, idx>NodeCnt’);

finish

operate check_invalid_end_1(testCase)

adjMatrix = testCase.graph_straight_seq();

startIdx = 1;

endIdx = -3;

expOut = -199;

verifyPathLength(testCase, adjMatrix, startIdx, endIdx, expOut, ‘Invalid finish index, idx<1’);

finish

operate check_invalid_end_2(testCase)

adjMatrix = testCase.graph_straight_seq();

startIdx = 1;

endIdx = 12;

expOut = -99;

verifyPathLength(testCase, adjMatrix, startIdx, endIdx, expOut, ‘Invalid finish index, idx>NodeCnt’);

finish

strategies(Take a look at,TestTags=“EdgelessTests”)

operate check_edgeless_graph(testCase)

adjMatrix = zeros(20,20);

startIdx = 1;

endIdx = 18;

expOut = -1;

verifyPathLength(testCase, adjMatrix, startIdx, endIdx, expOut, ‘Edgeless graph’);

finish

operate check_edgeless_start(testCase)

adjMatrix = testCase.graph_some_nodes_edgeless();

startIdx = 1;

endIdx = 4;

expOut = -1;

verifyPathLength(testCase, adjMatrix, startIdx, endIdx, expOut, ‘Edgeless graph’);

finish

operate check_edgeless_end(testCase)

adjMatrix = testCase.graph_some_nodes_edgeless();

startIdx = 3;

endIdx = 1;

expOut = -1;

verifyPathLength(testCase, adjMatrix, startIdx, endIdx, expOut, ‘Edgeless graph’);

finish

operate check_edgeless_graph_self_loop(testCase)

adjMatrix = zeros(20,20);

startIdx = 16;

endIdx = 16;

expOut = 0;

verifyPathLength(testCase, adjMatrix, startIdx, endIdx, expOut, ‘Self loop in edgeless graph’);

finish

strategies (Take a look at)

operate check_longest_path(testCase)

adjMatrix = testCase.graph_straight_seq();

startIdx = 1;

endIdx = 4;

expOut = 3;

verifyPathLength(testCase, adjMatrix, startIdx, endIdx, expOut, ‘Longest theoretic path’);

finish

operate check_unity_path(testCase)

adjMatrix = testCase.graph_all_edge();

startIdx = 2;

endIdx = 3;

expOut = 1;

verifyPathLength(testCase, adjMatrix, startIdx, endIdx, expOut, ‘Path size 1’);

finish

operate check_non_unique(testCase)

adjMatrix = testCase.graph_square();

startIdx = 4;

endIdx = 2;

expOut = 2;

verifyPathLength(testCase, adjMatrix, startIdx, endIdx, expOut, ‘Non-unique path’);

finish

operate check_no_path(testCase)

adjMatrix = testCase.graph_disconnected_components();

startIdx = 1;

endIdx = 5;

expOut = -1;

verifyPathLength(testCase, adjMatrix, startIdx, endIdx, expOut, ‘No path’);

finish

operate check_start_end_same(testCase)

adjMatrix = testCase.graph_all_edge();

startIdx = 3;

endIdx = 3;

expOut = 0;

verifyPathLength(testCase, adjMatrix, startIdx, endIdx, expOut, ‘Begin and finish index similar’);

finish

operate check_invalid_idx_empty_adj(testCase)

adjMatrix = [];

startIdx = 1;

endIdx = 1;

expOut = -99;

verifyPathLength(testCase, adjMatrix, startIdx, endIdx, expOut, ‘Degenerate empty graph’);

finish

strategies (Static)

% Utility features to create widespread adjacency graph matrices

operate adj = graph_straight_seq()

% Create the graph:

% 1—2—3—4

adj = [0 1 0 0; …

1 0 1 0; …

0 1 0 1; …

0 0 1 0];

finish

operate adj = graph_square()

% Create the graph:

% 1—2

% | |

% 4—3

adj = [0 1 0 1; …

1 0 1 0; …

0 1 0 1; …

1 0 1 0];

finish

operate adj = graph_all_edge()

% Create the graph:

% 1—2

% | /|

% |/ |

% 4—3

adj = [0 1 1 1; …

1 0 1 1; …

1 1 0 1; …

1 1 1 0];

finish

operate adj = graph_disconnected_components()

% Create the graph:

% 2 5

% / /

% 1—3 4—6

adj = [0 1 1 0 0 0; …

1 0 1 0 0 0; …

1 1 0 0 0 0; …

0 0 0 0 1 1; …

0 0 0 1 0 1; …

0 0 0 1 1 0];

finish

operate adj = graph_some_nodes_edgeless()

% Create the graph:

% 2

% /

% 4—3

%

% Nodes 1, 5, 6 are edgeless

adj = [0 0 0 0 0 0; …

0 0 1 1 0 0; …

0 1 0 1 0 0; …

0 1 1 0 0 0; …

0 0 0 0 0 0; …

0 0 0 0 0 0];

finish

strategies

operate verifyPathLength(testCase,adjacency,startIdx,endIdx,expectedResult,debugTxt)

% Execute the design

actualResult = shortestPath(adjacency, startIdx, endIdx);

% Affirm the anticipated

msgTxt = sprintf(‘Failure context: %s’, debugTxt);

testCase.verifyEqual(actualResult, expectedResult, msgTxt);

finish

Previous articleThe se7en lethal sins of Go together with John Gregory (Go Time #294) |> Changelog

Next articleDependency-based Take a look at Choice » Developer Zone

Equivalence Testing » Developer Zone

What’s equivalence testing and why do I want it?

Primary syntax

C/C++ code

.NET, Java, Python

Reusing unit assessments for equivalence testing

Overview

Equivalence take a look at for C code

Equivalence take a look at for Python

Conclusion

Code itemizing

shortestPath

tMATLABTests

What’s the Polar Decomposition? – Nick Higham

Pole Vault – 2024 Olympics replace » Man on Simulink

Inspiring Younger Communications Engineers: The Air Power Analysis Lab’s SDR Problem » Scholar Lounge

LEAVE A REPLY Cancel reply

Most Popular

#CoffeeWithRW: from Tech Author to Analytics Engineer

The Delegate RequestDelegate doesn’t take X arguments – Experiences with minimal APIs – blogs.cninnovation.com

Eleventy Starter Mission Updates

Tips on how to Set up an Entry Level

Recent Comments

ABOUT US

POPULAR POSTS

#CoffeeWithRW: from Tech Author to Analytics Engineer

The Delegate RequestDelegate doesn’t take X arguments – Experiences with minimal APIs – blogs.cninnovation.com

Eleventy Starter Mission Updates

POPULAR CATEGORY