[Regression] model-runner not able to execute convolution single node graph

[Verma-Rajat]

After integration of #3452, model-runner is unable to execute attached single node caffe2 graph.

Command: model-runner -m predict_net.pb -m init_net.pb -backend=Interpreter
-convert-to-fp16

Error log:
In 'op__4' From 'predict_net.pb'
input 0
ConvertTo
name : op__4
Input : float16<1 x 64 x 112 x 112>
users : 1

Conv_test_0.zip

Result : float<1 x 64 x 112 x 112>

Mismatching layouts:
Provided layout
Layout: NCHW [name = N : alignment = 1 : index = 0, name = C : alignment = 1 : index = 1, name = H : alignment = 1 : index = 2, name = W : alignment = 1 : index = 3]
Expected layout
Layout: NHWC [name = N : alignment = 1 : index = 0, name = H : alignment = 1 : index = 1, name = W : alignment = 1 : index = 2, name = C : alignment = 1 : index = 3]
From 'predict_net.pb'
Expected correct Glow canonical layouts for the graph
For comparison LHS Equal RHS with:
LHS: 0
RHS: 1
model-runner: ../lib/Converter/FunctionConverter.cpp:205: void glow::FunctionConverter::convert(): Assertion `function_.verify() && "Conversion led to invalid function"' failed.
#0 0x000000000080fd7a llvm::sys::PrintStackTrace(llvm::raw_ostream&) (../../../../glow/build_debug/bin/model-runner+0x80fd7a)
#1 0x000000000080dd0c llvm::sys::RunSignalHandlers() (../../../../glow/build_debug/bin/model-runner+0x80dd0c)
#2 0x000000000080de77 SignalHandler(int) (../../../../glow/build_debug/bin/model-runner+0x80de77)
#3 0x00007f92ba295390 __restore_rt (/lib/x86_64-linux-gnu/libpthread.so.0+0x11390)
#4 0x00007f92b941f428 gsignal /build/glibc-LK5gWL/glibc-2.23/signal/../sysdeps/unix/sysv/linux/raise.c:54:0
#5 0x00007f92b942102a abort /build/glibc-LK5gWL/glibc-2.23/stdlib/abort.c:91:0
#6 0x00007f92b9417bd7 __assert_fail_base /build/glibc-LK5gWL/glibc-2.23/assert/assert.c:92:0
#7 0x00007f92b9417c82 (/lib/x86_64-linux-gnu/libc.so.6+0x2dc82)
#8 0x000000000264609c glow::FunctionConverter::convert() /work/glow/build_debug/../lib/Converter/FunctionConverter.cpp:206:0
#9 0x0000000002645257 glow::convertFunctionToFloat16(glow::Function*, glow::PrecisionConfiguration const&) /work/glow/build_debug/../lib/Converter/Float16Converter.cpp:82:0
#10 0x000000000077d2eb transformForPrecisionMode(glow::Backend const&, glow::Function*, glow::CompilationContext&) /work/glow/build_debug/../lib/Optimizer/GraphOptimizer/GraphOptimizer.cpp:3198:0
#11 0x000000000077db25 glow::optimizeFunction(glow::Function*, glow::Backend const&, glow::CompilationContext&) /work/glow/build_debug/../lib/Optimizer/GraphOptimizer/GraphOptimizer.cpp:3273:0
#12 0x000000000087d9d4 glow::Partitioner::createDAGWithoutPartition(llvm::StringRef, std::map<std::__cxx11::basic_string<char, std::char_traits, std::allocator >, glow::BackendInfo, std::less<std::__cxx11::basic_string<char, std::char_traits, std::allocator > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits, std::allocator > const, glow::BackendInfo> > >&, glow::CompilationContext&) /work/glow/build_debug/../lib/Partitioner/Partitioner.cpp:401:0
#13 0x00000000008803c3 glow::Partitioner::heterogeneousPartition(glow::CompilationContext&) /work/glow/build_debug/../lib/Partitioner/Partitioner.cpp:673:0
#14 0x000000000088286a glow::Partitioner::partition(glow::CompilationContext&) /work/glow/build_debug/../lib/Partitioner/Partitioner.cpp:887:0
#15 0x0000000000689b26 glow::runtime::HostManager::addNetwork(std::unique_ptr<glow::Module, std::default_deleteglow::Module >, glow::CompilationContext&, bool) /work/glow/build_debug/../lib/Runtime/HostManager/HostManager.cpp:185:0
#16 0x0000000000534994 glow::Loader::compile(glow::CompilationContext&) /work/glow/build_debug/../tools/loader/Loader.cpp:517:0
#17 0x000000000055bafe main /work/glow/build_debug/../tools/loader/ModelRunner.cpp:65:0
#18 0x00007f92b940a830 __libc_start_main /build/glibc-LK5gWL/glibc-2.23/csu/../csu/libc-start.c:325:0
#19 0x0000000000531f49 _start (../../../../glow/build_debug/bin/model-runner+0x531f49)
Aborted (core dumped)

[opti-mix]

cc @shajrawi

[pjaaskel]

My #3815 is likely the same issue.

[mciprian13]

My #3804 also is related to layout errors (actually I have quite a couple of models which I cannot compiler anymore, giving the same error).

My proposal is until the layout checking is tested with more models/cases, the layout verification should be disabled by default.

[mciprian13]

@shajrawi, in order to help with this problem, I provide herein two full ONNX models which have some layout problems:
models.zip
I tried to compile them using the model-compiler front-end tool like this:

mkdir -p build
model-compiler -backend=CPU -model=model1.onnx -emit-bundle=build
model-compiler -backend=CPU -model=model2.onnx -emit-bundle=build

The problems arise around the Pad node and the MatMul node.

[shajrawi]

@mciprian13 thanks for the test-cases! i will take a look / use them to fix #3804 - this is very similar to, but not quite, the same issue as the one in here.