GitLab

I came across a pathological mesh example that triggered a heap corruption in version %OpenMesh 8.0.

The following is a minimal failing example, which will fail when run with AddressSanitizer (heap-buffer-overflow in OpenMesh::PolyConnectivity::add_face(OpenMesh::VertexHandle const*, unsigned long)).

#include <OpenMesh/Core/Mesh/PolyMesh_ArrayKernelT.hh>

typedef OpenMesh::PolyMesh_ArrayKernelT<>  MyMesh;

int main()
{
    MyMesh mesh;
    
    const auto a = mesh.add_vertex(MyMesh::Point{-6.95757 -0.418557  -7.25885});
    const auto b = mesh.add_vertex(MyMesh::Point{-7.03569 -0.128014  -7.26395});
    
    mesh.add_face(a,b,a);

    return 0;
}

When run with assertions enabled, this triggers:

OpenMesh/Core/Mesh/PolyConnectivity.cc:272: OpenMesh::ArrayKernel::FaceHandle OpenMesh::PolyConnectivity
::add_face(const VertexHandle*, size_t): Assertion `boundary_prev.is_valid()' failed.