Even through you do not use ListBox object, you should use some array where you safe the indexes of each added VObjects. For more understanding of internal logic of Vector Objects I recommend you to read the following article: Using Vector Objects
It helps you to understand how to work with Vector Objects.
If you want to remove VObjects from layer you should use this code:
For Each i As Aurigma.GraphicsMill.WinControls.IVObject In MultiLayerViewer1.CurrentDesigner.VObjects
MultiLayerViewer1.CurrentLayer.VObjects.Remove(i)
Next