With current Doslib 9.02, Win 10, AutoCAD 2017 I have this simple code:
(setq result (dos_msgboxex "Text " "titel" (list "a" "b" "c" "d" "e") 4))
It works fine and reurns the zero-based number of the button, but ...
When I press "Esc", it returns the number of the last button, although it should return "nil or an error" (see Help-file).
Who can confirm and / or solve this behaviour?