Here's another method that is pretty darn quick......and potentially dangerous, as it doesn't care if there are files in the folder or not...it will be deleted regardless.
(defun vl-rmdir (path / fso)
(setq fso (vla-getinterfaceobject (vlax-get-acad-object) "Scripting.FileSystemObject"))
(vlax-invoke fso 'deletefolder path 0)
(vlax-release-object fso)
)
edited to localize the fso symbol