buildman: Reinstate removal of temp output dir in tests

This was dropped my mistake. Reinstate it.

Signed-off-by: Simon Glass <sjg@chromium.org>
Fixes: d829f1217c6 ("bulidman: Add support for a simple build")
1 file changed