Ok... if anyone else runs into this, I was able to come up with something that made things work. Take a look at Icacls and the things it can do. Here are the commands that I ran in my application that might help someone. NOTE: AppPath + "\Output" and inPath are the same thing and they are both talking about a directory.
s.Execute("takeown /f " + AppPath + "\Output")
s.Execute("Icacls " + inPath + " /verify /t /c /l /q")
s.Execute("Icacls " + inPath + " /reset /t /c /l /q")
s.Execute("Icacls " + inPath + " /grant Administrator:(D,WDAC) /inheritance:e|" + inPath)
The first thing I do is take ownership of the directory (I'm an admin)
next, verify: finds all files whose ACL is not in canonical form or whose
lengths are inconsistent with ACE counts.
then, reset: replaces ACLs with default inherited ACLs for all matching files.
and last, grant permission: Will grant the user Administrator Delete and Write DAC permissions to file.
After doing all of the above, I don't have a problem deleting the directory, even if some of the file the attributes are mangled.
So, I hope this helps someone.