It was an interesting video, I usually enjoy linus' vids. A couple of things I noticed, having fans in the build sort of kills the point of passive cooling. The idea for passive cooling is silence, there are 4 fans in the case - 2 in the case itself, 2 on the gpu. So it's not a fanless system. Also, the rear exhaust fan is turned around as intake and with the proximity of the rear 'intake' to the cpu cooler tower it's not quite 'passive' either. Instead the cooling fan is just moved slightly further away from the cooling fins. The system at idle was pretty quiet, though at load it wasn't 'silent' at all. He went through an awful lot of trouble to build a 'quiet' pc, not a silent one. The biggest noise offender is likely the gpu, no real surprise there. It usually is, even if the fans can be turned off when the gpu is idling. Obviously if you have a higher end gtx 970/980 you're going to be using it for more than to provide a display to the screen. Gaming, rendering or gpgpu computing will put it under load and fan noise will be back in effect.
Try using the dark rock pro 3 with the fans in place. I can't hear them at full speed with an overclocked 4690k using an enthoo pro case which isn't a 'quiet' case with any extra padding or anything. Being at the very outside of the case, if done the same way as linus did it the two case fans will be louder guaranteed. Case padding won't help with fans that are directly in line with open air outside of the case and the first ones to be picked up by the ear provided the gpu fans are off/silent.
To get a better point of reference I just turned off every case fan leaving only the cpu, gpu and power supply fans running. Then I briefly stopped my gpu fan with my finger, the only tiny bit of noise than can be heard is the psu fan. I took the side panel off the case and had to all but put my head into the case with my ear directly next to the dark rock pro 3 to hear the fan at all. Be quiet lives up to their name. Since linus wasn't building a truly passive system in the first place and given how quiet the cooler is there's really no reason to remove the fans from it. My 4690k (oc'd to 4.6) is idling at 33-34c at 7% usage with an ambient room temp of 24c. That's with both cpu fans on and making no discernible noise with a completely opened up case on my desk less than 1m from my ears. Factor in the silent base case with the muffled panels and it really will have no effect other than require the work to remove the fans.
Edit: I ran a stress test with all the fans off and it's the first my cpu fans have ever revved up. They do make some noise though not bad. Had never noticed it before even while stress testing because I've never done so with every other fan off. Especially while pushing a decent overclock. So it is possible for them to contribute a little to the noise, though running an oc'd cpu under load with no case fans on at all is a highly unlikely scenario. They don't rev up like that with the case fans on. Maybe because with the rear exhaust so close to the cooler itself and the fan off, it makes a better wall than a vent. Heat was building up during the oc'd stress test with the exhaust fan being in the way not running.
The fans didn't rev up until the cpu hit 76c, a temp my cpu doesn't see during normal operation including gaming (more like low to mid 60's). I still think it would be best to leave the fans in place. In this instance, in spite of noise from fans running full out - they needed to. In other words, if they ramp up there's a need for it. Not having the fans would lead to an overheated cpu. I would rather know the fans are there to keep the system temps safe than have to bite my knuckles watching my thermals without them in the event temps rose that high with nothing to counter them. The potential running the cooler as a passive one is there for overheating and no sign of it unless you got a bsod or shut down (or thermal throttling) falsely thinking everything is fine because it's quiet.