The attenuator project has been going well, but we are still working on getting the signal isolation secure enough to take the WiFi signal level to less than about -80 dB. After a bit of research, we realized that our single-shielded SMA cables inside the attenuator were not really up to the task. They are made with RG-316 cable, which is only single shielded. It's hard to get solid numbers on how well it actually shields, but some similar cable was reported to shield at only -50 dB or so. For the attenuator to work well, we need shielding at > 90 dB.
It is actually fairly easy to order SMA to SMA cables that have double-shielded cable, I had just not realized the importance when ordering the previous batch. I put in an order for some cables using KSR-100 cable, which has an aluminium shield as well as woven shield, and claims > 90 dB shielding.
But, that will take a week or two to arrive... I had previously ordered a roll of copper tape, and thought wrapping my existing single-shielded cables with copper tape would add some much needed signal shielding. After a bit of fairly tedious work, all 6 cables were done and back in the attenuator.
I reconnected the AP and Station machines to the attenuator and ran some more tests. Throughput was around 340Mbps at lower attenuation levels, which is great performance. But, even when the attenuator was set to 95 dB attenuation, I still saw signal strength of around -71. This was significantly worse signal leakage than before! I then took some aluminium foil and wrapped the (fanless) AP and Station machines as completely as I could, being careful to crimp it in around the cables. After wasting several yards of aluminium foil, I was disappointed to see that I had at most shaved 1 dB off the leakage. I removed all the foil...
I knew that the SMA to U.fl pigtails inside the AP and Station were also single-shielded, and even thinner than the SMA to SMA cables. So, I took those machines apart and copper-clad those cables a well. While looking closely at the U.fl ends, I noticed that there is a small bit of cable that is without shielding. I tried to wrap this part with copper tape as well. I again connected everything together and ran some more tests. Now the signal level would go a bit lower, around -76 dB with full attenuation. This is still worse than before I copper-clad anything, but at least cladding the U.fl pigtails seemed to help a little.
At this point, I could only assume that adding copper cladding to the cables in the attenuator made them leak more instead of less. Perhaps the small amount of signal that leaks out of the cable shielding is then amplified by the copper tape I had added. I cut open a double-shielded LMR-195 cable and noticed that its foil layer was just under the woven metal shielding (and the woven shielding was under a rubber insulating layer). Maybe that makes all the difference. I removed all the copper clad cables and put the unmodified single-shielded cables back in place.
I directly connected some LMR-195 cables from the AP to the attenuator modules. I had to leave the Station modules using the single-shielded cable since I had only 3 of the correct type of SMA adapters for the LMR-195 cables. This brought total isolation back up to around -80 dB, right around where I started at the beginning of the day!
There are companies (such as rfmall.com) that sell double-shielded SMA to U.fl cables, so I ordered a few sets of pigtails from them. I'll try this all again when the new cables arrive.
In summary: WiFi signal isolation is not trivial. You have to pay close attention to cable types and connector construction, and solutions that appear obvious may actually exacerbate the problem!