How do you stay unattached emotionally to the couple you are swinging with?

Me and my husband have been soft swingers for years. We recently found a couple we both really like and have had a few full swaps with them. I am finding it hard to not become emotionally attached to the other male partner. I feel a genuine connection with this other man. Is this normal? Is it a dangerous path? Should we stop seeing them? PLEASE EXPERIANCED COUPLES ONLEY