You need to remain adjacent to them the entire time for your dance of weakness, yes. It is a little complicated but you can still drag them quite far - you don't have to pull them adjacent to you in the same direction you moved, so the two of you can swap sides as you cross the map together.