Handle and cancel values apply to the element being dragged. So if elements are grouped and you want to drag them as a group you need to follow the rules of the element being dragged.

You can only drag me from here

You can only drag me from outside here