Handle and cancel values apply to the element being dragged. So if elements are grouped and you want to drag them as a group you need to follow the rules of the element being dragged.
You can only drag me from here
You can only drag me from outside here