When a child role is removed from a parent, the RRSD processes the event with a single thread and blocks all other events in the queue. When the parent has a lot of user assignments this will take a considerable time to complete as the recalculations for each user are done in series.
These recalculations seem to be independent of each other and should be good candidates for multi-threading. This would reduce the time it takes to process the event.