I have read the following sentence in an Italian language learning tool:
Perché è dovuto succedere questo?
I thought that the auxiliary verb in this sentence would be "avere" because it is the auxiliary verb used with "dovere". However, "essere" was chosen as auxiliary verb; I guess that is because it is the auxiliary verb used with "succedere", but that seems quite odd to me. Is that right? If so, what is the rule behind this?
From Serianni, Grammatica, XI.38
The auxiliary of a verb governing an infinite (modal or phraseological verb) usually is the same as the verb it governs: «ho dovuto lavorare» (as it is said: «ho lavorato»), «sono dovuto uscire» (as it is said: «sono uscito»).
Although:
If the infinite is an intransitive verb - regardless of its specific auxiliary - the governing verb can use the auxiliary avere as well: «non aveva più potuto ripartire (Levi, Cristo si è fermato a Eboli, 22).
If the infinite is essere, the auxiliary of the governing verb is avere: «avrebbe voluto esser cento braccia sotto terra» (Manzoni, I Promessi Sposi, IX 76).
If the infinite is in the passive voice, the auxiliary of the governing verb is the one corresponding to transitive verbs, that is avere: «Agnese e Perpetua [.] avevan voluto essere impiegate ne’ servizi» (Manzoni, I Promessi Sposi, XXX 29; in the same way as one would have, with an infinitive in the active voice: «avevan voluto impiegare»).
Similarly, from the website of the Accademia della Crusca:
For what concerns the usage of the auxilaries with the modal verb, it is a somewhat intricated question, but solvable in practice following a few rules:
If one chooses the auxiliary of the verb governed by the modal, one is never wrong: for example "Ha dovuto mangiare" (like "ha mangiato"); "è dovuto partire" (like "è partito").
If the verb following the modal is intransitive, both essere and avere can be used: for example "è dovuto uscire" or "ha dovuto uscire".
If the infinite carries an atonic pronoun (mi, si, ti, ci, vi) one needs to use "essere" if the pronoun is before the infinite (for example "non si è voluto alzare"), "avere" if the pronoun is after the infinite (for example "non ha voluto alzarsi").
If the modal is followed by the verb "essere", the auxiliary will always be "avere": for example "ha dovuto essere forte", "ha voluto essere il primo".
So usually the modal verb uses the same auxiliary as the infinitive it governs, but there are a few cases where avere is used when one would have expected essere. In the example you give in the question succedere is an intransitive verb which normally has the auxiliary essere, so both essere and avere can be used by the modal:
Perché è dovuto succedere questo?
Perché ha dovuto succedere questo?
Although in my experience the second form is less common, and sounds slightly off.
You are right. When modal verbs are used, the auxiliary verb used is the same as the auxiliary verb of the infinitive.
With reflexive and/or impersonal verbs, things are different. It depends on the position of the verbal clitic in the reflexive verb. For instance, consider the following sentences built with reflexive verb lavarsi:
