I understand "agita la mano" is the proper way to say "wave the hand," but would like to know whether it is also correct Italian to say "ondeggia la mano"?
Not at all. First, it's not grammatically correct, since ondeggiare is not a transitive verb.1
And, secondly, even if it were (for instance substituting the verb with far ondeggiare), it would not be an idiom as “wave one's hand” or agitare la mano (or salutare con la mano) are.
Thirdly, (far) ondeggiare describes a specific, unusual gesture, not the fact that you're greeting someone. It's a bit like saying, perhaps, “wiggle one's hand”. Can you say it? Why not. Would you? Perhaps, in some special situation, to describe a particular movement someone does, but not in a usual, idiomatic way.
1 Thanks to user @linuxfansaysReinstateMonica for remarking this. It might be used transitively, but it's an old-fashioned or rare usage: see items 22 and 23 in the entry in Battaglia: https://www.gdli.it/sala-lettura/vol/11?seq=977