For example, there is a sentence in the textbook Nuovo Progetto Italiano:
Osservate le due immagini e dite quali differenze esistono.
Can I use "cosa" instead here? Thank you!
The main difference is that quale is an (interrogative) adjective, so it need a noun to which it refers (quale strada?, quali libri?...), while cosa – besides being a noun meaning “thing” – is a (interrogative) pronoun, so it stands in place of a noun, the object of the question: cosa vuoi? It's pretty much the same difference as between “which” (quale) and “what” (cosa).
You may also use che or che cosa, rather than cosa.
So, if you were to rephrase your example with cosa, you might want to say, for instance:
Osservate le due immagini e dite in (che) cosa differiscono
i.e., “...in what they differ”.
No, you can't. Quali is an adjective, referring to the noun differenze, and both together refer to the verb esistono.
If you substitute quali with cose you have two nouns, cose e differenze, without a connection between them and to the verb.
You could use instead say, dropping differenze:
[...] e dite cosa è diverso (not fully correct)
[...] e dite che cosa è diverso (better)
[...] e dite quali cose sono diverse (also good)