You have stumbled upon a very subtle point of Italian grammar. In fact the usage of the si constructions is bound by complicated rules. So complicated that someone wrote a research monograph on them:
d'Alessandro, R. (2007). Impersonal" si" constructions: Agreement and Interpretation (Vol. 90). Walter de Gruyter.
Most of what I'm going to write here is based on the above book. Unfortunately I have neither the time nor the skill to do an in depth treatment. I hope what I write will be enough.
And thank you for making me learn this overlooked part of Italian grammar!
Simplifying a lot, sentences with the si passivante are pretty much always ok:
Qui si fabbricano le case ([Someone] builds the houses here)
On the other hand the si impersonale is perceived as correct mainly when expressing habits and general descriptions (precisely, verbs in the Vendler aspectual class of activities). In particular, the direct object in this sentence is almost never allowed to have a definite article.
Qui si fabbrica case (Houses are built here)
while the following sentence is incorrect:
*Qui si fabbrica le case
Moreover, the si impersonale is less used by native speakers in the past tense, and in this case it is sometimes perceived as archaic or incorrect
?Qui si è fabbricato case
It is also worth noting that the grammaticality of sentences with si passivante and si impersonale varies a lot with the idiolect of the speaker. It seems that Florentine speakers perceive pretty much any combination of si + verb + noun as grammatical, while most other Italian speakers have much more stringent criteria.
Let's finally take a look at your sentences.
Sì, si leggono [i libri]
Sì, li si legge
Both of these sentences are grammatical, but only the first one is an answer to the question:
A scuola si leggono i libri? (Do you read the books at school?)
The second one, being a si impersonale, is an answer to
A scuola si leggono libri? (Do you read books at school?)
A similar situation holds for the past tense.
In the case of the si passivante, the subject, as usual, can be omitted if clear from the context.