You need to hash the annex to calculate the wtxid, which goes into the witness merkle root in the coinbase, which you check when validating a block though. You also hash it as part of taproot witness signature valuation, which is different from other witness data (eg inscriptions).