Are you implying that Prince Zhao is both dull and stupid?
No, he was aware that he had been framed, but with no evidence, and the fact that he had indeed had relations with her, he could tell there were traces of that. Moreover, the miscarriage of the child was a fact. He didn't believe that Imperial Censor Zuo would gamble with his own child, so he was currently unable to guess which link in the chain had caused the problem.
Such incidents were nothing new to him, but because it involved the Imperial Censor's wife, he'd been impeached and had suffered a severe loss of face before the Emperor. And with his own household causing an uproar, he was extremely agitated.
At this time, his subordinates asked him again about the Tasty and Pretty Fisherman restaurant—this was like a donkey kicking a steel board, resulting in a harsh scolding, naturally leaving him no mood to contend with Yan Qing.