I have all the understanding of your personal circumstances, and it must be hard getting through this.
But, here I am just telling you the legal position. Basically refusal is correct, as your partner has not exercised treaty rights for full 5 years. Whatever the reason behind not working is immaterial.
No, they do not expect him to work, but also they have no obligation to you if you expect PR. Once he stopped working, and you started working, you should have immediately took legal advice. If you took out a comprehensive health insurance, once he stopped working and you started working, then you would have qualified for PR. But you did not take any legal advice, so now you having to face this problem.
For any chance to get PR in your circumstances, you will have to spend an absolute fortune on solicitors, and wait years for a legal battle. Still no guarantee of success. Actually most likely it will result in a failure.
Get legal advice for the possibility of making another residence card application after getting fully comprehensive health insurance. Once you clock 5 years again, you will then qualify for PR.
My humble advice to everyone is, as soon as your circumstances change, research and find out, or take legal advice to ascertain if change of circumstances will have any effect of the immigration status. Do not just sit and wait until you get a refusal.
regards