I understand this is difficult for your mother. Since she is the main caregiver, she probably knows what she is talking about. First of all, you need to realize that your Dad may endanger others and himself, so with the babies at home, you may not be able to take him in either given his unstabe mood and being in this psychiatric place.
My father-in-law has late stage of Alzheimer's so I know what these elders go through given Alzheimer's. Everyone is different and your Dad seems to have a hard time to deal with it and he gets very anxious and psychotic. My father-in-law has now just moved to an assisted living home for mid-stage or late stage of Alzheimer's. This home is special in that they allow some sort of nusring care (with no feeding tubes but catheter is ok.) He could stay there until the end.
Before he moved, he had stayed home alone for 5 years with home care CNAs (certified nursing aids and caregivers.) It is expensive to pay for 24 hours home care, so this home is more economic. Also he forgot a lot (he forgot his sons in details but he still sort of knows my husband.) We had exhausted all the avenues at home and in the end he was almost dying at home. But after we moved him, he revived again and is eating well again in the new place.
An assisted living home for dementia is a controlled environment so he would not get lost in 24 hours and it is safe 24 hours. There are health care people in the home 24 hours.
I would suggest that you talk to your Mom. The assisted living home may accept him (not the regular assisted living home because he needs a controlled environment.) Note that some asssisted living may not accept him given his moods. Maybe that is why your Mom mentioned nursing home. Well, if no assisted living home accepts him, then the nursing home is an option although the skilled care home is more like an institution/hospital. The assisted living home is like a home with nursing care.
Before you consider moving him to your home, please consider the potential problem. My FIL is a gentleman and never had psychiatric episodes (never went to any psychiatric unit at all.) But it sounds like your Dad has lots of problems emotionally. It is a danger for your family. Unless you can make sure your house can separate your babies from him who might be in a bad mood, and you hire some part-time help (CNAs or caregivers), you would have a hard time to deal with him. Note my FIL looks normal outside so it is tricky that no one realizes he is in late stage. Same thing for your Dad, on the outside, it is tricky and you may think he is OK. But when he gets psychotic in your home, you will have a hard time to deal with it and then you may still need to send him to a nursing home.
Only you and your Mom know your Dad best, so please consider all options before you make a move. Frankly one always wishes to keep the elders at home as much as possible. But you also need to figure out if the private home is really good for him. At times, a nursing home or assisted living home can provide some respite and structure for the sick elders. Of course, there is also a potential that he may blow up in the nursing home and end up dying fast. It all depends on how he copes with it.
Please keep posted and let us know how we can help.