Camelot crumbles
Unfortunately, Sir Henry's fortunes could not sustain the magic that was Casa Loma. To finance expansion, Pellatt and Pellatt went further and further into debt. The one sure source of income from the monopoly of electrical power vanished when political decisions allowed for public ownership of electricity. In a futile attempt to restore his wealth, Sir Henry turned to land speculation. He was convinced that well-to-do Torontonians would rush to build homes around his castle.

However, in this case his entrepreneurial sense did not take into account the effects of World War I; during the war Canadians put their money into war bonds, not homes. After the war the economy slumped, tilting Pellatt and Pellatt into bankruptcy. The company owed the Home Bank of Canada $1.7 million - or in today's terms $20 million. With his stock worthless and his business debts out of control, Sir Henry was faced with a heartbreaking decision - a decision which he would always claim was made for him by the City's immovable tax assessors. Faced with an extraordinary tax bill, Sir Henry had no choice but to auction off his prized possessions for a fraction of their worth and to abandon his dream of a noble castle.

The Pellatts moved to their farm in King township in 1924. Lady Pellatt, weakened by the strain of the move and the financial difficulties, passed away later that year at the age of sixty-seven.

Though he lost a great fortune, Sir Henry never lost his spirit of philanthropy, a character trait for which he was honoured late in life. His service of fifty years with the Queen's Own Rifles was celebrated on June 27, 1926 with a march past of 500 men complete with the circling overhead of three military planes. When Henry died on March 8, 1939, thousands lined Toronto streets to witness his funeral procession. He was buried with full military honours befitting an old soldier who had given so much to his country.