Information from “Beyond our Memory: a history of Fitzroy Township”:

Came to Canada at the age of 19 in 1836. After his landing in Quebec, he sailed by small boat up the Ottawa River to Farm Point, where the boat had to be pulled to shore with horses and ropes. From there, he walked to Fitzroy Harbour and on to the 3rd line of Fitzroy where he cleared land and took up farming on Lot 7, con. 2. As a livelihood, he processed the ashes of his trees into potash. By the side of the creek near his house, he placed ashes in hollowed-out logs to make the potash which he sold to a dealer in Pakenham.

James convinced most of the rest of his family (including his parents and 6 brothers and 2 sisters) to come across from Ireland in 1838.

1851 census shows his age as 35, so his birth year would be near 1816. 1881 census shows him as 64, born in 1817.

Household in 1881 also had:

William James Story - born 1860 in Ireland

Thomas Story - born 1862 in Ontario.

1891 census has him in North Lanark, along with his wife, and daughters Margaret and Emma Eliza, and sons William David and Thomas.

Margaret Story - born 1858 in Ontario.

Eliza Emaline Story - born 1876 in Ontario. I had earlier information that an Eliza was born to James and Annette, but the birth year was 1847, so this would be a grand-daughter?

Anne Sterling has him passing way in 1898.