Born in Georgia to a mixed-race slave mother and Irish immigrant father, he identified and was accepted as White Irish American, as he was half Irish and majority European ancestry. When he was ordained in 1854, knowledge of his mixed-race ancestry was largely restricted to his mentors in the Church. (Augustus Tolton, a former slave who was publicly known to be African-American when ordained in 1886, is for that reason sometimes credited as the first African-American Catholic priest rather than Healy.)
Healy was one of nine mixed-race siblings of the Catholic Healy family of Georgia who survived to adulthood and achieved many "firsts" in United States history; his brothers Patrick and Alexander also became Catholic priests. James is credited with greatly expanding the Catholic church in Maine at a time of increased Irish immigration, and he also served Abenaki people and many parishioners of French Canadian descent who were traditionally Catholic. He spoke both English and French.