He served in the armies of France and of Spain while on the Continent after the fall of the monarchy and the execution of his father. He secretly married Anne Hyde, the daughter of Lord Clarendon in 1660, but continued his womanizing ways. Anne bore him two daughters, Mary and Anne. When Charles II returned to England and the throne, James became the Lord High Admiral and declared himself a Catholic in 1672.
Anne Hyde, the Duchess of York had also become a Catholic and died in 1671--James then married Mary Beatrice of Modena, a Catholic Italian princess.
His reputation for courage in battle suffered after the invasion of William of Orange. He panicked and fled for France. Before the Battle of the Boyne he suffered nose bleeds and did not execute a successful battle plan. The Encyclopedia Britannica in 1910 offered this harsh assessment:
"The political ineptitude of James is clear; he often showed firmness when conciliation was needful, and weakness when resolution alone could have saved the day. Moreover, though he mismanaged almost every political problem with which he personally dealt, he was singularly tactless and impatient of advice. But in general political morality he was not below his age, and in his advocacy of toleration decidedly above it. He was more honest and sincere than Charles II, more genuinely patriotic in his foreign policy, and more consistent in his religious attitude. That his brother retained the throne while James lost it is an ironical demonstration that a more pitiless fate awaits the ruler whose faults are of the intellect, than one whose faults are of the heart."
The line, "But in general political morality he was not below his age, and in his advocacy of toleration decidedly above it" does give James the credit he deserves although it does not go far enough. James did not just advocate toleration or tolerance; his Declaration of Indulgence addresses freedom of conscience for his subjects.
Also, as I have alluded to Edward Corps' The Court in Exile before, he seems to have repented both for the moral harm he did in being unfaithful to both his wives and for the political errors he made in ruling while he lived in France at St. Germain-en-Laye. James became prayerful and devout, and more sincerely lived up to his religious beliefs.