He was the only son of James Graham, 3rd Marquess of Montrose and Lady Christian Leslie. On 31 March 1702 he married Christian Carnegie, daughter of David Carnegie, 3rd Earl of Northesk. Together they had several sons, including William Graham and Lord George Graham.
Originally the fourth Marquess of Montrose, James was elevated to a dukedom in 1707, as a reward for his important support of the Act of Union, whilst being Lord President of the Scottish Privy Council. He was Lord High Admiral of Scotland from 1705 to 1706. He was Keeper of the Privy Seal of Scotland from 1709 to 1713 and served as Keeper of the Great Seal of Scotland from 1716 to 1733. He was also a Lord of the Regency for Great Britain in 1714, upon the death of Queen Anne. Furthermore, he served rather shortly as Secretary of State for Scotland at the time of the Georgian ministry of Lord Townshend. In 1719 he was one of the main subscribers to the Royal Academy of Music (1719), a corporation that produced baroque opera on the stage. He served as a Governor of London's Foundling Hospital at the time of its foundation in 1739. For much of his adult life he was Chancellor of the University of Glasgow.
Apart from his political career, the Duke is frequently associated with Robert MacGregor, who is popularly remembered by the name of Rob Roy. Graham was a primary creditor of MacGregor whom the latter blamed for his financial ruin; MacGregor then carried out a feud with Graham for some years.