Enslaver, Unionist, and legislator in the American Civil War
In 1858, Robertson returned to Richmond. He had owned 11 enslaved persons in the 1840 U.S. Federal Census, and at least 28 enslaved persons in the 1850 U.S. Federal Census. His father-in-law Capt. Francis Smith had enslaved 107 people in the 1840 census, of whom 20 were engaged in farming. In the 1860 census, Robertson owned seven enslaved persons in Richmond and at least 75 enslaved persons in Washington County, Virginia.
In 1859, Richmond voters again elected Robertson to the Virginia House of Delegates as one of their three delegates in the 1859-1861 session based on the representational changes after adopting the Virginia Constitution of 1850 and population shifts documented in the census. As Virginia struggled with the idea of seceding from the United States, Robertson was a staunch Unionist and tried to prevent its secession. He later characterized himself as a "friend to peace and the Union" and noted that he had actively opposed South Carolina's call for a Southern Convention in 1859. At the Henry Clay banquet in April 1860, at which former President John Tyler was present, Robertson gave "The Union" toast, followed by a short speech:
The Constitutional Union of the States" - The Union of the States is the harmony of the spheres. While obedient to the laws of their creation, they sing ever as they go 'glad tidings of great joy' to all the world. Rebelling against them, light and joy are swallowed up in darkness, and order falls back into primordial chaos.
After South Carolina and several other states started seceding in the winter of 1860–61, Robertson still advocated against Virginia following suit. On January 7, 1861, he presented the "Anti-Coercion Resolution", which rejected secession, but stated that if the Federal government used coercion toward the seceded states, Virginia would fight, which was duly adopted. When President Abraham Lincoln made his call for troops on April 15, 1861, Robertson believed the Anti-Coercion scenario met and was "from that time forth zealously active in all measures for the defence of his State." Virginia seceded after voters approved a referendum the following month.
Robertson continued as the leading vote-getter within Richmond's delegation to the House of Delegates for two additional terms; his legislative service ended as the Confederacy fell in 1865. In 1863, Robertson opposed and helped to defeat a bill to fix food prices, which he believed "fraught with the direst mischief", despite food riots in Richmond. When Richmond citizens presented a resolution asking their representatives to support a similar bill or resign, Robertson refused. When he found that his colleagues had already acquiesced, he resigned so as not to misrepresent his constituents. The House, however, requested that his resignation be withdrawn until the wishes of his constituents could be determined. A formal poll was held, and it was determined that a majority did not support the bill, so Robertson retained his seat.
