Later career
Rafferty returned to being an actor only. He had a small role in The Sundowners (1960), with Robert Mitchum and Deborah Kerr and played a coastwatcher in The Wackiest Ship in the Army (1960) with Jack Lemmon and Ricky Nelson. He guest starred in several episodes of the Australian-shot TV series Whiplash (1961).
Rafferty was cast as one of the mutineers in the 1962 remake of Metro-Goldwyn-Mayer's Mutiny on the Bounty, starring Marlon Brando. The filming of Bounty in Tahiti dragged longer than six months but it restored him to financial health after the failure of his production company; it enabled him to buy a block of flats which supported him for the rest of his life. Rafferty dubbed the film The Bounteous Mutiny.
In 1962, the 6 foot 5 inch actor was socialising with fellow expatriates in a London club when they were joined by an Australian who acted as doorman, and unbeknownst to Rafferty, was a professional wrestler. Claiming he was being ignored after helping them get in the doorman was so argumentative that Rafferty was provoked into accepting a challenge to 'step outside'. In the severe beating that followed he sustained deep grazing across his face and suffered a myocardial infarction (he had not been aware of having a heart condition until the incident) costing him the chance at roles in two major film productions.
In 1963 he recorded a long play record with Festival Records (FL-31015) titled A Man and His Horse, narrating a selection of works from Australian verse composers including Banjo Paterson (1864–1941), Adam Lindsay Gordon (1833–1870) and Will H. Ogilvie (1869–1963).
Rafferty appeared in some episodes of the series Adventure Unlimited shot in 1963. He played the Australian Prime Minister in the Australian sci-fi TV series The Stranger (1964) then travelled to England and appeared in eight episodes of Emergency-Ward 10 (1964). While in England he was in The Winds of Green Monday (1965) on British TV.
He travelled to the US and guest starred in episodes of The Wackiest Ship in the Army (1965) (as a different character to the role that he played in the movie version). This led to further offers to work in Hollywood on television shows; he played a Union soldier in The Big Valley (1966) with a noticeably Australian accent. He was also in episodes of Gunsmoke (1966) and Daktari (1966). "What else can I do but look to America for my future when there is still no assistance or help from the government," said in April 1966.
Back in Australia Rafferty had a good part in the Australian-shot comedy They're a Weird Mob (1966) a big local success. He returned to Hollywood to appear in episodes of The Girl from UNCLE (1967), Tarzan (1967) and The Monkees, as well as the Elvis Presley movie Double Trouble (1967) and the adventure tale Kona Coast (1968)
Returning to Australia he guest-starred in Skippy the Bush Kangaroo, Adventures of the Seaspray (1967), Rita and Wally (1968), Woobinda, Animal Doctor (1970) and Dead Men Running (1971). He continued to make films such as Skullduggery (1970).
Rafferty's final film role was in 1971's Wake in Fright, where he played an outback policeman. (The movie was filmed mainly in and around Rafferty's home town of Broken Hill.) In a review of the film, a critic praised Rafferty's performance, writing that he "exudes an unnerving intensity with a deceptively menacing and disturbing performance that ranks among the best of his career".
His final performance was in an episode of the Australian war series Spyforce (1971).
Hours before he died, Rafferty was offered a prominent role in a film The Day the Clown Cried by Jerry Lewis which was never completed or released.
