Rimmer's career began with 1957's The Dangerous Age, a romantic drama, before appearing in Dr. Strangelove (with James Earl Jones), The Bedford Incident (with George Roubicek and Phil Brown) and The Dirty Dozen (with John Hollis and Roubicek). In 1965 Rimmer began his role as Scott Tracy, one that began in 1965 and with which Rimmer was involved until his death.
Rimmer also appeared in several Bond films, most notably an extensive part in 1977's The Spy Who Loved Me, released the same year as A New Hope and also featuring Anthony Forrest and Roubicek. One of his roles, in the earlier Live and Let Die, was a voiceover part. Other film appearances include brief roles in 1978's Superman (with Terence Stamp) and its sequels (the former with fellow frequent Bond collaborator Clifton James), Hanover Street (with Harrison Ford and John Ratzenberger), Gandhi (with Geraldine James), Morons from Outer Space (with Mark Lewis Jones) and Anastasia: The Mystery of Anna (with Julian Glover).
Later in his career, Rimmer appeared briefly in Batman Begins (with Liam Neeson) and Dark Shadows (with Christopher Lee). He also appeared in the Star Wars fan project The War of the Starfighters, which also featured Caroline Blakiston, Garrick Hagon and Michael Carter. His final credited roles, after over 160 appearances on television and in films, were in the animated show The Amazing World of Gumball and an indie film titled DARKWAVE: Edge of the Storm. Rimmer also published two books, an autobiography in 2010 and a novel in 2014, and appeared on-stage with the Royal National Theatre, which has hosted numerous other Star Wars alumni in various roles.
Rimmer passed away on March 29, 2019 at the age of 89. He was survived by his wife and sons.