Resembling dusty volumes of spells, Derek Jarman’s private sketchbooks are magpie-like collections of paraphernalia assembled into large albums painted matt black and gold. Their highlights have been gathered together in a new publication that offers a window into the obsessive inner workings of the director’s mind over the two decades he spent in the trenches of independent filmmaking, before his death from an Aids-related illness in 1994. These talismanic books were the bedrock of his projects – the first creative skirmishes that would later become the films Caravaggio, The Last of England and The Garden, music videos for Pet Shop Boys and The Smiths and myriad other projects, some of which were glimpsed and abandoned on these pages.

“For as long as I knew him, there was always a sketchbook on the go,” writes his frequent collaborator Tilda Swinton in the introduction. “It was unthinkable that there would not be – like the kettle on the boil, the chain on the bicycle, the shilling in the meter.” A Slade-trained artist, Jarman peppered the books with colourful sketches, collages and poetry written in scratchy fountain pen. A flurry of odds and ends build a vivid picture of their creator: a newspaper clipping from the day he and friends covered Cliff Richard’s Rolls in “it’s cool to be queer” stickers; Polaroids of Swinton and others in his shifting circle of fancy-dressed partners-in-crime; pressed sprigs of mimosa (a favourite flower which would later adorn his coffin); a scarlet feather from an elaborate costume. A £10 note taped into one page represents his fee from the BBC for directing 1989’s War Requiem, while a briskly typed letter from Mary Whitehouse announces a “new low in television viewing” following the broadcast of his cult punk film Jubilee in 1978. (“Something to do with an erect cock, which she’d obviously never seen before,” writes James Mackey, the producer of some of Jarman’s most famous films, waspishly).

For as long as I knew him, there was always a sketchbook on the go

Derek Jarman’s Sketchbooks distils the many strands that made up Jarman’s work – humour, torrential creativity, romanticism and the palpable political anger that burned fiercely during the dog days of Thatcherism, most evident in his defiant stance as part of LGB-rights group OutRage! On the last and most beautiful pages are simple washes of Yves Klein blue inscribed with gold pen, the seed of the haunting film Blue, conjured as Jarman was losing his sight in the final months of his life. Viewing the books in today’s newly turbulent times, it’s hard not to feel how deeply his voice of creativity and dissent is missing from the fray.

Derek Jarman's Sketchbooks is published by Thames & Hudson on October 7.