When I interviewed Daniel Craig for his first outing as James Bond, his pecs were ripped, his clothes elegant, his manner intolerably genial. The whole package was that of an alpha male at ease under interrogation by a babbling slob with no muscle tone. God, I hated him. And then he did something that cheered me up. He bent down to sip a banana milkshake and almost skewered himself with the straw. “Oooh, I nearly put my eye out,” he said in a satisfyingly effete voice.

Evolutionary biologists suggest that the emotion of schadenfreude I felt then, that pleasure in witnessing a tall poppy cut itself down to size, stems from early societies that were dependent on cooperation and so needed to be egalitarian. That said, if Bond had actually put out his eye with the straw, I wouldn’t have delighted in his misfortune. Probably.

But there’s more to schadenfreude than that. Tiffany Watt Smith’s favourite YouTube video features a cute toddler reaching out to a fluffy kitten, ostensibly to stroke it. Instead, the toddler punches it on the nose. The cat then rears up and swipes the child with its paw. Why is Smith’s resultant giggling socially defensible? “If you found this funny,” she writes, “you may be, in fact, celebrating the toddler learning an Important Life Lesson.” Whatever you need to tell yourself.

Watt Smith argues with Freud, who suggested that when babies laugh at adult incompetence, it’s not because they have a sense of humour, but because they have a taste for gloating and triumph – or schadenfreude. After visiting psychologists at the University of London’s beguiling-sounding Baby Laughter Project, she comes to another conclusion: when babies laugh they are taking pleasure in surprise at the unexpected happening. That’s why they giggle at peekaboo and delight in adult fails. Laughter has a cognitive element: it helps us to see the world afresh and to learn. Such is what she calls good schadenfreude.

I much prefer bad schadenfreude. The boss calling himself “Head of Pubic Services” on an important letter. The joy remainers took in the apostrophe in the slogan “Brexit Mean’s Brexit”. Giggling over the MP who called Jane Austen “our greatest living author”. I enjoyed it when Watt Smith, a university lecturer, happily tells us there’s an article suggesting students who complain about their lecturers are bad in bed. Even though that reads like wish fulfilment.

I especially loved the story about Dutch schadenfreude at the nation that gave us the word. Towards the end of coverage of the 2010 football World Cup semi-final, ratings peaked on the German TV channel ADR. Why? Dutch viewers, realising Germany was poised to lose against Spain, were tuning in in their hundreds of thousands to hear German commentators describe their nation’s defeat.

Supporters celebrate at the end of the World Cup semifinal match between Germany and Spain in 2010.

Cause for Dutch schadenfreude … celebrations at the end of the World Cup semifinal match between Germany and Spain in 2010. Photograph: Christophe Simon/AFP/Getty Images

Watt Smith is at her best when she relates cases that flirt with bad taste. Racing driver Stirling Moss once told an interviewer why he thought so few women competed in Formula One:“I think they have the strength, but I don’t know if they have the mental aptitude to race.” He then walked off into an empty lift shaft, fell and broke his ankles. “Did you smile?” Smith asks her readers. Too right I did, though a split second later, I felt a twinge of self-disgust. If he’d just sprained his ankles, maybe I’d have had less compunction. Appealingly, Watt Smith isn’t above this grubby fray. She recalls feeling jealous of her husband. He was writing a book in another room as she struggled to complete this one. Then he got an email from an editor hailing the superb manuscript he’d submitted, while she got one from her publisher asking where the hell hers was. She yearned to take joy in his misfortune, but even after scrutinising his email she couldn’t find any cracks in the praise. After she confessed these feelings he, “being a very kind person”, laughed and then the two bonded by giggling over a successful writer getting their book trashed in a newspaper review. Schadenfreude – keeping couples together.

Which brings me to my schadenfreude shame. Eight years after I gave Joanne Harris’s novel Coastliners a stinking review in these pages, she took her revenge by creating a sleazy Guardian journalist called Jeffrey Stuart. But here’s the thing. Who felt more schadenfreude? Harris by trashing me in her fiction, or me for realising that my review must have really hurt if she had to take such an elaborate retaliation? Either way, the other week I found three copies of her book in a secondhand shop. I examined them and found all had their spines broken a third of the way through, as if each reader had given up 70 pages in.

But I now realise my schadenfreude in finding these discarded books was just the joy of a petty man who can’t write a novel, not even one as terrible as Coastliners. Which is why I’d like to apologise to Harris for starting this mutually demeaning schadenfreude spiral 16 years ago.

Nietzsche called it the joy of the impotent and insisted we should avoid it if we seek, as we ought, to overcome our shortcomings. Watt Smith, by contrast, concludes that in itself schadenfreude is “ethically ambiguous”. Yes, the 256 million of us who’ve watched the video of a dad getting kicked in the nuts by a toddler may not be better people as a result, but experiencing this emotion has, she suggests, its uses. It can make you feel good if you’re feeling inferior, highlight life’s absurdity, create a soothing solidarity among fellow failures, help take down a smug arsehole.

On that last point, consider the suffragette Annie Kenney. “If you were my wife I’d give you poison,” yelled a man during a suffragette rally. “Yes, and if I were your wife I’d take it,” Kenney snapped back. Who could not enjoy the heckler’s misfortune?

But are we living, Smith worries, in a Spitegeist where joy in others’ misfortunes becomes toxic and risks obliterating serious political debate? “If there was ever an environment that would leave us jonesing for a new hit of justice, it is our digital age.” For “justice” in the previous sentence, read schadenfreude, and think of TV host James Corden playing on it when he tweeted that the theft of Kim Kardashian’s jewels from a Paris hotel was not a fit subject for others’ joyful tweets at her misfortune. He had a point, but I can’t help feeling conflicted: anything that gets Corden moralising can’t be all bad.

Watt Smith wrote her book as part of a research project for the University of London’s Centre for the History of the Emotions. It is filled not just with gags and shaming confessions, but chastening thoughts – nowhere more so than when she challenges the orthodoxy that schadenfreude has its limits. “Flirting with the morally questionable, testing the limits of where we ‘ought’ to stop, is an exciting game,” she argues. And not just a game but the basis of, say, Frankie Boyle’s career. But my pleasure curdled when I saw the author photo and blurb. Look at her with her smug scarf, clever hair and face, not to mention a Twitter handle referencing her doctorate. How I hope I’ll be stumbling into discounted piles of her book come January. But it is so delightful that probably won’t happen.

Good grief, that last sentence was hard to write. Taking joy in others’ misfortune is easy; applauding their achievement without rancour much more difficult.

Schadenfreude: The Joy of Another’s Misfortune is published by Wellcome Collection. To order a copy for £8.59 (RRP £9.99) go to guardianbookshop.com or call 0330 333 6846. Free UK p&p over £10, online orders only. Phone orders min p&p of £1.99.