Hello friends,

This week Hannah and Charles got to hang out in person at the AI for the rest of us Summer Social in Central London. It was absolutely buzzing! Thank you to everyone who joined us for the celebration and to Rezonant for sponsoring the event!

Despite the noise we managed to capture the energy of the community on our live stream. Hannah lost her voice from shouting and then attempted Bonnie Tyler’s “Total Eclipse of the Heart” with an authentic gravelly tone at karaoke before finally calling it a night at 3am.

Don’t worry, our meetups are usually far more educational and less rowdy. We’ll be back in September, October and November for the next wave of events. Let us know what you’d like to learn about and we’ll start lining up speakers.

Have a wonderful week,

Hannah & Charles

Software Factories At Enterprise Scale ​ Software Factories at Enterprise Scale: A Federated Platform for Agentic Development — a free whitepaper by re:cinq. ​ Get the free whitepaper

What’s Hannah reading this week?

Two things I didn’t expect to be googling this week. “AI urinals” and “10 Downing Street hot podium guy”.

I’m not sure whether this is just a funny meme or a real incident but I had to giggle when I read that someone had showered themselves in pee when they mistook a hand dryer for an AI urinal they had read about. I can confirm that AI urinals are in fact real and they scan your wee for hydration stats and other health markers. What a world we live in!

Apparently one of the technicians at 10 Downing Street has become the unexpected highlight of our political press conferences. Known as “hot podium guy” he has outlasted 5 Prime Ministers. I am only just learning about this internet legend due to the unusual decision of our latest prime minister to break with tradition and not use a podium to make his debut speech. If there’s no podium then where is the “hot podium guy”?... Bitter disappointment swept the internet. Probably.

Yes, here in the UK we have a new Prime Minister, Andy Burnham, and with it some interesting changes in the heart of government. Most notable for this audience is the elevation of Kanishka Narayan, Minister for AI, to the cabinet. This is good news, we need leaders who are fluent in technology and can shape ambitious, realistic and practical policies.

Less good news however was the annihilation of DSIT, the Department of Science, Innovation and Technology. At moments like this I look to people I respect at the intersection of technology and government to try and understand the impact. Dr Laura Gilbert who founded the Incubator for Artificial Intelligence at 10 Downing Street shared her thoughts on LinkedIn this week, commenting on how disruptive this thrashing is for the teams impacted, who were just beginning to settle down after one sweeping change when another hit them.

If you want to build a high performing team you need to set them up for success. They need to be able to focus on the problem you’ve given them to solve. I feel for the teams who are expected to “keep calm and carry on” yet again while they are caught up in relentless reorganisations and thrashing priorities.

This week the biggest news story in my technology bubble was the “rogue” OpenAI agent that hacked Hugging Face. I have been on a full rollercoaster of emotions around this story from full apocalyptic catastrophisation, to awe and excitement, to “maybe it was all just a marketing stunt” cynicism.

In one of the security communities I am a part of, the chatter was equal part humour, fear and cynicism too.

If you haven’t been following the story it’s a mind-blowing example of what frontier large language models can do today. OpenAI were testing a new model and, in pursuit of its goal to pass a benchmark test, the agent found a way to bypass OpenAI’s sandbox to access the internet. It then attempted to obtain test solutions by hacking into Hugging Face’s database.

I suppose stealing the answers is one way to pass a test. It’s not the ethical way to pass a test but we’re talking about computer programs here, they do not have a moral compass. “Pass the test” is the goal. “Steal the answers to the test” is an efficient strategy.

This is terrifying. This was an AI agent with a seemingly innocent goal. What if the goal the agent had been set was to inflict damage? What if the target was less technically-accomplished than Hugging Face?

What if there were open models freely available to anyone in the world with ever-increasing cyber capabilities without the strict guardrails that you experience on OpenAI, Anthropic or Google’s platforms? Oh wait… that’s not theoretical, that’s actually happening.

Over my coffee this weekend I started to think about cyber-criminals, because I know people who have been hacked and I know that the hackers have absolutely got away with it. Sometimes with millions. Cyber-crime is easy now, you don’t need to be an expert, all you need is access to a powerful large language model. I worry that these baddies also now have the ultimate defense if they do get caught - “My agent went rogue. It wasn’t me, it was my agent.”

What does that look like now we have a high-profile example of an agent that stole information without ever being prompted to. Should the human supervisor of the agent still be held accountable in court if their intent was not criminal but the agent's actions were? How do we know whether the human supervisor had malicious intent if the prompt looked innocent on the surface?

We need answers to these questions now. This is a problem today.

There are also two sides to this story. The agent that attacked Hugging Face and the agents that detected, defended and forensically analysed the attack after the event. As we’ve previously shared in this newsletter, you can’t use Anthropic’s Fable or OpenAI’s Sol for sophisticated cyber security defense work, their guardrails prevent it. Hugging Face and every other security team in the world are locked out of using frontier labs and have to use open models. These are mostly Chinese models of course.

The White House has been making noises about banning Chinese models but they obviously can’t. Your attackers have access to this technology, you have to have ways to defend against it. You cannot deny your cyber defense teams access to this tech or you’re at a significant disadvantage. Even Jensen Hueng has waded in to make sure the US doesn’t make a mistake here, issuing a statement about the importance of open models.

This week OpenUK launched the third edition of their AI Openness Report which provides commentary on the role of Chinese open models as well as the political and commercial dynamics that underpin open technology. You can read my section on page 36 where I focus on the intersection of AI, Developer Tools and Open Source.

What’s Charles reading this week?

In the US, a federal judge approved Anthropic's $1.5 billion settlement with authors. This is the largest copyright class-action settlement ever, and follows an earlier ruling that training on books was fair use, but that Anthropic's piracy of those books (rather than buying them) likely wasn't. Authors will get roughly $3,000 per work (four times the statutory minimum), and the numbers suggest broad buy-in: 91% of eligible authors and publishers filed claims, and only 350 opted out.

The judge did trim the lawyers’ cut, though. Their fees went from a requested $300M (20%) down to about $101M (under 7%), and the three lead plaintiffs' service awards were cut from $50,000 to $15,000 as they were deemed "unreasonable". Most late opt-out attempts were rejected, including one from entertainment lawyer Donald Passman, though two authors were granted late opt-outs. Anthropic, for its part, says it's glad to have the fair-use precedent settled and to be closing the matter.

Underneath all the numbers the question is, if an AI company takes someone else's creative or intellectual output and trains on it without asking, who exactly has been wronged, and what do they get for it? Interestingly that same question, i.e. who's being wronged, and by what, is just as unresolved a few layers up the stack, where it's not authors and publishers doing the complaining, but Anthropic itself.

The US frontier AI model companies are essentially in an arms race against the Chinese Open-Weight models. There are model companies from other parts of the world, of course, but they aren’t generally as competitive. The US companies are in front, but not it has to be said by all that much. It was perhaps remiss of us, but we didn’t cover Chinese start-up Moonshot AI's latest large language model K3 when it was released a couple of weeks ago. It is the largest open source model on the market, with 2.8 trillion parameters. It now ranks No. 1 on the Frontend Code Arena coding leaderboard, above Anthropic’s most powerful model, Claude Fable 5, though other benchmarks show that whilst K3 approaches the most advanced US models it doesn't quite match them.

Earlier this year, Anthropic accused Moonshot and other Chinese AI labs of illicitly distilling Claude models to train and improve their own. Distillation in AI is a method where a large, complex AI model trains a smaller, faster model to mimic its behaviour. It transfers core reasoning skills and output probabilities from a "teacher" model to a lightweight "student" model. The technique allows you to produce a model whose smaller size makes it more efficient; better from a carbon/energy point of view and also more suited to running on smaller hardware like a mobile phone.

This week The Register’s Simon Sharwood reported that Donald Trump’s Assistant for Science and Technology, Michael Kratsios, has accused China’s Moonshot AI of creating K3 by distilling Anthropic’s Fable. But whilst distillation is likely part of the process, if K3's performance actually rivals top models as some benchmarks suggest, then distillation alone does not explain the rapid advancements.

Given this it is perhaps unsurprising to see the US Bureau of Industry and Security, an agency within the US Department of Commerce, investigating whether Chinese firms like Moonshot are accessing advanced US AI chips. If the investigation formally concludes that Moonshot improperly trained its AI models on those from a US AI lab like Anthropic using US chips, the Commerce Department could add the Chinese lab to its entity list — a BIS-enforced designation that restricts how foreign companies can access US technology like advanced chips. However, as Hannah points out in her section, this is fraught and very unlikely to work.

There’s another interesting question here though, which Ben Thompson writes about on Stratechery. “Why exactly is it [distillation] bad? After all, what are large language models but the distillation of all of the knowledge on the open Internet, scraped by the frontier labs and distilled into the models that are themselves being distilled? Who is exactly being wronged here?”

The big US model providers certainly want to try to hobble the Chinese labs since, if they are ever to turn a profit, they need to be the only viable option. Somewhat ironically, if the big US firms do end up properly compensating authors, artists, composers and others for the work they pirated, it could yet end up being in the interests of the creative industries to have the US firms succeed. $3,000 isn’t much for full rights in perpetuity for a novel, but it is more than nothing.

Updates

Summer Hols We'll be back in September for our next in-person meetup Join the community

Follow us on LinkedIn Bite sized nuggets of AI learning! Follow

Follow us on BlueSky Bite sized nuggets of AI learning! Follow

Catch Up On The Conference Subscribe now and don't miss all the latest recordings! YouTube Channel