AI chatbots could help mental health but also reinforce harmful thinking

By Laura Thomas, Curtin University | Reviewed by Irfan Ahmad

Image: Aerps.com - Unsplash

Artificial intelligence chatbots could help people better understand and manage their mental health, but they could also reinforce harmful thoughts and increase emotional dependence, a Curtin University researcher warns.

Published in Digital Health, the paper compares AI conversations that follow two possible paths: one that helps people explore their problems and consider new perspectives, and another that traps them in a cycle of repeated thoughts and reassurance.

Author and Professor of Mental Health Warren Mansell, from the Curtin School of Population Health, has proposed a new framework to explain why AI conversations can have very different effects on people depending on their emotional state, what they hope to gain from the interaction and how the chatbot responds. Professor Mansell was the first to evaluate a therapy chatbot, back in 2013, and his research has focused on trying to measure and understand how a person’s attention either narrows or broadens during a conversation.

Professor Mansell said chatbots could be helpful when they helped people to explore their feelings and question their assumptions, rather than simply agreeing with everything they said.

“AI conversations can move in very different directions depending on what a person brings to the interaction and how the chatbot responds,” Professor Mansell said.

“For some people, a chatbot may provide a space to express their feelings and explore problems without fear of judgement. For others, repeatedly receiving reassurance or having their existing beliefs confirmed could make it harder to see things differently.”

Research into AI and mental health has produced contrasting results. Some studies suggest chatbots can reduce symptoms of depression and anxiety, while others have raised concerns about harmful responses, emotional dependence and AI reinforcing users’ existing beliefs, even when those beliefs could be damaging.

Professor Mansell said the same chatbot could potentially help one person while having a negative effect on another or even affect the same person differently depending on how they use it over time.

For example, someone experiencing anxiety might use a chatbot to explore their worries and consider alternative ways of looking at a situation. However, if the conversation repeatedly focuses on seeking reassurance or confirming their fears, it could narrow their thinking rather than help them work through the problem.

The paper suggests the risks could be particularly concerning for people who are already vulnerable and most in need of psychological support.

While AI chatbots are often accessible, inexpensive and available at any time, Professor Mansell argues their benefits may depend on whether they encourage meaningful reflection or simply provide immediate reassurance to keep users engaged.

He said developers should design chatbots that encourage people to think through their problems, ask thoughtful questions and consider different perspectives, rather than simply telling users what they want to hear.

The proposed framework also highlights the importance of understanding how conversations develop over time, rather than judging a chatbot’s effectiveness solely on whether users feel better immediately after an interaction.

Professor Mansell recommends people approach AI conversations with a clear purpose and consider how their use of chatbots fits alongside other priorities and relationships in their lives.

The proposed framework is yet to be tested, and further research is needed to establish why some AI conversations may support mental health while others could increase psychological distress.

As AI becomes an increasingly accessible source of mental health support, Professor Mansell said understanding these differences would be important for developing safer and more effective technology.

The paper ‘Up the mountain or down the rabbit hole? A simple rubric to conceptualise the benefits and risks of AI conversations as dynamic changes in awareness’ was published in Digital Health.

Read next: EFF Calls for Digital Platforms to Balance Interoperability and Security
by External Contributor via Digital Information World

EFF Calls for Digital Platforms to Balance Interoperability and Security

By Bill Budington and Rory Mir, Electronic Frontier Foundation (EFF) | Fact-Checked by Irfan Ahmad.

Image: FlyD - Unsplash

An open internet where people can freely build, share, and remix each other's work is a powerful engine of grassroots innovation we’ve all benefited from. In recent decades, however, corporations have taken control of an increasing share of this work, giving them a greater ability to stifle challengers and limit our choices. When regulators address this by mandating that tech giants play fair by requiring interoperability, the gatekeepers respond with a tired refrain: Playing fair puts users’ security at risk.

Unfortunately for these trillion-dollar companies, this gotcha isn’t the end of the conversation. Regulators can ensure that meaningful choice and security thrive.

Lawmakers have been catching up, recognizing that the best way to rein in big tech is to require these oligopolies to open up their platforms to independent developers. As EFF explained six years ago, mandating interoperability helps prevent the misuse of people’s data and enables more privacy-respecting alternatives. Tech giants, in contrast, have been arguing that our data is safer when they have exclusive control.

Tech Giants Claim Interoperability Means Insecurity

The European Union’s Digital Markets Act (DMA) put interoperability into law under Article 6(7): “gatekeeper shall allow providers of services and providers of hardware, free of charge, effective interoperability with” a designated Core Platform Service they operate. The language describes an unambiguous requirement to remediate vendor lockouts and anti-competitive behavior of the major service operators—it is not optional.

Apple claimed that enforcing this obligation in mobile app stores could put customers at risk and undermine trust in its security and privacy protections. It’s an argument reminiscent of Apple’s fight against the right to repair, and in both instances the company is fighting to maintain its restrictive ecosystem to stop people from finding better deals elsewhere. Apple has long cultivated a more corporate-controlled experience with the iPhone, but this argument is now also being used for the relatively-more-open Android devices from Google.

In the same year Apple argued against the DMA, Epic Games sued Google for its restrictive terms of service and extractive policies for revenue generated in apps. Google predictably alleged that opening up its platform to other app stores it couldn’t vet would allow harmful apps to be installed. The case was ultimately decided this year in the U.S. federal appeals court for the Ninth Circuit, with the court rejecting this argument, dealing a blow to Google’s feudal security arrangement, and paving the way for third-party app stores to be delivered as apps via the Google Play Store.

Interoperating Secure Systems Can (and Should) Exist

Policymakers cannot take tech giants’ claims at face value. Companies have every incentive to resist interoperability requirements, because interoperability lowers practical barriers and reduces switching costs for consumers adopting competing services.

Even where companies offer credible claims from internal technical and security experts, independent experts rarely have the access and information they need to test those claims. Companies have selectively made arguments that security is harmed by interoperability, often in contexts which are to their advantage. These companies are happy to let a general misconception flourish which frames security as inherently opposed to interoperability.

This narrative is wrong. We’ve seen that when interoperability and information security are both set as essential and non-negotiable criteria, secure interoperable systems result—systems that don’t sacrifice the rich features we’ve come to expect of the services we rely on.

The cross-platform rollout of Rich Communication Services (RCS), and now Message Layer Security (MLS) on top of it is a perfect example of this, creating end-to-end secure messaging by default between Android and iOS. For RCS and MLS, Google was conveniently on the other side of the interoperability argument, campaigning Apple to adopt the standard to overcome the diminished “green bubble” experience iPhone users had when texting non-iPhone users. Once Apple finally agreed to interoperate by implementing common standards, all users benefited from improved messaging security. In cases where legitimate security concerns are raised, these can be assessed and remedied on a case-by-case basis, without sacrificing user agency.

Striking the Right Balance

Retrofitting interoperability into already secure closed systems shouldn’t be done carelessly: it requires regulators strike the right balance of pressure and leniency to allow the time and precision necessary to make these renovations. Any components developed with the assumption of privileged access by trusted parties (e.g. your browser engine) are particularly vulnerable to serious security risks when opened up to unaccountable third parties.

By adapting interoperability and information security as essential design principles, we can expect more safety, but also more freedom.

With regard to the DMA, when security concerns are legitimate, EFF urges the European Commission to work with companies and outside experts to offer guidance in technical implementation and provide clarity on regulatory compliance. This approach paves a path forward that balances the interest of legal oversight and the needs of the companies alike.

Policymakers wishing to uproot these unfair platforms of control must do so with care and limit the impact this disruption can have. But security concerns must not act as a lever to indefinitely delay compliance. Closed systems should be impartially evaluated, and if concerns are valid, regulators can allow more time and guidance to be given to companies implementing these changes. One possibility is a tiered approach: offering leniency during an initial period where systems need to be retrofitted to abide by new requirements, followed by a subsequent and ongoing stricter observance of fair play.

Once interoperability and information security are adopted as essential design principles, further development with these rights-enhancing principles in mind will more easily follow. The result will be an internet the world deserves—where we can expect more safety, but also more freedom.

Read next:

• You Can’t Solve AI Risk Without Solving the AI Evidence Problem

• Why calling for a ‘kill switch’ is the wrong way to think about reining in AI

by External Contributor via Digital Information World

You Can’t Solve AI Risk Without Solving the AI Evidence Problem

By Jen Rosiere Reynolds, Tech Policy Press | Reviewed by Irfan Ahmad.

Image: Sasun Bughdaryan - Unsplash

In September, AI executives voiced fears of human extinction. President Trump moved quickly to sign a “morally binding” framework for AI companies to self-police.

The gap in urgency between these two events, just days apart, leaves the public largely in the dark about AI risks. Researchers, for their part, are limited in their work by industry opacity. The situation is reminiscent of research on social media, which has long struggled with access to platform data. Today, we are at risk of forgetting lessons from that work in the face of a technology that is much more powerful, even putting aside speculation of so-called superintelligence.

AI companies’ restrictions on data and models give them significant power while limiting the ability of independent researchers or the public to assess the technology’s impact. Self-policing can work if corporate and public incentives are aligned or there is external pressure shaping behavior. External pressure can only happen if outside researchers or the public have enough visibility to evaluate impact. When companies restrict transparency, how can the public or government know if any slow-down or oversight is appropriate?

Digital trace data is one example of a useful research resource. It is, simply put, a record of online activity. Usually it is not originally intended for researchers but created for other purposes, like advertising or broadcasting a political opinion. The number of followers in a Facebook group and Google’s AI overview of search results are both digital trace data, but neither are inherently created for research.

Still, researchers have shown that this type of data can answer questions across the spectrum of human experience, including human trafficking recruitment, investor herding, acts of free expression, illicit drug market activity, and propaganda dissemination, using methods ranging from large-scale quantitative analysis to surveys and case studies.

Despite two-plus decades of research on social media, researchers are still unsettled on answers to important questions about harm or harm-mitigation. This is in part because existing resources still do not provide durable research access. An analysis of 113 unique researcher requests at the Princeton Accelerator, where my colleagues and I build and maintain tooling for research on the information environment, points to strong demand for a relatively small set of capabilities: researchers want to be able to analyze digital trace data independently, preserve it, and sometimes share it with partner researchers. That may sound like a basic requirement for scientific research, but in the social media space the problem is nowhere near solved.

While most people’s lives have become more digital, many researchers have moved away from digital trace data because it is too difficult to access or work with. Today, there are seven types of resources which researchers can draw on to study the information environment. None of these resources are without challenges; they are scattered across providers and hedged with restrictions on who may use them and for what.

The first type is company platforms and systems, to which access is usually mediated through restricted APIs or controlled research environments. Four years ago, the majority of peer-reviewed publications on the information environment focused on X/Twitter before X increased the price of academic access from free to $42,000 per month. Even when companies offer data access programs for researchers, it is limited, granted at the companies’ discretion, and often on a one-off basis. In most cases, the data cannot be retained. The lack of comprehensive and persistent access makes replicating or validating findings challenging.

Survey panels, the second type, are expensive to produce and limited to specific questions asked during a snapshot in time.

The third type, for-profit data providers, includes services which are often prohibitively expensive and don’t provide scientifically weighted datasets. As a result, researchers and readers (including policymakers) risk drawing conclusions about the data that is accessible, rather than the subject under question. The fourth type, open-source and free tools, are more accessible but suffer from the same lack of representative data.

The fifth type, archives and preservation efforts, helps; but coverage remains uneven across populations, platforms, and data types. The sixth buttresses and facilitates the others: researcher safety, legal, and ethics support includes guides, toolkits and advisory services to researchers for legal and ethical risks, intimidation, harassment, and the psychological impact of content. Researchers have faced these challenges in practice, including from social media companies directly.

That leaves the seventh resource: Europe tried to fix some of this through article 40 of the Digital Services Act (DSA), which gave European researchers a legal pathway to access social media data. In practice, though, the route is still slow and messy. Approval can take months, and companies are failing to comply. X has issued denials without explanation. TikTok has returned data with gaps. Meta omitted European politicians because they didn’t have 25,000 followers. As a result, the European Commission has issued preliminary findings against Meta and TikTok and fined X €120 million. If companies do not comply with legal penalties, a more flexible, self-policing, structure seems unlikely to be successful.

AI is even more difficult to study and evaluate than social media — and on multiple dimensions. Privacy concerns around chatbots are acute: because use feels like a private conversation with something that might feel human, users are often unusually vulnerable with AI chatbots. Still, with consent, researchers can study those interactions. But the training and datasets behind the model and the numerous decisions developers make to bring it to a customer, such as safety rules and the ongoing A/B testing, remain invisible.

There have been nascent steps to address this. For instance, the European Union just designated ChatGPT a “Very Large Online Search Engine,” triggering requirements for transparency and researcher access. But conversations where the chatbot adds its own input don’t appear to be covered.

AI might deliver enormous net benefits; it may even be a national security imperative. But that does not mean we shouldn’t systematically study and measure AI, even if the most sensitive capabilities remain restricted for safety and security reasons. Even if data access under the DSA improved, there would still be gaps and limitations, especially for non-Europeans. A global experiment is underway, where we can’t understand the effects and the companies are writing the rules.

There is currently no sanctioned route to independently study AI systems or tools at scale — not to mention their impact on people, economics, and society. After there is, then whatever is decided about guardrails and pace can be optimally informed. The United States has failed to create a data access pathway for studying social media and its effects. Will it do so for AI, even as it becomes an increasingly critical source of information?

This story was originally published by Tech Policy Press.

Read next:

• Americans Distrust AI Advertising, With 43% Feeling Duped, Yet 54% Plan AI-Assisted Holiday Shopping

• A critical flaw affecting 70% of all websites has the cybersecurity world panicked
by External Contributor via Digital Information World

A critical flaw affecting 70% of all websites has the cybersecurity world panicked

By Andrew Cullen, The University of Melbourne | Fact-Checked by Irfan Ahmad

Image: Ben Griffiths - Unsplash

Details have been published of a critical flaw in the programming language PHP, which is the backbone of more than 70% of all websites. Left unfixed, the flaw would allow hackers to easily take over and compromise millions of websites and the information held on their servers. In response, multiple European governments and operators including the Port of Rotterdam have taken websites offline.

The Australian Signals Directorate, America’s Cyber Defence Agency, and similar bodies from the United Kingdom, Europe and Japan have all warned web developers to apply urgent updates to their core systems, and to consider taking down websites until safety can be guaranteed.

The organisation behind the free, open-source language has published updates that patch the flaw, but performing the updates on the millions of systems that use PHP will take time.

What is PHP?

PHP is often called “the backbone of the web”. It’s a programming language – a way to give instructions to computers – that has been around since 1997, and it allows websites to manage user inputs and process information behind the scenes.

The power of PHP is its versatility. Websites use it for everything from logging users into accounts and running online stores, to requesting data from databases and powering millions of blog posts and news sites.

PHP runs on the server rather than your personal computer. This means most of us interact with PHP every single day without ever seeing it.

A timeline of concerns

In August, a cybersecurity researchers from a company called Wiz used their own tools and the latest versions of Google’s Gemini and OpenAI’s ChatGPT AI models to trawl through PHP’s massive codebase.

The Wiz team were searching for hidden programming weaknesses that an attacker could chain together to take over a system running PHP simply by sending a carefully crafted request over the internet.

Within a few days, they found ways to break into PHP-powered systems from Drupal and WordPress which are used in huge swathes of the web.

Wiz contacted the PHP security team to let them know about the flaws they had found. By late September, the PHP developers confirmed that the issue existed, and quietly began working on a fix.

In cybersecurity, the responsible thing to do when you find a bug is to keep it private until a patch is ready. This is because public disclosure could provide a detailed map for hackers to exploit.

This appears to be what the PHP developers were planning. However, yesterday a user on popular industry website Hacker News let news slip about the security flaws before any patches were ready.

Following the leak, PHP’s developers rushed to publicly confirm the issue on October 21, releasing emergency patches ahead of schedule.

Implementing the patches is far from simple. Because PHP sits underneath thousands of third-party plugins, frameworks and managed hosting services, websites are only secure once every individual layer of the supply chain integrates and deploys the PHP fix.

Why does this keep happening?

Much of the internet is built on foundational technologies that were developed and maintained by hobbyist developers. Many rely on goodwill and donations rather than robust corporate funding.

Many of these tools were first developed decades ago. They were never designed to safely support a world where millions of websites and billions of devices use them to handle sensitive and private data.

Compounding this issue is the fact that maintenance is rarely a priority for organisations. All too often, companies have only the most basic knowledge about all the pieces of their supply chains – either physical or digital.

Even in high profile cases such as this one, many businesses don’t even realise what software they depend on, let alone have any idea of how secure it actually is.

What should you do?

This issue is mainly one for people who maintain websites or manage technical infrastructure. For those people, the priority is clear: patch systems immediately when updates are available, and consider taking down critical systems from the internet until you can be confident about their security.

For everyday internet users, these vulnerabilities affect the servers hosting websites rather than your personal device. However, because some sites may be temporarily compromised before they can patch, it is wise to be cautious about entering sensitive personal data on unfamiliar websites over the next few days.

Major cloud providers and security firms are rolling out emergency filters, but the global scramble to secure the web is just beginning.

Cybersecurity teams around the world are rushing to contain the damage. But the scope of this issue should be a wake-up call.

Governments and businesses cannot ignore the resilience of their digital supply chains. Just as manufacturers track the suppliers of the physical components they depend upon, we should expect that digital infrastructure also traces every system and framework it depends on.

Gathering this information is the first step for businesses and governments to truly understand how heavily their digital viability relies on underfunded, volunteer-maintained code. Until businesses treat funding open-source foundational tools as a critical security investment rather than a charity, these vulnerabilities will keep happening.

Andrew Cullen, Guest Research Fellow, School of Computing and Information Systems, The University of Melbourne

This article is republished from The Conversation under a Creative Commons license. Read the original article.

Read next: New paper explores how chatbot health guidance leaves no trace for clinicians or regulators, and calls for transparency safeguards

by External Contributor via Digital Information World

New paper explores how chatbot health guidance leaves no trace for clinicians or regulators, and calls for transparency safeguards

By David Hermanovitch, Binghamton University | Fact-Checked by Irfan Ahmad

Image: Salvador Rios - unsplash

As more and more people turn to AI chatbots for quick, 24/7 medical advice, a new perspective paper co-authored by faculty at Binghamton University explores the risks this practice carries that are largely invisible to users – and healthcare systems.

The paper, written by researchers at Binghamton University, Stanford University, Texas A&M University, and Indiana University, and published in Nature Health examines what happens when a chatbot gives someone bad health advice. The authors argue that the harm is hard to catch because the conversation stays on the AI company's platform, the person has no place to report the problem, and outside researchers cannot measure the damage.

“We’re getting medical advice from these chatbots, but nobody—literally no one—is looking at this. If there’s an error or an issue, there’s just no way for people to know,” said Kaicheng Yang, an assistant professor in the School of Computing at Binghamton University’s Thomas J. Watson College of Engineering and Applied Science and one of the paper’s authors.

The researchers gave the example of a 60-year-old man who asked ChatGPT how to cut chloride from his diet and swapped table salt with sodium bromide. The man ended up being hospitalized with bromide toxicity after experiencing hallucinations and paranoia. The issue this paper highlights is that because the man received this advice from a chatbot and was unable to retrieve the conversation, his doctors had no way of knowing exactly what the man was told, making it nearly impossible to piece together exactly what happened.

Because there is no system of transparency in place for AI companies as it relates to the visibility of health advice it generates for patients, this paper asserts that this built-in, systemic lack of visibility is a key aspect of AI-generated health information because it actively prevents oversight. “Harms are not only possible, but structurally hidden from the clinicians, researchers, and regulators who could otherwise detect and correct them,” the authors note.

This paper also emphasizes a key facet of chatbots generating answers for people with health questions – they can be wrong. Chatbots can generate erroneous answers confidently, oversimplify information, or draw from outdated information/false claims. However, the authors also note that those risks can trickle down to those who are not actively seeking answers. With the prevalence of AI summaries in search engines and the integration of AI in social media platforms like X and Meta, you do not have to be actively searching for an answer to be given an incorrect bit of information.

“Sometimes I’m not even looking for health information, but just by browsing my social media feeds, it’s there. It just shows up, and we believe that could have undesirable outcomes, especially if there’s medical misinformation or state actors trying to manipulate the online discussion,” Yang said.

Regardless of whether a person is actively looking for answers or comes across them incidentally, the authors of this paper argue that the harm rarely leaves traces that clinicians, regulators, or researchers can verify.

To combat this, the authors propose several methods to increase transparency and responsibility for AI companies. They note that AI companies should give users access to their own health conversations to provide them the opportunity to share this information with clinicians so they can better investigate the pathway that led to a harmful event. They also recommend that those companies develop a disclosure system to allow health guidance to be flagged, reported, and investigated.

They further recommend that social media companies take more responsibility by enhancing clear labeling of AI-generated health content, withholding that content until it is vetted by medical governing bodies, and similarly, that search engines should only utilize vetted information in summaries. Policymakers, the authors suggest, should also extend physician malpractice liability to AI chatbot companies.

“We do not think this is something we should rely on the companies to do, because their incentive is always to make more money,” Yang said. “Building such a system goes against that incentive. So we have to have some kind of third-party monitoring system, an independent evaluation.”

Read next: University of Bath Study Questions TikTok’s Response to Reports of Suspected Illegal Drug Sales
by External Contributor via Digital Information World

University of Bath Study Questions TikTok’s Response to Reports of Suspected Illegal Drug Sales

By University of Bath | Fact-Checked by Irfan Ahmad

Bath study finds that TikTok's in-app reporting system fails to act on most reports made against accounts advertising illegal drugs for sale to UK users.

Image: Solen Feyissa - Unsplash

The vast majority of TikTok accounts identified by researchers as connecting drug sellers with potential buyers in the UK remained publicly accessible 14 days after being reported for suspected breaches of the platform’s rules on illegal drugs, according to a new study led by the University of Bath.

The findings raise questions about how effectively social media platforms respond to reports of illegal activity and whether the safeguards envisaged under the Online Safety Act are working in practice.

The study team, led by Professor Chris Pudney from the Department of Life Sciences at Bath, carried out what they believe to be the first independent real-time audit of how a major social media platform responds when ordinary users report accounts suspected of facilitating the sale of illegal drugs.

The new study is published in the International Journal of Drug Policy. Also involved in this work are researchers from University College London and the University of Manchester.

Over a five-month period from February to July 2026, the team identified 455 UK-facing TikTok accounts that offered to supply drugs. These accounts made references to cocaine, synthetic cannabinoids (commonly known as spice) and controlled medicines. To qualify for the study, all accounts had to show a drug-related cue, UK targeting and an offer or route to supply.

Testing the system available to ordinary users

The researchers used TikTok's standard in-app reporting function – the same process available to any member of the public who encounters suspected illegal content on the platform.

They then recorded TikTok's response and checked whether the accounts remained accessible.

The results were stark. TikTok returned an explicit ‘no violation’ decision for 95.2% of reports. After 14 days, 83.5% of accounts remained publicly accessible. This was true for different categories of drugs and different forms of promotion.

Under the Misuse of Drugs Act 1971, supplying or offering to supply a controlled drug is a criminal offence.

In addition, the Online Safety Act 2023 classifies drug supply offences as priority offences. Since 17 March 2025, regulated user-to-user services have been required to take proportionate measures to prevent users encountering priority illegal content, and to operate proportionate systems and processes designed to minimise how long such content remains available and swiftly remove illegal content once they become aware of it.

TikTok's own Community Guidelines also prohibit the trade, marketing and promotion of illegal drugs. The study examined content that the researchers argue should represent a relatively straightforward test of the platform reporting system: accounts openly displaying drug-related imagery or terminology, targeting UK users and signposting an offer or route through which purchases could be supplied.

Professor Pudney said: “This is a straightforward test of a reporting system. These were accounts we identified because they displayed drug-related cues, targeted UK users and contained an explicit offer or route to supply. Yet 95% of our reports came back ‘no violation’.

“It’s one thing for a platform to report very high proactive removal rates. It’s another for an ordinary user to point directly to apparent drug-selling accounts and be told, repeatedly, that there is no violation. Our findings show a serious weakness in the reactive reporting pathway we tested.

“The Online Safety Act treats offering illegal drugs for sale as a priority offence. These findings do not by themselves establish legal non-compliance, but they do raise an obvious question for Ofcom: are the systems and processes platforms have in place actually effective when users report this kind of content?”

Easy to find, hard to remove

The study was not designed to estimate how common drug-selling content is on TikTok. However, the researchers note that hundreds of eligible accounts were easily identified through ordinary searches using widely known drug-related terms and sales-related language.

The study found that most change in account accessibility occurred within the first two days, after which removal was limited. Every report classed by TikTok as ‘no violation’ was returned exactly 30.5 minutes after reporting. Accounts that survived the initial period were unlikely to be removed later.

Professor Pudney said: "We could not determine whether decisions were made by automated systems, human moderators or a combination of both. However, the fact that every ‘no violation’ decision was returned at precisely 30.5 minutes suggests that at least some automation is likely to be involved in the process."

He added that timeliness matters because harmful content can accumulate much of its exposure early in its life cycle.

Are online safety regulations fit for purpose?

The researchers stress that their findings should not be interpreted as evidence that all drug-selling content escapes detection by TikTok.

The platform reports removing large quantities of violating content through its own proactive moderation systems.

However, the Bath-led study examined a different question: what happens when a user directly reports an account that appears to be involved in illegal drug supply.

Professor Pudney said: “Platforms often publish impressive statistics showing how much content they proactively remove, but these tell us very little about whether a platform acts when a user directly reports apparent criminal activity. That gap should be tested, not assumed away.”

The researchers are calling for independent, offence-specific audits of platform responses to test whether online-safety systems work in practice when users directly report suspected illegal content.

Professor Pudney said: “Ofcom should be independently testing whether the systems that ordinary users rely on to report suspected illegal content work in practice. If an ordinary user can repeatedly point a platform towards apparent drug sellers and receive ‘no violation’, that is exactly the kind of real-world performance a regulator should want to understand.”

Update: TikTok Responds

A TikTok spokesperson told Digital Information World in an email:

“We strictly prohibit the sale or promotion of illegal drugs on TikTok. Whilst the full data set from this research has not been shared with us, we have removed violative content and accounts. We continuously improve our moderation systems, and since this study was conducted, we have strengthened in-app reporting."

Read next: Why calling for a ‘kill switch’ is the wrong way to think about reining in AI
by External Contributor via Digital Information World

Why People Resist Change: Study Identifies 12 Reasons, Including Fear, Effort and Ethical Objections

By Radboud University

Change often meets with resistance, such as during reorganisations or when we are asked to eat less meat. Researchers at Radboud University investigated this resistance and identified twelve reasons why people resist change. They also developed a scale to measure why people resist a specific change. ‘It is important to take people’s concerns about change seriously.’

Almost all change is difficult: adapting your lifestyle, a reorganisation at work, or starting to use gender-neutral language. When behavioural change is required, resistance is therefore often encountered. As a result, it can sometimes take a very long time for a change to be accepted – and sometimes it doesn’t happen at all. It is therefore important to understand where that resistance comes from, according to Radboud University communication scientist Marieke Fransen. Fransen and her colleagues investigated this and identified twelve reasons.
Twelve reasons

‘People who resist change are often dismissed as difficult, but that resistance often stems from entirely legitimate concerns,’ says Fransen. ‘For example, people are sometimes afraid that the change won’t work, or that they don’t have the right skills to adapt their behaviour, or that the change will restrict their freedom. It is important for organisations, managers and communications officers to understand these concerns. Then you can address them.’

The researchers developed a measurement tool called the ‘Reasons for Resistance to Change Scale (RRCS)’. The RRCS consists of 36 questions that can be used to measure exactly why people feel resistance to a particular change. The researchers identify the following twelve reasons:

1. Doubts about effectiveness: people are not convinced that the proposed change will work.

2. Lack of personal relevance: the change does not seem relevant or in the interests of those affected.

3. Ethical objections: the proposed change goes against their principles.

4. Lack of autonomy: the feeling that the change will restrict their freedom or their ability to make their own choices.

5. Fear of social judgement: fear that others will think less of them.

6. Doubts about those driving the change: questions about the competence of the people implementing the change.

7. Preference for the status quo: people are satisfied with the situation as it is.

8. Uncertainty about their own abilities: fear that they will not be able to successfully adapt to the change themselves.

9. Effort: the change requires too much time or effort.

10. Lack of inclusion: a feeling of having too little influence on decision-making.

11. Entitlement: people find the change unfair or feel they deserve better.

12. Fear of the unknown: uncertainty about what is about to happen creates an unpleasant feeling.
To measure is to know

To develop the measurement tool, researchers used data from questionnaires completed by 2,341 participants from the United States and the United Kingdom.

‘With many changes, people say they feel they have too little autonomy, so that reason comes up frequently,’ said Fransen. ‘But beyond that, the reasons why people oppose a particular change vary widely.’ For example, in the case of a new vaccine, ‘fear of the unknown’ scored very highly within the study population, whilst people who resist the idea of eating less meat primarily prefer the status quo and do not feel that this change is personally relevant to them.

Image: https://ift.tt/I71gQRd — CC BY

‘If you know which reasons are the main causes of resistance, you can take appropriate action,’ according to the researcher. ‘If people are wondering whether the change will actually be effective, you can share convincing evidence of this with them through clear communication. And if people are afraid that they themselves do not have enough skills to make the change, you can offer support or training.’ Sometimes the solution does not lie in communication, but rather in adjusting the change itself, making it easier for those involved to accept. For example, if people feel excluded from the decision-making process, it may be a good idea to give them a say after all.

‘There will always be resistance,’ says Fransen. ‘That’s not a bad thing. Resistance can actually help us achieve something better. But rather than speculating about why people aren’t on board with a particular change, it’s better to measure it.’

This research was made possible in part by a grant from the Centre for Strategic Communication, Logeion. See also the previously published white paper (in Dutch): https://www.logeion.nl/wp-content/uploads/2025/08/20250602_csc_whitepaper_05_WHITEPAPER-2.pdf

Literature reference: Fransen, M. L., Vandeberg, L., & Denkers, A. (2026). 12 reasons why people resist change: Development of the reasons for resistance to change scale (RRCS). Basic and Applied Social Psychology, 1–14. https://doi.org/10.1080/01973533.2026.2741816.

Read next: 

• Four AI engines were asked the same local question every day. They rarely agreed

• How doubting your doubts may increase commitment to goals
by External Contributor via Digital Information World

Four AI engines were asked the same local question every day. They rarely agreed

By Terry Carter, Founder NameBeam

Reviewed by Irfan Ahmad

Ask ChatGPT which roofer to call in Dallas, TX and you get a name. Ask Perplexity and you may get a different one. Ask again tomorrow and the answer can change. NameBeam, a small Wyoming company, decided to write all of it down.

Since September 1, 2026, NameBeam has asked four AI engines (ChatGPT, Perplexity, Gemini and Claude) the same buyer questions in 46 US market-and-trade pairs, once a day. Every answer is saved, hashed and recorded. The counts are in a public file and the raw answers sit behind each market.

In the window from September 1st - October 11th, 36 of the 46 markets had two or more engines naming at least one business. In only 14 of those 36 did two or more engines pick the same business as their most named answer. The other 22 markets split. A buyer in those cities gets a different first name depending on which app they open.

Roofing shows the split most clearly… Take residential roofing in three cities. In Dallas, Perplexity named Arrington Roofing on 27 of its 37 answer days (https://ift.tt/av9N6h7). In Denver it named Gates Enterprises on 35 of 37. In Miami it named Extreme Roofing on 26 of 37.

ChatGPT and Claude named none of those 3 companies on any of their 40 answer days. Gemini, asked fewer times because of its quota, named them on 0 of 8, 0 of 9 and 0 of 10 days. Same city, same question, same six weeks. One engine returned a steady name. Two returned with nothing for it at all.

Across all 46 markets, Perplexity named at least one business in 37 of them. Gemini did in 32, ChatGPT in 24 and Claude in 9.

When an engine did name a business, how steady was it? Perplexity's most-named business showed up on 56 percent of its answer days on average. Gemini's was 72 percent across the 40 markets where it answered. ChatGPT's was 20 percent and Claude's 7 percent.

The method is plain. A business counts for an engine on any day the saved answer names it in full. Each engine is measured against its own answer days, so a quota-limited engine is not punished for being asked less often.

What the record does not claim

The record does not say why an engine picks a company. It does not say that a website change or a review caused a name to appear. It records what was said, on which day, by which engine, and keeps the text so the count can be rerun. Namebeam sells that record to agencies and platforms that serve local businesses, not to the businesses being measured.

The practical point is simple. "Does AI recommend us?" is not one question anymore. It is four, and they can have four different answers. A dated record is the only way to know which.

About the author

Terry Carter is the founder of Carter Enterprise LLC in Sheridan, Wyoming. He runs Namebeam, a public record of how AI engines name local businesses. Every figure he publishes carries a date and a link readers can open.

Read next: 

• Scientists Outrank Government Officials but Not Business Leaders, as 56% Question Their Independence, Bias or Relevance, Study Finds

• AI Models Reported 88% Confidence but Were Correct 79% of the Time, Study Finds
by Guest Contributor via Digital Information World

Scientists Outrank Government Officials but Not Business Leaders, as 56% Question Their Independence, Bias or Relevance, Study Finds

By Joe Stafford, University of Manchester

Fact-Checked by Irfan Ahmad

People more likely to see scientists as credible experts when they discuss technology than when they comment on wider social issues.

A study of more than 6,000 people in the UK and Sweden suggests that although scientists are generally regarded as credible, their status as experts is more contested when they discuss issues such as health, education and employment.

Researchers from The University of Manchester and KTH Royal Institute of Technology in Sweden found that scientists were perceived as more credible than government officials, but were no more credible - and in some circumstances less credible - than businesspeople and campaigners.

The findings challenge the assumption that scientific expertise automatically gives academics greater credibility when they contribute to wider public debates.

Scientists don't automatically have the advantage

The researchers wanted to examine whether people actually regard scientists as more credible than other groups when presented with the same claims about society.

Rather than simply asking people whether they trust science, the study used a survey experiment in which participants were asked to assess statements attributed to different types of experts. These included an academic scientist, businessperson, government official or issue advocate.

The statements covered technological and non-technological issues, allowing the researchers to examine whether the identity of the person making a claim affected how convincing it appeared.

The study involved nationally representative samples of adults in the UK and Sweden. Data was collected online, with 6,198 complete responses ultimately included in the analysis. Each participant assessed 8 different combinations of statements and expert types.

Technology makes a difference

The researchers found a clear difference depending on what scientists were talking about.

When discussing the societal consequences of technological developments, scientists' credibility increased relative to other expert groups. But when the same set of experts was discussing non-technological social issues such as health, education or employment, businesspeople and issue advocates were perceived as more credible than scientists.

The researchers suggest this may reflect the different ways people understand scientific expertise. Technology-related questions may be seen as areas where scientists possess particularly relevant specialist knowledge. Wider social questions, by contrast, can involve competing values, political judgements and forms of knowledge outside academia.

Why might people distrust scientists?

The researchers considered three potential sources of distrust. People were asked whether they viewed university scientists as being:

• closely allied with vested interests rather than independent;
• ideologically driven rather than open-minded; or
• insufficiently focused on issues relevant to society.

Around 56 per cent of all respondents agreed with at least one of these statements: 37 per cent viewed scientists as influenced by vested interests, 37 per cent considered them ideologically biased, and 33 per cent believed scientific knowledge had limited relevance to the types of societal issues examined in the study.

Image: Research Policy 55 (2026) 105640. Anders Brostrom, Cornelia Lawson, Mabel Sanchez-Barrioluengo. CC BY

The researchers found that these three forms of distrust were associated with the credibility disadvantage experienced by scientists.

They also found that the specific credibility gap between technological and non-technological issues was largely driven by people who expressed these concerns.

Rethinking the role of scientific expertise

The researchers say the findings highlight the difference between public trust in science in general and the credibility of scientists when they act as experts in specific public debates.

The study suggests that people may broadly support science while being more selective about whether they regard scientists as credible sources of advice on particular social questions.

“Scientists are increasingly expected to contribute to debates about some of the biggest challenges facing society, but our findings suggest that this does not automatically translate into greater credibility,” said Professor Cornelia Lawson from The University of Manchester. “This presents an important challenge for universities, scientists and research funders seeking to ensure that research has an impact beyond academia.”

“It also raises wider questions about the changing relationship between science and society, and what we now expect scientists to contribute to public debate.” — Professor Cornelia Lawson

The research was conducted by Anders Broström of KTH Royal Institute of Technology and Cornelia Lawson and Mabel Sanchez Barrioluengo of the Manchester Institute of Innovation Research, Alliance Manchester Business School, The University of Manchester.

Publication details

This research was published in: Research Policy

Full title of the paper: Contested credibility: Public perceptions of scientists as experts

DOI: 10.1016/j.respol.2026.105640

URL: https://authors.elsevier.com/sd/article/S0048-7333(26)00231-3

Read next: Kids and teens are going to AI chatbots for advice. How can parents make sure this is safe?
by External Contributor via Digital Information World

Kids and teens are going to AI chatbots for advice. How can parents make sure this is safe?

By Amy Dawel, Australian National University.

Fact-Checked by Irfan Ahmad.

Image: Nicola dowie - Unsplash

Last week, Australian Federal Police commissioner Krissy Barrett warned artificial intelligence (AI) chatbots are not a sufficient replacement for human support for children. The comment was in response to a recent Adelaide University study that found more than one in three Australian children turn to AI chatbots for support when they are sexually victimised online, or otherwise tell nobody at all.

While there can be benefits to children using AI, such as help with learning, or practising social skills, there is also a risk they will go to AI chatbots and assistants for emotional support over parents and other trusted adults.

In Australia, around half of children aged between 10 and 17 who use AI are turning to it for personal and social reasons. This includes seeking advice on sensitive topics, talking about their feelings, and just “hanging out”.

So how can parents help their children to use AI safely?

Why are kids turning to AI chatbots over people?

One practical reason why young people are using the tech is it can provide them with answers around the clock. This means a chatbot might be a natural go-to when a child wants a quick answer and a trusted adult or friend isn’t available.

Another reason is pure curiousity. Like adults, young people are experimenting with AI and trying to understand what it does and how it can make their lives better.

Seeking deeper connections

Sometimes the reason for talking to chatbots is deeper though.

It might be that a young person wants information about something they feel embarrassed about raising with their parents or friends. As teens explore their identity, they begin to ask more sensitive questions. This may make them more likely to use AI for personal reasons than younger children.

For others, the reason is an unmet need for connection. Reaching out to chatbots for personal reasons is more common for those who are already vulnerable.

A 2025 Danish study found loneliness may be a precursor for high school students seeking AI companionship. Other studies suggest a lack of connection to parental figures may also lead to teens talking to chatbots.

What do kids miss out on when they choose chatbots over people?

A major concern is that, if AI use becomes excessive, it may displace real-world interactions and create unrealistic expectations of human relationships.

This problem is linked to sycophancy – the tendency of AI to be overly agreeable.

AI interactions lack the natural friction that happens in human relationships. This warping of social experience is a particular worry when it comes to young people because they are still forming their understanding of how relationships work. We know real-life human social interaction is crucial to teens’ development.

AI lacks context

Another concern is the quality of the advice AI provides. And it’s not just its capacity to spread convincing misinformation that’s a problem.

AI lacks the full context of a child’s life. Parents and carers are often much better placed to provide the nuanced and age-appropriate guidance their child needs.

There are also more subtle impacts. When a child poses questions to an AI chatbot over a parent, they are missing out on day-to-day moments of connection with family. While each miss may be small, the accumulation over time can truly matter.

This is especially important for children. Research shows strong and secure bonds with parents in the preteen years support kids as they navigate the challenges of adolescence.

Here’s how can parents and carers can help their child experiment with AI and stay safe.

1. Explore AI together

Find a moment to sit down and explore AI tools on a device with your child. You might:

• engage in brainstorming ideas with the software

• test a chatbot’s ability to correctly source information

• ask the AI about itself with questions such as “do you have feelings?” and “can you think?”

• ask AI to generate and edit a short story with your child as the lead character

• ask AI to tell a joke

• play a game with an AI chatbot, such as twenty questions.

One AI game that is both fun and educational is the “catch the AI out” game. The idea is to test AI on topics that you know or can fact check. Points are scored each time you catch the AI make an error.

2. Ask your child questions

While exploring these AI tools and functions with your child, ask them questions to help them think more critically about the tech such as:

• what do you think AI is good at?

• what can humans do that AI can’t?

• where do you think AI gets it’s information from?

• how often is AI wrong about something? How can you check this?

• what do you think an AI chatbot does with the information you tell it?

Their answers can open up conversations that help your child enjoy AI while understanding its limitations and how to stay safe.

For instance, if your child says AI does everything better than humans, you can help them brainstorm all the things people have to offer – hugs, for starters.

Or, if your child isn’t sure what AI does with their information, you might talk them about privacy risks.

3. Remind your child you are there to support them

It’s important to encourage your child to come to you for advice on difficult topics instead of AI.

To help children feel more comfortable, maintain a curious mindset whenever they raise a sensitive topic with you. Research shows teens who perceive their parents as more accepting tend to share more. Try to take a moment to listen to them before jumping in with advice too.

You can also ask non-judgemental questions about your child’s AI use to show you’re engaged. This can be as simple as asking them how they’re feeling about a particular AI interaction, or what they’re currently using AI for.

Talking about AI with your child should be an ongoing conversation.

4. Keep up to date with the tech

One of the challenges for parents is that AI is rapidly evolving. Australia’s eSafety Commissioner offers resources that can help people keep up to date.

We are currently researching how children use AI chatbots and how it impacts their lives as they enter their teen years, through one-to-one conversations with them. If your child is aged between 9 and 12 years, and is chatting with AI, you can register their interest.

Amy Dawel, Clinical Psychologist and Associate Professor, School of Medicine and Psychology, Australian National University

This article is republished from The Conversation under a Creative Commons license. Read the original article.

Read next: UK Age Checks Prompt Privacy Warnings Over Potentially Risky VPN Services

by External Contributor via Digital Information World

UK Age Checks Prompt Privacy Warnings Over Potentially Risky VPN Services

By University of Edinburgh

Fact-Checked by Irfan Ahmad.

People’s online behaviour has changed in response to legislation to prevent children from viewing harmful content – raising possible privacy and security risks, research suggests.

Image: Erik Mclean - Pexels

A new study examined how individuals have responded to age verification checks enforced by the Online Safety Act (2023) and evaluated the potential risks from bypassing the rules.

The Online Safety Act is a set of UK laws and duties that require social media companies and search engines to protect users, especially children, from illegal content and harmful material.

Significant increase

Researchers from the University of Edinburgh, in partnership with the Universities of Newcastle and Durham, studied data from Google Trends and Reddit, a forum-based social media platform.

They found that UK users’ online discussions about Virtual Private Networks (VPNs) – tools used to circumvent the new restrictions – increased by 145 per cent, whilst UK Google searches for VPNs rose by 147 per cent, using statistical analysis to compare with a typical day.

Discussions relating to the Act in UK political communities on Reddit also increased by 1,481 per cent.

Age regulations

The increases were concentrated around 25 July 2025, when age checks for adult websites and some social media platforms came into force, researchers say.

Whilst online discussions do not translate directly into VPN usage, the team used this as a proxy for interest, as they were not able to access data on the number of VPN downloads.

Age verification may require users to disclose sensitive information about themselves, such as facial scans, identity documents or payment details.

Increased risks

A VPN creates a secure, encrypted connection between users’ devices and the internet. It hides individual IP addresses and jumbles data so internet providers cannot track a location or online activity.

However, people who seek to gain access to restricted sites without submitting personal data by using VPN services may be putting themselves at risk, experts say. This is because some high-risk VPN services track, retain and share user data with third parties.

The increase in VPN interest matters because people using these services may not understand the difference between low and high-risk providers, experts say.

Researchers say there is a need for greater understanding of the privacy risks of VPNs to help keep everyone safe online.

“The sharp rise in interest in VPNs highlights an important unintended consequence: users seeking to avoid sharing sensitive information for age verification could unknowingly use services that collect and sell their data.” — Dr Tuğrulcan Elmas, Lecturer in the School of Informatics, University of Edinburgh.

“It surprised us that people trust VPN providers over the companies who have been certified to run the checks, especially as user interest also increased for high-risk VPNs. We urgently need to raise awareness of the privacy risks of VPNs and highlight that not all VPN providers are created equal.”  — Dr Marc Juarez, Lecturer in Cyber Security and Privacy, School of Informatics, University of Edinburgh.

“What stood out from our findings was that users did not frame VPNs as a way to bypass restrictions, but as a way to preserve privacy amid identity checks, facial scans and surveillance. Future policies should account for how people actually respond and address the concerns that drive them towards circumvention.” — Dhyey Mehta, Undergraduate student, School of Informatics, University of Edinburgh.

Read next: AI Models Reported 88% Confidence but Were Correct 79% of the Time, Study Finds
by External Contributor via Digital Information World

AI Models Reported 88% Confidence but Were Correct 79% of the Time, Study Finds

By Scott Morrison, University of California

Fact-Checked by Irfan Ahmad.

Image: Mirella Callage - Unsplash

It is no secret that AI chatbots hallucinate. Lawyers have filed AI-generated court briefs citing made-up cases. Students have turned in papers footnoted with studies never published. Reporters and researchers alike have seen chatbots confidently name sources and statistics that simply don’t exist.

Don Moore, a UC Berkeley Haas management professor who has spent his career studying human overconfidence, went into his research on AI assuming this kind of confident fabrication was a “perversity” unique to machines—the behavior of a system that hadn’t yet learned to reason properly. But the deeper he delved, the less it looked like a machine problem at all. AI chatbots, he came to realize, are trained to be compliant, agreeable, and quick to produce an answer rather than admit they don’t have one. Humans, when pushed to appear competent and cooperative, do the same thing.

“It’s not that different from when humans are confident and wrong,” says Moore, the Lorraine Tyson Mitchell Chair in Leadership and Communication. “We do it routinely.”

That realization sits at the heart of his new working paper, currently under review at the journal Cognitive Science. Coauthored by professor Jacob Bien of the USC Marshall School of Business and recent UC Berkeley data science graduates Noam Michael, BA 26, and Daniel BenShushan, BA 26, the study tests 11 leading Large Language Models (LLMs)—including models from OpenAI, Google, Anthropic, and Meta—on their ability to know what they don’t know. The research finds that the LLMs they studied are, on average, overconfident. And like people, the AI models grow more overconfident as questions get harder and oddly under-confident when questions are easier.

The findings raise uncomfortable questions about trust. The paper notes many users already rely on chatbots heavily and unquestioningly, trust that may be misplaced if a model’s confidence outpaces its accuracy. Moore fears this same blind spot could show up in self-driving cars that don’t know when to hesitate, or in autonomous weapons systems already in development, where an AI’s failure to register its own uncertainty isn’t just misleading but dangerous.

Calibration versus accuracy

Most people judge a chatbot’s answers in binary terms: they are right, or hallucinations. Moore’s research measures not whether AI gets things right but whether it knows when it’s likely wrong. Psychologists call this calibration—holding a belief with a level of confidence that actually matches how likely it is to be true. A well-calibrated mind, human or machine, sounds sure about the sun rising tomorrow but will hedge on a coin-flip election.

This is different from accuracy. A model could get 90% of answers right and still be badly calibrated if it claimed near certainty on every one, including the 10% it gets wrong. Poor calibration leaves users unable to tell which answers deserve a second look. Part of the problem is rooted in how large language models work: they generate text by predicting likely next words rather than looking up verified facts. Every response, confident-sounding or not, is ultimately an educated guess.

To measure AI calibration with precision, Moore’s team devised a new test they call LifeEval. It gives an AI a person’s sex and age, asks it to guess how long that person will live, and checks its stated confidence against government life-expectancy data. By tweaking the person’s age or how close the guess needs to be, researchers could make the test harder or easier without changing anything else about it—solving a problem with older tests in which hard and easy questions differed in too many ways at once. LifeEval lets researchers isolate one clean question: as difficulty changes, does AI’s confidence rise and fall the way it should?

Across all 11 models tested, the pattern was strikingly consistent. On average, models reported 88% confidence in their answers but were correct only 79% of the time, a nine-point overconfidence gap. The researchers suggest AI exhibits the “hard-easy effect” for the same reason people do: confidence is a noisy, imperfect signal of real performance, so it drifts toward a default middle ground, leaving models overconfident on hard questions and too modest on easy ones.

Newer “reasoning” models, built to deliberate and self-critique before answering, were meaningfully better calibrated than faster chat models, though still imperfect. Preliminary comparisons, Moore says, suggest AI’s overconfidence is somewhat milder than typical human overconfidence, a finding he is continuing to study.

The core issue, Moore’s research suggests, isn’t that models perform badly; it’s that they don’t know what they don’t know, any more than people do. When a model wanders into unfamiliar territory, it tends to answer with the same assurance it uses on questions it handles well, leaving users unable to distinguish the two.

“What does overconfidence look like?” asks Moore. “It looks like confidence.”

The training trade-off

Part of the explanation, Moore argues, traces back to how these models are built. Before public release, AI systems go through “reinforcement learning from human feedback,” or RLHF, a fine-tuning process in which human evaluators rate model responses, nudging the system toward answers people prefer. The trouble is that people tend to prefer confident, agreeable answers over hedged, uncertain ones. That preference may be training overconfidence directly into the system—models emerging from RLHF may be more compliant and more assured than before training. It exposes a real tension between an AI built to be liked and one built to be accurate, and Moore suggests the likely resolution isn’t a single fix, but different classes of AI for different purposes.

“My guess is that there will be different sorts of AI systems, some designed for support and therapy and others designed for industrial applications, whose goal is to be useful,” says Moore. “Useful systems should avoid flattery and sycophancy whenever possible. They should speak the truth directly and as accurately as they know how.”

In the meantime, users can take small steps on their own. Moore says he routinely prompts his own AI assistant to accompany each answer with a calibrated probability and to flag its own uncertainty rather than present every answer as certain. “It’s pretty good about noting—here’s where the evidence is good, here’s what I’m sure about—but you might want to check this,” says Moore.

Users, he adds, should treat AI as they would any fallible source of information. “Trust,” he says, “but verify.”

Read next: Who’s paying whom? The circular money machine behind the AI boom (or bubble)
by External Contributor via Digital Information World