Online reviews can expose hidden social ties, increasing risks of phishing attacks

By Hope Reese, University of Texas at Austin

On the surface, posting online reviews looks like a win-win activity. It helps both businesses we like and people who might frequent them.

But although posting reviews can benefit others, it can also put us all at risk, according to new research from the McCombs School of Business at The University of Texas at Austin. It can inadvertently expose our personal connections and make us and our online friends more vulnerable to cyberattacks.

The research focuses on a particular kind of email attack called spear phishing. A bad actor impersonates someone the target user trusts. The actor tries to trick the target into sending money or disclosing confidential information, such as passwords or bank accounts.

Screenshot of Yelp homepage: DIW - CC BY

In recent years, phishing has become big business, says Yan Leng, assistant professor of information, risk, and operations management at McCombs. From 2021 to 2023, she reports, the FBI’s Internet Crime Complaint Center received nearly 1 million complaints involving $305 million in losses.

“Social influence is a good thing,” Leng says. “The problem is that network data could be leaked.”

Exploiting Connections

By network data, Leng means information about our social relationships with people we’re connected to online. They can be on social platforms such as Facebook and review platforms such as Amazon, Google, or Airbnb.

Unlike Facebook, most review platforms don’t list a person’s friends. But Leng suspected that a phisher could figure out someone’s friends from their online behavior, such as reviews they wrote and ratings they provided.

She investigated, with Yijun Chen of the University of Melbourne; Xiaowen Dong of the University of Oxford; Junfeng Wu of the Chinese University of Hong Kong, Shenzhen; and Guodong Shi of the University of Sydney.

The researchers used the business review platform Yelp and covered 4,299 reviewers from Louisiana and Pennsylvania in 2020. On Yelp, unlike many review platforms, both texts of reviews and lists of friends were publicly accessible.

That allowed Leng to cross-check her work. First, she analyzed people’s reviews to deduce their networks of connections with other users, as a cyber-attacker might do. Then, she compared those presumed connections with their actual lists of friends.

She found that an attacker could correctly identify 49% of users’ social relationships purely from their online behavior, while incorrectly labeling only 10% of unconnected pairs as connected. When that false-alarm rate rose to 20%, an attacker could correctly identify even more relationships: up to 63%.

Why? The most salient clue was the lengths of Yelp reviews. When two people follow each other, Leng found, there’s an observable relationship between the lengths of their reviews.

If my friend writes longer reviews on Yelp, she explains, I follow suit and also write longer reviews. Another kind of relationship is that, if my friend writes longer reviews, I write only a short review, which can complement theirs.

Once a spear phisher infers a user’s relationships on Yelp, they can send scam texts or emails to that person’s friends, Leng says.

Volume matters, she adds. The more connections a scammer can identify, the more impersonation attempts they can make, and the higher their returns on the costs of targeting and sending emails. For the Pennsylvania data, returns soared from 109% for 500 attempts to 1,098% for 10,000 attempts.

Adding Noise Discourages Scammers

Unfortunately, Leng says, existing privacy laws, such as the European Union’s General Data Protection Regulation, don’t fully protect against this kind of privacy risk. Even if a platform doesn’t publish users’ friend lists, bad actors can infer them from seemingly harmless behavioral data.

To start with, platforms should evaluate whether they’re creating this type of risk, she says. Besides review platforms, e-commerce marketplaces and media-sharing platforms can be vulnerable.

Then, they should develop safeguards to reduce risk. One strategy, she suggests, is to add noise, which she defines as “small, carefully designed random changes to the data before it is released.”

A platform could subtly alter the text of reviews, to vary their lengths without changing their meanings, she explains. That could blur the behavioral fingerprints that reveal relationships without making the data useless for the platform’s own analytics.

In simulations, the researchers found the strategy reduced economic incentives for cyber-attackers, sometimes even making returns negative.

How much noise to add could be a challenge, she cautions. Each platform could choose a privacy budget, based on how much information it needs to retain and how much social privacy risk it is willing to accept.

One thing is clear to her: The current level of privacy risk is unacceptable. Says Leng, “The platforms need to protect not only what users disclose, but also what others can infer.”

“When Behavioral Data Betray Users: A Diagnostic and Protective Framework Against Social Interaction Leakages” is published in Information Systems Research.

Fact-Checked by Irfan Ahmad.

Read next: 

• AI agents can now remember and hackers can ‘poison’ their memories — a new cybersecurity threat

• Ask for More Pay and Get Penalized, Stay Quiet and Fall Behind
by External Contributor via Digital Information World

Do AI recommendations beat a vet’s? Here’s what pet owners say

Is AI reliable when it comes to your pet’s health? According to a new survey, conducted online between April 22 - April 28, 2026, more than two in five (43%) use tools like ChatGPT for recommendations on their pet’s health and nutrition.

Nearly half (47%) mainly use AI for more unusual questions such as “Can pets see ghosts?” But the study of 2,000 pet owners, commissioned by Darwin’s pet food and conducted by Talker Research, found more health-based questions averaged seven times a week.

Pet parents will ask AI about nutrition (55%), symptoms or illness (54%), pet care products (52%) and general curiosity about behavior (50%).

Almost three in five (59%) have made changes to their pet’s care as a result of AI recommendations. And 90% of pet owners say AI and online resources have made them feel more informed. But 50% admit researching pet care decisions online sometimes makes them feel unsure.

Veterinarians still come in first for pet owners’ most trusted source for health and nutrition advice (71%), and just 14% trust AI tools the most. If the two contradict each other, 64% will listen to their vet over 12% that choose AI.

Overall, 30% of pet owners agree using AI increases worry for pet health, mainly when it comes to health (47%) and nutrition (33%) .

The study also asked pet owners how they try to verify info they find online, and as most trust vets the most, the majority of pet parents (57%) rank professional credentials as the best signal for trusted info. After that is personal experience (34%), scientific studies (31%) and AI-generated responses (11%).

“AI is changing how people gather information, but it doesn’t change the responsibility that comes with caring for an animal,” added Hagenson. “The most confident decisions tend to come from balancing expert advice with firsthand understanding of your pet, rather than relying on any single source.”

This firsthand understanding is shared by many as 89% said AI can’t replace a pet owner’s intuition. But trust will only become more challenging as 52% believe AI will be a normal part of pet care within the next five years.

Fact-Checked by Irfan Ahmad.

Read next: 

• How AI could make us work even harder

• As Bots Gain Authority Over Workers, Does Human Oversight Still Matter?
by Guest Contributor via Digital Information World

How AI could make us work even harder

Malte Jauch, University of Essex

Image: Vitaly Gariev - Unsplash

Technological progress has taken more than its fair share of the workload from humans over the years. Washing machines clean our clothes, tractors plough the fields and computers sort out the admin.

Meanwhile, economic growth and redistribution of wealth means most workers in industrialised countries do not need to spend most of their waking hours on gainful employment. The 12 hour working day that was common during the 19th century has been largely reduced.

So will developments in AI help us gain even more free time?

Not necessarily. In fact, AI might actually cause working hours to increase.

One reason for this has to do with the effects of AI on people’s bargaining power in the workplace. If workers are fearful that their jobs might be lost to automation, research shows they may be inclined to increase their efforts as a way of signalling to employers that their labour is still valuable.

My research on the rat race suggests that this kind of response is to be expected.

Employers must continuously decide whom to hire, promote or dismiss. In making such decisions, employers seek to reward productivity. But productivity is often hard to measure, so employers rely on alternatives metrics such as the amount of time spent working.

Checking whether a work station is still occupied after everyone else has gone home is much easier than assessing the accuracy of a complex project. And from a worker’s point of view, increasing the time they spend at work seems like an effective way of setting themselves apart from other (apparently less motivated) workers.

But when everyone pursues this strategy, it becomes self-defeating. No one gains a positional advantage, while everyone works more.

The introduction of AI to workplaces can turbocharge this dynamic. Existing AI tools have already automated some tasks, like data entry, creating an expectation that entire roles will be redundant soon.

Against this backdrop, the rat race becomes more cut throat. Workers reasonably expect large scale restructurings of their companies and seek to make sure that they will retain their jobs in the process. Researchers recently found evidence for this, noting that workers in AI exposed industries like financial services and IT are literally switching off their office lights later than those in less exposed industries.

Another problem is that employers can overestimate the extent to which AI boosts workers’ productivity. The availability of AI tools to improve efficiency creates an expectation among employers that workers can do more than they could before in the same amount of time. However, when these expectations are overblown, they can lead to additional work intensification or longer work hours.

Working nine ‘til…?

Finally, the adoption of AI causes significant frictions. Established routines are overhauled, existing roles change and new roles are created. Workers must be trained in the use of AI tools and must learn to work alongside them.

But what makes this particularly challenging is that existing AI tools are continuously being improved and new tools developed at a rapid pace. All of this places additional demands on workers’ time.

It may be sobering to contemplate that AI might not improve our lives by relieving us from toil. Overwork and burnout might even increase over the coming years.

But that situation is by no means inevitable or unavoidable – especially if future control over AI’s design and implementation is not left to employers and large AI firms.

It is possible in theory to design and implement AI in ways that improve workers’ skills, enhancing their areas of expertise, making their work more valuable and thereby enhancing their bargaining power with employers. An empowered workforce would be able to bargain for favourable working conditions, including working time patterns that fit with people’s lives and preferences.

And if developers were to prioritise augmentation and empowerment, AI could be the ultimate tool that helps to tackle widespread resentment and disenchantment with our working lives.

Malte Jauch, Lecturer in Management and Marketing, University of Essex

This article is republished from The Conversation under a Creative Commons license. Read the original article.

Fact-Checked by Irfan Ahmad.

Read next:

• How Remote Interviews Are Changing What Hiring Managers Look For

• As Bots Gain Authority Over Workers, Does Human Oversight Still Matter?

by External Contributor via Digital Information World

Perfectionistic leadership can affect employees beyond work, contributing to work-family conflict and presenteeism

By University of East Anglia

Image: Mushvig Niftaliyev - Unsplash

Employees under pressure to meet exceptionally high standards are more likely to experience conflict at home and work while unwell, according to a University of East Anglia study.

Researchers found that employees who perceive their leaders as perfectionistic are more likely to experience work-family conflict.

That conflict, in turn, increases the likelihood of presenteeism, where employees continue working despite being ill.

The findings suggest that leaders who impose unrealistically high standards on others can affect employees' wellbeing, both at work and at home.

The results, uncovered by academics from UEA, Sheffield Hallam University and the University of Liverpool, are based on a study of 483 employees across 102 teams in UK universities and colleges.

Prof Annilee Game, from UEA's Norwich Business School, said: “While the link between leader characteristics and employee wellbeing is well known, we uncovered what connects leader perfectionism to presenteeism.

"Our study is among the first to show how leaders' unrealistic expectations can fuel work-family conflict, driving employees to work even when unwell.”

The researchers suggest that demanding leadership behaviour can drain employees' time, energy and emotional resources, leaving them with less capacity for family responsibilities and increasing stress across both domains.

Prof Ana Sanz Vergel also from UEA's Norwich Business School, said: "This study reveals the powerful influence of leadership and the blurred boundary between work and home.

"Leaders' perfectionistic expectations spill over into employees' home lives, making it harder for them to cope with family demand, and the consequences are substantial.

"Conflict between the work and family domains depletes employees' energy to the point that they lack the resources to engage in health-protective coping strategies.

"Instead, they may turn to what appears to be the easier option but is ultimately a harmful one. We need to challenge the notion that presenteeism is the answer."

Conducted over six months, with data collected at three points in time, the study adds to growing evidence that leadership styles can influence employees' wellbeing in ways that extend beyond the workplace.

Dr Rahul Goel, Senior Lecturer in Organisational Behaviour and Human Resource Management at Sheffield Hallam University, who led the study, said: “We were interested in understanding what happens when perfectionism is directed towards others, particularly when it comes from people in positions of leadership.

"Because perfectionism is often seen as positive, our findings highlight that leader-driven perfectionistic expectations can harm wellbeing and life outside work.

“Organisations should therefore reconsider which leadership behaviours and norms they promote, rather than assuming higher standards always improve outcomes.

"I hope this research encourages organisations and leaders to reflect on how expectations are communicated and the potential consequences of creating an environment where leaders expect perfection from employees.”

'The perfect storm: How leader perfectionism fuels employee work-family conflict and presenteeism’ is published in the journal Work & Stress.

Fact-Checked by Irfan Ahmad.

Read next: Essential digital technologies are critical infrastructure – they are not always built and managed that way
by External Contributor via Digital Information World

“It’s not cheating, it’s quiet efficiency”: What is AI watermarking, and how should we respond to it?

By Julian Koplin, Monash University

Image: Nahrizul Kadri - Unsplash

On 2 August 2026, the European Union’s new AI transparency rules came into force. Under Article 50 of the EU AI Act, text generated by AI will need to be marked so that it can be more easily detected.

Anthropic is the first major AI lab to implement this requirement. Claude, its flagship chatbot, has begun embedding invisible ‘watermarks’ in the text it writes. Many other AI companies have agreed to do the same, so Anthropic probably won’t be alone for long.

This move towards transparency is a good thing. But it also carries a risk.

We need a shared social understanding of which uses of AI are a moral problem, and which are not. Until we have this, there is a danger that we will treat AI watermarks as a sign that the human ‘author’ has done something wrong – and dissuade people from using AI for those purposes for which it is genuinely useful.

How does AI watermarking work?

According to Anthropic, the company embeds ‘watermarks’ in AI-generated text by subtly steering the word choices made by its AI models.

Large language models work by generating text one word at a time. Sometimes, there are several candidate words that the model thinks would work equally well. For example, it might be unclear whether a wombat is best described as ‘stocky’ or ‘sturdy’. Previously, this tie would be settled using the digital equivalent of a coin-flip. Now, however, Anthropic uses a pseudo-random algorithm to select one word rather than the other. 

To readers, the outputs appear normal. However, those with access to the algorithm used to guide these choices can check whether a passage of text matches the word choices that Claude would make.

No single word choice is definitive proof of AI use; even if describing the wombat as ‘stocky’ is exactly what Claude would have done, it is entirely possible that a human writer simply happened to choose the same word. However, a long passage of text might contain hundreds of these apparent coin-flips. If all of them come up heads, the most likely explanation is that the coin was loaded.

The reaction

Following the news, many Claude users cancelled their subscriptions and took to social media to complain. Some are worried that if a watermark is detected in their own writing, others will wrongly assume that the AI has done all the work – rather than, say, making a minor contribution to their writing process. As one Reddit comment put it: “I gave the instructions, context, decisions, and countless refinements, Claude was [just] the tool.”

Not everybody is sympathetic to these complaints. As another commenter put it: “The only reason you wouldn’t want [watermarks] is to lie to people.”

Sub cancelled

- you scrubbed the internet and claimed it content as yours
- if I paste something i genuinely wrote and ask to reformat or help fix grammar, it’s now captured as all AI?

Get a grip

— Harshil (@harshilbud) August 15, 2026

This is an understandable reaction. There is something that looks self-serving about wanting to enjoy the benefits of AI while concealing that you have done so – especially when the internet is awash with AI-generated slop. If AI watermarks make it harder to get away with producing low-effort online content, this is a good thing.

Yet the frustrated Claude users get something right. The presence of an AI watermark cannot tell us how the human ‘author’ has used AI, nor whether their use of it ought to be condemned.

What does an AI watermark mean?

As Anthropic’s documentation states, AI watermarks tell us just one thing: whether the content was “processed by Claude”. They do not tell us how much influence Claude had over the ideas in the writing. This is important to understand, since people use AI in many different ways, and they are not all equally deserving of condemnation.

Consider, on the one hand, typical examples of ‘AI slop’: vacuous AI-generated LinkedIn posts listing productivity hacks that are already well-known (“Try the pomodoro technique!”), and university essays produced by ChatGPT (and submitted without changes) after students upload their assignment instructions to the system. These AI outputs are distinctly low effort; their human authors deserve little credit for them.

On the other hand, imagine a scientific paper produced by a researcher who is not a native English speaker, and who has asked Claude to help render their hard-won ideas into fluent English. Or imagine a philosopher who has turned to Claude for copy-editing advice and decided its suggestions would make their argument clearer or more accessible. I write philosophy myself; I can confirm we are bad at this.

In this second set of cases, the human author has done a large share of the work. They also deliberately set out to communicate something that they, the human author, thought mattered.

These are not typical cases of ‘slop’. Arguably, they represent some of the best applications of generative AI. However, because Claude contributed to the phrasing of these texts, they will carry the same watermarks as a list of generic productivity hacks wholly produced by AI. 

What does the absence of an AI watermark mean?

The presence of a watermark cannot tell us whether the ‘author’ has behaved badly. Equally, the absence of a watermark does not mean the ‘author’ produced the work themselves.

As Anthropic acknowledges, editing Claude’s outputs can replace the distinctive word choices on which its AI detection efforts depend.

This means two things. First: a human ‘author’ can have Claude generate a piece of writing for them, then make some simple word substitutions to wash away the watermarks. They might contribute nothing to the text but a few synonyms, yet pass the watermark test with flying colours.

Second: a committed cheat can spare even the slight effort associated with thinking of synonyms by using a second AI to do this for them. AI ‘humanisation’ apps – already a cottage industry – tweak AI outputs so that they will not be flagged by current AI detection tools. No doubt these apps will soon be able to launder watermarked text.

Alternatively, a dedicated AI cheat could turn to a service that does not watermark its text. Such services will presumably continue to be offered by companies operating outside the EU’s jurisdiction, as well as via open-weight models that individuals can run on their own hardware.

What next?

AI watermarking will bring greater transparency around AI use. Whether this transparency is an unmitigated good depends on how thoughtfully we respond to it.

At a time of growing AI backlash, I worry that watermarks will be treated as a clear sign of wrongdoing, even when the human author had good reason to use AI. I also worry that if watermark detection comes to be seen as the authoritative test for AI use, technically savvy AI users will be even better able to escape suspicion than they are now.

Many people want nothing to do with AI. This is a respectable position; some people have legitimate objections to how large language models were trained, and others are understandably concerned that relying on AI might erode their ability to think for themselves. And yet, many people do wish to use AI, and it is important to recognise that not all the things they might use it for are pernicious.

We need to build norms that distinguish these uses from the low-effort ones, and to make sure that watermarks are not seen as incontrovertible evidence of wrongdoing. One starting point might be to look at whose ideas, perspectives, and values are driving the writing. When AI helps people express something that matters to them, it has achieved something worthwhile; when it is used to replace human thinking, it has definitely not.

We also need to avoid outsourcing our judgements about human authorship to AI detection tools. Already, many of us are developing a kind of ‘sixth sense’ for the prose produced by popular AI models; a well-tuned ear will hear a phrase like “It’s not cheating, it’s quiet efficiency” and sound the alarm. Reading for traces of human thought (or their absence) will matter even more in an era when watermarks offer a test for AI use that looks objective but can be gamed.



A watermark can only ever tell us that a tool was involved. Precisely when we should welcome the use of this tool remains a question for us humans to decide.

Fact-Checked by Irfan Ahmad.

Read next: AI agents can now remember and hackers can ‘poison’ their memories — a new cybersecurity threat

by External Contributor via Digital Information World

AI agents can now remember and hackers can ‘poison’ their memories — a new cybersecurity threat

Abbas Yazdinejad, University of Regina

Artificial intelligence systems are starting to do more than answer questions. New AI “agents” can remember information from previous interactions, plan a series of steps and use digital tools to complete tasks.

Memory is part of what makes these systems useful. But my recent research, conducted with my colleague Hadis Karimipour at the University of Calgary, shows that memory can also create a security weakness that is easy to overlook. Think of an AI agent as an assistant that keeps a notebook of what it learns. Each time it completes a task, useful information can be written into the notebook and consulted later.

Now imagine that someone manages to slip a misleading instruction into that notebook. The attacker may not need to take control of the AI directly. The agent can continue working normally for some time. But days — or several interactions — later, it may open its notebook, retrieve the poisoned information and treat it as something it previously learned and can trust.

This is known as memory poisoning and the important part is the delay. A poisoned AI agent may not immediately behave like a compromised system.

An attack that waits

Many familiar cybersecurity attacks produce effects relatively quickly. A malicious link is clicked, malware executes or a stolen password is used to access an account.

Memory poisoning can work differently.

In our research, we examined 2,614 simulated multi-step attack trajectories involving memory-enabled large language model agents. We studied four types of attacks: chain poisoning, policy rewriting, backdoor triggering and slow drift.

Rather than asking only whether an attack succeeded, we examined what happened to the agent over time. That distinction matters.

Imagine someone secretly adding a sentence to an employee’s notebook saying: “Requests from this person have already been approved.” Nothing necessarily happens when the sentence is written. The employee might complete several unrelated tasks normally. The problem emerges later, when a relevant request arrives and the employee consults the notebook.

Like a notebook, an AI agent’s persistent memory can carry information from one interaction to the next. (Pexels/Sergey Torbik)

An AI agent with persistent memory can face a similar problem.

Our experiments showed that some attacks remained difficult to distinguish from normal behaviour through much of an interaction and became apparent only later. In particular, slow-drift and backdoor-trigger attacks could evade evaluations that looked only at individual steps until their effects appeared in later interactions.

Checking once may not be enough

This creates a problem for how we test AI security.

Suppose a security team examines an AI agent immediately after it encounters suspicious information. The agent appears to behave normally, so the interaction is judged safe.

That may be like inspecting a notebook immediately after someone has inserted a misleading entry but before anyone has acted on it. The absence of immediate harmful behaviour does not necessarily mean the attack failed.

Our results also showed that the risk did not always increase in a simple straight line. Some attacks produced what we call “non-monotonic” patterns: behaviour could appear more concerning at one stage and less concerning at another before the attack ultimately developed.

This means testing an AI agent one prompt or one interaction at a time can miss part of the picture. Instead, security evaluations may need to follow the agent across a sequence of interactions — essentially watching the whole story rather than examining individual photographs.

A new security problem

This issue is becoming more important as AI systems develop from chatbots that respond to individual questions to agents designed to perform longer tasks.

An ordinary chatbot conversation can often be treated as relatively self-contained. A memory-enabled agent is different because yesterday’s information may influence tomorrow’s decision.

When an AI agent can also use tools, the consequences of a poisoned memory can extend beyond generating an incorrect sentence. An agent might eventually use remembered information when deciding what action to take, which resource to access or which instruction to follow.

This doesn’t mean that memory-enabled AI agents are inherently unsafe. Memory provides important benefits: it allows an agent to maintain context, learn user preferences and work on tasks that cannot be completed in a single interaction.

But it changes what defenders need to protect.

Securing the prompt in front of an AI system is no longer necessarily enough. The information that the system carries forward may also need protection.

From snapshots to stories

There is a simple lesson from our research: When an AI system has memory, security also has a memory.

If an attack can be planted at one moment and activated much later, evaluating only the moment when malicious information first appears — or only the moment when something goes wrong — can leave out what happened in between.

That is why we argue for trajectory-aware security testing: evaluating how an AI agent’s behaviour develops across multiple interactions rather than judging each step in isolation.

For non-specialists, the idea is perhaps easier to understand without the technical terminology. If you want to know whether an assistant’s notebook has been compromised, you cannot simply watch the assistant write one page.

You also need to pay attention to what the assistant remembers — and what happens when it eventually opens that notebook again.

Abbas Yazdinejad, Assistant Professor, Department of Computer Science, University of Regina

This article is republished from The Conversation under a Creative Commons license. Read the original article.

Edited by Irfan Ahmad.

Read next:

• AI agents are all the rage—but research shows they leak private data

• TikTok's design makes it harder for young people to think critically about the mental health content they see
by External Contributor via Digital Information World

AI agents are all the rage—but research shows they leak private data

By Alicia Roberts, Wake Forest University

Wake Forest professor uncovers significant security issues with large language models.

Image: Unsplash - Brecht Corbeel

Before you prompt AI to answer another question or perform another task, a Wake Forest computer scientist wants you to know it could expose your sensitive data.

Ying Zhang, an assistant professor in Wake Forest University’s Department of Computer Science, studies security in software engineering. Her latest research, “How Your Credentials Are Leaked by LLM Agent Skills(opens in a new tab),” explores how large language model (LLM) agents make data vulnerable to attacks.

• LLM agents are autonomous AI systems that analyze circumstances to plan and execute multi-step actions to achieve a goal.
• A skill is a reusable software extension that gives an LLM agent new capabilities. By installing skills into LLM agents—like Claude Code, Codex or Cursor—developers can teach agents to perform specialized tasks, such as accessing a database or analyzing domain-specific information.
• The credentials in danger of exposure include secret identification meant to allow two systems to talk together safely, without creating access to sensitive data.

The exposure happens through the use of third-party AI agent skills. The tasks the skills perform could be anything from creating a presentation using your notes to scanning financial documents for compliance issues.

And, while credential leakage can happen either unintentionally or maliciously, the end result is the same—unauthorized access to private data, Zhang said.

“When the agent skill is flawed or developed with malicious intent, it will steal your data and maybe pass it back to a remote server to be used in some malicious way.” — Ying Zhang, Computer Science Department

Zhang, a corresponding author on the study, will present her research at the International Conference on Automated Software Engineering, Oct. 12-16 in Munich.

How do data leaks happen with AI agents?

According to Zhang’s research, these leaks happen in two ways:

• Malicious: When creating the skill, the developer includes instructions to steal your credentials and access your private data.
• Unintentional: The developer doesn’t use secure coding when creating the skill, and unknowingly gives attackers access to credentials and data to exploit.

“A lot of skills have credential leakage problems, and there are also malicious skills being developed and distributed,” Zhang said. “Through our work, we are helping detect these skills and remove them from the open-source market.”

How pervasive is this problem?

The research team behind this study used 17,022 randomly selected skills to create 170,226 outputs. They used skills available on SkillsMP, the largest open-source AI agent skill marketplace. It provides access to more than 1.6 million skills.

The researchers found:

• 520 affected skills.
• 1,708 security issues in those skills.
• 10 leakage patterns.

Moreover, 89.6% of leaked credentials were immediately exploitable.

When the researchers alerted SkillsMP to the problem, all malicious skills were removed, and most of the vulnerabilities created by negligent coding were fixed.

Why does it matter?

Zhang said the findings point to two problems in software development and AI:

• Too many developers don’t know how to integrate security into their software. They lack knowledge in the security domain.
• Developers also tend to push out a product before giving security the consideration it needs. Zhang contends security should be part of planning from the start.

“When I train my students, I teach them that security is a critical component in their software design,” she said. “Every feature they develop, they have to keep security in mind.”

The rapid evolution of AI and the rise of AI-assisted software development called vibe coding makes securing people’s data even more important, she said. In this new era of software engineering, amateur developers don’t understand software security. And inexperienced developers can’t rely on AI to address security, either.

What can stop the leaks?

More than anything, Zhang wants to see security intuitively integrated from the initial software design stage—not just after a breach.

AI creates new security issues that developers and researchers must address, she said. Researchers need to devise a standard for data safety that AI developers must meet. Developers need a tool that analyzes AI agent skills for safety issues. And users could use assurances, in the form of a regulation or contract, that the apps they download are safe.

Reviewed by Irfan Ahmad.

Read next: 
by External Contributor via Digital Information World