Wigan Council has launched a new AI-powered virtual assistant designed to help residents navigate adult social care services 24/7. The council claims the tool is a “safe, reliable source of information” and has been “rigorously tested” with local residents and its digital partner, Agilisys. While AI (Artificial Intelligence) assistants can improve access to public service information, independent research has highlighted the potential for inaccurate or biased outputs, raising questions about how such systems are evaluated and monitored.
How the virtual assistant works
The assistant allows residents to ask questions at any time of day or night, promising instant, accessible responses. Cllr Keith Cunliffe, the cabinet portfolio holder for adult social care, stated the AI will complement the council’s existing front-door call line, which receives more than 13,000 calls each year. Agilisys, the council’s digital partner, said the assistant forms part of its ethical AI programme, and is designed to provide value to residents without using data for machine learning.A resident quoted in the press release described feeling reassured about future care options after using the assistant.
Questions around accuracy and oversight
Independent research from the Ada Lovelace Institute, however, highlights the challenges AI systems can present in social care settings. Their report, Scribe and Prejudice?, examined AI transcription tools used by social workers and found that these systems can generate inaccurate or misleading outputs if not closely monitored. While Wigan’s virtual assistant is a different type of AI, the findings raise relevant questions about how councils verify the accuracy of automated responses and ensure resident safety.
The report notes that many AI tools in social care are deployed without comprehensive independent evaluation, while oversight mechanisms vary widely. This raises a question for Wigan Council: what did its “rigorous testing” involve, and was any external or independent assessment undertaken?
The press release repeatedly emphasises “rigorous testing” and “reliable information”, yet no independent evaluation or verification is referenced. Email correspondence with Professor Karen Yeung’s PA (Prof Yeung is an AI ethics expert) confirms that independent assessments are often necessary to validate AI tools used in social care.
Professor Yeung’s team directed attention to the Ada Lovelace Institute report, highlighting that even systems used in professional social work settings can fail to meet reliability standards without careful oversight.
Ethical considerations for public service AI
Cllr Nazia Rehman, cabinet portfolio holder for finance, resources, and transformation, highlighted that Wigan Council has a robust ethical framework to use AI responsibly. Agilisys’ head of AI Incubator, Peter Rogers, added that the council had been “an exceptional customer” in developing ethical AI products.
However, the Ada Lovelace Institute stresses that AI systems in social care need to be assessed for accuracy, transparency, accountability and inclusivity as well as ethical compliance. Its research found examples of transcription AI misrepresenting information in critical social work records, illustrating the potential consequences of errors in systems used in care settings.
The recommendations and caveats of the report are:
- It is unclear how much oversight is needed to review the outputs of AI transcription tools (the ‘human in the loop’) and to evaluate how well they work. This needs further empirical exploration.
- AI transcription tools can significantly reduce the time spent on administrative tasks. Any efficiency gains should not compromise high standards of quality and fairness in critical decision-making.
- Current uses of AI transcription tools risk harmful hallucinations and misrepresentations of people’s experiences to be entered into statutory care records.
- The use of AI transcription tools challenges existing accountability structures in social care. Social workers are currently responsible for the additional work of ensuring the accuracy and quality of outputs.
- Safeguards on the use of AI transcription tools in social care are inconsistent across and within local authorities, with no consensus on the contexts where these tools pose unacceptable risks and should not be used.
- There is very little evidence on how AI transcription tools impact the experiences and outcomes of people who draw on care. Our evidence suggests that these tools can lead to harm in some instances.
According to the council’s press release, all data supplied to the virtual assistant is held securely and is not used for decision-making or machine learning. While this reduces some risks, it does not eliminate the possibility of inaccurate or inconsistent advice if the underlying information is incomplete or outdated.
Potential benefits and limitations
Despite the concerns, virtual assistants can provide valuable support for residents seeking quick guidance outside office hours. The council notes that the AI complements human support rather than replacing it, preserving access to the front-door call line for those needing personal assistance.
This is important because AI outputs should not substitute for professional judgement. Errors or omissions could have serious consequences for people relying on social care services. The press release also does not address how the service will support residents who are digitally excluded.
Greater transparency would allow residents to assess the council’s claims for themselves. Publishing information about testing methods, accuracy, monitoring and any independent assessment would provide a clearer picture of how the system is being governed. Without that information, the council’s claim of “rigorous testing” is difficult for the public to independently assess.
What residents should know
Residents can access the virtual assistant via Wigan Council’s website. While it may offer convenience and immediate guidance, residents should remain aware that:
- AI responses should not be treated as a substitute for professional advice
- The council has not stated whether the system has undergone independent external auditing or verification
- Residents should continue to use the council’s call line for complex or urgent enquiries.
The council’s approach reflects a broader trend of adopting AI in public services, balancing innovation with potential risk. How successfully Wigan Council addresses accuracy, oversight, and ethical considerations will determine whether the tool truly adds value.
An ambitious attempt
Wigan Council’s virtual assistant could make it easier for residents to access information outside traditional office hours. But its usefulness will ultimately depend on the accuracy of the information it provides and the safeguards surrounding it. Greater transparency about testing, monitoring and independent assessment would allow residents to judge for themselves how reliable the system is.
Being sole carer for a dementia patient
A sombre insight into the exhausting and lonely daily reality of looking after a parent suffering from dementia
The price unpaid carers pay
Four out of five unpaid carers in the North West are taking on the bulk of caring responsibilities for people...
Bylines Network Gazette is out!
The latest Bylines Network Gazette is out now. Become a Friend of Bylines Network or upgrade to a paid Substack subscription, and you will get access to a vital collection of citizen articles and stories.
Journalism by the people, for the people.
