The report was issued under Government Auditing Standards. OTI disagreed with the findings and recommendations; the entry distinguishes the agency’s assertions from the auditors’ tests and conclusions.
The chatbot stayed in beta while users and auditors found inconsistent answers.
New York City’s Office of Technology and Innovation launched the MyCity chatbot for business questions in September 2023 and expanded it to government services in March 2025. The Comptroller’s audit found that the chatbot did not consistently provide accurate information and that small changes in wording or capitalization could change the result.
The audit also examined how performance was measured and supported. OTI asserted that the chatbot achieved more than 95% accuracy and nearly zero hallucinations. The auditors found that the chosen denominator produced a higher rate, recalculated August 2025 results at 84.8% to 92.7% when based on questions asked, and noted that responses categorized as having room for improvement were excluded from OTI’s accuracy rate.
More than 2,200 questions were asked in July and August 2025. Of the 70 users who submitted thumbs-up or thumbs-down feedback, 50—71.4%—were dissatisfied, and 19 said the response was not helpful.
OTI’s weekly analysis included 48 questions the chatbot should have answered. It did not answer 23, including questions about childcare and government services.
The audit team received different answers to identical questions. Rewording a sentence, or changing “NYC” to “nyc,” also changed whether useful information was returned.
OTI said it conducted red-teaming and other tests. The auditors said the one-page 2023 summary supplied lacked enough detail to assess whether inaccurate or inconsistent responses were tested or how often testing occurred.
OTI added a beta disclaimer stating that responses might be inaccurate or incomplete and said there were no plans to move the chatbot out of beta. The audit recommended structured testing to identify inaccurate or inconsistent responses and inform future use.
The audit’s findings are attributed to the Comptroller. OTI disagreed with all seven recommendations related to project management and contract oversight and maintained that its chatbot testing and accuracy were stronger than the audit concluded.
Performance claims were not governed as evidence. The reported accuracy rate changed materially depending on the denominator and which response categories were counted. A metric cannot support deployment unless its population, exclusions, method, reviewer and decision threshold are fixed and reviewable.
Known defects were not tied to a visible release gate. User feedback, internal reports and independent testing showed inconsistent or missing answers. The record supplied to the auditors did not demonstrate how those findings could block, constrain or withdraw the public service.
“Beta” was used as a warning, not as a control. A disclaimer may tell users a system can fail; it does not make the system reliable, ensure that affected users understand the risk, or establish who must act when evidence shows the failure is recurring.
Exposure floors are recorded, not averaged down.
The chatbot provided public information about government services and benefits. It did not make final eligibility decisions, but inaccurate or inconsistent answers could still shape whether people understand, pursue or abandon a service. That creates elevated operational and public-interest exposure.
Ownership, testing & control readiness
Atlas establishes who owns the public representation, who accepts residual error and who can stop or constrain the service. Strata defines test coverage, acceptance thresholds, metric integrity, content-change regression, user-feedback escalation and evidence for continued deployment.
Can we demonstrate, with retained test records rather than an accuracy claim, that our public-facing AI gives consistent and supportable answers across realistic user language?
Ask to see the test population, failed cases, denominator, exclusions, release threshold and decision owner. A percentage without those fields is not deployment evidence.