New York City government chatbot advises businesses to break laws

New York City's new AI chatbot aimed at helping businesses navigate the city's robust regulatory environment seems to have a few screws loose.

The chatbot, which was launched last October, is a tool sanctioned by the city's government to serve as a point of advice and reference for businesses within the municipality. While the bot has largely been a helpful resource for many, there have been numerous instances when it answers a query with potentially misleading or flat-out illegal answers. 

For example, someone asked the chatbot if a restaurant could serve cheese that had been partially eaten by a rat. The bot answered that, yes, "you can still serve the cheese to customers if it has rat bites," according to The Star. The bot also added that the business would need to "inform customers about the situation." 

Other responses that flaunt New York City regulations include stating that business owners can take cuts of their worker's tips (which is illegal), obscure the pricing of funeral services (which is outlawed by the Federal Trade Commission), and refuse to accept cash payments in a retail store (which has been required in New York City since 2020), among other violations. 

It should be noted that no AI chatbot service is 100% accurate, and even human employees can make mistakes. However, the chatbot's failures (limited as they may be) present an interesting conundrum for business owners in New York City. Since the city officially sanctioned the chatbot, the question of liability, should a business owner follow the bot's prescription, has been raised.

Microsoft, which provided the Azure AI framework for the New York City chatbot, stated that it was tweaking the bot to "ensure the outputs are accurate and grounded on the city's official documentation," according to The Star. 

New York City mayor Eric Adams defended the chatbot in a press conference earlier this week, stating:

Anyone that knows technology knows this is how it’s done. Only those who are fearful sit down and say, "Oh, it is not working the way we want, now we have to run away from it all together." I don’t live that way.

As of press time, the chatbot is still up and running.

Buy an Amazon Echo Dot (5th Gen) from Amazon.