Index of AI assistants of public authorities
The International Laboratory for Digital Transformation in Public Administration staff members Elizaveta Dyakova and Anna Sanina compiled the Index of AI assistants for Public Administration. The study examined international experience, including the analysis of seven reputable indexes and methodologies used by the United Nations, the OECD and others to assess the development of artificial intelligence and e-government. Having identified common approaches, the researchers created a model of indicators specifically for evaluating government chatbots.
The developed Index of AI assistants for Public Administration includes four key components: regulatory standards, technology, ergonomics and functionality. The authors attributed the compliance of the AI assistant's work with legislative requirements, data security standards, ethical standards and personal data protection to regulatory standards. The technology component includes the technical capabilities of the chatbot: infrastructure reliability, compliance with modern cybersecurity requirements, and the ability to scale under load. Ergonomics refers to user-friendliness from the point of view of the average user, such aspects as the clarity of the interface, ease of communication with the bot, and the availability of the service for different groups of the population. By the term functionality authors cover practical usefulness of a virtual assistant: how fully and effectively it answers questions, performs the stated functions and solves the problems of citizens.
Each key component have four sub-components with a set of specific metrics. To determine the weight of each component in the composite index, the Laboratory staff conducted a survey among 25 experts – specialists in the field of public administration, digital technologies and direct experience working with chatbots. The experts ranked the components by importance, and based on this, the authors of the index determined the weighting coefficients of the components. As a result, "regulatory standards" gained the most weight (34% of the index's weight), "functionality" (24%) and "technology" (25%) made a slightly smaller contribution, and the "ergonomics" component turned out to be the least significant (17%).
The study also included a survey of 134 citizens about the assistants' functionality, convenience of their use and usefulness. At the same time, the authors collected objective data on the capabilities of each chatbot, the availability of certain functions, compliance with established standards, etc. The study covered chatbots on portals of public services, taxes, transport, social support, education and housing and communal services. According to the results, all chatbots showed an "average level" of work, with the exception of the "Robot Max" from the gosuslugi.ru, which scored a "high level". It should be noted that all the analyzed state chatbots are at the proper level of security and confidentiality. All government chatbots tested by researchers are capable of providing good quality work, according to the analytical report.
