An AI assistant for more than 60,000 employees: HR, integrations and a staged rollout

The project started with HR’s problem of scattered systems

The assistant came from an initiative of the HR department at a large global pharmaceutical company that operates in several dozen countries. HR matters at this company are spread across many systems, for example knowledge bases, payroll systems and benefits systems. Employees did not know which system to turn to, what they would find there and what they could do there.

HR wanted to reduce the complexity of this setup and put it plainly: if IT did not do it, HR would find a way and solve the problem on its own.

The speakers fairly often see the opposite situation at clients. The board or a director says “we need AI”, and IT wonders what it can do. IT usually starts with its own knowledge bases instead of discovery with the business. This is the source of an opinion heard in the market: 90% of AI rollouts fail because they bring no value.

When the Protopia team joined the rollout, the project was already fairly well described, had a sponsor and business owners, and the company ran an AI platform. The client decided it needed someone from outside who had seen agentic rollouts. With these systems, experience means a year or two, and context problems or hallucinations appear sooner or later.

The assistant knows who the user is

In this project, the assistant is a chatbot frontend that knows the user’s context and makes HR processes easier for them. The name “assistant” sets it apart from an agent, understood as an autonomous component that connects and automates processes in the background. Unlike a classic chatbot without context, it knows the user’s country, role and contract type: an employment contract or a B2B contractor agreement. This decides what the person can do in HR matters.

The assistant integrates with systems that have run at the company for years, for example the HR ticketing system. Tickets cover, among other things, maternity or sick leave, a training request, a role change and an employee transfer. Around them sits a huge, scattered knowledge base (KB): some procedures are local, some are global. This raises questions such as whether a manager with employees in three different locations has access to all the KBs or only to some of them.

The scope grows in stages along the roadmap

The assistant takes on one area after another, because the product roadmap does not try to put everything in at once. The client has many systems, so the starting scope had to be narrowed.

Knowledge base integration came first, probably the most urgent need: the assistant was to answer in the user’s context. Example questions: how many days of leave do I have, am I eligible for a given benefit, can I get a company car. The assistant searches these knowledge bases for the user and builds the answer in the language of the question.

The second area is tickets. The assistant first tries to resolve the matter from the KB. When the user still does not know what to do, or already knows that a ticket is needed, the assistant offers to fill in the right ticket with them and shows where it is. There are many ticket types.

The third area is employees’ annual plans. Together with the manager and the assistant, the employee drafts a plan with SMART goals and reviews it. The roadmap beyond that is broad, and for now it is a set of intentions: learning, recruitment, employee development and internal promotions. The target is for the assistant to be a central entry point that ties the HR systems together.

The speakers see that clients expect a superagent that solves every problem from its first day in production. From experience they know that this does not work. In this project the work is iterative, split into tasks and separate agents.

The assistant says plainly what it does not handle

From the first contact, the assistant states what is possible and what is not. The speakers see that clients are often afraid to say what an agent can and cannot do. Here, with each new MVP release, the organization holds a large town hall meeting and tells the first users what the assistant can do.

Some ticket types have complex forms with dynamic drop-down menus and too many dependencies. The team decided that the assistant does not handle them. It is more efficient for the user when the assistant says it cannot help with this matter and gives a link to the form, which the user then fills in on their own. This way the team avoids hallucinations. The Pareto principle held for tickets:

Marek Grabarz: “We can handle 90% of the functionality with 10% of the effort, and the other way around.” (translated) 16:28

The assistant connects systems, and processes stay at the source

The assistant orchestrates processes and does not have to duplicate them. When a process is very complex and locked inside the source system, the team describes it to the assistant in general terms, without step-by-step instructions. The substance of the process stays on the system side.

Existing systems such as SAP or SharePoint do not integrate in an obvious way. You have to work out how to connect to them, where their maintenance teams are and where in the organization the knowledge of the target systems sits.

Delegated access is an example of a low-level problem. If the system is SAP, for example, a user sees their own annual goals and e-learning in it, and a manager also sees their employees’ data and what the employees have declared. The assistant may see only the data that the user has access to, so it must replicate the roles and permissions from the source system (RBAC). This was the first obstacle the team had to solve with the client.

Marek Grabarz: “Most of the work is around identity, around integration, around identifying those APIs.” (translated) 27:12

The rollout reaches ever larger groups of users

The stage called the first pilot run is production with limited reach. At the time of recording, it had more than 1,000 active unique users.

Probably the day before the recording, the team got approval to go to production with the full scope described above. This is still not a full rollout: the next stage covers 10% of employees, which means thousands of people. The target, probably at the start of July, is for the assistant to reach all countries and all employees. At the time of recording, no major obstacles to this plan were visible.

Tickets, returning users and feedback measure success

The first measure is the number of tickets, especially user requests, meaning questions to HR like “how do I do this or that”. The assistant should resolve the matter at the knowledge base stage, so that it does not end as a ticket and HR staff do not answer such questions in chats. The team tracks changes in the number of tickets and in their types. The second measure is return visits: whether users are willing to come back to the assistant. The third is whether the assistant really solves problems. The feedback built into the platform and the assistant shows this.

The feedback has two dimensions: business and technical. The user rates whether the session solved their problem, and whether it did so quickly or only after many questions and steps. From the feedback, the team finds the cause: a problem on the assistant side, a misunderstanding of how the assistant works, or gaps in the knowledge bases. A team fixes the knowledge bases on an ongoing basis, because they sometimes hold old or incorrect versions of content. Errors happen, but a lot of the feedback says: great, just add more features.

When errors come up, the team uses a monitoring platform with tracing. At the pilot stage, the team can pull the full chat content, the call context and the called functions, and the session ID is kept. After the pilot, tracing in production will be narrower, because the assistant handles sensitive HR data, sometimes about pay, sometimes about health.

Cost covers the project and tokens, and some benefits are hard to measure

The cost of an agentic solution is the sum of many items, including the project cost and the tokens that the whole system uses. You can try to measure the benefits through organizational efficiency and user satisfaction, but the value of many such processes cannot be measured in money.

In a large organization, it is not always clear where the right ticket is and which one to file. This is what people call tribal knowledge. Take a financial analyst with an HR matter. Instead of waiting for days, writing emails and pinging someone on Microsoft Teams, they can resolve it quickly with the assistant and return to their own work. Such uninterrupted work time is a benefit that is hard to measure.