“You had to set it up,” he says. “And the computing power required to run a 70 billion model is significant. We set it up ourselves, deployed a server, deployed the model and then there was the usage beyond that.”
Azure now offers a pay-as-you-go option where customers only pay the token cost, but there are still setup costs for organizations looking to deploy on-premises models.
“In an ideal world, that would be the best scenario because you’re no longer constrained by token costs,” he says. “The only costs you pay are for the infrastructure. But you still need the computing power and other things, like networking.”
Supervisory costs
As new-generation AI moves into production, the monitoring required can be another unexpected cost. Many systems require human intervention or expensive engineering safeguards to verify accuracy, reduce risk, or ensure compliance.
“I don’t think we expected regulation to come so quickly,” says Sreekanth Menon, global head of AI at Genpact. “When generative AI was introduced, it became the top issue at the government level, and all governments woke up and said we need regulation.”
The EU law is already in force and work is underway in the US. “Now companies have to take this into account when developing AI, and that comes at a cost,” he says. But the regulations are not a bad thing, he adds. “We need regulations so that AI decisions are good and fair,” he says.
Regulatory compliance after the systems are built is also costly, but companies can plan ahead by implementing good AI governance systems. Ensuring the security of AI models and associated systems is also a cost that companies may not be prepared for. Running a small-scale production test not only helps companies identify compliance and security issues, he says, but also helps them better estimate other incidental costs, such as those for additional infrastructure, search, databases, API and more. “Think big, test small and scale fast,” he says.
AI proliferation
In the past, traditional AI might have taken a year or two of experimentation to get an AI model ready for use, but general AI projects are evolving rapidly.
“The basic models available today allow companies to think about use cases quickly,” says Menon. “Now we are at a stage where we can think about an experiment and then quickly move into production.” He suggests that companies hold off on doing all AI projects at once, establish a cost mechanism and clear goals for each project, then start small, scale carefully and invest in continuous training.
“Upgrading your qualifications comes at a cost, but it helps you save on other costs,” he says.
Matthew Mettenheimer, associate director at S-RM Intelligence and Risk Consulting, says he often sees the use of artificial intelligence spreading across organizations.
“A CIO or a board wants to implement AI across the entire organization, and before they know it, there are a lot of spend and use cases,” he says.
For example, S-RM recently worked with a large consumer goods manufacturer that decided to push AI enablement across its organization without first establishing a governance structure. “And every single department immediately went to work trying to implement generative AI,” he says. “There were overlapping contracts with different tools for different parts of the organization, which really drove up the spend. The marketing department was using one tool, the IT team was using another. Even within the same department, different teams were using different tools.”
As a result, the company kept paying for similar services, with each group having its own contracts, and efficiency not increasing by working at scale. And people were getting subscriptions to new generation AI products that they didn’t know how to use.
“There were a lot of good intentions and half-baked ideas,” he says. As a result, IT spending has increased massively, he says. Companies must first understand where new-generation AI can really make a difference. Then, companies should build their projects step by step and in a sustainable way, rather than buying as much as possible. Some areas where companies should hold back on spending are use cases that could be culpable for the organization.
“As an insurance provider, if you use AI to determine whether or not a claim will be paid, it can introduce some liability if the AI mechanism is not used or calibrated properly,” says Mettenheimer. Instead, prioritize use cases that free up employees for more complex tasks.
“If someone spends five hours a week updating the same spreadsheet, and you can reduce that time to zero hours a week, that person really has more time to be more productive,” he adds. But if checking the AI’s work product takes as much time as it saves, that doesn’t really make the work more efficient.
“Generative AI is a really powerful and incredible tool, but it’s not magic,” he says. “There’s a misconception that AI can do everything without manual processes or validation, but we’re not there yet.”
He also advises against carrying out AI projects for which perfectly good solutions already exist.
“I know of some cases where people want to use AI to feel like they have a competitive advantage and be able to say they’re using AI for their product,” he says. “So they put AI on top, but they don’t get any benefit other than just being able to say they’re using AI.”
“Executives can’t wait to get started with next-generation AI,” said Megan Amdahl, SVP of partner alliances and operations at Insight.
“But without a clear goal in mind, they may spend a lot of time on cycles that don’t produce the results they hoped for,” she says. For example, clients often focus on small use cases that improve efficiency for a small number of people. That can sound like a great project, but if there’s no way to scale it up, you can easily end up with a sea of point solutions, none of which delivers real business impact.
“Here at Insight, we were selective about which team we wanted to target to improve help desk feedback,” she says. One strong use case had a team of 50 checking the status of customer orders. But not only was the team small, the people were located in low-cost-of-living locations. Improving their efficiency with generic AI would have had some impact, but not a significant one. Another team created bills of materials for customers and was much larger. “We decided to go with a team size of 850 instead so it would have a broader impact,” she says.
In addition to selecting projects with the greatest impact, she recommends looking for projects that have a narrower scope in terms of data requirements. Take, for example, a help desk assistant with generic AI.
“Don’t investigate every question the company can ask,” she advises. “Limit the questions and observe the answers you get. This will also reduce the amount of data you need to retrieve.”
Organizing data is a big and costly challenge for companies using AI. The data should be clean and in a structured format to avoid inaccuracies. She recommends that companies deciding which generation of AI to implement first consider projects that focus on generating revenue, reducing costs and improving brand affinity.
