Companies are embracing generative AI, but challenges remain

Less than two years after ChatGPT’s release, enterprises are showing strong interest in leveraging generative AI for their operations and products. Dataiku and recognitionA survey of 200 senior analytics and IT leaders from large companies around the world revealed that most organizations are either spending significant amounts of money to explore generative AI use cases or have already implemented it in production.

However, the path to full adoption and increased productivity is not without hurdles, and these challenges present opportunities for companies offering generative AI services.

Heavy investment in generative AI

Survey results released today by VB Transform highlight the significant funding being put towards generative AI initiatives: Nearly three-quarters of respondents (73%) plan to spend more than $500,000 on generative AI within the next 12 months, and almost half (46%) plan to allocate more than $1 million.

However, only one-third of organizations surveyed have a dedicated budget for generative AI initiatives, with more than half funding generative AI projects from other sources, such as IT, data science, or analytics budgets.

It’s unclear how funding generative AI will impact departments that would otherwise benefit from it, and the return on investment (ROI) of such spending remains unclear, but progress in large language models (LLMs) and other generative models doesn’t appear to be slowing, so there’s optimism that the added value will eventually justify the cost.

“As LLM use cases and applications grow across the enterprise, IT teams need an easy way to monitor both performance and costs to get the most out of their investments and identify problematic usage patterns before they significantly impact revenue,” the survey reads in part.

a Previous survey Dataiku’s research found that companies are considering all kinds of applications, from improving customer experience to improving internal operations like software development and data analysis.

Persistent challenges in implementing generative AI

Despite the enthusiasm for generative AI, integrating it is easier said than done. Most survey respondents reported that they have infrastructure barriers to using LLMs as they would like. In addition, they face other challenges, such as regulatory compliance with local laws like EU AI law and internal policy challenges.

The cost of operating generative models also remains a barrier. Hosted LLM services such as Microsoft Azure ML, Amazon Bedrock, and OpenAI API remain popular choices for exploring and creating generative AI within organizations. These services are easy to use and abstract away the technical difficulties of setting up GPU clusters and inference engines. However, token-based pricing models make it difficult for CIOs to manage the costs of large-scale generative AI projects.

Alternatively, organizations can use self-hosted, open-source LLMs that can meet the needs of enterprise applications and significantly reduce inference costs, but this requires upfront investment and in-house technical talent that many organizations do not have.

Tech stack complexity further impedes the adoption of generative AI, with 60% of respondents saying they use five or more tools or software for each step in the analytics and AI lifecycle, from data ingestion to MLOps and LLMOps.

Data Challenges

The advent of generative AI hasn’t eliminated existing data challenges for machine learning projects. In fact, data quality and usability are the top data infrastructure challenges facing IT leaders, with 45% citing this as their primary concern, followed by data access issues cited by 27% of respondents.

Most organizations have a wealth of data, but their data infrastructure was created before the era of generative AI and did not take machine learning into consideration. Data resides in different silos and is often stored in different formats that are incompatible with each other. It needs to be pre-processed, cleaned, anonymized, and integrated before it can be used for machine learning purposes. Data engineering and data ownership management remain key challenges for most machine learning and AI projects.

“Despite all the tools organizations now have available, people have yet to master the quality (and usability) of their data – is it fit for purpose and meets the user’s needs,” the survey states. “It’s ironic that the biggest challenge of the modern data stack is actually not modern at all.”

Opportunities amid challenges

“The reality is that generative AI will continue to change and evolve as different technologies and providers come and go. How can IT leaders get in while remaining agile and adapting to what’s next,” said Conor Jensen, field CDO at Dataiku. “Attention is turning to whether this challenge, along with spiraling costs and other risks, will outweigh the value creation of generative AI.”

As generative AI continues to move from an exploratory project to a foundational technology for scalable operations, companies that offer generative AI services will be able to support enterprises and developers with better tools and platforms.

As the technology matures, there are many opportunities to simplify the technology and data stack for generative AI projects, reducing integration complexity and allowing developers to focus on solving problems and delivering value.

Businesses can also prepare for the wave of generative AI technologies, even if they haven’t yet explored the technology. By running small pilot projects and experimenting with new technologies, organizations can identify pain points in their data infrastructure and policies and start preparing for the future. At the same time, they can start building in-house skills to ensure they have more options and are better equipped to take advantage of the technology’s full potential and drive innovation in their industries.