Three Challenges with On-Premises AI

Getting artificial intelligence off the ground for business is on the rise. Forbes Advisor estimates that 72% of businesses are using AI for at least one business function in 2024.

For many businesses, using the hybrid cloud is a secure-enough, cost-effective solution. For others, AI on premises offers an appealing combination of control and high-level security, but is it worth it?

The answer: Maybe. It really depends on what your team is willing to do, and whether that’s worth it.

With several decades of collective experience helping customers make the best choices for the best ROI, the AI Advisory team has recently helped customers make AI a reality. As such, the ERP Suites team tends to recommend against doing AI on premises unless several specific conditions are met.

By the end of the article, you’ll know whether or not doing AI on premises is a good fit for your business.  

Gut Check: Do You Need An AI Solution of Any Kind? 

AI, GenAI, LLMs – emerging technology is sexy, for sure, but the process to get it off the ground is involved and complex. Your team should establish a baseline before making any AI decisions:

•         Are you in a market or field that can benefit from adopting an AI solution?
•         What are you looking to gain that AI can help with?
•         Do you have the necessary resources to start an AI journey?

The ERP Suites AI Advisory team has a proven GenAI methodology that can help you answer these questions. Whether you do or don’t work with a third-party, undertaking a GenAI implementation is a multiphase process that deserves close consideration.

You’ll also need to determine whether an on-premises solution is viable for your business.  

The AI ROI Might Not Be Worth the Expense

Implementing AI on premises isn’t cheap. Developing your solution requires a hefty upfront investment in hardware, infrastructure and setup. These are the main costs you’ll encounter:

• Hardware
• Software
• Networking equipment
• Cooling systems
• Ongoing maintenance

Specialized servers, GPUs (graphics processing units) and CPUs (central processing unit) are all necessary components for AI. Companies that produce GPUs, like NVIDIA, can charge a premium for their products. A single NVIDIA A100 GPU costs between $10,000 - $15,000.

AI requires an enormous amount of energy to operate. You’ll need to account for that energy generation, plus cooling costs so your data center doesn’t melt down. This can often add thousands of dollars per month to overall expenses.

After the initial setup, you’ll need ongoing maintenance and support. Keeping hardware and software up to date, managing security patches and updates, and troubleshooting issues – these all require dedicated IT staff and specialized expertise. If you don’t have enough human capital to take that on, you could be compromising your whole AI endeavor. You’ll also want to consider how AI fits in with your JD Edwards ERP system, or your existing CRM system.  

The companies that can afford the various expenses of an on-premises AI solution tend to be private, with large budgets. For most small to mid-size businesses, these resources are not available or realistic.

Bottom line: The escalating costs of an on-premises AI solution often aren’t worth the potential ROI. This type of solution is better suited for a niche population with large amounts of capital to invest.

AI Isn't Your Company's Area of Expertise

Some organizations can juggle specialties within their field without a dedicated team. This doesn’t work for AI – at least not yet. AI requires highly skilled staff that most companies just don’t have in place. There could be a future where every company has an AI team in place, but we’re not there yet.

Undertaking AI necessitates building and training AI models, a highly complex endeavor. If you’re doing this on premises, you’re going at this alone, whereas folks using a hybrid cloud solution can take advantage of existing AI models.

Then there’s the people portion. The demand for skilled AI professionals, so far, outpaces the supply – though AI is growing as a study concentration in universities. If and when you do assemble an AI team, they’ll need to have a diverse set of skills that go beyond machine learning expertise.

They’ll need to be proficient in:

• Data engineering
• Software development
• System architecture
• DevOps
• Many other skills.

By contrast, cloud-based AI services often come with a deep pool of talent and expertise. Your team remains free to focus on core business competencies. Most organizations tend to take advantage of ready-made AI teams.

Bottom line: AI requires highly specialized experts to make the costs worthwhile. If you can’t afford the human capital, you might not be able to afford an on-premises AI solution.

Your Resources Can't Keep Pace with AI Development

Scaling up is a major consideration when investing in any kind of solution or product. Besides an immediate payoff, you want to know you can count on returns far into the future. With a cloud-based AI solution, this is as simple as the click of a button. Not so with on-premises AI.

With the rapid pace of AI development, hardware that worked six months ago might be obsolete tomorrow. Take the NVIDIA A100 mentioned above. Released in 2020, it was quickly followed by the NVIDIA H100 in 2022. The H100 is significantly faster, with enhanced efficiency and performance for AI tasks. But wait – what about TPUs (Tensor Processing Units) and FPGAs (Field-Programmable Gate Arrays)?

These new devices could evolve even faster than traditional GPUs. And what about the underlying AI technology? Is your system equipped to handle increasing workloads? Do you have all the necessary resources in place to not only evaluate that consideration, but make any required changes? Do you have access to increased capital, to invest in new requirements?

Bottom line: You need to ask yourself if an on-premises AI solution will still be competitive in the next five years, not just the next four quarters.

Any AI Solution Requires Careful Consideration

AI solutions are becoming increasingly expected in the workplace. By and large, most teams turn to a cloud-based solution. But if you’re still curious about on-prem, it’s important to know about potential pitfalls.

Learn more about how on-premises AI and cloud-based AI compare.