OCI Generative AI Agent

Table of Contents   

1. Welcome & Speaker Intro

2. OCI Generative AI Agent Service — Scope & Principles
3. Why Agents Are More Than Just LLMs
4. Agent Infrastructure & Structure
5. Use Cases for Agent Service
6. Benefits of the AI Agent Service
   
7. OCI Generative Agent Service and Platform
8. Key Components of the Agent Platform
9. RAG as a Tool in the Agent Platform
10. SQL or NL2SQL Tool in the Agent Platform
11. Why Choose OCI Generative Agents
12. Demo Overview
13. Demo Walkthrough
14. Closing Remarks and Q&A

Transcript

Welcome & Speaker Intro

OCI Generative AI Agent Service — Scope & Principles

Thank you everyone for your time for today's presentation. We're going to talk about the OCI generative AI agent service and how it enables our customers and the capabilities we deliver with the platform that we just released a few weeks ago. Before we continue, as usual, Oracle safe harbor statement.

The unique approach of Oracle is that, while there are already many agentic products existing in the world and on the internet, Oracle is focused on providing something that binds and integrates tightly with our products. If you're an OCI customer, you should be able to create your agentic applications and use cases, implement them, and run them directly on OCI—without needing to develop or deploy third-party systems. At the same time, the system must integrate seamlessly with our apps, databases, and enterprise applications.

Oracle also emphasizes flexible deployment options and fully managed capabilities. In most cases, you don’t need to build or maintain anything yourself. For developer-centric approaches, we provide an agent development kit library that works with the fully managed service, minimizing deployment overhead and ensuring easy scalability, monitoring, and security once in production. Security and privacy are foundational: the agent service runs with models on OCI, ensuring that no data leaves your region or tenancy.

Additionally, we’re committed to offering a broad set of models for customers to select in the future. The overarching goal of the service is to enable enterprises to deliver value quickly by streamlining workflows, implementing agentic use cases with minimal overhead, and providing a virtual assistant that can help employees or support business processes. Building these systems can be tricky since LLMs can be unpredictable, require grounding, fine-tuning, and prompt tuning to adapt effectively to specialized use cases. They also demand infrastructure capable of scaling with GPU resources while remaining cost-effective. Our service is designed to directly address all of these challenges.

Why Agents Are More Than Just LLMs

Now to understand the AI agents and why is it actually more than just LLM? Just to give you a few examples when you go to a agent service and you just say what is my cash flow last week, if you just use LLM it is not that this large language model does not have the ability to know that. So the agent services elevate the large language models by providing access to additional systems to be able to do that. For the first question, for example, you're going to need access to your data and you're going to need that access securely. Or if I just ask turn off light bulb at 9:00 p.m. for example, it has to create a task that needs to be scheduled to execute that task at a specific time and then it needs to invoke also some API to perform that task. The same also with the rest of the task where you just say analyze some proposal against my policy. So for example here it needs to extract data for some existing policy document information and then together with the input that you provided it has to analyze this information to provide your response. Or it should be also capable to follow instructions and be able to execute for example sequentially or parallel tasks. So all this is something that other agentic platforms have provided and also the agentic platform that we are providing at OCI has those capabilities.

Agent Infrastructure & Structure

At the high level what the agent is, it’s the agent infrastructure and the structure of the agent. At the core is a large language model that we fine-tune and prompt and optimize and specialize for react-capable tasks. So it has reasoning capabilities, it can act on tasks, it can have different personas, it can plan actions, you can also through your input tell them in what sequence to execute actions if it's necessary. We have a user interface for communication, we have an easy way to get started to chat with the agent in the OCI console, but at the same time we provide also APIs and ADK library—agent development kit—for easy development. The agent platform has a long and short memory. We have the ability to work with tools, you can modify prompts, and of course we have access to knowledge bases that you can define that could be various abilities. For example, just working with your existing PDF documents where we can completely ingest them and store them without you needing to do anything. So as a fully managed service, or the knowledge bases could also be something that can access third-party systems or databases for example. All this you see here we are providing as a fully managed service at OCI. So when you build your application with our platform you do not need to think about how to host that, how to run that, how to scale it, how to utilize it, how to secure it. Everything is part of the OCI agent service.

What can you do with the agent that we're providing? Some of the low-hanging fruits or some of the high-level tasks that you can implement is of course the agent can understand and interpret your queries. It can use additional knowledge that you provide to the agent also for that. It can determine necessary actions and also execute those actions. It can retrieve data if necessary. It can, as already said, execute those actions also not only against OCI but of course also against third-party systems.

Use Cases for Agent Service

Some of the use cases that we are working on and we are enabling with the agent service, you can see on this slide. The spectrum of use cases that you can implement in the various business sectors and operations is really vast. Of course large language models still have some challenges but as you have seen over the past few years they exponentially get better and better and the reach of use cases that we can implement with these models increases basically every quarter. Of course when implementing those use cases you have to be very cautious about the limitations of the large language models which also exist in the agent service like hallucination for example or maybe not the ability to follow some tasks. And this is something where our service tries to achieve a high quality and provide you the assurance, and we test for that with our science teams. Every time we have an incremental optimization in an agent service we make sure that our baseline of quality is always at the level necessary for implementing enterprise applications.

Benefits of the AI Agent Service

Potential benefits of the AI agent service and what you can use it for in your enterprise are significant. Very common is a boost in productivity—that’s probably one of the tasks that we see, for example, also at Oracle with a lot of benefits. With the agent service, it is finally a place where business users can access their database and talk to their database directly. This promise has existed for a very long time and still business users need database experts to write their queries. Now with the agent service, for most of the low-hanging and operational execution information that you want to access from your database, business users can do this through the agent service with the natural language to SQL capabilities. Agents can also help write code, execute operations in sequence, and help with preparing demos, presentations, brainstorming ideas, and accessing third-party documents or APIs. This makes life much easier and really boosts productivity.

Agents can also help reduce human error and cost. With human error you still have to be careful, so we implement our agents always in a way that there is a human in the loop when necessary and always with the ability to track what the agent executed and what the agent has done so that you can go back and validate these actions. But due to automation of repeating tasks, the workforce can now concentrate on other types of jobs and not necessarily have to do these repetitive tasks. Finally, agents can improve contextual relevance—especially with the ability to provide additional documents and call APIs. Now when you do research or want to grab specific information from your company, you have the ability to wire the agents with those documents and APIs, which can help you in your decision process.

OCI Generative Agent Service and Platform

And now we go to the OCI generative agent service and the agent platform that we released end of March and which we of course continue to improve. Some of the features that you will see and that I’m going to demonstrate in the presentation are things that we continue to elevate, but some of the key goals for the agent platform are to have simplified development of the agent service and faster time to market. We are releasing an agent development kit library for people that are more developer-concentrated or oriented in the way they develop agents. This library works with our agent platform in the cloud. When you finish defining your agents and your tools in your workflows, they are also automatically created in the agent service in the cloud so that you can directly execute them without having to think about additional deployment.

We are opening our agent platform for customizations to give you flexibility in implementing your business cases and business use cases. You can now provide more customizations to the agent and of course the ability to define tools which could be a function executing tools or, in the agent, we’re releasing a feature for the agent service to call API endpoints directly through the open API schema. So if you have REST endpoints that are running something from your application, the agent can access them directly automatically by configuring the necessary endpoints.

Key Components of the Agent Platform

Some of the key components of the agent platform already mentioned a few times is of course the core—the brain of the agent platform—which is the large language model. Then there are the tools. We have two types of tools: pre-built tools like RAG and SQL that we had before. Now they are part of the agent platform. You no longer need to create only a RAG agent, only an SQL agent, or another separate agent and bind them together. Now you can have one agent and select all of these capabilities, and this agent can work with all of them at the same time and orchestrate tasks automatically between those tools. So one agent can now have RAG capability and SQL natural language-to-SQL capabilities at once, and in addition, within the same agent, you can configure functions and API calling tools to directly call API endpoints.

Of course, as I mentioned already, this is a fully managed cloud-native agent platform. We orchestrate all of the tools automatically based on the user input question or inquiry. The platform supports multi-turn chat experience with context retention for up to seven days, custom instruction capabilities, and human-in-the-loop options that can be enabled or disabled depending on the use case. And of course, as always, there is tight control on security and governance. For example, when you configure your API endpoint, we provide a wide range of security capabilities for how to configure the agent to call it. It could be via an API key, resource principle policy authentication in OCI, OAuth authentication, or propagating user authentication and delegation from the client if necessary. Full-spectrum security in OCI is integrated with the agent platform.

RAG as a Tool in the Agent Platform

On the RAG side, the capabilities that we had before we kept, but now they exist as a RAG tool within the agent platform. Of course, we continue to work on enhancing accuracy when we pull information from different relevant sources and to reduce hallucination. We have better context handling in the new RAG service, and we continue to open up for more and more customizations. You can now specify custom instructions for the RAG, tell it how to answer specific questions, or in what format to answer them.

We also continue with our knowledge bases and the ability to fully automate ingestion. We have a fully managed pipeline to ingest documents so that you don’t have to think about how the ingestion pipeline should work, how to chunk the documents, how to vectorize them, or where to store them in a vector database. We provide the option to completely automate this for you.

Some of the key new features in the RAG tool in the agent platform are a fully managed and scalable advanced ingestion pipeline with a vector store, the ability to specify custom instructions, and support for one agent to work with multiple knowledge bases rather than only one. You can also provide security policies that identify which user groups have access to which knowledge base. We are enhancing retrieval quality with out-of-the-box hybrid search, which combines lexical and semantic search, as well as paragraph-level citation.

We now also offer multilingual support, multimodal parsing for graphics and charts within PDF documents, metadata ingestion with filtering by metadata, and cross-region access for Oracle databases. This is very important because in many cases, agents and generative services are available in regions where GPUs exist, but your established data region might be elsewhere. With cross-region support, you are still able to access your information from the agent service.

SQL or NL2SQL Tool in the Agent Platform

Another tool, the SQL or NL2SQL, which previously existed as a separate agent, is now available as a tool within the agent platform. Again, you can now use this together with RAG and other tools at the same time. It really aims to make it easy for people to access their databases and extract information by simply asking questions in natural language. The specialized fine-tuned model, developed in-house at Oracle, converts natural language to SQL, queries the specified databases, and returns the information.

We provide a wide range of abilities to improve the quality of the SQL generated. You can provide schemas, additional information about tables, and we have self-correction to help reduce errors. SQL works very similarly—it utilizes the same capability from the agent platform but adds additional tools for SQL generation, explanation, self-correction, execution, schema linking, in-context learning, and custom instructions. You can also provide extra details about the tables and columns to improve results.

In addition, you can select between two models—a small one and a larger one. The small model is very fast at generating queries and is more suitable for databases without very complex structures. The larger model is more capable and is valuable in situations where there is a very complex database schema. Finally, you can also select between two SQL dialects: SQLite and Oracle SQL.

Why Choose OCI Generative Agents

Why choose OCI generative agents? I’m going to show you also a demo video about three and a half minutes long to see some of the capabilities and what you can do with the agent service in a very compact format, which I hope will help you understand the capabilities of the service and what you can do. But again, just to quickly repeat, of course, there is seamless integration and a very easy way to run scalable performance powered by OCI infrastructure, offering high reliability, elasticity, and scalability. We are continuously working on improving accuracy with every release. We make sure that the quality of the responses gets higher and higher.

Of course, this is a fully integrated OCI native service, meaning we have full integration with Oracle databases and the broader Oracle ecosystem within Oracle Cloud.

Demo Overview

Let’s move on to the demo quickly. One second. So I’m actually very fast and ahead of schedule. So let’s start the demo. I wanted to create a customer support agent. This demo is going to show you, in a very compact form, what is a typical use case you can implement with the agent service and how we implemented it and put everything together. It will make human customer support agents more efficient.

To start, I created a simple autonomous database instance with the following schema. As you can see, I have a customer table, a support agent table, a tickets table, and a couple of helper tables. I’ve used the generative AI service to generate data for this demo. Next, I created my agent using the OCI agent service. As you can see, my agent has three pre-built tools to work with. The SQL tool is connected to our autonomous database instance where we store our support ticket data. I’ve shared my database schema with the tool and allowed the tool to execute SQL queries on my behalf. I’ve also shared more detailed information about the various table columns for better results.

Next, we have the RAG tool which was trained on a set of knowledge-based articles I auto-generated using the generative AI service. These articles were uploaded as PDF files to a storage bucket. The last tool is a custom tool which I configured to execute an internal API to send emails. The custom tool can be configured to make function or API calls to automate workflows and retrieve information from additional data sources.

Now that my database and agent are ready, I used Oracle Code Assist to create my application. I created a fairly detailed description of an application with a backend and frontend. Code Assist generated the outline of the application as well as all the necessary code and configuration files. I followed the instructions to create the structure of the application and ran the various commands to install the required packages. I adapted the initial code to my needs and here’s the resulting application.

Demo Walkthrough

First, I’ll pretend to be a call center manager and ask a question like, “How many tickets do we have in the system?” As you can see, the agent figured out that for this task, it needs to use the SQL tool, which generated a SQL query to retrieve the data—all without the user having to know anything about databases or SQL.

Next, I’m going to ask who is the busiest support agent right now. Thanks to the magic of LLMs and natural language to SQL, the agent was able to understand that “busiest” in this context means the agent with the most tickets assigned to them, generate the appropriate SQL command, and extract the correct information.

Digging deeper, I can ask which tickets are assigned to Gabriel White. Now, is Gabriel White the support agent? I’ll ask to see the information for ticket number eight. I see that the ticket is talking about network speed issues. In order to resolve this ticket, I’ll see if we have any knowledge-based articles which explain how to troubleshoot such issues.

For this query, our agent will use the RAG tool which was trained on knowledge-based articles stored in object storage. As you can see, the agent came back with the relevant information as well as a direct citation and link to the original file. Now that we have a solution for the customer’s issue, I’ll ask the agent to send an email to the customer with the suggested troubleshooting steps.

Using Oracle Autonomous Database, the OCI agent service drawing on multiple internal data sources, the generative AI service, and Oracle Code Assist, I was able to streamline the workflow of customer service agents easily and effectively.

Closing Remarks and Q&A

And that’s it. I was pretty fast with my presentation today, but I hope this was insightful and thank you very much for your time.

[Music]

Let’s see if we have—go ahead, Manuel.

Justine: Just checking to see if there are any questions. I don’t know if the audience would like to ask Lud Mill any questions. Now is the time to post your questions on the chat or the Q&A tab. So maybe we’ll give you guys a couple minutes, few minutes for folks to chime in with any questions.

That demo video is really well done. That was nice. I like the time lapse fast forward.

Yeah, you didn’t—very compressed, but I hope it kind of uh it’s capable to show the power of the agent service and what you can do with it.

Yeah, maybe while folks ask questions, any particular area, you know, or maybe any particular services Lud Mill that you’re seeing are getting adopted at a higher rate that you can share, a little bit more popular, more being more consumed at this point in time?

Uh, what services do you mean, man?

So you know in your—you talked in the video there was RAG, there’s different services. Are there ones in particular—Code Assist is a very intriguing one, even in our space and JD Edwards—but just curious to see if you’re seeing adoption in any particular ones at a higher rate.

So basically what you see in the demo, this has the highest rate of implementation right now because it’s very practical. So you have a RAG, you have a database, and you have custom tools execution at the same time because most of the customers already have documents and they want to be able to extract information or chat with the documents. And you know in the past if you remember usually you have internally in the company some system where you ask a question in search for information, it gives you 20 results, and now you have to go through all of the results to identify where the relevant information is.

And especially challenging when the relevant information is spread to several documents. Now with the RAG, what they do is drop the documents at the object storage. We ingest them and they’re ready to go. They don’t have to do anything. Then of course we are Oracle. At work, we have a lot of customers with a database already and so many of the questions against the database can be already done through the agent, as you can see. So we just generate the queries automatically and show that. We even go one step further for some implementations where people—customers—would like to show also graphics within the chat. So that’s also no problem to do.

And then of course every enterprise customer already has APIs somewhere, right? Established APIs. And so all three components—being able to automatically call APIs and orchestrate them, and then also have a RAG and SQL in the same system and put them all together—already enables a wide range of use cases, as I’ve shown in one of the slides. That could be anything basically from that point. I am for example a very big fan of the API endpoint calling feature. This is one of the kind of things that gets a lot of attraction at the moment because every API has a swagger, or now it’s called an open API schema, and in the agent you just need to upload that schema and that’s everything you need to do.

And of course you have to configure the authentication, how the agent should authenticate against the API. And now I no longer need to do anything to work with my application. I don’t need to write a single line of code. I just need to talk. I can give an example with OCI—everything you do at OCI has an API. Let’s say that—how I create the bucket, to see what is in the bucket, to see the documents, to upload the document to a bucket, etc. So now I can just show the agent the API and now I can just converse with the agent to do these operations for me and I don’t need to write any Java or Python code. It’s just calling the APIs automatically and it’s executing these operations for me. So this is a feature that people get excited a lot about as well. 

Absolutely. No, that’s certainly exciting. Certainly exciting internally for us as well as conversations we’re having with customers, Lud Mill. So that resonates with what you’re saying and in fact we have one of your colleagues talking this afternoon about the Code Assist functionality. He’s going to take it a level.

Great. Code Assist is really great. Currently, it’s one of the most used capabilities within Oracle. It is a fine-tuned model by Oracle, by the way—it’s not a publicly available model. But, you know, Oracle is—what people don’t realize—Oracle watches a lot around security and a lot about, for example, the code that our Code Assist generates. Is that proprietary code or not? What do we do in that case? So we really want to go through all legal kind of hiccups to make sure that we get all approval to make sure that we generate code that it’s not going to be against, you know, other companies and affect proprietary code, etc.

And then what we actually at some point want to also introduce is this into the directly into the agent service so that the agent service can write also code and be able to execute that code also directly as well.

Fantastic. Fantastic. Well, I think the only other question, you know, comment I’d like to make—and if you have any additional color you could share—right? You talked about, you know, rapid rate of iterations and improvements to the services and a variety of different factors, the LLM as well. And I know just, was it two or three weeks ago, you guys came out with the new release. You guys are constantly right on that effort of enhancing the services, the LLMs, and making it more—whether it’s more feature-rich or like you’re saying, more accurate, more precise. Is that still the case?

That’s still the case, yes. And there are various improvements that we’re going to do in the next six months. For example, for the RAG, we’re going to increase the number of files that we support and we can ingest. We should be able to ingest code, SDK documentation. And then for the SQL, for example, we’re improving and adding the ability to work what we call enterprise schema. Sometimes the schemas can be very large and so it’s just very hard for SQL—for large language models—to deal with that. And there are various techniques of what to do in that case, and we are going to provide abilities there that we can work with 300, 400, 500 megabyte large schemas and be able to optimize it in a way that delivers—that the agent automatically delivers the relevant schemas and then identifies the queries for that.

For example, agent platform of course is going to continue to improve. We’re also going to open up all of the tools in the platform for more and more of the developer persona, meaning that people will be able to configure even more. For example, we have a prompt template already, but maybe we’re going to open up also the system prompts. We’re going to open up the models to select multiple models, the parameters of the models. We’re probably going to have one option that is going to be like the default option which we optimize for everything—you can use it out of the box. But if you want to go for another model, and you want to try another model, and you want to test another model, let’s say for RAG output generation, you want to try—I don’t know—LLaMA 4 or something else, we’re going to have also other models that you can select and you can try to see if the RAG agent works with that model better.

We will not be able to test for all those models—I think, I hope people realize here that it’s really very challenging to assure quality on all models. So there is going to be—we will continue to have one model on which we specialize. Currently, for example, the LLaMA models, we’re very good working with them and optimize them and fine-tuning them. I also have to say the models reach already a level—like, for example, the LLaMA 4 model—it was a bit disappointing maybe the results, but actually it’s a very versatile model and it’s very easy to tune. And with some additional work in tuning, it reaches already the GPT-level quality that we see on top of the table of the arena. I’m sure that Meta will probably have follow-up releases on that. So the models already kind of reached such a good level that most of the models deliver almost the same quality results.

Fantastic. Fantastic. And then maybe one more question, because this has come up recently at this event with some customers we’ve spoken to and then other customers before the event that we’ve had conversations with and helping them adopt AI, and it’s in the RAG and document understanding. So I’ll put you a little bit on the spot, but you know, feel free to share what you can, is support of additional document types—whether it’s the Microsoft family of, you know, documents or other types of documents. Is that in anywhere in the roadmap to expand those?

Yes. Yes, it is. It’s just that for us and for our customers at the moment it was easier to convert these documents in PDF and then keep the—something that is very complicated in the RAG that people probably don’t realize is how you extract the information from a document. If you just get a PDF document as example, you have the text that could be a different layout, formats, different tables, they could be images, they could be links, could be very complex tables inside, and it turns out that you can use different techniques and you have to use different techniques to reach a very high quality of extraction of the information.

For example, and every technique has a pros and cons. For example, if we just use OCR, it is good in extracting layout, but it doesn’t extract information that—for example, if there is a text that is bound to a link, the OCR does not see that. For just one example, very often there is information that is hidden that you have to read the PDF directly to be able to do so. And so we try to reach this super good quality level of extraction information of PDF files before we jump to another format.

And so for most of the customers it turns out it’s not a big problem to convert existing documents in PDF and we ingest them. But at some point, we definitely have on the roadmap to also be able to ingest Office documents, for example directly also documents, for example from SharePoint directly or other sources so customers do not need to copy those documents.

Yeah. Fantastic. No, that’s great stuff. Thank you for those insights, Lud Mill. I think that exhausted the questions that I had for you. So, unless you have any more comments. Uh, whoops. I think we might have lost Lu Mill.

Thank you, Lud Mill, for all the insights. Thanks everyone for participating. Oh, there you are.

Yeah. Well, thank you again, Lud. We appreciate your time, your insights, and we will stay tuned for new stuff that you and your team are developing and announcing. So thanks again. Everyone, thanks everyone for attending.