This episode explores how JD Edwards organizations can balance AI autonomy with human oversight. Drew Robb discusses Human in the Loop AI, including how organizations can determine when AI agents should inform, recommend, act with notification, or operate autonomously. The conversation covers risk, confidence levels, approval thresholds, testing, governance, auditability, and AI readiness, emphasizing that AI authority should be earned through measurable performance while keeping human judgement in the decisions where it matters most.
Table of Contents
- Introducing Human in the Loop AI
- AI Agent Autonomy Levels and Human Oversight
- AI Agents vs. Traditional JD Edwards Automation
- Trust, Risk, Testing, and the Dangers of Too Much Approval
- A Framework for Deciding How Much Authority AI Should Have
- When AI Should Act, Recommend, or Require Human Approval
- Confidence Scores, Exceptions, and Governance Across the Business
- Piloting AI Agents, Measuring Performance, and Building Auditability
- AI Readiness, Ownership, and the Path to Responsible Autonomy
Introducing Human in the Loop AI
Introduction: Why AI Agent ROI Matters in JD EdwardsShouldn't AI agent be allowed to approve an invoice, place an order, or update a forecast without anyone reviewing its decision?
And when does requiring human approval protect the business, and when does it simply create another bottleneck?
Today, Drew Rob from ERP Suites joins us to explain how JD Edwards customers can determine which decisions AI should make, recommend or escalate.
By the end, you'll have a practical framework for giving AI the right amount of authority without giving up human accountability.
Welcome back to Not Your Grandpa, JD Edwards, the podcast that helps JD Edwards users make practical decisions about modernization, automation, and emerging technology.
I'm your host, Nate Bushfield, and today we are talking about how AI is moving beyond answering questions and generating reports.
AI agents can now monitor information, identify issues, trigger workflows, and take action inside enterprise system.
That creates an important question for every organization, though. How much authority should this AI actually have?
Should it automatically resolve routine discrepancies? Should it recommend what happens next? Or should it stop and ask a person for approval before touching a transaction?
Joining us today is Drew Rob from ERP Suites.
Drew has spent a great deal of time exploring how AI can connect securely with JD Edwards, work through orchestrations and support real business processes.
Drew, welcome back to the show. It's been a while. It's nice to see you again.
But before we get into the Human in the Loop AI, give us a little background on your role and the AI work that you've been doing with JD Edwards.
Yeah, absolutely. Thank you, Nate. Happy to be back on the podcast. It has been a while and and really exciting times, especially with our organization of European suites.
We're moving into really agent development and AI agents inside JD Edwards and it's really taking off. So it's been really exciting times for us.
So for me, my role was, you know, AI advisor and really still is, you know, advising customers, educating customers on AI and when to start to implement, especially the agents now.
But I've also been doing since we've been ramping up a lot of work on the back end infrastructure and implementation of agents to quickly and seamlessly get them out to our customers so they can start using them in in their day-to-day jobs and in whichever ones they decide to, to pick from our off the shelf or, or customization agents that we're also building out with customers as we talk more and more customers and see what would be beneficial for them inside of JD Edwards.
So a lot of still educate, educating on front facing, which was I absolutely love still talking to customers about that with getting into the nitty gritty of the back end implementations to really make sure we build out strong AI, secure agents with autonomy and human and lupus.
We're going to talk about today for our customers to make their business successful.
So thanks again for having me, Nate, and looking forward to the conversation today.
Yeah. And I think that you're one of the best people to have this conversation with because you've not only seen the front end when it comes to interacting with customers and interacting with their business, but you also see the back end of what goes into this AI agent idea and product.
That you can actually give a little bit of insight on both sides of it on where it was, where it's going and what we should be allowing it to do now.
So let's establish the foundation first because Human in the loop AI can sounds like a technical architecture term when it really is just a business decision about authority and accountability.
So let's start here. What does Human in the loop AI really mean and why should JD Edwards leaders care about it now?
AI Agent Autonomy Levels and Human Oversight
Yeah, I mean, really Human in the Loop is really about providing human feedback, review and approval inside of your AI powered agents in different processes.
Now, you know, inside of JD Edwards, you know, we already really do have built in approval routes and workflows that control transactions we'll talk about a little bit later with our sales owner agent.
But we can do that with, you know, making purchase orders decisions, especially on like an out, like it reaches like a certain threshold. Like that's very important.
But it's really used to really add a decision maker into this entire process, right? You know, it's not as simple as just whether or not AI can perform an action. It's whether the organization is prepared to authorize AI to perform that specific action.
And what sort of autonomy do these AI agents need?
Now inside of ERP suites, we are building out a continuous fleet of agents and we're building a lot of agents. I'm not going to put a number on it, but there's so much going on here that we're building out and human in a loop. And human judgement plays a huge factor when building out these agents as well.
You know, and, and inside of our agents, which I'm going to at very high level describe and, and there's definitely demos we can show, you know, customers in the future is building out the autonomy via a dashboard.
So the user has the ability to set the autonomy level levels inside of the agents that we can either, you know, set them to four different levels here, right?
So level 0 is like we like to call it in our agents or assist. So these are really the autonomy levels that are just info only, no action, you know, read only queries reporting it can you can show up in the chat window such when we ask for questions, right?
But it really just provides insights to the user of when they are using our AI agents, right?
The second autonomy level goes a little bit deeper and it's recommend, right? It recommends different approvals that may be required. You know, it's the fault for all right operations.
So if we really think about this, it pops up in a little window saying, do you want to actually do this, right? Do you actually want to complete this action JD Edwards and potentially make this change and update JD Edwards And that's via orchestrations.
But the human still has the ability to say yes or no to that recommendation, right? It doesn't just automatically do it.
Now level 2 is the ACT plus notify. So this is when we auto execute that action and notify autonomously.
And we'll talk a little bit later about our e-mail agents that send emails out to users, but that's really autonomously when this action occurs, do this right when, when this orchestration goes through, send this e-mail to this user, right?
And it's really trusted operations there and it's under a certain threshold, but just really talking about, you know, the autonomy of that.
And then lastly is really just fully autonomous. So that's our Level 3, right? And and it's really for low risk and high volume operations.
And we'll talk about a little bit later of how you kind of integrate into your AI agent, the autonomy and what's going to take, you know, the initial testing with the agent before actually adding on to fully autonomous and kind of our stepladder.
But we'll get into that a little bit later.
But really it's those 4 layers that we have built into our AI agents and the users are able to set them on command for different agents and different actions in that agent.
That's what that's what we really talked about too, is, you know, we may have a sales order agent that can can upload a sales order and you can do a full autonomy for that because it sends it in.
But you know, maybe we have to connect it to certain advanced pricing features or different things. And we might want recommend set for that or showed or send me an e-mail before accepting this. Right.
So there's just different areas and we'll get into other examples of VR agents about that.
But it's really going to come back come down to the fact that you can set different actions inside the specific agent as well, different autonomy levels to have human in the loop.
AI Agents vs. Traditional JD Edwards Automation
Yeah, you bring up the autonomy there. So how does an AI agent differ from the automation that JD Edwards users are already using currently?
Yeah, and that that's big with like a lot of JD I was the users using orchestrations, which again, that's a big part of, you know, our agents is leveraging orchestrations to connect to your data via the AIS server.
But traditional automation, it generally follows, you know, predetermined specific rules, you know, specific conditions that may occur to perform a defined action and you follow that specific workflow and complete that action.
Whereas as we talked about and I travel a lot and talk about this is our AI agents are, you know, they're to interpret less structured information and really go through the step by step.
But it's really like what if scenarios, right and use various tools other than orchestrations, such as Gen.
AI length F, you know, different techno.
I'm not getting into that.
You know, the whole art section.
That's another discussion.
But it's the overarching thing is it's really about pursuing a specific goal, right?
And the ability to really handle more complex tasks and make decision paths based on that specific goal to what we want to accomplish, right?
You know, think of the AI agent and, and we have it on our website is, is really a digital worker next to inside JDL where it's, I can ask you to ask trivial workflows inside of that JDL environment, but take it a step further as we're talking about provide different recommendations and, and all that just depending on the various use case.
So real quick example, or, you know, as sales order agents, you know, looks at that unstructured, you know, customer PO, you know, it uses a specific OCR technology, but then it classifieds and routes exceptions.
You know, that judgement that we have for the various exceptions.
It there's too many things that may go on, too many customers that may be involved that there's really not fixed rules you can set for that.
So that's what really separates agents from the traditional automation.
This is what if scenarios that branch out to actually figuring out and defining goals and then learning on itself and getting smarter and smarter as well.
See, I mean the really it comes down to, you know, the agent just having more freedom, you know, to select actions, you know, be able to monitor, be able to provide escalation rules as well.
And again, that provides, you know, when you escalate to the next person in the process and send that e-mail out, you know, they can approve and send back to the agent.
So that's where in different in different facets of the agent in different steps in the workflow, human in the loop can be at any point in time, right?
It can be on the second step, it can be on the 4th step, right?
And it still will be involved again, like as the agent gets smarter and become more fully autonomous, you can see, and we'll talk about later about, you know, the overflow of notifications and why that's a bad thing.
So we'll get into that later.
But yeah, it it's really can kind of be wherever they're mate.
Trust, Risk, Testing, and the Dangers of Too Much Approval
Yeah. So like, does requiring this human approval mean that the organization doesn't really trust AI or is there something more to it?
No, absolutely not. And we can go back to, you know, ability to, to test the agents and, and really it all comes back to the data behind the agents. And we can get into that a little bit later as well, but it's really around risk control, right?
It's not an indication that AI is fails, failed, you know, business, especially JD Edwards companies are already using approval thresholds, you know, with various different employees raising up to managers, you know, for certain things such as purchase orders and financial transactions, you know, and that's been that approval process, right?
Going up to a manager is similar to what the approval process should be. You know, maybe for AI there, right? You know how it should be treated similarly to that, you know, the authority really should match the risk in common consequence, especially at the beginning.
We don't want to give AI for full authority to do high risk transactions or changes inside of your JD Edwards system. Because again, we are updating the JD Edwards system based on these transactions that you would just doing any sort of automation work inside of JD Edwards.
So it's really going to come down to, you know, increasing confidence inside of, you know, AI adoption, having leaders more confident in what they're doing, but also having those sensitive decision making processes also being kept in check, right?
You know, it's really about trust being earned through the continuous testing of the AI agents and based on measurable performance of that agent.
And that's what we're also working at ERP suites to build out is more observability, not just what task the AI agent is doing, but also how well it's performing.
And, and that's something we're, it's our next step of building out.
And, and then we already have a lot in there. Don't get me wrong. Like you can look at our a, a backwards, see the observability and, and what specific tools and using it, what's task it's doing.
But the performance of KP is, are also on our dashboard. And how well is, you know, how well we doing? How, how well is our from that process earlier, prior, how much better are we getting?
You know, we can, we can talk about this in more depth. You know, I know Nate has a podcast schedule later to talk about specific AI agents, especially our finance ones.
I'm about our AP agent, Like how, how much quicker can we can we, you know, can we send in those invoices and through to JD Edwards and, and seamlessly do that without manual review? Like how much confident are we with those?
And that's just that's just the the fact that, you know, if we want to measure the performance of the agent so we can add more autonomy to it later to have less routine checks throughout the process.
See, that's interesting though, because as someone that doesn't really trust AI to the full extent that a lot of people, and I'm sure a lot of people out there right now are thinking, yeah, I don't trust this.
Like there's so many holes that they've seen with open AI that's out there right now. That is a very different version than what ERP Suites is doing.
But why shouldn't companies require a person to approve every AI action when there is a level of uncertainty that goes into AI agents right now?
Yeah. And that and that, I've kind of mentioned it before. Like it it can turn into, you know, and, you know, the approvals can really turn into just another notification system.
And a lot of times when we do a day-to-day jobs will get annoyed with multiple notifications. So I think almost too many we have says like, can we turn these off? We don't need these anymore, right?
And that's what really AI turns into. It's not necessarily a productivity tool that can help you in your day-to-day lives and provide recommendation that turns into just a notification tool that you're overwhelmed with, whether it be sending via e-mail or or another chain.
Teams Webex, like we, we don't want that. We don't want or even JD Edwards applications because we set things up with our agents too. It's something we don't want, right?
We want it to be a productivity rule. We don't want employees to spend too much time reviewing the routine recommendations that go every day, that come up every day.
We want to turn those autonomous after you know the agent and has been trained up enough and we are confident with what it's coming back with the output, right.
And that's when we talked about continuous training here, you know, in that day-to-day work manually.
Yeah, excessive alerts can cause, you know, we talked about before approval for teach from a manager and having way too much approvals and there's too many people involved in the age.
And when it's supposed to help with the productivity, it's almost like you're babysitting it without needing the babysit it, right.
If there's no reason to because it's it's it's creating the right output.
You know, they should really be concentrated. These human approvals should be on accepted exceptions on certain cases.
You know, when a price point is too high, why is it too high? You know, consequential decisions, various things like that.
And that's when we talked about especially with the ability to, you know, provide different autonomy levels in the human interlude processors, low risk, repeatable actions or typically what candidates for autonomy at first, those are what they will look at, right?
And we talked and talked about before with the sales order entry, right? It's something that, you know, it's easy, it's pretty simple. We're going to upload a sales order. That's autonomy, right?
And that's automation, which we can do. But even providing some of those recommendations, as it gets to know more customers and how they work and what they buy, then those can become more recommended autonomous actions as well.
So really those low risk jobs. But again, eventually, eventually it can comfortably autonomous, right? But it's all about the training.
And then you can set that again as a user in the dashboard.
So, and I really liked how you brought up the productivity side, because an AI agent is supposed to take less time. It's supposed to speed up some of these processes.
And if you're getting a notification for every single step to approve it, does that actually speed it up? Probably not.
You're probably still doing the same amount of work. You just have someone else coming to you with every single question they've ever had.
And you know, for the micromanagers out there, that might be a good thing for, but for some of these miniscule tasks that even a micromanager would be like, why am I taking a look at this?
It doesn't make a lot of sense for them to have to do every single step.
And you're right in the way that until they're completely trained up, maybe you do want that in the beginning just to make sure that everything is working correctly.
But that is what the test phase is for. You have that before you actually roll it out. So that's it's a little bit different there.
But what risks would you say arise when a company maybe skips that test phase and they go too quickly, like to the Foley autonomous AI?
Oh, yeah, this is this is big.
And actually one thing I wanted to bring up before that need, and this is very important, what we've run into with customers even before skipping the autonomy.
We have to really talk about the data aspect of it because we need a lot of data to train these AI agents before we can start adding autonomy to them. Because I'll even get the right results.
So just remember, data cleansing is always important thing. It might come up in every podcast.
So before we before we even set the autonomy limits, are we going to be able to even get to autonomous with that?
So I just wanted to bring that up before anything.
But it's really about, you know, really providing yeah, that that that prior assessment for the data cleansing, you know, so the AI may act on, you know, complete data rather than incomplete or outdated or incorrectly interpreted data.
I mean, that's really important, right.
And just having the agents, you know, being able to, you know, connect to, you know, multiple.
Oh, no, sorry, sorry.
We might need to get that part.
All right, All right, Yeah, we can.
I'll read the question again.
Yeah.
And then yeah, I can just go through the value.
It's all right.
So.
So I ask it again.
I'm not going to use the way that I'm about to ask it.
I'm going to cut that part out.
So same, same thing.
I like it.
I liked it.
All right, You're all good.
You're all good.
All right.
All right.
Ready.
Yep.
All right.
Cool.
So what risks arise when a company moves too quickly towards fully autonomous AI?
Yeah. I mean, a big risk that comes up is, you know, you know, we might have an agent that might be permitted to execute, you know, multiple connected action, right?
And they might need to do multiple different sub tasking.
And that in itself is, is not a good thing because what you're going to see here is the agent may make a mistake early on, right at task one or two.
And then it will just keep going. It will just keep going through, which will then, you know, it will become a whole domino effect in the end, right?
And really what happens is, is the organization has an issue to struggle and to determine, you know, who's accountable for that decision as well.
You know, what's the sort of level of autonomy, autonomy and observability, right?
And that's why I mentioned before, you know, European suites is really looking at putting even more work into the observability of agents.
Not just what I said when it talks about listening to cast, but how well is it doing to make those strategic recommendations even raise the level of autonomy to from assists to recommend to automate, right?
And this and there's four different levels we talked about before and it really comes into the fact that, you know, we we just need to make sure there's no in inadequate logging, right?
You know, it can make it an outcome and especially if the AI makes a mistake, because that's bound to happen.
And I'm 100% perfect, right?
It makes it even harder not having logging as well to reverse that AI driven mistake and what actually happened in the process.
So, so it's very important and those are just big risks that we really need to think about.
And, and that's why, again, we're building out the autonomy levels, the observability and the logging as well to know who's doing what.
Even if there's a misapproval or something that might show up in the AI agent and someone approved something they shouldn't have proved and then the agent took it with them.
You know, we had that log in of the various users and who made the changes to right with the human in the loop.
Did the agent make the change or did the human make the make the change right?
A Framework for Deciding How Much Authority AI Should Have
So the objective isn't to put a person in front of every single action, or to remove people entirely. Objective is to put human judgement in the right places.
But let's make this practical. What framework should JD Edwards customers use to decide whether AI can act autonomously or make recommendation or even require an approval to do an action 100% made?
And that's, and that's a fantastic question. And and I'm going to be honest, it's going to be customer by customer basis, right?
Every customer.
And that's why we need those discovery and assessment calls early on because every customer is different, their AI strategy is different, their appetite for AI maybe digit different.
Do they want a simple AI agent or do they want a very complex AI agents?
Is is a culture readiness there like for the business age or business analysts are working side by side with these agents every day, going to going to be ready and able and willing to accept these ages.
And that's our biggest conversations we have with customers too, is like, are they ready to accept ages?
And that's what we have in the whole IAI ready this assessment.
We have all this with customers because we want to make sure it's not just throw a throwaway agent or agents are good, like they're great and they will help JD out with productivity.
But the biggest thing is going to be the ability inside the system.
So that's very important to really look at too, right?
You know, the only one also look at two companies may, you know, buy similar agents and they may have like different timelines for building out those agents.
So allowing AIAI agents to form some tasks on their own.
You know, we might need to build a different agent that's that's a little bit different for each customer, right?
And that's why we talked about it earlier with the off the shelf agents we have versus the the semi customizable with building a repository of agents that we can reuse and implement very seamlessly.
But that's still very important.
I think it's also important as well made to mention other factors that might come into play here with the framework, you know, financial impacts, right with with building out the agents.
You know, the ability as well when we talk about autonomy, autonomy, the ability to reverse an action, you know, this one is very important too.
You know, really around the AI confidence or uncertainty with agents.
You know, I'll, I'll confident you and AI and your business, right?
Is it going to be able to perform the way it's going to perform?
Is there going to be uncertainty, uncertainty with this results Is that and that's why we need to test it a lot more.
And we also need to look at again, we have a nice to meet you up in security regulatory and policy requirements.
Who has access to what data inside the agent is also very important before setting the autonomy levels.
And the framework around it is, is if we use JDL with security mixed with our OCI security.
So really we already just take those rules in security measures we already have in JD Edwards and, and append them on to our agents that we built out for the various uses that might be interacting with the agents that we build out.
One other thing, you know, we, we really want to talk about too is, is, is, you know, the framework around it is it's kind of going back to the autonomy level, right?
What authority will the agent have right?
So still going back to the four levels, right?
The ability to inform, right, gather information explained what's found recommends to the AI proposes the action, but the person still decides, you know, the approval, right?
By exception, AI acts within defined limits and escalates exceptions and then act autonomously.
AI completes the action and records the outcome.
I think it's really important to highlight these.
Again, just knowing that we talked about the framework, like to me these are the four autonomy level frameworks.
Every agent needs to add no matter what.
And all of the agents we have built out have this autonomy level.
And, and just a quick example of this is if we look into, we have a financial reconciliation agent that matches transactions and proposes adjusting general entry.
So there's your recommend phase or routes, anything material for human review before it even posts it.
So just talking about the level of autonomy, like this one's different from the sales order automation, their sales order update agent that will update a sales order.
This one's different because it's actually providing recommendations and providing human in the loop.
So again, it's going to depend customer by customer, financial impact, what's the budget?
What's the confidence and uncertainty with the end users, with the company itself?
Or just remember, there's four autonomy levels and they can be changed throughout every agent, right?
And be set for different tasks inside the agent.
So I think it's important to explain how that all comes full circle.
When AI Should Act, Recommend, or Require Human Approval
Yeah.
Speaking of tasks, though, what kind of Judy Edwards tasks are strong candidates for autonomous AI action?
Absolutely, yeah. And and we've, we've kind of mentioned it before, it's really around, you know, the highball you repetitive activities with, you know, consistent inputs and clear success criteria.
You know, we talked about before it's actions with low financial or operational impact, you know, tasks that can really be quickly detected, right?
And errors can be reversed pretty easily.
If the AI agent were to make an error, there's not too much that would go on and not too many maybe tables that would be hit that need to be changed.
Or, you know, if it's sent out to a specific customer, like maybe you know, an e-mail sent out to a customer with, you know, a sales, you know, a shipment option or something like that's something that might not be our first priority with the setting, the autonomy levels, it might be levels, it might be later on.
Have a, have a few more examples here.
You know, just the ability to, you know, classify, as I mentioned before, sales orders and purchase orders, you know, routing request approvals through purchase order approval via, you know, an e-mail agent, which we have built out as well.
That's one of our support agents.
We have, you know, creating drafts and reports and summaries inside of the agent chat window to provide recommendations.
We have an inventory allocation agent.
So that's very helpful to have the human in the loop aspect.
And you know, the, the, the supplier or enough supplier, the, the, the coworker, they're working with the agents to do the, oh, the scheduler, sorry, the, the scheduler to provide inventory allocation recommendations that the agent provides.
And yeah, as I mentioned before, I scheduled reminders with emails, you know, when a journal entry has been submitted inside of JDE, we'll be able to see that journal entry, you know, potentially updating a non critical cast.
But really it comes down to the, the most important thing, right?
It's always starting with a specific use case through a detailed design discovery phase, right?
Rather than giving your agents system wide access, right?
Start with that specific view because we've already had some conversation with customers like, well, why don't you just why don't we just connect to our entire JD?
I would say, so why don't we move data over those guys?
The I'm like, no, no, let's start with one specific thing and we can start the autonomy levels and this can grow out and you can buy different agents.
So you can do various agents can do different tasks.
I think that's really important also to really think about, right?
You know, we really start small and grow.
Yeah, exactly.
And I think it's really important just to know that, you know, we really are positioned ERP suites, positions, our JELS agents as, you know, being good decision makers with predictive insights, with intelligent automation inside of that for the existing JDL was process.
So we're not changing our process unless we need to, right.
There's still process improvement that makes it done beforehand, make the AI agent more successful, but we're not going to do that if we don't need to to start to go out these agents.
So when would you say an AI should make a recommendation, but leave the decision to a person?
Yeah, this, this is very important too, right?
And, and there's a few factors we need to really look at here.
And again, it's going to be customer by customer basis.
I still want to preference that agent to agent basis, but it's really when an agent can, and this is what we see, right?
It's, it's really when an agent can make meaningful, accurate analysis, but decisions require business context or judgment, right?
So it can make the analysis, but there's still some approval that may go in because there's different nuances.
Because again, like we said before, the agents aren't just automated actions, right?
They're not just like specific, it goes a lot into what we've talked about earlier with the what if scenarios, right?
So it really comes into several strategic implementations with maybe several different options that may show up for making those those various recommendations, right?
It may, it really affects the planning, right, but does not demand in the media system action is, is what we think is really important for the decisions we make with AI to make a recommendation, right?
It's it's something that like we talked about before, it's not sending specifically to a customer or something like that.
And it's, I think that's a good place to really start and and showcase the ability of the agents to make strong recommendations to aid the aid the humans in their day-to-day jobs is what it really comes down to, right?
So like forecasting adjustments, inventory optimization we talked about earlier that's, that's very important.
You know, potential cash flow projections, suggested maintenance times as well.
If we get more technical with that, you know, our inventory allocation agent that we built out is specifically built to old recommendations and not be fully autonomous for risky calls of the JDL and changes that will eventually go to, you know, the shipments be shipped out to the supplier via inventory.
So it will even make changes to inventory when there shouldn't be changes in May, right?
You know, and, and going another way, right, with our financial planning agent, you know it, it generates forecasts and performance statements with, you know, various narratives and explainable, you know, actions that go into that.
So if you, if you can see that, and that drives their decision making, right?
The data's driving the decision making rather than just a specific final number.
It gives them various different avenues to look at.
So again, going from the finance inventory, like that's what we're doing, we're giving them options, right.
And then they are there because they owe the strategic subject not expert to make that decision in the long run.
So yeah, I mean, it really comes down to, you know, the human reviewers should be able to, you know, understand what's coming back from the agent, but they are they to there to make the decisions because they understand the business the best, right.
Yeah, which makes a lot of sense.
And maybe to go a step further into that different level of approval or recommendation level, what JD Edwards decisions could normally require explicit human approval.
Yeah.
And we really talked about it's, it's those ones that are super cost when you may even manually make a decision that you seen daily lives without agents or if the agent makes the decision, it's it's really those possibly irreversible actions, you know, those very high volume transactions, You know, the decisions are governed by, you know, governed potentially by sensitive data, you know, internal policy, you know, contracts, regulations, you know, and the actions really affecting, you know, different employees, customers, I mentioned it before, supplier relationships, right?
There's probably is there's that whole dynamic, that political dynamic when you talk about, you know, bigger industries out there, They need their product now.
They need their product now.
They're big accounts.
We can't have recommendations for a smaller account come in, their shipment gets sent out while the bigger ones just sit in their way and then you lose the account.
Like those factors come into play, which is why we need humans to make sure, OK, this is a good inventory adjustment, but this isn't our priority A supplier and we can build that into the agent as well.
Like these are priority A, these are priority B.
This probably C customers, but that's something very important to also look at, right And I mentioned it before, like over the threshold, large payment options right on board and changing a supplier, you know, really you know, changing privileged user access.
We don't have a lot that deal with security.
It's more business process, but that's also a huge one that and when you're building out agents, anything to do with security while we need approval before we do that any changes made.
And then, you know, as our, you know, we with our specific agents as well, like the reconcile agent, you know, you know, material differences and write offs before posting entry.
We mentioned that one before, right?
And then, you know, inventory allocation, so stock cancellations, you know, constrained stock commitment or sent to the planet rather than them just acting.
I think that's that's very important to look at it and really looking at, you know, processing unauthorized purchases.
Again, we're not changing stadium core functionality with these agents, but we want them to aid you and your and your business and your decision making.
And I think that's the most important thing, that fully autonomous is a very important thing, but it might not be every single action that the agent takes need to be fully autonomous.
Some actions can and some avenues can within a specific agent, especially if there's a lot has a lot of feature, functionality, capabilities.
But not everything needs to be autonomous.
And these are the ones.
Oh exactly.
Like there's still decisions that need to be made by a true human that understands the business to a level that an AI agent doesn't right now.
Not saying that that can't change in the future, but at least for right now, yes, having a human in the loop can help your business not only make more confident decisions, but also speed up a lot of these processes that maybe an AI agents can't do itself.
Confidence Scores, Exceptions, and Governance Across the Business
But how, how should a confidence score and even exceptions influence the approval process here?
Yeah, with the confidence score, this is this is big, right?
You know, it really comes down to the company's specific thresholds, though.
Like how confident are they with the AI agent process, right?
And we've kind of mentioned it before, you know, does the agent have high confidence and low risk, right?
Are we confident it's giving you the right outputs but also has low risk?
This is when they can start to act autonomously.
You know, do we have moderate confidence monitor risk, OK act, but only within certain limits, certain thresholds, maybe certain approvals that will approvals that will be involved in the process or do we have low confidence with the outputs maybe or it's a high risk?
Do we need to escalate it a little bit further, right, with the agents, right?
And it's really just it's a way to trigger review, right, to request more information and, and the ability to stop the agent process before it goes haywire, right?
And, and, and does too many steps in the process.
Well, we have no way of turning back of what it's done, right?
We do have a way, right?
You have the logging, don't forget the logging.
We still have the logging.
The problem is you got to go back and make all those changes that the AI agent they have done through the sub task, it's completed, right?
So, you know, we really just got to be cautious with that, right?
You know, really have to look at really things when we look at, you know, exception criteria, you know, should include, you know, unusual amounts or missing data or conflicting records, new suppliers, abnormal timing, you know, behaviors outside the expected patterns.
These are something that we really need to get alerted on from the agent to have humans get involved and make sure we're getting the right information we need to get right and continuously checking, testing the agent through various AI models, different authority levels.
I think that's very important.
Or sales order agent actually has a fuel confidence level will flag anything that is outside some threshold and, and send over to ACSR for review.
So there's where the e-mail notification comes into play, right?
And and really there's a confidence bar with our inventory allocation agent.
You know, if it clears over the confidence bar, we have too much inventory, right?
It defers to low confidence for other people.
So I think that's the most important thing, right with this question is just understanding that, you know, we need confidence scores in every agent.
We need sort of those KPIs that keep track of how they are doing, but also the confidence of we're getting the right data back.
And then that's built into the agent as well, right.
How confident is it reading the sales order?
How confident is it with this inventory allocation it it's built in, right.
And and then that's The thing is we still have to train you so taste test them and still to make sure it's been data in the end.
Yeah.
So would you say that the same approval model should apply to every single department or are there differences here?
No, no.
And this goes back to even we can even go as deep as as agents specific here, right?
They're all going to have different autonomy levels.
And we talked about this before, right?
And, and you know, it's, it's really important to just understand that we can fine tune those different autonomy levels to a certain threshold that we have, but also understand that's it's process specific rules.
But we, all of our agents still live under the same governance framework though, right?
All built into the same infrastructure framework, which is very important.
So even though the autonomy may be different for different, not just every agent, but every subtest, we still have implemented through the same OCI technology, various models, they're all the same.
So even if they're what can feel and act different.
They still are all under the same framework and guidelines, right?
And they do, but just know they do not apply.
Every appointment, every department can be different.
Every agent can be different, Right.
Yeah.
So it's important.
Yeah.
No, no, I was just going to say, like, once those authority levels are truly defined, the next challenge is making sure they continue to work after the AI really enters production.
A model that looks safe in a workshop still has to prove itself with real JD Edwards data and real business transactions.
So how should an organization introduce human in the loop AI and determine when it is safe to give the system more autonomy?
Piloting AI Agents, Measuring Performance, and Building Auditability
Yeah, and this really comes back to starting out with one specific defined use case.
We really talked about this before talking about, you know, what's the what's the value add from this AI agent, right?
How's it going to help to eliminate cost?
What are the expected outcomes?
What are the goal, the specific goals for the AI agent, right?
And it and it comes into the sole fact that, you know, a good example of this is our sales order agent, right?
You know, in a specific guard rails where it's fenced in to, you know, right through approved orchestrations, but never directly to specific tables, right, and it sends everything to orchestrations, right.
We have confidence levels at flags for human review inside of JD.
I was as needed and and this really eliminates the the fact for hallucinations that might show up in our agents area agents, which is very important.
You know, you know, we really have the ability to compare various different recommendations inside the agents might not just get once the recommendations they'll give you 3.
So what, what's the best choice?
And that's where the human experience comes into play, right?
And, and especially, you know, measuring the accuracy of the agency, the exception rates, false positives, what's the really time saved through the downstream business outcomes.
You know, ERP suites is always going to recommend that we begin AI agent adoption with human oversight no matter what, you know, and we're going to progress towards greater autonomy with governments and auditability also built in.
That's also auditability.
I don't know if I mentioned it's also in our agent dashboards, every single one of them.
So you can export the document, do the auditing as needed, you know, for auditors that come through your business as you start to use AI agents, which is a very important aspect as well to look into.
Yeah.
So what would you say a pilot should look like before AI is really allowed to take action in JD Edmonds?
Yeah, I mean, a pilot again, it it still goes back to, you know, a very narrow process, you know, a clear owner with a definable outcome, right?
A definable goal, you know, document the current workflow.
You know, various approval points that are needed inside of JD Edwards, which are always there.
And then common exceptions are also needed to train the agent.
Remember to always give access to that specific narrow use case and not your entire system.
Run the AI apparel with existing processes before allowing it to escape on execute on the entire process is very important.
Also, you know, review where a recommendation is needed, right, where a human intervention is needed for human decision and why, you know, establish A blowback plan for production deployment.
But also remember to our way our agents are set up is it's dev test prods, same with JD Edwards.
Make sure you've done a lot of testing, a lot of development testing.
Make sure you don't skip those steps, right?
And and really define the business to what are the success you want to get out of the AI agents, right?
Is it faster cycle times, fewer errors, less manual works?
Obviously a big one with automation.
So that's yeah.
So what would you say?
Like what evidence should these leaders require before reducing the human review side of it?
Oh yeah, this is, this is great.
And it's, it comes back to this really good logging, right?
And, and, and seeing like error laws that come up and then having the agent understand when errors do occur, right.
And, and it's really as well as with the errors, it's, it's the KPI dashboard or KP is that come up right?
Consistent performance between a sample of like similar transactions and they keep doing the same thing over and over and over again.
It's correct, right?
It's continuous in repetitive transactions that you keep, you keep testing, right, right.
And it's just that confidence level, right, that of the actions before.
And if you see something off, that's when you escalate it for even if you right.
So, and, and I think that's, yeah, important to just understand that results should be monitored over time.
You know, the various processes, the data, the various business conditions, these can all change throughout the time.
So it's important to continuous monitor everything and continuously say that we have observability inside the agents as well as auditability inside the agents, which can always be checked.
Yeah, and that's perfect segment, I guess there the audibility, right, What should be included in an audit trail for an AI assisted decision?
Yeah.
And there's, there's, there's a lot, right.
Probably the biggest thing is the specific tools, which we have no entire age, the specific tools of various integrations or the JD Edwards orchestrations being used know what's being involved in every single transaction.
You know, what data has been involved when you initiate that process, What, what specific AI recommendations or selected actions may have happened as well as, you know, what's the final transaction, you know, the resulting business outcome.
I think if we, it's very important to that, that provide, you know, just a few examples, right.
So, you know, sales order agents distinguishes between the, you know, the agent extracted document versus the CSR value validated document.
And you can really see the percentage of, you know, extracted autonomous agent uploads versus the CSR manual uploads and, and keeping a log of those records as well with those various reason codes that might be needed.
And and with the reconciliation, you know, the ability to have audit read documentation to drill back into source transactions that may have been done by the agent when they become often enough that we can do autonomous interaction inside JD Evans, which is very important.
So, yeah.
AI Readiness, Ownership, and the Path to Responsible Autonomy
And when it comes to the confidence, who should own that decision to give an agent more authority 100%.
So it's kind of a really a whole company aspect of this, right?
And, and who's going to, who's going to own it, right?
So obviously the business process owners and business analysts should be accountable for the various business process decisions it makes, right?
IT should be reliable for the architecture of the integration, reliability and supportability.
Security should definitely monitor the identity access credentials, the system permissions.
So we don't go haywire here and people are touching on touching things they shouldn't be outside the agent that we've developed for you again, but our agents are go because it provides JDL with security.
So just remember that that's already built in.
If you already have that, you know, finance and internal teams are there to help audit the the agent and the AIT with our help, right?
We are the vendors.
The AIT will provide performance evidence, you know, to assess business risk with the agent and continuously monitor and support the agent ownership.
And you mentioned this before, it was great.
I love that you mentioned this me and just push the production, right?
All of these different phases, even though it may be developments going on with building an agent for you, these different facets and these responsibilities need to be established during development testing before the agent get pushed to production.
That's super important.
Like, OK, the agents there, but does anybody know their responsibility once it's out there?
Because it can't, again, it's not fully autonomous.
It doesn't it's not all correct right at the beginning.
We ought to treat it.
So that's that's very important to understand that these human interactions are very important and not letting our agents from granted basically.
So what would you say the first step should be for AJD Edwards customer after listening to this episode?
Yeah and it comes down to the first step of what we've had with our conversations with our customers is conducting AI readiness assessment.
You know, we still got to cover data flashes agents are we got to cover data integration, security, governance, expected billion business value is huge because we've already talked to customers who if it, if I try to add AI integration, AI integration projects and they've thrown away the AI AJ because it didn't work.
And then you lose all confidence with future AI agents.
So very important to assess the business value.
What's the point of this?
We can't just be implementing AI just to implement AI.
We talked about this in past podcast.
That's the same with agents.
Even if the technology is new or indifferent, you know, it's, it's really just avoiding the beginning with a broad question of where I can use AI versus we got to get down to the specific narrow process.
What problem are we actually trying to solve?
What's the measured desirable outcome?
And again, identifying 1 repetitive decision or approval process might be a good place to start.
That has impacted substantial employee time.
We talked about before the end of the inventory allocation agent, a planner may spend weeks doing it, but with recommendations it may turn to 8 different 8 hours or so a day, right?
You know, really focus on routine cases with high impact decisions.
You know, decide which portions the AI can inform, recommend, execute or autonomously adapt.
That's also important is a set of the autonomy levels.
Now they will probably start well, but they can increase as time goes on and really document the current approval process inside JD was we we again are leveraging our AI agents with JD Edwards current approval process starts holds those business rules.
So that needs to be documented clearly, emphasizing explained to A to a vendor who is helping you implement AII.
Just want to say the answer isn't that AI should always decide or always ask permission.
The answer depends on the risk.
It depends on the evidence, depends on the controls surrounding each specific business process.
And that's the main thing to get out of this episode is that it depends on your situation.
Not there isn't a company out there that's the same as other company.
Everyone has different permissions.
Everyone has different pain point areas that you should look at to see if you can automate with AI agents.
But if your team is evaluating AI agents for JD Edwards, ERP suites can help you identify the right use cases, define practical approval boundaries, and build a road map from human review to responsible automation.
The goal is not to automate everything, it is to give AI the right level of authority for each decision while keeping your data, processes, and business protected.
Visit erpsuites.com today to connect with the ERP Suites team and evaluate where human in the Loop AI truly fit into your JD Edwards strategy.
But that's a wrap on today's episode of Not Your Grandpa's JD Edwards.
The biggest take away is that AI authority should just be earned, not assumed.
Let AI handle routine low risk work, require human judgement when the consequences are significant, and use measured results to determine when greater autonomy makes sense.
Drew again, shout out to you for joining.
I know you're a busy guy.
It's great to have you back on this podcast.
Seriously, it's been great talking to you and learn a little bit more about where humans should truly interact with AI agents.
But if this episode was helpful, subscribe, leave a like and share it with someone on your IT, finance, procurement, or even your operations team.
But until next time, keep modernizing, keep asking better questions.
And remember, this is not your graph, but JD Edwards.
Topics:
