Tim Gasper [00:00:05] Hello and welcome. It's time once again for Catalog& Cocktails. It's your honest, no- BS, non- salesy conversation about enterprise data management with tasty beverages in hand. I'm Tim Gasper, longtime product guy, data nerd, customer nerd, all those things, joined by Juan Sequeda. Hey, Juan.
Juan Sequeda [00:00:20] Hey, Tim. How are you doing? I'm Juan Sequeda, principal scientist at data. world, and as always, it's a pleasure. Take a break, middle of the week, let's go chat about data. You see now we're talking a lot about AI. And we got a special, special episode today.
Tim Gasper [00:00:34] We do.
Juan Sequeda [00:00:35] I do want to be, again, in the honest no- BS style here at Catalog& Cocktails. Our guest is Brandon Gadoci, who is the VP of AI ops at data. world. Now, again, all transparent, no BS. This continues to be non- salesy because we're not going to be talking about data. world. We're going to be talking about the stuff that Brandon is doing that happens to be data. world, but it's actually something that we're really excited to go talk about because we believe every other company should be doing this too. And with that, I'm going to stop because I'm going to introduce Brandon. I'm going to let Brandon introduce himself, and we'll dive into all of this. Brandon, how are you doing?
Brandon [00:01:13] I'm doing great. Thanks for having me. Very excited to be here today and chat. I am the VP of AI operations here at data. world, as Juan mentioned. It's been a position that I've had for maybe six months now, and I can say quite honestly, it's the most exciting, satisfying work that I've done in my career. I'm happy to share about what that looks like and see where it goes.
Tim Gasper [00:01:36] That's awesome.
Juan Sequeda [00:01:37] Fantastic. That's a bold statement, by the way. I'm looking forward to unpacking that together.
Tim Gasper [00:01:41] Yes.
Juan Sequeda [00:01:41] Well, there's a lot of things people will be hearing is like, ah, this sounds like a BSE thing, and let's go unpack this. But hey, let's kick it off. What are we drinking and what are we toasting for today? Brandon.
Brandon [00:01:53] I've got a great big mug of water.
Tim Gasper [00:01:58] I was really hoping you were going to say Wild Turkey. I was really hoping.
Brandon [00:02:01] Nope, nope, we just water at this point.
Juan Sequeda [00:02:06] Any particular toast? Anything you want to toast for?
Brandon [00:02:12] I guess what's on my mind is mostly the solar eclipse. Yesterday was a little bit more powerful than I thought it was going to be, and so it got me thinking a ton about just being thankful for my family and where we're at, and just everything that's been happening in my life. So I'll toast to the solar eclipse.
Tim Gasper [00:02:32] Love that.
Juan Sequeda [00:02:36] We had an opportunity to go see it. It was kind of, I saw pieces of it. It was cloudy, but it was still really fun, surreal experience. Like, oh, everything got so dark and then-
Brandon [00:02:46] Yeah, we heard people cheering from all around the neighborhood. It was a really interesting experience.
Tim Gasper [00:02:53] A very human connecting experience and throughout history. So yeah, I'll cheers to that as well. I'm drinking my coffee. For those who are listening, it's unfortunately not yet a cocktail- o'clock, but we'll get there soon enough. But yeah, cheers to the eclipse. That was cool. I've never been through a total eclipse before, so that was a unique experience.
Juan Sequeda [00:03:16] That was fun. So yeah, cheers to the eclipse and then it just gets people together. All right, we got so much to chat about. Let's kick it off. Brandon, honest, no, BS. What the heck is this AIOPS stuff?
Brandon [00:03:30] Well, I'll say we're kind of figuring it out as we go, and as Brett always says, we're kind of writing this history as it happens. So for me, the story started out with an idea that I ran by our co- founder and said, Hey, why haven't we done this? The next day after sharing that over a happy hour in Slack, he shot me a message that said like this. It was a link to a chat experience that would recommend content and help people better navigate learning. So I was blown away and then kind of ran from there, started showing it to people inside the company, and they were like, this is really cool. I could use it for this, I could use it for this. That sort of launched this path of me getting to understand how people work inside the company, what their friction points are, and then going to work myself and trying to figure out ways to build tools or solutions for them that would make their job easier and make them more productive. So what AIOPS looks like now, it's a process by which I go and interview different people at the company. I find out how they work and where their friction points are, ask a series of questions. I go back and mock up some solutions, I'll come back to them and say, do you think this would make your life better? They say, yes. We build it, we test it, we launch it, and then we iterate. So that has touched every part of our company, from marketing to partnerships, to sales, to customer success. So yeah, that's what AIOPS is right now.
Tim Gasper [00:05:05] Yeah, no, being here at data. world, I've had the opportunity to see all the different ways you've been able to apply things. And one thing that I want to zoom out and unpack here, which I think is super interesting, Brandon, is I think a lot of the industry has a perception around this phrase AIOPS, which I think they think of it more how they think about things like MLOPS or DevOps. If you think of this technical field around orchestrating the infrastructure and the management of AI and things like that. But I think that your role and your focus is quite a bit different than that even though we're kind of using this phrase here or AIOS or AI Operations is not that sort of infrastructure approach. It's a very business first use case, first kind of approach to AI. Can you unpack that a little bit and the thought process and how different that is?
Brandon [00:05:54] Yeah. It's funny because the term came across my radar when Rachel Woods, a friend, had started talking about it way back in, I want to say October maybe of last year. Then I started to Google search what is AIOps? And like you said, Tim, all I did was run across kind of an IBM definition of it or some infrastructure talk around AI operations. And to me it seemed that it was more than that. It is not just talking about what are the technologies that we need to put in place, but what are the operations, what are the people problems that we need to solve? A lot of times when I'm doing these interviews, I'll say I'm in AI operations, but flip those two terms. It's really operations first and then how does AI support that?
Juan Sequeda [00:06:41] That's a huge one. People will get the words AIOPS, but I love how you said it. We're in operations first and we're going to figure out how to use AI to go do that. You're going off and talking to people and interviewing. What does that look like? Is there a style, a template of questions that you have? Because I feel like this is something that should be repeatable and stuff.
Brandon [00:07:06] To be really honest, I've written down a few things over time, but interviewing and just keeping a freeform conversation is something that I just enjoy doing. So I sort of follow something I learned a long time ago, the what method. So whatever I'm talking, why is that important? What does that mean? Where does that go? Who does that? So I try to just keep asking questions and keep asking questions. It's not uncommon in these interviews that someone comes to the table with no sort of idea. They're just coming to have the interview. I wouldn't be surprised if half of them were like, oh man, I got to take this interview with the AIOps guy, but almost to a T, every time somewhere in that interview, either I or they come up with something. Now that you got me thinking or what can we do this? Or what about, I didn't think about would this be possible? So sort of that moment of like, oh my goodness, this is now something that can happen where in the past maybe it hasn't. In fact, I would say that one of the things that I'm realizing is that AI in general is allowing people to create at a pace that wasn't possible before. We know that, but I think we've been trained to... what you're seeing a lot of is this unpacking of pre- trained thinking. You'll hear me say maybe later that people have to learn to unspeak Google. We kind of grew up in that time. But when it comes to this innovation, I think people have been trained like, oh, I have this idea. Well, that will never happen because it's too much of an investment. And AI is now kind of unlocking this pent- up innovation that people have had over the years. So we have that aha moment in that interview. So to answer your question specifically, I don't have a pre- trained template that I use. I kind of just follow the conversation and try to be a good interviewer.
Tim Gasper [00:08:50] How do you know when you're seeing a use case where you're like, oh, wow, this is something that AI or gen AI could be really helpful with?
Brandon [00:09:01] It's interesting because there's sort of three levels that I've discovered of where AI can help. There's these quick wins, low- hanging fruit. There's then the next step when I'm calling version two of the apps I'm building where you're maybe getting a little bit more sophisticated, you're introducing things like a database behind your solution. You're introducing agents, which is really just multiple AIs working together. Then there's kind of this third stage that really gets into preparing enterprise data and understanding how to handle hallucinations and all of that stuff. So oftentimes when I'm talking to somebody in my head, I can say, okay, well that could be done in the low- hanging fruit section, or no, we could do that with just hooking up a database and a couple of different agents working together. Or man, this is really outside of what we can do at this point. So I'm a team of one here at data. world doing this, and so I have to prioritize projects like that. So could this be something that has enough value for the amount of work that I would have to put into it to do it? But there is some use cases where you're just talking to people and you're like, AI has nothing to do with this. They just need a Kanban board.
Juan Sequeda [00:10:10] That's the honest obvious right there. It's like, no, you need to organize yourself.
Brandon [00:10:15] Yeah, people can go too far on the AI thing, just like any other technology when you get excited about it and you have to kind of step back and be like, what am I doing? The big thing now that I run up against is what's a sort of Chat GPT or Open AI or the offer out of the box, and then what can we do ourselves and what should we build? What should we sort of augment those types of questions?
Juan Sequeda [00:10:41] I like that these quick wins. Basically it's like the easy, the medium and the hard. So does that mean that you should always be looking for these quick wins, these low hanging fruits, and then they always go on that?
Brandon [00:10:53] I think there is a ton of that to happen. We made a ton of progress with a pretty simple implementation of AI. So we were using Streamlit out of the box just for a good chat experience. Then what we would do is interact with a vector database and have that vector database send back resources from our company that were potentially interesting in the conversation, but we'd inject that directly into the prompt and then the bot would use. So that was a really quick win. Vector database, Streamlit application. The next stage of, okay, how do we put a database behind it? How do we interact with multiple agents? And getting into states and all that kind of stuff is sort of the next evolution to it. But I do think there's a ton of quick wins out there in areas like marketing, in areas like partnerships, in areas like customer success. What hinders some of that is a company's particular stance around data and where it can live and how do we have to control all this and what we put in. So there's several different factors to look at, but yes, plenty of quick wins I think.
Juan Sequeda [00:11:56] So let's dive into those. So for people who are listening, they're like, let's go dive into what are the things that are just right in front of you that are great opportunities in those different domains that you said like marketing customers and so forth, what are those types of quick wins that would be transferable to any type of organization?
Brandon [00:12:14] You might have to cut me off on some of these responses here because I think a ton of-
Juan Sequeda [00:12:19] Let's go.
Brandon [00:12:20] When it comes to the world of sales, we live in a world where it costs twice as much money and it takes twice as much time to get half the attention of a potential buyer as it did 10 years ago. There's an active defense being put up against the attention of people combined with the data privacy stuff that's happening. It's getting more and more expensive to get quality people into a sales pipeline. So creating an experience using AI that is sort of chatbots 2.0. We saw companies like Drift and Intercom over the last 10 years do really well with these chat interfaces, but it was either real- time person or guided conversation. I think that what the AI is doing is saying, Hey, you're in control. If you give them an experience where they can chat with the AI about your company, you are in control. You lead the conversation. We're here behind the scenes if you want to have a chat, but you allow people to accumulate the information they need at their pace to make decisions when they want to make the decisions. That positions you as a company to be sort of in line with them versus banging on their door and sending emails and whatever. I think in the sales arena, there's a lot of opportunity there. I think in marketing, a lot of marketing is writing copy. It's not uncommon for marketing companies to be putting out hundreds of landing pages. So creating or using an AI to help craft these pages, or at least version one, we created a bot that would basically spit out a landing page if you gave it a topic, a company, an industry that was optimized for SEO and ready to be plugged into a website, tons of opportunity with copywriting I think on that side. On the customer success side, I think there's empowering your agents to know where all your best content is. A lot of companies I've worked at have stuff spread out all over the place. So how do you more quickly get the customer success directors the information they need to make a better customer experience? Yeah, I'll stop there. I can go into probably some more, but I think there's a lot of stuff like that that can be done fairly quickly. When I'm talking quickly, I'm talking in afternoon to spin up one of these apps.
Tim Gasper [00:14:42] Yeah. Can you talk a little bit about the time process here? Because I think a lot of folks who maybe aren't as steeped in AI application development or some of what's possible right now might look at some of these generative AI applications and say, oh, well that seems like a big investment. I'll need to put a big team behind it. It could be months and months before we see the value. This sounds risky. What's been your experience in your best practices around how to actually get some value from these things?
Brandon [00:15:11] So I saw a PWC report that came out a couple of months ago that was talking about how companies, especially FinServe companies in this case, should work with AI inside their companies. They had a whole slide dedicated to the people structure. You have a director reports to the board and you have these other team leads, and then they each have a data scientist and they each have a UI person. And at scale, I think that's amazing. That's very expensive. But part of the magic of AI is that you can do so much with so much less. So for me, I'm a little bit of a different archetype. I can code and I do design and I can do sales and I can do marketing, so I can get a whole bunch done by myself. But really it's like anything else. There was a time that I was helping my brother. This is maybe a terrible analogy, but I was helping my brother remodel his house, and I know nothing about remodeling and flooring and all that kind of stuff, but as we sort of stripped away the different things in his house, I noticed a lot of mistakes and cover ups and imperfections and things that somebody else did. It's kind of like, oh, well, once you tear something down and demystify it, then you realize that you can make progress just like anybody else. I think AI is no different. So once you kind of demystify some of these new terms that come out and people are saying a lot of words that don't really mean what we think they mean. So we're kind of defining the vocabulary as we go, but once you strip some of that down, you can kind of realize that this is anything else. In fact, I would say that programming or creating AI driven applications is, and Dave kind of introduced me to this concept, it's more akin to negotiating or having a conversation with the technology than it is this kind of binary if this then that thought pattern that we have in the past. So back to that breaking these mental patterns that we have. When you think about AI application development, if you can just get a few of the key concepts, you can really start to make some progress quickly.
Tim Gasper [00:17:16] That's super fascinating there. Go ahead, Juan. Yeah.
Juan Sequeda [00:17:22] That was a profound analogy. I really liked that. At the end of the day, it's like you think it's so complicated, but no, just start little by little and you can go figure it out.
Brandon [00:17:31] Yeah. If you can just also wrap your head around that at the highest levels, at the Jeffrey Hinton 60 minute interview levels founder of AI neural networks, he even says that, Hey, we don't know what happens when it goes into that middle layer and comes out. So if those guys don't know, I can make some progress as well.
Tim Gasper [00:17:51] Yeah, the programming paradigm here is very different. You kind of just said it there in your previous segment there in terms of it's much less deterministic. So what's that experience been like as you're working with LLMs and trying to... almost like you're training the intern and trying to tell them, Hey, you stay within these boundaries and here's your script that you need to leverage. What's that experience like?
Juan Sequeda [00:18:21] To add to that, can you just walk us through what you could do literally in an afternoon? What are the things you'd be doing? How would you be doing that?
Tim Gasper [00:18:29] If you were trying to tackle one of these applications, what's that process you're going through?
Brandon [00:18:33] Yeah, so first part of that question was... Tim, what did you say?
Tim Gasper [00:18:43] Is around as you're-
Brandon [00:18:48] Working with the AI?
Tim Gasper [00:18:51] Yeah. What's that experience, that programming, that experience of negotiating with the LLM that you're going through this process?
Brandon [00:18:57] It's equally magical and sometimes frustrating because the nature of the LLM is to produce something that didn't exist. So there's times when you're negotiating with it and it just won't listen. So you have to kind of figure another way around it. A lot of that's the prompt engineering piece of what people talk about, but it is different because anything's possible. Again, deterministic sort of legacy way that people do things has been this, if this, then that sort of thing. Even the most complex state machines are, even if this answer is here, then go there. So what AI allows you to do is to really just be much more free- formed in the way that you're interacting and building these applications. So I think we all over corrected in the beginning to like, okay, chat will be the UI of the future. I don't think that's the case because some people need a more guided user experience in there. But to then answer your question, Juan, there is some foundational things that can be done to make AI application development easier. One of the problems that we ran into early on is we created this chat experience on our website for potential customers of data. world. They would come in, and I mentioned this in the beginning, and they would have this conversation, we'd recommend content to them, and then they would accumulate that content and then go on their way. But what we did to do that is we actually went out and scraped our own website using a Python, I think it was Beautiful Soup or something, and we got all of that content down. We then, and I say we, this was really Brian's work that he did after that sort of first interaction we had at that happy hour. He would get the content from the scrape. He would then take a summary. He would go to OpenAI and ask for a summary of that content. Then he would take that summary, send it back to OpenAI and get the embeddings. So the embeddings are the numerical representation of where this content lives in the AI verse. Then we had a huge file inside of that application that was about 32 megs of JSON. That was all of our content, URL description, title and all that. That's what we would use to feed the app. Well, that process is not sustainable if you've developed 8, 9, 10 apps because now you've got all these files that you need to update anytime a piece of content changes. So another guy at our company built something we call a vectorizer, which was basically a service that would ingest this stuff nightly and then expose it to an endpoint. Then you could build apps on top of that. So when I say you could do it in the afternoon, assuming you've sort of figured out some of these fundamental pieces, then you're really cranking. You're picking up one application, putting it down somewhere else, changing the prompts, changing the way it interacts with the vector database and the content, and then creating a new experience. So that's how our CEO always says the solutions that we're creating now are solutions. People ask, how much faster are you creating it now than you would've been able to? And it's like, you can't even say that because it would've never been created in the past. There just wouldn't have been enough time to invest and get these things up and running. So how do you calculate ROI on? It would've never have been created and it was created. I don't know, but that is what I'm finding quite a bit.
Juan Sequeda [00:22:13] That's an excellent sound bite right there, because people are always obsessed. We should be, right? How do we know that this is being successful? How are we measuring this stuff? But I think we are really pushing the barrier here, the frontier that we're doing something that hasn't been done before. So then leads me to ask two questions. One is, how do you suggest people get the buy- in to go start doing something like that? And what is the minimal set of people resources that are needed? You're talking that you need people who to go set up something, understand how to go set up a vector database for these types of stuff and so forth. What is the minimal stuff that you can need to do, and how would you get people to get buy- in for this?
Brandon [00:22:57] Yeah, maybe this isn't a good answer, but my particular personality has always been to just do it and then show people, make it. Then it's very hard to get people motivated behind somebody else's idea. I don't know how many times I'm like, Hey, wouldn't it be cool if blah, blah, blah? Even if they're excited, they're not as motivated as you, but when you show somebody and there's this sign of momentum and they see what is possible, then people get motivated. I gave a talk a long time ago where I said, Hey, I just created something called chair pants. They're pants that you can sit in, they turn into a chair. Does anybody want to invest in that? The audience, nobody raises their hand. Then I said, what if I told you that these are RDM production? We've got to deal with QVC, we air next month, what do you think now? People kind of sat there and it was like, well, that's because there was momentum. So if you can create some sort of momentum, then people want to get on board. But talking about it and saying, can you please support me, has never been as successful as me going out and actually creating something. That's the first part. The second part in terms of what is the minimal amount that you can do? Well, that bar just got lowered with Chat GPT and these apps too. So I would challenge people that think, well, I don't have... Our natural inclination is people is if somebody says, look, what I did is to say, well, I can't do that because X, because Y, because it's easier to admit we can't do something than to try and fail at it. So I'd encourage people now really with the tools that they have available to see how much they can do with their AI assistant, but to specifically answer the question, I think some level of coding is necessary to understand how these different things work and demystify it, and then some understanding of the business. So understanding how to interact with people, how to have empathy around what they're going through, and then having a bit of coding skills is enough to sort of get something off the ground with your AI assistant next to you.
Juan Sequeda [00:25:09] So you have a very unique background and you combine all these things. I wonder do we need some sort of a unicorn style person who can do so many of these different things to get this or need a couple of people or?
Brandon [00:25:27] Yeah, I would think in most situations, if you combine somebody up who is an operations person with someone who's a coding person, even with part of their time just to get some things off the ground and some momentum at a loose set of goals, " Hey, could you go and talk to the marketing team about if there's anything that you think AI or and custom application development could help with?" And then turn them loose and see how it goes. But yeah, I think with two people, you could start to make a pretty good dent.
Tim Gasper [00:25:59] So maybe a more business minded person paired up with somebody who can do some fast iterations, some coding, things like that, that duo could do a lot.
Brandon [00:26:09] I think so. Empowering them to act like consultants within the company to go and just say, " Hey, what would make your life better? And please clear your mind before we have this conversation because we can do things now that we couldn't do in the past."
Tim Gasper [00:26:25] Yeah, no, I like that. So we hit around some roles and we hit about low- hanging fruit. You mentioned technology a couple of times as we've gone through this. Do you see sort of a best practice stack that you've started to see get implemented here? You mentioned a database. I assume you're talking Vector database. You mentioned a vectorizer, right? I'm curious if you see, here's my toolkit.
Brandon [00:26:58] Yeah, and I'll tell you my toolkit right now. I'll tell you where I came from and then what I'm doing right now. But I'll preface this by saying I'm not a... and you guys know this, you work with me. Nobody would look at me and say, you're a coder. You're a developer. I can do that stuff, but it is not my natural skillset. But I've taught myself and used AI and I've become somewhat proficient at it. So the stack that we came from, I mentioned before, was a stream- lit application. Then it wasn't even a vector database. It was a JSON file. Eventually we moved that to this vectorizer service that somebody else created. I'm not exactly sure what that was built on, but my stack recently, and I'm trying to actually rebuild part of that vectorizer. My stack right now for better or worse, is I create a react front end. I use a Node Express backend and MongoDB behind the scenes. I don't have having to deal with database migrations and all that, and it kind of happens on the fly. Then Mongo actually just released a vector search capability too. So that's my sort of quick go- to stack for spinning stuff up.
Tim Gasper [00:28:14] Nice. That's quick and easy. It's basically sort of a web stack just focused on being able to churn quickly out these AI applications.
Brandon [00:28:27] Another crazy thing. In the past, there's always this sort of, I'm a developer, so I'm not a designer, I'm a designer, I'm not a developer. Those lines have blurred over the years. But what's crazy now with AI is if you're a developer and not a designer and you ask the AI, " Hey, create an interface, it's a chat interface, give me the HTML code and style it using Tailwind CSS," it'll look better than most of the things that people have put together in the past. So it's pretty cool.
Juan Sequeda [00:28:55] I think one of the honest no BS takeaways here for me is no more excuses please. Yeah. If you're like, oh, that's not for me, the bar has now changed.
Brandon [00:29:11] I think what we're seeing is it's kind of a unique experience to watch from my side because people interact. On one side of the house there's people like me and a couple of guys that I know that are so in this and realizing what this means for productivity and learning and the way other side, there's people that think it's going to take over the world and it's an evil presence and all of that. Then most people are kind of in the middle, but the excuse of I don't know how to do it or I didn't have information, is being chipped away at because if you want it's there within seconds and if it didn't work, ask it again. You know what I mean? So yeah, for me, AI has aligned with my personality very well because I can get the information I need very quickly.
Juan Sequeda [00:30:00] So one of the things is, we talked originally of like, oh, how would you start off and like, oh, I need to show the ROI, but how do you show the ROI based on if I didn't exist before? But you have posted and you've been blogging a lot about your progress around everything you've been doing here, and you have talked about the productivity increase and you've actually measured things. So what is the process of... How are you defining these interviews or whatever to go figure out what is being measured for productivity gains and what are those productivity gains that you're seeing?
Brandon [00:30:33] Yeah, so yeah, I kind of tongue in cheek made that how do you measure ROI? But the reality is that inside of a business, you need to figure out how you're going to manage ROI because that's how you get future investment. So I think there is a couple of different ways that you can do it. One is if you're building these applications, just using kind of standard practices for tracking whatever your metrics solution would be. So all of our apps to Mixpanel. We also have a lot of these apps hooked up to Slack channels where you can get some anecdotal feedback about what's coming in. But the state that we're at, which is very early in this process, the best way I could figure out how to measure this was just through a simple survey and then structuring those survey questions with some sort of intelligence behind them. Meaning you can take a survey and get any answer you want, but making sure that your questions are leaving room for the truth to come through. So we've done a couple of different surveys and what we found is that the people that you're not going to launch a tool and everybody's going to use it. You still have these adopter. Some people, change management is tough. So for the people that are engaging with the tools that we created and each tool is so bespoke, it doesn't benefit everybody. But for the people that are engaging with them, we're seeing a self- reported productivity lift of around 25%. That's a pretty big number just to throw out, and you hear all of these stories about some companies reducing their costs by 70% on customer support tickets and all these other. So for us, it's been a series of surveys at this point that we take on a quarterly basis and then compare the results. Some of the goals we have internally are about broadening the use at this point, not necessarily creating more productivity gains, but how do we empower more people and build more tools to suit their needs?
Tim Gasper [00:32:31] So you're not necessarily looking to get 25... you said 25%, is that right?
Brandon [00:32:35] 25%, yeah.
Tim Gasper [00:32:36] So you're not necessarily looking to go from 25 to 30 or 35 or 40. You're looking at, well, what's the next business process I could roll this out to? What's the next group I could bring this to you, right?
Brandon [00:32:45] Yep. That's the stage we're in now, and I think I've created seven or eight different tools at this point. I would say three of them get very high usage. So it is a process of learning, okay, that wasn't a hit. Well, why? Then going back and figuring it out. Sometimes it's a tweak. Sometimes they don't need it as much as they think they did or whatever. That's all part of the process. But currently we're at, okay, this worked for a couple of different select groups. How do we get more groups involved in that?
Juan Sequeda [00:33:15] This is interesting because people start saying, oh, I have this problem. They're like, well, giving you a solution for it. So either you figure out why the solution is not working or it's like then that really wasn't a problem then.
Brandon [00:33:27] Yep. That's a lot of it too. There are people that are very excited about the art of the possible and what we can do, and then at the end of it's like, that is cool. I'm just not using it as much as I thought I would use it. But hey, that's part of every software development I would imagine.
Juan Sequeda [00:33:43] So what I'm realizing here too is that this is one of them is to improve productivity. How can we use AI to get to improve the operations? But the opportunity here thing is if I zoom out, so that you are really under codifying and cataloging basically the processes within an organization and figuring out what is actually hard, what is actually easy, where are clearly the bottlenecks, how can we go groove things? So at the end of the day, it makes total sense that this is the title AI ops, but it's more operations first and how this becomes the secret tool for the CEO's office of like, God, this is how we're going to be so effective in our company and be much more faster productive.
Tim Gasper [00:34:31] You're analyzing all the business processes across the organization and looking for optimization opportunities.
Brandon [00:34:38] I really find two things. One, I find that there's two or three groups working on something similar.
Juan Sequeda [00:34:43] Oh, that's the other one too.
Brandon [00:34:46] And instead of creating a solution, I'll get all those people in a room and say, " Okay, how do we solve this across the board?" And then other things you run into are true sticking points that AI can't help with, but are real problems that need to be solved. For instance, in a customer success scenario, there's a lot of ideas around like, okay, can we route these tickets that come in? And if they're an easy response, we auto respond with AI with a human sign- off. But if they're not, they get raised to this and then we send them this article. Then you start asking, " Well, okay, where is all your content? You have 15,000 documents. Where do those live?" And oh, some of them are in Microsoft Word, some of them are in Google Drive, some of them are in our JIRA instance. Then, okay, well, how do we get all those organized and understand what they are? Especially how do we look not for PII, but for just stuff we wouldn't want to be sharing publicly. And it's like, all right, well that'd be great. I know how to do it. We send it all off to an LLM and it'll help us organize and sort what that content is. It's like, oh, but we can't send the sensitive information to the LLM to tell us it's sensitive. So how do we get, okay, well now we got to get into, do we host our own LLMs? So there's certain you go deep enough sometimes you're like, well, we can't solve this with this set of tools I've been using, but it brings up a bigger conversation around how should we organize and clean and understand our data?
Tim Gasper [00:36:13] Some of the same problems that plague people also plague AI. It's not like, oh, all the knowledge is disorganized. Oh, well, let's make an AI app or the information's organized. I can't make the AI app work. Right?
Brandon [00:36:34] Yep. I think what I see here at our company is I am overlapping with a lot of what the product is working on too. So I often kind of come together with the product and say, Hey, I'm doing this and this and this and what are you doing? I'm doing this. Okay, should we merge those things together? Well, not yet. Let's talk about that in a quarter. So then I go off and create and then we come back together. So there's a great deal of inspiration and collaboration that happens through that process. But yeah, you're exactly right. The old trope of crap in crap out holds up, especially in this world.
Juan Sequeda [00:37:14] Throughout this podcast over the years, Tim, I've always brought up, we need to have the, what do I call it? The psychologist, the person who's interviewing people, understanding how the business works and so forth, right? Therapist, there we go.
Brandon [00:37:32] Data therapist. That's an interesting-
Juan Sequeda [00:37:34] Yeah, I think one of the things we're planning to go do too, Brandon, I think is have all our transcripts of all inaudible and people go search that. So you'll search for data therapists and we'll find the episode or we go find that. I think this is a really important thing because you want to be able to understand what the typical example, what do you mean by customer? Oh, there's so many different national customers, so let's go talk about this. Let's go catalog. So that's the therapy part. And B, have empathy with people. It's like, yeah, I get it. This is so complicated and so forth, and you're doing that work already, and I just find that this is a huge opportunity for companies who are forward- thinking, want to be forward- thinking. You can now start, you have your data and analytics type of things you want to go do you want to be able to increase operations, the efficiency of your organization, you want to be able to go bring AI. All of this is merging. I'm seeing this merge together and it's a big reminder that I think data and AI, they work hand in hand. And you think about it within an organization, it should probably actually end up reporting to the COO office because it's all about operations, the brain of the organization, how we could do this better. Anyways, this is a very enlightened conversation because I'm seeing a lot of huge opportunities of how companies can evolve going... which leads me to, before we wrap up going to our lightning round, I was like, what's next? Where is this all going?
Brandon [00:39:09] Who knows? It's happening so quickly. I think you'll see sort of a similar pattern that you've seen with other technologies. At the end of it, this is a technological advancement and every time that happens, you have people who think it's terrible and people who think it's great first time fire was ever created. I'm sure somebody is like, we can cook something great and somebody else is like, that's going to burn down the house. What do you do? So there's this kind of first stage and then what we see companies typically do, I think, and this happened a lot with the explosion of data at McKinsey staff that flies around every couple years that people quotes a couple years old, but it's really older that 99% of the data that was created in the world was created in the last three years. So companies rushed to get their hands on it and then they sort of figure out what they were going to do with it. So they invested in data lakes and data warehouses and all those, and then they said, " Okay, well now how can we turn that into competitive advantage?" Then they invested in the data scientists, right? Then they invested in technologies around. I think we're seeing a lot of the same stuff with the AI. Nvidia, which is a video game company, was fortuitous enough to make a change towards AI and their parallel processing and they're sold out. Their stock is through the roof. So it's evidence that companies are acquiring the technology they need to be able to run their own LLMs and all of that. Then I think they'll invest in the people and the process and follow suit with what we've seen in the past. I think we're a little bit ahead being a younger company and trying to move fast, but I think you'll see the role of AI ops start to pop up as we're describing it inside of companies in the next two years.
Juan Sequeda [00:40:57] Do you think it's going to be... so you're saying it's going to be more of the smarter, smaller startups? What ranges and actually is it going to be more tech companies or you think actually non- tech companies are going to see the opportunity here or I don't know?
Brandon [00:41:12] Yeah, I think some industries will move faster than others, but I don't think there's a single area that AI won't have some sort of impact. So true to form, you'll see smaller companies move faster and bigger companies move slower for good reason data security and policies and the way that they do things and the amount of stuff that they have. But I wouldn't be surprised if the term AI operations or you're already seeing some Chief AI officers and stuff pop up here and there. So I think it would be a shame if AI operations was tucked under some IT function and just lived there. I think elevating it to an organizational level provides a ton of leverage as you start to do these exercises and understanding the business and where everything is and tying the pieces together.
Juan Sequeda [00:42:06] All right Tim, I think we've got to go to lightning round. This has been a fantastic conversation because it is really given me a lot of aha moments and excited about where the opportunities for companies to actually be leaders and change and do something revolutionized. I mean literally do something that's going to change the way we've always done.
Brandon [00:42:28] inaudible with this. I don't know what they're waiting for.
Juan Sequeda [00:42:29] I'll tell you another one but all right, I'll kick it off. First lightning round question. As you've developed these AI applications, what do you see as the biggest limiter? Is it the hallucinations or something else?
Brandon [00:42:45] I'll answer that in two ways. One would be wrangling the hallucinations and trying to get the information that prompt engineering to a level that is not just giving you the answer you want once, but doing it over and over again. Then I would say the second biggest thing, or equally is having some sort of infrastructure set up that allows you to move quickly. So whether that's your stack or your vectorizer or cleaning your data or knowing where it... having that infrastructure set up correctly.
Tim Gasper [00:43:16] Nice. All right. Second question for you. So one thing that I wonder is as more SAS applications start to come out in the AI space, one example I know that we've been running into, we're seeing around is this thing like Glean where it is a natural language search on your files. Do you see in this battle between kind of SAS AI applications and then more of the bespoke applications that you're building, is one side of the equation going to win out?
Brandon [00:43:49] That's a really good question and I've thought about this a lot. I don't know the answer to that question because the ability for people to spin up these solutions and as a SAS model is fantastic and quick. I don't know that because of the advances in AI and being able to build bespoke applications at much less of an investment. So I don't know what that looks like going forward, but if we look at take sales as an example, there's got to be a stopping point to investment at some point, meaning there are so many tools out there for a sales organization to use to layer on. There's your CRM and then there's your third party intent data and then there's your information data. So you have all these layers and every year something comes out and we have to buy it, now it costs$ 80,000 to get a deal into the qualified stage of the pipeline. So at what point does that explosion become too much and it's cheaper now to create these bespoke applications internally?
Tim Gasper [00:44:46] Yeah, that's fascinating. We could have a whole conversation about that. The cost dynamics as well as the rate of tech innovation. This is going to be interesting to see it play out.
Juan Sequeda [00:44:58] Then the organization's going to be nimble enough and who say, I am going to invest in these AI ops. Well, I think it's actually be cheaper for me to actually go do this. So it's a bit of a, I'll build it myself, but it's a balance of build versus buy. I could buy all those things, the foundational ones and I'll buy some of the data, whatever to go do that. I think this is going to be interesting. All right, next question. Should the leaders of AI ops end up reporting to the COO, the CFO, or actually just go end up to the technical side, end up to the CTOs?
Brandon [00:45:32] I'm probably biased, but I would say the COO. That's the way that we have it set up here. I think for me, it puts an emphasis on the people and the process first. Then how do we use technology to support that? I'm assuming it could be successful in different places. My personal opinion is I like it reporting to the COO and the conversations we have around this are not technical. It's about impact, it's about people, and I think that's valuable.
Tim Gasper [00:46:01] All right. Final lightning round question for you, Brandon.
Brandon [00:46:03] Yeah.
Tim Gasper [00:46:04] As you do these interviews with people to figure out the right AI apps and how they can make an impact, I see you acting a lot like a product manager. Do you see a key skill set for a leader of AI operation needs to be almost like a product management skill set?
Brandon [00:46:24] I don't know that I would pigeonhole into product management or this or that. I would say more from a personality type. I think curious people who are good communicators would do well in this role. So naturally being curious and wanting to know more about things, but then also on the communication side, being able to structure the conversation in a comfortable way so you can get that information. But also the communication piece is important for interacting with the AI understanding language and how words and orders and those sort of things. So I would say the two most important curiosity and communication.
Tim Gasper [00:47:01] That last piece that you just said there is deeply fascinating. I think maybe something that this whole every knowledge worker in the world is going to have to think about is not only now do I have to be a good communicator to people. I actually know how to be a good communicator to the LOM.
Brandon [00:47:19] I'll tell you this and I'll leave you with this. I mentioned in the very beginning, learning to unspeak Google. It's funny, I watched my son who grew up with all this technology. He's 15 now, and the way he types into Chat GPT is the way we all type into Google. So we learned, we didn't know it, but we learned how to search. Next time you're doing a Google search, watch how you type. It's not a sentence. It's a series of weird words put together. We don't have to do that anymore with AI. You can just speak. So that transition is going to be interesting to watch.
Juan Sequeda [00:47:49] Yeah, this is the human behavior part. So Google changed... this world before Google, right, Alta Vista and all these stuff is it was learning how to do advanced search and go into the advanced search stuff. You could go register in a community college thing to go learn how to go learn Booleans and stuff. It was 93, 94 and then Google changed that. So we went to keyword search and now we're have to go through this paradigm shift and we'll see how that happens.
Brandon [00:48:23] Yeah, it's not uncommon for a question of mine to be... I've recorded my voice before just rambling about something and then hit send. That's way different than optimizing keywords.
Juan Sequeda [00:48:35] This will be interesting. I remember Tim last year, we got lucky. We went to an event with Tim and we asked him, " So with Chat GPT, do you think that's going to change the interface to the web? Now it's going to be conversational instead of keyword through search." I was surprised to hear his answer saying yes.
Brandon [00:48:56] See, I don't know. I think it will trend that way, but I have been kind of discovering a subset of people that don't want that burden. They would rather check some boxes and hit some dropdowns and stuff. So I think it will be a combination. But yeah.
Juan Sequeda [00:49:10] We'll see. All right, Tim, take us away with takeaways.
Tim Gasper [00:49:15] Yes, so much good stuff here. So we started off with honest no BS, what is AI ops? You start off with like, well, we're figuring it out, but it's really about trying to leverage this generative AI technology to make a business impact, to optimize for various business processes to make someone's job easier. You talked about some different examples where you would go and you would talk to different parts of the organization, sales, marketing, customer success, learn how they're doing things, find friction points, figure out how to improve their functions, get the buy- in needed. We clarified that there's a difference between the kind of AI operations that you're doing. I think then some other ways that AI operations, I'm doing some air quotes here as being used out in the space. I think a lot of people think of it purely technically like ML ops, DevOps, but this is really around people process first, impact first, value first, and then technology is a supporting pillar for that. When you talked about interviewing all these people, you really were trying to have a free flow conversation and how important this was to really understanding what people need, the curiosity that you talked about and really learning how to unspeak Google. You mentioned that a few times throughout. Our generation, we train ourselves on keywords. That's just how we think. This is a new paradigm. How do you know when AI can be used where you said there's three different levels, there's these quick wins, this low hanging fruit. There's a little bit more advanced stuff where maybe now you need to use a vector database or some multi stage agents. Then there's the third level where you really want to interact with enterprise data. I think that's an interesting kind of framework, almost a maturity model to think about. We talked about quick wins and how important that is. One example is in marketing you have to create lots of landing pages. That's a great application of AI that can have a big impact fast. And you said that you had a stack that you kind of worked with here that started with things like STREAMLIT and a Vector database, but it's been evolving and you talked a little later about how that stack is evolving. Then finally, before I pass it to Juan here, how do you start? You said create momentum, so you just have to get into it, you just got to start doing it. The bar is now lower than it ever has been before because of Chat GPT. There's some coding required, but not a ton. You said you yourself don't consider yourself a coder, although obviously you have some skills there and maybe you're not a unicorn, right? Or you don't have a unicorn that you can put into this role where at least at least pair somebody on the business side with somebody on the technology side that could be a really powerful duo to lead AI operations. There's so much more. You said act like consultants. I thought that was a great phrase, but Juan, over to you. What were your takeaways?
Juan Sequeda [00:52:10] Yeah, so interesting on your stack has evolved. Right now, a lot of this can be learned by using Chat GPT and they go program around it. So react is your front end, right? Mongo DB, Mongo now has all this vector search. So it's really easy to get this up and running. Measurements. So the reality is the business is that you do need to figure out how you're going to measure the ROI around this stuff. So when you create these apps, you're implementing tracking web analytics like mixed panel, putting things into Slack, and you get some anecdotal feedback in there and you really have these surveys. So you're really structuring the questions to get the truth around the impact. Understand that productivity gain with what you've been seeing is around a 25% productivity gain, and not necessarily looking to drive a higher percentage. So oh, that 25, let's get that to 30 and 35. It's right to focus in on just increasing the adoption, the usage of that. Then just understanding new business processes or departments where you can go in and expand that use cases. I noted there that when you don't see the results, you expect either neither to drive the adoption or that solution doesn't fit, or the problem wasn't really strong enough to actually deal with it. So I think that's something that we start then literally understanding more of our business. So you start cataloging the processes of the business and finding those problems or those opportunities. Interesting findings that you have while talking to people is that you find that there's two different groups doing the same thing, right? There's duplicate work, duplicate processes. There are things that are real problems, but the AI can't really solve that needs to be escalated in different ways. As you are looking to empower the AI, the same problems that plague people, plague also AI, you have to deal with the sensitive data, the private data to consider and so forth. At the end of the day, where is this all going? Let's acknowledge that this is, I like the analogy like fire. Oh my God, this is amazing. Oh my god, I can bring down our house. So there's two different routes if think about it. Your point is that AIOps will be seen more and more in a couple of years. Obviously smaller companies will move fast and it would be a shame if it's under it. How did we do anything we missed?
Brandon [00:54:08] I thought you did great. I don't know how you did that. Amazing.
Juan Sequeda [00:54:12] No Chat GPT was involved.
Tim Gasper [00:54:14] Brandon did not create an app for us to do this.
Juan Sequeda [00:54:17] But we have been trained for almost four years. All right, to wrap up here quickly, Brian, three questions for you. What's your advice about data, about life? Whatever you want. Who should we invite next and what resources do you follow?
Brandon [00:54:35] Advice. I'll expand on something I said earlier is there's a lot of people who want to do a lot of things and have already defeated themselves in their mind, and this advancement in technology is going to empower a lot of people to do things they couldn't do. So lean into that would be my advice. Who to have next? Rachel Woods. Have you had Rachel on the podcast yet?
Juan Sequeda [00:55:04] We haven't. I've been reaching out to her.
Brandon [00:55:06] Yeah, Rachel's phenomenal. She's a very forward- thinking person in this space and knows a ton and has had a lot of success. So she's been a good friend and mentor on this process as well. So I'd say Rachel. What was the third question?
Juan Sequeda [00:55:20] What resources do you follow? People, blogs, magazines, whatever.
Brandon [00:55:27] This will be a good way to end. I created a bespoke application for myself that goes out and uses something called ExaSearch and then also Google custom search to find news related to AI operations every day and summarize those into blurbs that I read. So they're coming from all over.
Tim Gasper [00:55:47] That is a good way to end things off. You created an app for that.
Juan Sequeda [00:55:53] I love it. Brandon, thank you so much. I'm really excited that we had this conversation because it's going to... hopefully everybody who's listening. They're seeing there is a huge opportunities and you're leaving money on the table and time on the table. Get that back and let's be more productive. So thank you so much. Cheers.
Brandon [00:56:10] I appreciate it.
Juan Sequeda [00:56:11] Cheers, Brandon.
Brandon [00:56:12] Have a good one.
