Tim Gasper [00:00:00] Hello, everyone. It's time once again for Catalog& Cocktails, presented by Data. world. It's your honest, no BS non- salesy conversation about enterprise data management, with tasty beverages in hand. I'm Tim Gasper, longtime customer guy, product guy, data guy at Data. world, joined by Juan Sequeda.
Juan Sequeda [00:00:18] Hey Tim, it's Juan Sequeda, the principal scientist here at Data.world. And as always, it's a pleasure. Today, we are actually now live officially on a Wednesday because last week we did it on a Thursday night and whatever, and middle of the week, end of the day. And super excited to keep the whole day- to- Day Texas stuff going. We had a phenomenal weekend. Truly, if you were not in Austin last Saturday, you missed out. I needed to clone myself with so many talks and so many people to go talk to. And one of the talks that we had at Data Day Texas, I was super excited it all lined up, and she's here today is, Jessica Talisman, who is an information architect at Amazon who has so much experience in taxonomies and library sciences, obviously information architecture, a very long career around this. Jessica, it is super, super awesome to have you here. How are you doing?
Jessica Talisman [00:01:07] I'm doing great. Thanks for having me.
Juan Sequeda [00:01:10] Well, I'm super excited. Let's kick it off. What are we drinking and what are we toasting for today?
Jessica Talisman [00:01:16] Well, I am drinking coffee, thanks to a free mug from Data Day Texas. Yeah, coffee with a little bit of coconut milk in it.
Juan Sequeda [00:01:27] That's a good one.
Tim Gasper [00:01:28] That sounds good.
Juan Sequeda [00:01:30] And what do you want to toast for?
Jessica Talisman [00:01:32] I want to toast to the exchange of ideas. I think that was what was so dynamic and amazing about Data Day Texas, it was an ongoing dialogue, nonstop thinking and it's exchange of ideas and learning. That's what I'm toasting to. Yeah.
Tim Gasper [00:01:50] I love that.
Juan Sequeda [00:01:51] Cheers. Tim, how about you?
Tim Gasper [00:01:54] I am drinking a martini but it's made with Empress Indigo gin, which has... What is it called? A butterfly pea blossom, which means that it has a blue color when you don't add anything to it. But if you add lemon, an acid to it, it actually makes it turn a purpleish pink color.
Juan Sequeda [00:02:23] Oh, look at that. Very cool. This is a live experiment.
Tim Gasper [00:02:28] It's a little party trick.
Juan Sequeda [00:02:28] For everyone here, not only listening to us, you should watch the video to go see that.
Tim Gasper [00:02:32] Yeah, went from blue to pink. Pretty cool little gin there. I thought I'd do a little trick for today. I will cheers to the exchange of ideas as well. And actually, unfortunately I wasn't able to make it to Data Day Texas this weekend. And so I'm feeling all the FOMO. I want to catch up on all the slides and all the good things that came out of that. Such an amazing crew. Cheers to the exchange of ideas.
Juan Sequeda [00:02:54] And I just went to the bar and I said, " Well, there is Aperol and there's gin." I just Googled a couple of things and yeah, just an Aperol gin tonic. Actually, it's a pretty smooth, relaxing drink. And just a reminder, this is great, why we need to go to conferences and fortunately we get to have the possibility to do this in person because a lot of the hallway conversations, the happy hours, the dinners, that's where a lot of the bigger exchange happens, not just at the talk in the five minutes for Q and A. Cheers to that, and hopefully people get the opportunity to have more face time with all the folks that you get to interact virtually. Cheers.
Jessica Talisman [00:03:30] Cheers.
Tim Gasper [00:03:31] Awesome, cheers.
Juan Sequeda [00:03:33] All right. Our warmup funny or warmup question here is, what's your favorite organizational activity, something you like to go do or involves organizing?
Jessica Talisman [00:03:45] I think it's organizing junk drawers. I love organizing junk drawers and making sense of that mess. I am known for buying little organizing trays and going after it in drawers at home, my kids' drawers, everywhere. That's my task.
Juan Sequeda [00:04:08] I can see that, I'm actually going right now, yeah, my drawers need some organization.
Jessica Talisman [00:04:13] There's no such thing as NA or other, if you care about it, it's going to go somewhere.
Juan Sequeda [00:04:18] Miscellaneous is not okay.
Jessica Talisman [00:04:19] No, it isn't.
Tim Gasper [00:04:20] I don't care about my drawers, they're in such disarray. Jessica, you would not like my drawers.
Jessica Talisman [00:04:26] I'm happy to help you out.
Tim Gasper [00:04:28] All right, inaudible.
Juan Sequeda [00:04:29] Tim, how about you?
Tim Gasper [00:04:31] For me, my favorite organizational activity is to organize my desk, because the room can be all messy, clothes can be everywhere, whatever, but as long as my desk is nice and neat, as long as my pile of books is in a nice neat pile... I know for some people they would say that's a mess. No, that is an organized pile I would say, then I'm happy.
Juan Sequeda [00:04:52] Is that an oxymoron right there, organized pile?
Tim Gasper [00:04:54] An organized pile, yeah, maybe so.
Juan Sequeda [00:04:57] Well, mine is to organize my wine, to have my little cellar with all my fridges and where to put things and then maybe I'm going to change things a little bit here or is this really up to date and everything? And then there's a lot of ...
Tim Gasper [00:05:08] Is your collection pretty big now?
Juan Sequeda [00:05:09] Yeah, I ran out of fridges and rack space. Yeah. It's a good problem to have, I guess. I don't know. Well, my wife's pregnant too so she's not drinking wine. Yeah.
Tim Gasper [00:05:25] You're accumulating it faster than usual.
Juan Sequeda [00:05:27] We're accumulating and then I'm like, " I really don't need a... " I just ordered a case of this stuff. I really didn't need it. I don't have a place, I'm not drinking. Anyways. Whenever folks in Austin reach out to me who like wine, " Let's go drink some wine." All right, let's kick it off. Honest, yes, what the heck is information architecture and what does that have to do with AI?
Jessica Talisman [00:05:48] All right. Information architecture is the act of organizing information. It could be controlled vocabularies, taxonomies, thesauri, even goes into ontologies, but it's creating methods and systems for organizing information. And usually, that's done for the benefit of a front end experience. And it can be backend experience as well, internal operations, but it helps to support findability and discovery. It involves research and it involves cleaning data and knowing the stakeholders, knowing the platforms. It can be pretty technical. A lot of people think that information architecture is simply designing browse experiences. And that is one facet, maybe information architecture in the UX/ UI experience, but the more global information architecture is really about organizing holistically.
Tim Gasper [00:06:47] Yeah. I'm familiar with people talking about information architecture quite often in the context of a website. You go there and you're like, " Hey. Well, here's this section and here's this section." And then of course there's hierarchies. Sometimes you go into a section and there's more sections. I think probably a lot of folks who are listening who aren't familiar with the field in more detail probably think of websites. But what are some examples of other kinds of information architecture that maybe people don't think about as often?
Jessica Talisman [00:07:18] Even database structures can be information architecture structures. And so other systems, you mean outside of the enterprise or outside of a business, you might...
Tim Gasper [00:07:30] Yeah, it could be even outside of enterprise. Yeah.
Jessica Talisman [00:07:33] A grocery store, any store that has those signs or that gives you some map or idea of where things live, that would be one example. Obviously e- commerce, you see it everywhere. Even medical systems, tax systems, those all follow taxonomies or information architecture systems. Call centers, being able to route calls appropriately for customer service. Beyond chatbots, if chatbots are going to work, and this is dipping into the AI realm, there has to be some logical map to the domain.
Tim Gasper [00:08:14] Yeah, let's explore that more. What is the connection between information architecture and AI? And then specifically, you said chatbots. It sounds like there's something especially specific around chatbots that this ties to.
Jessica Talisman [00:08:31] Yeah. Chatbots on a very simple level, chatbots need a taxonomy in some information structure in order to help the person using the chatbot. It can route calls, it can route communication beyond phone calls obviously, so that you're giving AI essentially, or large language models, a map of internal systems. An enterprise, an organization, a company's information is unique to that company, it's like a fingerprint. There's no way for AI or a large language model to know how to make sense of any input without having that map. It knows what topics, what subjects, what actions, what event structures exist in order to post or classify information appropriately to the right node on a taxonomy.
Juan Sequeda [00:09:31] What I am really excited about how you kicked it off here is that you gave a very specific definition of information architecture, organizing information, including controlled vocabulary, thesauri, ontologies, with a method and system to organize information for the benefit of the front end but also the backend. And then when we go into the backend, you said even database structures. I want to get more into the AI but before we even dive deeper into the AI, information architecture, let's jump into data architecture and how this is related because be honest, no BS around all this stuff is that people throw out these words and people are more data architects and then you're information architecture, and they're making stuff up. Okay, let's demystify this and I think also acknowledge that these definitions, maybe we won't agree on them. And that's fine. Be honest. What is our definition for these words?
Jessica Talisman [00:10:27] Data architecture could be exclusively schema based definitions for a database. If you're thinking of a Postgres SQL or a SQL database, you're creating the architecture for those schemas, for the columns, for the tables, for ETL and ELT as well. And so if you're constructing schemas and looking at entity resolution from a database perspective, obviously you have to make sure it all checks out and that the flow of information can happen from the database and flow through the system in order to be useful to everyone else in the organization that's handling or managing data. Now, obviously not everyone is in SQL databases, relational databases. The bigger question is how do you make that data then usable to everyone else in the organization? Because not everything is in a SQL or Postgres database.
Juan Sequeda [00:11:27] And then the other big topic there is connecting this to the enterprise architecture. I just really wanted to inaudible. Let's start connecting this all together.
Jessica Talisman [00:11:35] And the funny thing is I've worked in all of these domains, so it depends on where you get hired. It seems like a moving, shifting landscape. And so enterprise architecture is usually centralized and deals with helping to structure the entire infrastructure of an enterprise, of an organization. How the enterprise communicates throughout to make sure that how the data basically on a higher level flows. Now, I have seen enterprise architect also advertised in job postings as a sales type job, which is interesting. If it's a sales type job and you're dealing with enterprise architecture, perhaps you're analyzing how a certain product that you're trying to sell works with the overall data architecture of an organization, and analyzing that.
Juan Sequeda [00:12:32] Now, tying this back to AI, one of the things that we are seeing is that with AI, you can do all these... We focus on the chatbots. I can just ask questions on all these documents, all these text that I have, but if you really want to connect this to how your organization is thinking, how they organize their information, well, you need to understand how it's being organized. What are these vocabularies? What are the taxonomies? What are all the semantics? And if you don't invest in that... And I think also the other part is like, " Well, it's already my data, my database schemas." I'm like, " Well, there's this whole... That's structured for... "
Jessica Talisman [00:13:13] A disconnect, yeah.
Juan Sequeda [00:13:13] There's physical stuff and then you're like, " How do people actually think?" I'd like to open the floor to you, what are the disconnects that you're seeing in what people think like, "Oh, this is okay. This is easy." ... when in reality it's much more complicated.
Jessica Talisman [00:13:30] Yes. Number one, what I'm seeing is people that are trying to derive an information architecture such as a controlled vocabulary taxonomy from ChatGPT, their starting place being trying to get an AI model to render a taxonomy ontology out of input that may be a presentation, it may be a collection of documents. And I'm not sure that that's a realistic methodology in ad tech. It's very common for organizations to just ask blank questions. It's like writing a blank check but you don't know where it's posted to. If you ask a large language model to create a taxonomy for you that's consistent and validated, last I checked LLMs do not have validation machines for checking information architecture components such as taxonomies or ontologies. Again, because that's a fingerprint of your organization, a collection and a holistic view of your entire organization, including classification schemes definitions, ontologies, knowledge graphs are for the win, you're able to truly structure what is unique and what is specific about your organization because how your organization defines something may be totally different from the way that the rest of the world or other organizations define that. LLMs don't know that. They can learn but that again, takes you telling or inputting that information in order to derive that meaning.
Juan Sequeda [00:15:16] But we can use it as a... And I think we should start using it for some inspiration to not start from a blank slate. I think that's an important...
Jessica Talisman [00:15:25] Exactly, brainstorming for sure. But I've been in situations where you're trying to create a structure and you ask ChatGPT for example, because that's the most widely used, and you would ask for a taxonomy say, that represents health and medical. And if you know how LLMs, and especially in the case of ChatGPT, is trained and what the training data is, that's a pretty good indicator of what sources are being used in order to deliver you information or to deliver you that taxonomy. For example, it trains on NIH, it trains on all the government sites, ChatGPT does specifically, it trains on all of that open access information. And so if it's going to create you a taxonomy specific to a certain domain, we're just taking medical as an example, I have seen more than once these examples or these starting places blow up as like, " That looks awfully familiar. It's the health and human services browse menu." Or you start to see these patterns exist. Now, as a starting point, if you're able to look at what an LLM perceives as level one, level two, level three in a taxonomy, that brainstorming or that starter bit, you can go back and say, " Okay, this makes sense. Where did these exist in our information environments?"
Tim Gasper [00:16:54] That's super interesting. And it makes a lot of sense that you're essentially... You're asking something relatively specific out of something like ChatGPT or an LLM, and of course it's going to find that part of its training corpus that it thinks connects, and it may not be reflecting your unique fingerprint.
Jessica Talisman [00:17:14] No. I want to give this example of there actually being pre- work before you input and start to try to derive meaning. If you actually go through the exercise of taking all of the publicly available taxonomies that are most widely used throughout the internet, that would be GS1, the brick schema. And then you'd be looking at Google and Facebook Marketplace and IAB and all of these taxonomies that are most widely used. Do an analysis and figure out what's the same across all those taxonomies? What's unique across all those taxonomies in the level one and level two, because that's the highest representation. And start to look at the patterns but also start to look at your data, your information structures if you have them, your taxonomies, your hierarchies, and you can actually start to see very transparently how the answers that are derived from an LLM or specifically ChatGPT, because of the instance of training on the worldwide web, how there's so many similarities and there's almost a mirroring effect. It's not so much of a mystery. If you know what the training data is, then you can assume from step A through your end point, that that information is obviously going to play into the answers you receive.
Juan Sequeda [00:18:45] Let me highlight here this comment that we just got here, " Agree with this comment about the taxonomy specific to a domain, yet there is such a wide spectrum of deployments that are not the case for a level one start. Many are very complex and might need to be trained on your own, compared to the others."
Jessica Talisman [00:19:01] Yes, yes.
Juan Sequeda [00:19:03] What I find interesting from this comment here if I understand this correctly, is that there's so much out there that very specifically you can go find on the web. It's not just go to GPT, actually go to the original store. And then it's interesting to see where the overlap is. And part of that is that should be if you're starting off, that should be an inspiration. And then you say, " Okay, that's stuff that all these other experts have done. There's so much agreement. How does that compare to what I'm going to do?" And I'm going to guess here, but it's almost an 80/20 rule, 80% of that type of stuff is probably going to be relevant to me but then there's going to be another part that's going to be specific. And then you get at that top level and then when you get more specific, then that's going to be again, specific to you. Is that a valid assessment?
Jessica Talisman [00:19:52] That's a valid assessment. You do need that whole idea of the wide spectrum of deployments. And it's not always the case. The complexity of a domain does need specific training. I think that's what I was saying before, is you need specific training in order to render anything meaningful. And so we look at the advent of RAG models just as an example, of RAG implementations. Well, that's lending more context in order to derive more meaning and more specificity from an LLM because you have to surround your instance and your prompts obviously and your output specific to the context of your instance.
Tim Gasper [00:20:42] Have you found that things like RAG architecture is helpful to get more useful information architecture assistance from AI? Is it helpful, and how helpful is it?
Jessica Talisman [00:20:58] It's very helpful because you have the opportunity to present to your information in a structured fashion and then also surround the context with input that perhaps is less structured but has the context within the documents, or the input. Basically, I see it as almost a curation tool as well. And it helps to align the instance and use cases because otherwise... For example, if you don't have coverage of specific topics that are relevant to your organization, how is that output from an LLM, from AI, going to deliver that information?
Juan Sequeda [00:21:45] Let's dive into this right now, I want to get into some examples. What happens in your opinion, your expertise, if you do not invest in information architecture, you start building all these AI and RAG architecture. I'd love for you to come up with some examples like, " We're not going to go do it. Here's what's going to happen, good and bad. And then here's what we're going to go invest in it." And I would love to get an example of what it means to do information like, " Here's a taxonomy or whatever." And so forth. And what are the pros and cons on that stuff.? Let's get into that.
Jessica Talisman [00:22:17] Yeah. It's interesting because we talked about deriving some structure out of an LLM as a starting place. Well, I have seen firsthand when this happens that if you don't have an organized project where you are determining from a single source, meaning that there's one person in charge of determining what these level ones are, I have seen it such that there's splitting or fragmenting of categories, meaning that there's... I'm going to give the example of home and garden, which is a pretty common domain, and home improvement. And so there tends to be this wishy washiness, if that's an official term, about where to post information. Going back to that point is okay, so you've created two parallel verticals in terms of an information structure, home and garden and home improvement, and all of a sudden you're putting anything having to do with plumbing under home improvement and home and garden. You miss the exactness or the preciseness, which is a problem because your base structure is flawed or faulty. There's no way. It could be one or the other, it could be both. That's fine. But then that flows through to analytics dashboards, that's another problem. Then you get fractured categories where it exists in two things. Are they the same thing or different things? Plumbing and home and garden and plumbing and home improvement, are they the same thing? You end up seeing that problem start to replicate itself throughout your entire system. It becomes a non- starter because you end up having a major dirty data situation and having to reconcile entities across a system. You've just added a problem to your system rather than improving.
Juan Sequeda [00:24:16] To play devil's advocate here, wouldn't it... Why do I need to go organize home and garden versus home improvement to know where plumbing is? At the end, these systems can find it. At the end of the day, the goal here is to go find and discover.
Jessica Talisman [00:24:38] If I'm launching a campaign, for example, I work in ad tech, just give this example, I'm launching a campaign and my campaign's called home and garden and I decide to curate things because there's context, that's what we've been talking about, is information architecture lends itself to context. It's by the parent/ child relationships that we understand the context of a certain topic or subject. Home and garden, if they're different, there's disambiguation settings that you can use with parentheses saying, " This is only relevant to home and garden and this is only relevant to home improvement." ... as an example. Well, if you launch two separate campaigns into two separate ad groups and those things are not disambiguated but you intend them to be the exact same thing, plumbing and home improvement and plumbing and home and garden are the exact same thing, you need that level of specificity and you need to be aware of that so you can track revenue for example, and performance because they may perform. It's good to know that those things are split, that that category is split between two verticals because perhaps it's going to perform better in one vertical than the other. And that's just in that example. But another point is that it matters for clarity for external customers, if this really, really matters, that external customer may have an ask or deliverable and is unaware of that confusion in the information matrix. It does lend itself to trust and clarity disambiguation, those things are very important. That trust certificate, it really matters obviously in business and especially when you're dealing with money.
Tim Gasper [00:26:25] Yeah. No, this is super interesting. As you go through some of these examples of where there can be problems... As you started to talk about some of this though, using your example around home improvement versus home and garden, there are certain situations where it makes sense for them, contexts in which it makes sense for them to be thought of together. But then there are other contexts where it doesn't, such as for example, who is the persona that I care about? Well, in home and garden, maybe I care more about the gardener, whereas with home improvement, I care more about the plumber and the handyman and things like that. And so a lot of what you've been saying around information architecture, and even going back to the original thing that we talked about around a site map or browsing around a website, they tend to be pretty hierarchical, where it is forking and forking and forking. But there are situations where hierarchies don't really make sense, maybe especially in the context of AI. I'm curious about how do we navigate that? I guess, what does information architecture look like if it's not a hierarchy?
Jessica Talisman [00:27:48] I guess, you could say a controlled vocabulary at base. If we are going to take away the hierarchy, the best that you can do is a controlled vocabulary. And if you want the added benefit of reconciling and at least having a field where you can have acronyms and synonyms, that is at least a place to start. It may be a flat list but again, us humans operate, and I know that we're not AI, but US humans operate with hierarchy in our brain, decision trees. And if we think that decision trees are not part of AI, then we're sorely mistaken.
Juan Sequeda [00:28:31] I'm totally with you here, Jessica. And I think this is one of the first times that Tim and I may have a bit of a disagreement here on probably the need or requirements when it comes to information architecture. I'm saying this based on your comment, Tim. I'm like, " Well, what if there's some things that don't need hierarchies?" At the end of the day, if it's explicit or implicit, it's there, the hierarchies exist. Now, the question is do we want to invest in making them explicit or not? And I think my point is like, " Okay, let's not boil the ocean. Let's create a hierarchy of everything. You should be driven by some of the use cases." And that's why I was asking about when should I start investing more in having these semantics and this information architecture? And look, if you're going to do things like search and want to go find things, then you could make an argument depending on the type of stuff that you have that you don't need to invest so much in taxonomy and stuff. But then at some point you're like, " There is so much stuff." And when you start people complain, " Well, I can't find things. I search for something, I get so many results back." Then you're like, " Well, now how do I figure that out?" I think that's an indicator. That's one thing. And second, I really love what's very specific is I need to go map this to questions that have to be very accurate. For example, on revenue, I need to go categorize the amount of revenue of home and garden versus home improvement because I'm going to invest more. At that point, people are going to be fighting, " Oh, no. This thing is over here and this thing's over." People are going to be fighting then to go put their semantics in there because you're hitting the bottom line, money. I need to know how much more space in my factory I need to go do this, depending on how I'm organizing my space. I think it really depends on, at the end of the day, the bottom line and stuff. If like, "Oh, I need to go find something." And here's a POC and just a couple people think it's cool, yeah, don't invest in it. But if you're going to put it and it gets bigger, people start complaining about this stuff, then I don't know, this is where I'm like, " Yeah, more data and compute is not always going to solve the problem."
Jessica Talisman [00:30:47] And that's expensive.
Juan Sequeda [00:30:48] I want to spend a little bit more time and invest in that semantics. Anyways, I'm going off a rant here.
Jessica Talisman [00:30:54] Yeah, it's inaudible.
Juan Sequeda [00:30:55] Do you agree, back me up or push me off the cliff here?
Jessica Talisman [00:30:59] No. It's expensive to just be hoping for some answers and just trying to make sense of what you've received as far as output. And then to make sense of that... First of all, it's expensive for the model to run, we're looking at economy. And so it's been proven time and time again, and that's why vector databases are very helpful, is for helping to cluster and to direct learnings and trainings and be able to post. It's one of those things, if you look at how the internet's structured, if you want to have a site that registers with Google, you're going to have to do it. Say you have a Squarespace site, it's going to have to be done in maps because they like to know where you are and what you're doing. But then you have to choose from a taxonomy. Same is true for Instagram, same is true for almost every single platform. They need taxonomy in order to place your business on the map and in reality within the internet. When we turn on our computer, that's an order of operations, there's a hierarchy there. We have file and folder systems. If we were to look at a base structure and think that, " Okay. Well, ChatGPT or LLMs are basically statistical." Well, we had LLMs well before ChatGPT, this is not new. And best practices are to have a taxonomy. If I were to train some model and to train it to... I don't know, say that I'm creating some decision tree or a workflow using BERT, which is a large language model, I have to have a taxonomy or else it can't decide what to post to, what decision to make.
Tim Gasper [00:32:50] I super agree that we need taxonomies, but I guess, where my mind goes, and I'm curious, Jessica, as well as Juan, if your response to this is it doesn't really matter, it's not an applicable example. But let's go into the example of a file share environment. I have my own Google Drive and I manage a hierarchy of as much as possible, the digitization of my different files. And certain things are pretty easy to classify like, " Oh, I got my W2. Okay. Well, let me stick that in the taxes folder." But then you've got, " Oh, I went to the doctor and now I have a receipt from when I went to the doctor. And I have a folder called health that's at the same level as taxes. Now, do I stick my health receipt into the health folder or do I stick it into my taxes folder because I want to make sure that I take advantage of it as part of my taxes?" I'm curious about does that matter? Ultimately, does AI care? We don't have to be overly exclusive with our hierarchies, the point is just to give a map.
Jessica Talisman [00:34:03] I personally would create medical receipts folder, that's just me. And that would be a sub folder because that level of specificity is helpful. Being able to help ...
Tim Gasper [00:34:17] I can tell you're an architect and I'm not.
Jessica Talisman [00:34:21] It is helpful. And nesting folders and understanding... I go as far as to also include the year in that as well. But that gets into also mapping systems and being able to map and resolve data entities across an organization. What's one person's receipts could be another person's just plain medical. The level of granularity and the way that we organize things across our organizations within those organizations can vary greatly. There's a huge disparity in most organizations, and that would be the role of enterprise architecture for example, or a role for enterprise architecture or data architecture. It's important.
Juan Sequeda [00:35:09] I want to highlight this comment here. Search needs to tolerate the fact that some people think tomatoes and mushrooms are vegetables. Revenue reporting might need a strict taxonomy. We can support both. And I think I highlight this because there are going to be areas where we don't have to be pedantic but there's some other areas where we must be pedantic because of regulations. Part of a taxonomy is what do we define as PII and PHI? That needs to be fixed, and there's no discussion about it because we can get fined on stuff. No, no, no, no, no. This is it, period. And some other stuff like tomato, tomato, mushrooms or avocados or whatever.
Jessica Talisman [00:35:55] Well, and to resolve that, yeah, I agree that search needs to be able to tolerate it. And that's why I build thesauri structures, because I can have a dedicated schema field that prepares for that with NLP, there's different constructs. And so if you're going to resolve tomatoes being a vegetable or where it's placed or where it's categorized and all of its different names and spellings, you can do so in that type of structure. I think what's important is that we're not constrained to just a taxonomy structure. Knowledge graphs become knowledge graphs because they can resolve those entities. It's the very founding principle of information architecture, is yeah, we resolve entities.
Juan Sequeda [00:36:38] Sorry, Tim, and I, we always... For people who don't know how this works, we have a shared Google Doc, we got our Slack going in the background, and then people have seen us do it live. We're taking live notes, that's how we do our summary. But Tim is like, " This is fascinating. I need to go up to the lightning round, time is passing so quick. We need another hour." I'm like, " Tim, that's what you missed out." Actually, I missed out talking to Jessica at Data Day Texas. That was one of the things I was really bummed about. We bumped into each other at a bar at 11:00 PM inaudible. But anyways. Okay, we've been throwing out these words, taxonomy, thesauri, controlled vocabulary, ontologies, knowledge. All right, Jessica, give us the Jessica definition of all these things.
Jessica Talisman [00:37:21] Okay, of taxonomy? Go.
Juan Sequeda [00:37:24] Let's do controlled vocabulary, thesauri, taxonomy, ontology...
Jessica Talisman [00:37:30] Okay. Controlled vocabulary is usually a flat list of terms or concepts within an organization. I love them with definitions, that makes it a glossary.
Juan Sequeda [00:37:42] Oh, hold on, that's important. Flat list of terms, that's a controlled vocabulary. And then you take the controlled vocabulary plus definitions, and that turns it into a glossary.
Jessica Talisman [00:37:52] Glossary, yeah. Or some people called a data catalog, which would include whatever its canonical term is and map it to the database. We can go that far but you'd still have definitions possibly. Then you go into taxonomy, taxonomy is a hierarchy with parent/child ...
Juan Sequeda [00:38:10] Wait, hold on. Doesn't thesauri go ...
Jessica Talisman [00:38:13] Thesauri. Taxonomy first, then thesauri.
Juan Sequeda [00:38:14] Okay. You see, thank you. Well, I've seen it the other way around but...
Jessica Talisman [00:38:18] Yeah, it could go either way but there's a difference programmatically. Taxonomy is the hierarchy of terms. It can be represented with a lightweight ontology to indicate parent/ child relationships, and you can carry the definitions through. But thesaurus, if you take that hierarchy, include the definitions, resolve and allow for relationships that traverse the hierarchy that would be related or related to, that becomes a thesaurus. A thesaurus helps, it also captures alt labels or aliases, which would be synonyms and acronyms, and can host other languages as well. You can start to integrate NLP at the thesaurus stage. Ontology are the constructs or schemas that help to resolve entities, create relationships, add context. You have properties, classes, relations and attributes within an ontology that basically is giving your taxonomy and thesaurus wings.
Juan Sequeda [00:39:22] That's a beautiful t- shirt right there, an ontology gives a taxonomy thesauri wings.
Jessica Talisman [00:39:33] Yes.
Juan Sequeda [00:39:33] Okay. And then above ontology, is there more or...
Jessica Talisman [00:39:38] Knowledge graph. A knowledge graph, if you put all of that specialness into a knowledge graph, then you're not only defining the controlled vocabulary, the taxonomy, the thesaurus with all the definitions carried through, and the ontology to make the connection, the context and define the relationships between things in your knowledge graph.
Juan Sequeda [00:40:04] All right, I think this is a perfect segment, it's 40. We'll go back. Minute 37 to 40, great definitions here around this. One of the things we wanted to talk about is also the relationship between information architecture all the way to knowledge graphs. What is your recommendation here for organizations and actually people listening right now who are thinking, " Oh, I'm not doing anything with information architecture and I should be doing... And I'm hearing all this stuff about LLMs, AI and knowledge graphs are involved, and I'm hearing semantic layers, ontologies... " There's just so much stuff right now. What's your recommendation for folks to take it easy and start getting educated, and how should they start?
Jessica Talisman [00:40:57] I highly recommend that everyone just download the ANSI Z39 standard for monolingual vocabularies. I know it seems daunting but that standard is openly available and it defines all of these things, including giving you a purview into how to structure things with code, really, really high level code and schemas and ontologies. But the whole idea is building and creating structures for information retrieval systems. And it's important to follow the steps. I think that so many of us are so rushed to get to the end result. The bad news and the good news is this is not fast, information architecture is not fast. Understanding some of the basic rules to what makes a taxonomy a taxonomy, what makes a controlled vocabulary a controlled vocabulary, will help you avoid some of the pitfalls. There are some pitfalls to creating these structures such as creating recursive loops, machines and people don't like recursive loops. That's a very common mistake.
Juan Sequeda [00:42:11] We're talking about people's roles here, do you think that we're going to go ... Do we need more information architects or is it... Coming from the data world, people are like, " I'm doing data engineering." Or these roles like analytics engineering. There's all these roles or titles people have but they're doing things that involve semantics. Is it these particular existing roles they have to upskill themselves with this or you think this is a brand new thing that people should be focusing on and we should just have more information architects in the world?
Jessica Talisman [00:42:47] I think both. I think understanding leads to support, and that's one of the problems, is for us information architects that are in organizations and trying to structure information, most often we're the lone information architect. Now, I will tell you, I went to two years of graduate school to learn what I know. And I concentrated in informatics, specifically in structuring information for catalogs, for knowledge graphs, the whole deal. We are somewhat shy to hire library information science folks into enterprises, and when we do, there's just one person and they're very underfunded. What I see is this struggle where people feel like, " Oh, I can just read a book or I can just do it overnight and I'll get it." I get it. And that is not necessarily the case because if you look at the construction of Wikipedia, Wikidata, DBpedia, some of these open sources that AI trained on, for example, if we look at the domain of libraries, if you've ever been to the internet archive, there's a methodology. This is something that's been going on for 3000 years, it's not new. Data engineering from my understanding, I don't know, 50, 60 years. I think there's a lot of crossover, sure, and a lot of understanding, but there's a shared understanding that needs to be had. It can't be done in isolation. Information architects and information scientists need data engineering and data science folks to collaborate with because it's important to have both views and both sides of the coin present. You need to be able to translate things through all systems and information architects cannot do that alone. I would say hire more information architects. Ask me, if you want to find some really, really special people in the information science and library science domains, they definitely know what they're doing.
Tim Gasper [00:44:44] Reach out to Jessica, she has some great people to connect you with. I want to ask you one more question before we get into some of our lightning round and wrap up and things like that, which is that we've got a lot of folks that listen who are data practitioners, who are data leaders, who are trying to drive positive change in their organizations. And what would be your recommendation to them thinking about information architecture in general but also in support of AI. If you were going to say, " Hey, here's a couple of takeaways." Some homework for our listeners, what would you give them as homework?
Jessica Talisman [00:45:29] Look into structured data and really what that means. Some people will stop at the definition of data structure for a SQL database. Look at the extended definition of what structured data means in terms of what is available on the worldwide web, what AI trained on. And we haven't had enough discussion about training data structure. We have to assume that if training data exists and that data is structured similarly across the spectrum, why are we not pulling that from the back to the front? Why are we not integrating that into the way that we speak and the way that we are structuring data for AI? I've heard so many complaints throughout the past year about having unstructured data and, " Let's try to structure it. I'm just going to take Google taxonomy or Facebook's marketplace taxonomy and I'm just going to do it that way." But you're missing a step, and you're missing that critical step that is going to make the difference between it being super expensive, moderately expensive, or manageable. If we're complaining about cost, if we're complaining about training data, if we're complaining about structured data, then it's important to understand structured data beyond a relational database.
Juan Sequeda [00:46:54] I really appreciate this call because I think when it comes right now to a lot of the AI work and chatting with your data and so forth, is focusing on the unstructured and then on the data side, it's only on database schemas, but there's much more than that. Before we go to the lightning round questions, I do want to have this question that came up. In this definition, is an information architect an ontologist?
Jessica Talisman [00:47:14] They can wear the hat of an ontologist.
Juan Sequeda [00:47:16] Okay.
Jessica Talisman [00:47:17] Yeah, yeah.
Juan Sequeda [00:47:21] I agree. Just to call out a book, one of my favorite books I tell, is Semantic Web. This is a good one. I am very, very extremely honored and proud that I get to work with Dean Allemang every day.
Jessica Talisman [00:47:35] Yeah, it's pretty amazing. Yeah, and that's the thing, is we've created these really narrow... Amazon just recently changed most of the titles of taxonomist to ontologist. What hat or what term the enterprise decides to name what I do or what information architects do is really up to the enterprise.
Juan Sequeda [00:47:55] This is a great call out, and I think talk about where are jobs going next thing. We always look at the big tech, the giants, what they're doing. This is really interesting to know. Amazon is now calling them... All these roles, you're ontologists.
Jessica Talisman [00:48:07] And inaudible makes a good point, by the way, about structured data queries being cheaper than a call to an LLM. I think that's a good point because then you have this luxury. If you do structure your data correctly and you do have a knowledge graph, then you also have something to validate your queries with or your LLM output with, because that becomes a source of truth.
Juan Sequeda [00:48:35] Oh, my, there's so much I want to go dive. We just opened up a whole inaudible going out into databases more, about ontologies and stuff, but we got to start wrapping up here. Jessica, let's go to our lightning round questions. And I'll kick it off. Question number one, is the primary role of information architects going to shift to focus on helping AI?
Jessica Talisman [00:49:00] I think we are already there, to be fair. It's just there aren't enough of us and our voices seem to be quiet.
Juan Sequeda [00:49:11] Hopefully discussions like this is making the voice louder and people inaudible.
Jessica Talisman [00:49:14] Absolutely, and just invest in information architecture. If you care about your data, then show that you care by hiring and involving information architecture into your system. Usually you see these jobs and information architecture appear within an organization when it's, " Oh, shit. I can't find anything. Our data's a mess. What do we do?" It's the fire and then you get in there, " We have to fix it right now. How fast can you do it?"
Juan Sequeda [00:49:42] That was the takeaway of the research that we did working with Dean, is like, " Hey, you want to do all this chat with the data over your structured data? Well, guess what? It's actually going to be accurate with knowledge graphs and semantics and all this stuff."
Tim Gasper [00:49:53] Yeah. Your comment really resonates with me, Jessica. An aha moment that pops in my head is that I feel a lot of organizations invest way more in data architects and infrastructure architects than they ever do in information architects. And that was probably always an issue but it's becoming a bigger and bigger issue as AI becomes more important and relevant.
Jessica Talisman [00:50:20] Yes. And remembering that information architecture is not simply a browse structure in the front end of a website because sometimes you'll see those UX and UI activities happening in isolation, regardless of what shape the data is in.
Tim Gasper [00:50:34] IA people who could be helping on the data side, on the AI side, are being excluded and left to working just with the marketing folks.
Jessica Talisman [00:50:44] Yeah. And then we get into fun mapping tasks and all of a sudden new careers evolve that are just simply mapping entities on the back end from a SQL database to the front end of a site, which is just maddening in some ways.
Juan Sequeda [00:50:57] Barely scraping the circuit.
Tim Gasper [00:50:59] Yeah. This was an important aside, a worthwhile interruption of the lightning round. All right, second question, is a flat glossary, no taxonomy, no ontology, no hierarchy, useful?
Jessica Talisman [00:51:13] It is, it is. It's a starting place, especially if it has definitions. And that glossary can add context. I think anything that adds context... Now, personally, I don't think it's enough but it is important because at least you can agree upon nomenclature, naming conventions. If we're all speaking the same language, I think that's step one. It is useful. What I call home improvement, you might call home and garden.
Tim Gasper [00:51:41] Yeah.
Juan Sequeda [00:51:42] Yeah. All right, next question, is investing in knowledge graphs going to become a given for every organization trying to work on AI?
Jessica Talisman [00:51:51] Yes. And I say that only because you've seen all of the work that's going on with vector databases and with RAG, which is all great, but a more economical way to manage that is knowledge graphs, plus you have a natural validation source by having knowledge graphs. Now, yes, in the future but again, that is contingent upon hiring more information architect type people, like ontologists or having a inaudible.
Juan Sequeda [00:52:20] This is a fascinating position right now. And I'm with you, I'm with you. But again, you can make the position... The reason why you want to go invest in knowledge graphs is because the amount of compute and even just the query execution stuff is going to be cheaper if you go do that investment upfront because otherwise you're just running cycles.
Jessica Talisman [00:52:42] It already happened.
Juan Sequeda [00:52:43] It is already happened. I think that's one but then you're like, " Well, to get there, I have to go invest in all these people, invest in this time to go do that." It's almost a chicken and egg, but hopefully we can quickly break this. I was actually having a discussion with some other folks, is well, you go all these cloud data warehouses, there's a compute behind this. And you don't know exactly if it's generating the right query, it's going to generate all these different large queries, it's going to be expensive to go do that. How much are you actually considering the compute costs for that? And they're like, " No, right now it's ... I guess, right now it isn't." But then later on you're going to get the bill and like, " Oh, shoot. All this AI is costing me the compute of just doing the AI thing but then the query execution itself later on... " You got me, I got inspiration to do some other new work. I love this, thank you.
Jessica Talisman [00:53:29] If you get a vector database and you have that running with your LLM, those two in combination are very expensive. That's where the costs just collect exponentially. And if you have some flaw in your system, you see systems where you're racking up these huge bills, is that worth it for the customer? Would you rather invest your money rather than compute time? Wouldn't it be better to invest it on organizing and knowing what you have as an organization? I don't know, I guess that's up to the organization.
Juan Sequeda [00:54:05] Yeah.
Tim Gasper [00:54:06] Yeah. There's a nice aha moment here where I'm really seeing that... I think some people think of AI like free lunch.
Jessica Talisman [00:54:15] Yeah.
Juan Sequeda [00:54:16] Oh, yeah.
Jessica Talisman [00:54:16] And it's an oracle, it's like Zoltar. We think of it as this magical box, that it's just going to answer all of our questions and even with accuracy, and that's not always the case. Again, if you want accuracy in your LLM and want to try to minimize hallucinations, knowledge graphs are your answer.
Tim Gasper [00:54:38] Love it.
Juan Sequeda [00:54:39] inaudible.
Tim Gasper [00:54:39] All right, last lightning round question. I'm going to ask you to prognosticate a little bit here or make a prediction, do you feel as long as we invest in information architecture and building out our knowledge graphs, that we have all the ingredients for AI today? Or are we missing something?
Jessica Talisman [00:55:02] That's a good question. I think we do have all of our ingredients if we invest in information architecture. Again, I go back to if you dissect training data, what ChatGPT for example has been trained on, if we look at that instance, if you look at training data and the structure of information, what has made ChatGPT or other LLMs so successful? It's because they have information architecture, it's following the Wikidata taxonomies and ontologies and roadmaps. And so when we sit here and we ponder, " Oh, wait. What is in ChatGPT? What's making it work?" We know what's making it work, it's the training data. And so if we are able to match information architecture with equally, an information architecture, then even though it seems abstract, we're speaking the same language, we're using the same constructs in order to classify output and to derive meaning.
Tim Gasper [00:56:04] This is fascinating, it's now really clicked for me what you're saying here, Jessica, that why is ChatGPT so fricking magical? Because of all the IA work that it was trained on.
Jessica Talisman [00:56:22] Absolutely. It can't occur in isolation, it's not deriving... Like we say, it's a statistical model. Well, that statistical model references an architecture, has to, or else if someone asks a question, what is it going to do? Is it going to find every word that's close to or near home and garden? That's not as reliable. You have to have a base structure and understand the relationships between things as well.
Tim Gasper [00:56:50] That makes sense.
Juan Sequeda [00:56:54] I strongly believe that's the case, I can imagine that is the case. Now, I'm sure that if you go talk to the large language model folks, the transformers folks are like, " No, it's just more data, whatever." But then the real experiment is what if I gave you a ton of crap that has no information organized? Let's see how you do. How about we eliminate Wikipedia or just fuck up all the structure, the connections inaudible. Let's see how you do.
Jessica Talisman [00:57:25] Yeah, even start with taking those widely available structures, take the Getty as the source. Openly available, you can download it. Take NIH as taxonomy and ontology. Use that as your base structure, then try to use an LLM and just compare the differences.
Juan Sequeda [00:57:44] People have been asking me, " Juan, on the research that you've done, why are you seeing a lot of the accuracies increasing with knowledge graphs?" And I'll be public about this right now because I've been organizing my thoughts, is I don't have the evidence, I don't even know how to set up an experiment about this, but just our language, the English sentences and stuff follows a format of a graph, the subject, predicate, object. But then your database schema is not necessarily. I postulate that that's a reason why things inaudible.
Jessica Talisman [00:58:18] Absolutely. And those graphs are not... Large language models are not necessarily... You can use SQL. I know that you've done some work with using SQL. And sure, but you need to take it that one step further.
Juan Sequeda [00:58:35] Yeah.
Jessica Talisman [00:58:36] Yeah.
Juan Sequeda [00:58:37] All right, so takeaway time. Tim, kick us off. Take us away with takeaways, so much.
Tim Gasper [00:58:45] I'm going to have to find a way to summarize this. All right, we took so many awesome notes here. We started off with honest, no BS question of what is information architecture and how does it tie to AI? And Jessica, you provided an amazing definition. I couldn't possibly replicate that definition. For those of you that are just listening to the takeaways, listen to the full episode. You'll hear Jessica give an amazing definition of it. But in summary, it's around organizing information, controlled vocabularies, thesauruses, ontologies, methods and systems to organize information for the benefit of... You said, often the front end experience, but can also be the backend experience. And for AI, both of those are super relevant. It supports findability and discovery, it requires that you understand the context and the stakeholders. Even things like database structures are information architecture but it's a certain kind optimized for certain use cases. And you even provided a wider aperture of different kinds of information architecture, like going to a grocery store. There's an information architecture there of what aisle are things, how have they been categorized, where does it live, medical systems, tech systems and so on. And how does it tie to AI? Well, you said chatbots needs some way to have a map, an information system so that they can access information in a way that makes sense. And enterprise knowledge that an organization has needs to have some structure to tell AI how to classify things, how to know how to interact with that knowledge. And every organization is different, use the phrase or the word fingerprints pretty often, that every organization has its unique fingerprint. Enterprise architecture specifically is around the structure of the infrastructure of an enterprise organization, how it communicates, how the data at a high level flows, versus data architecture, which can be data schemas, could be how the information is being used by different systems and applications. We have a few different definitions here, and I know in Juan's takeaways, I think he's got some more definitions that you talked through, but it really helped to create some clarity between these different concepts. And finally, for my takeaways, I'll say that you did a great job outlining some of the gaps that are in place right now between information architecture and what we're trying to do around AI. People trying to use ChatGPT to create taxonomies are going to find that what comes out are the things that it was trained on, and not the unique fingerprint or unique context of your organization. And so you're going to hit some barriers unless you start leveraging RAG architecture and knowledge graphs and other types of things that are going to help to feed in more of that context. Knowledge graphs are for the win you said. LLMs can help with brainstorming and can assist, but they cannot drive. You need to really consider and do your own research on what kinds of things are the sources for ChatGPT, and really think about what's going to be able to give you the kinds of results that you're looking for, and not just the cookie cutter types of things. And so much more but I'm going to pass it over to you, Juan, what were your takeaways?
Juan Sequeda [01:02:02] Well, continue here. We had this great discussion when it comes to like, " Hey, so what happens if you do or you don't have information architecture when you have all these AI or RAG architectures?" Throughout the episode, we talked a lot about home garden versus home improvement. I think that's a lot of people to go back and listen to the episode and catch that. At the end of the day, look, for some things it's not that important because you can search for things, but we really discussed that in the moment that search becomes harder, it's harder to go find things, it's because you fractured that. And that's the importance of it. Another thing is when you really want the clarity, like, " Hey, you're going to actually map revenue to one of these things. You need to have a clarity of is it going to be home garden or home improvement? It's going to be very different." Customers may expect different things, and at the end of the day, this leads to trust. We talked a lot about hierarchies, and I think this was a question, how much do we really need hierarchies or not? And the whole point is that hierarchies, taxonomies, they're all over the place already. You need to place things such that you know how to go use them. Hey, you're going to put something on Google, on Instagram, whatever, you have that dropdown menu and they know exactly what that goes to. There's a hierarchy of things. Heck, your file structure is on your computer. There's that taxonomy behind things. And went through that example, do I put this medical receipt under my tax folder, my receipt folder, and then either/ or? Or might as well just create a new thing called a medical receipt that can go to both.
Tim Gasper [01:03:24] I've got some work to do this weekend, reorganizing my Google drive.
Juan Sequeda [01:03:26] See how we put this all back in? Part of your organizational activities, who likes to organize their file structure on their computer? All right. We did definitions, controlled vocabulary, a flat list of terms, a glossary is a controlled vocabulary plus your definitions. And in a data catalog, you actually take that and you connect it to the actual database schema elements. The taxonomy, you have that hierarchy, the parent/ child relationships. Thesauri is the hierarchy plus related to relationships, alternative labels, which are the synonyms, you have other languages. And then you get into the ontologies, which are the schemas that you can create relationships. You define classes, relationships between the classes, attributes. And I love this quote, " Ontology gives taxonomy and thesauri wings." And you start putting all these things together, you really create that knowledge graph that provides that context specifically now for AI. And how do you get started? Well, first it's to understand, yeah, information architecture is not so fast always. You have to understand the rules. There's pitfalls to avoid, like don't create recursive loops. Machines don't like recursion either that much. We need more information architects, we need organizations shy to hire information architects. And we need this. Think about it, data in its modern sense is just a couple of decades around, while the library of science has been going on for centuries. And finally, homework for listeners is look into structured data. Jessica, how did we do? What did we miss?
Jessica Talisman [01:04:56] You did not miss a thing, I don't think.
Juan Sequeda [01:04:59] Well, again, as we always say, this is thanks to you, we're just repeating the stuff that we heard from you. Thank you so much for sharing all the valuable insights for everybody. All right, to wrap it up, throw it back to you. Three questions. What's your advice? Who should we invite next? And what resources do you follow?
Jessica Talisman [01:05:21] What was the first one again?
Juan Sequeda [01:05:23] What's your advice about data, about inaudible.
Jessica Talisman [01:05:25] My advice is if we're looking for starting places or that specific book, just start to deconstruct the taxonomies that are openly available, and just start to run analysis on that. We're data folk. You guys would have no problem determining...
Juan Sequeda [01:05:43] Another one I love to go share with people is schema.org, so much ...
Jessica Talisman [01:05:45] Schema. org is great, and just understanding that and structure that as a taxonomy because there is one in there.
Juan Sequeda [01:05:50] Yeah. Who should we invite next?
Jessica Talisman [01:05:53] Oh. Who should you invite next? Have you had Ole On?
Juan Sequeda [01:05:57] Yeah.
Jessica Talisman [01:05:58] Yeah. Oh, darn it. I'm going to have to think on that. Yeah.
Juan Sequeda [01:06:05] All right, and then just post on LinkedIn inaudible.
Jessica Talisman [01:06:08] Actually, Mary Bates, do you know who that is? Or Richard... Yeah, Mary Bates.
Tim Gasper [01:06:15] I'm not familiar.
Jessica Talisman [01:06:16] I think that would be interesting.
Tim Gasper [01:06:17] Okay.
Juan Sequeda [01:06:18] All right. And then finally, what resources do you follow? People, blogs, books.
Jessica Talisman [01:06:26] I follow inaudible. The Discipline of Organizing, I really love as a book. It's very abstract. Every organization I go to, I try to lend that advice. And it is very abstract, I'll have book clubs. I think book clubs are very, very important just to help to get resources and information out there. It's a super library and science type book, the Discipline of Organizing, but it has that abstraction that helps to take you from the beginning to the end of constructing vocabularies, taxonomies, ontologies. And I think that's a great resource.
Juan Sequeda [01:07:05] All right. Well, Jessica, thank you so much. Just a quick reminder, next week we will have a guest, Paco Nathan. I'm actually going to be live with him in a castle in the middle of nowhere in Germany. That'll be a fun inaudible.
Jessica Talisman [01:07:20] Oh, nice.
Juan Sequeda [01:07:20] We're going to be talking about all things around bringing innovation, open source data stuff, and obviously knowledge graphs and LLMs into the real world and all that. That'll be a fun discussion. Jessica, thank you, thank you, thank you so much.
Jessica Talisman [01:07:31] Thank you both.
Tim Gasper [01:07:33] This was amazing, cheers.
Juan Sequeda [01:07:37] inaudible conversation.
Jessica Talisman [01:07:37] Cheers.
Juan Sequeda [01:07:37] Cheers, everybody.
Special guests
