S3E05.5 - Minecraft (Bonus - Teaching Computers to Game)

#Minecraft #MinecraftBasalt #NeuralNetworks #ArtificialIntelligence #AI #TeachingComputersToGame #BoardGames #Science #SciComm

Summary

In our final minisode about teaching computers to game, we leave the tabletop behind and move on to Minecraft and even the real world. We're also back with Dr. Prithvi Akella, who helps us understand how Minecraft and other digital games provide more open-ended platforms to work on AI models, along with what an "AI agent" actually is (no, not a spy--well, _probably_ not a spy) and how they're used to run tasks in both the game and real worlds. We also talk about what large language models actually are, how they and vision-based models work, and happens when you let a thousand AIs loose on their own Minecraft server. So get ready to punch some wood in our final minisode of this series for Gaming with Science.

Timestamps

00:00 Introductions
01:04 What is Minecraft?
03:23 Teaching AIs to play Minecraft
07:18 AI agents and LLMs
10:45 Letting AI loose in Minecraft
17:49 No more games for AI?
20:54 So what about us humans?

Full Transcript

(Some platforms truncate the transcript due to length restrictions. If so, you can always find the full transcript on https://www.gamingwithscience.net/ )

Jason Wallace 0:04
Jason, hello, and welcome to the Gaming with Science podcast, where we talk about the science behind some of your favorite games. In today's minisode about teaching computers to game, we'll be talking about Minecraft and the next frontier of machine learning. All right, everyone. Welcome back to Game with Science. This is Jason. This is Brian, and we are once again joined by our special guest, Dr. Prithvi Akela, who is here to help us understand machine learning and AI and the world of Minecraft today. Prithvi, can you do a quick introduction for the people who may have forgotten since last week?

Prithvi 0:36
Hello, everyone, nice to speak to you all again. My name is Prithvi. I finished my PhD from Caltech about three years ago, where my emphasis was on validation of learning enabled systems with a goal of trying to make sure that these systems function more reliably and safely in general practice.

Jason Wallace 0:51
All right, and in our final episode of this four part mini series on teaching computers to games, we have left the realm of board games and gone into computer games, so we are going to be talking about Minecraft today.

Brian 1:02
Finally, a real game.

Jason Wallace 1:04
I don't play Minecraft. Minecraft is a game with very pixely art. I see my daughters playing, and they seem to have lots of fun building farms and villages and making artwork and rugs and stuff in it, and it looks like digital Legos, and that's about all I know about it. So I'm going to pass it to Brian to explain to us what is Minecraft.

Brian 1:23
Sure, I think digital Legos is actually a great analogy for Minecraft. So, as a player, you'll spawn into a big wilderness expanse all made out of blocks. It's been around for over 15 years at this point. It was released in 2011 so that's very long legs for a video game. It's had routine updates throughout the time that have sort of kept people interested, add new things, new features. Basically, you can collect the blocks in the world, all these natural resources, craft them into other things, build structures, build castles, stuff like that. You are supposed to eat food at night time, monsters will spawn, so you have to like make yourself armor and weapons to protect yourself. It's generally a sandbox game, in the sense that, like, the player usually is the one who decides what they want to do. It's sort of very open-ended, which is probably why a lot of kids enjoy it's very creative. Again, it's like imagine you had an infinite box of Legos without having to worry about things like gravity, and also you get to fight monsters at the same time. One of the key things about Minecraft, though, is that each world is procedurally randomly generated, so based on a seed and a bunch of noise maps, as you keep walking, the world is technically unbounded. Obviously, you can't actually do infinite, because your computer will explode at a certain point, but as you keep walking in a direction, there will always be a next horizon, a next hill, a next forest, a next ocean, all based on this seed procedural number map, there is technically an end to the game, where, like, you can get to a state where the game will play its end credits. You have to do some pretty esoteric, sort of obscure things to get to the end of the game, which involves, like, building multiple interdimensional portals, collecting resources from two specific monsters, finding a rare structure underground again, using more rare resources, and then fighting the dragon boss at the end of the game. I think maybe that's one of the things that makes it interesting as a challenge is every time you spawn into a Minecraft world, unless you have artificially plugged in the exact same seed, that world will be unique and different. You can always beat it, but things will not be in the same place, the resources will not be in the same place. The way to beat the game will not be in the same place.

Jason Wallace 3:23
And I think this is part of the appeal of why games like this were the next frontier after Go was mastered, is because Go and other board games have very finite states. It's all bounded in this board. You know exactly what the pieces are you can work with, and you have very clear goals. In chess, it's to capture the opponent's king. In Go, it's to capture the most territory on the board. Minecraft, as I understand it, does not have specific goals like that. There are many, many things you can do, but not really anything you have to. There's no single goal for the game, and so that makes it, to my mind, one step closer to reality, where we have this massive unbounded sphere we all live on, and we are trying to do all sorts of things, and so it seems like Minecraft is sort of like a stepping stone to being able to get these AI agents to work in the real world, is to first train them in a simpler world that has known rules, but not quite as many of them, and if someone messes up in Minecraft, no one dies,

Jason Wallace 4:17
which could very much happen in real life,

Brian 4:19
so when you say teaching an AI to play Minecraft, what are we trying to get the AI to do?

Jason Wallace 4:24
So, there's a challenge called MineRL, so Mine RL, or MineRL BASALT, which was a sequel to it, where they had certain goals in mind. Did you have a chance to look these over?

Prithvi 4:33
I did have a chance to look them over, and actually, there's been significant advancements beyond Simple RL for training these models to play Minecraft or play a lot of the Starcraft or these other types of games, which are not open-ended per se, like in the previous episode, right? We talked about how we used games as a way to train these models or figure out better ways to train these models, because as human beings it is natural for us to learn the world through these games, but granted, a lot of the tasks that we otherwise expect ourselves to achieve, or a lot of the tasks that we otherwise have to do on a daily basis are relatively open-ended, in the sense where there is perhaps at the end of the day some criterion which determines the end of either a game or a task that we're doing in our offices or a specific function we need to do in a factory, for the sake of argument, but they are a little bit more open-ended, and so then as we get to now trying to train these models in Minecraft or Starcraft, or any of these slightly more open-ended games, or open-ended sandbox scenarios, if you will. It stresses our ability to make good ways of training these models, so that they can adapt, learn, figure out optimal actions in these now more little open-ended settings. And then, to the point that you mentioned, where we started with MineRL or MineRL, we've now moved all the way to transformer-based architectures like Google Deep Minds Dreamer, or I think OpenAI also had a VPT, so a Vision pre-trained transformer that basically just by feeding a transformer architecture a number of images or a number of videos of people just playing Minecraft,

Prithvi 5:56
it actually just trained the system to play Minecraft in and of itself, which is absolutely wild,

Brian 6:01
which, considering how much YouTube Minecraft content is already out there. I'm sure there was a deep training set to work from

Prithvi 6:08
a very deep

Jason Wallace 6:09
part of the goal of some of these contests was to get that level of training where you didn't have to run the computer through Minecraft 10 million times to find a strategy, because that's not what humans do. We watch someone else play, and the article I read pointed out you can take a human child and show them a 10 minute video on Minecraft on how to mine a diamond, and they will get it. And the idea is, how can we get computers so that they can learn like this? Now, in reference to our previous conversation on Go, this does mean that you tend to copy the existing strategies, so you may not end up with Minecraft from Mars, like we did with Go from Mars with AlphaGo Zero, but it does much more efficient if someone's already found a workable strategy. You don't need to waste the resources just reinventing the wheel.

Prithvi 6:50
Completely agreed, and this actually is, in my opinion, a phenomenal branch of work that we're trying to undergo, specifically in the context of robotics, but also agentic systems in general, where figuring out ways to make these systems function more reliably with minimal amounts of data, minimal amounts of compute, so that way it doesn't cost so much money to train, so much energy to train, and allows for these systems to be a little bit more explainable, because we break them down ourselves. I find as a fascinating area of work, which I imagine will probably be longer than the 20 minute conversation.

Jason Wallace 7:18
Now, you mentioned agentic systems, and we've mentioned AI agents a few times. Can you explain to us what exactly is an AI agent? I hear people use those all the time, but I don't really have a good definition of what we mean by an AI agent.

Prithvi 7:32
Sure, so perhaps before defining an AI agent, I should first define an LLM. So, an LLM just stands for a large language model. These are the things that underlie, say, Chat GPT, which I imagine many listeners have otherwise interacted with. What is an LLM? A large language model, generally speaking, and at a relatively high level, it's just some machine learning model that you give it some words and it spits out a bunch of other words. At the highest level, this is basically all an LLM is. Again, like Chat GPT, you type something into Chat GPT, those are a bunch of words, it spits out a bunch of words that are likely to follow the words that you have spit out,

Jason Wallace 8:04
as understand it. These actually grew out of translation software, basically like predicting what the next word would be

Prithvi 8:11
exactly,

Jason Wallace 8:11
and people realize that you didn't have to be translating to do that. You could do that with normal conversation, and you could actually create conversations that way. Is that basically correct?

Prithvi 8:20
Yeah, that's actually exactly correct. The original paper, through the original "Attention Is All You Need" paper by Vaswani and his co-authors, as well. That original paper was actually dedicated towards machine translation. So, basically, how do I translate from English to another language, or any language to any other language? And then, yes, to your point, Jason, I don't necessarily need to only use these architectures for translation. I could, as long as they understand the pattern behind what words come after what words, also start generating words that otherwise would follow in a paternable sequence from the words that I provided to the model as well. Then what an agent is, is an agent effectively wraps around these LLMs and allows for these LLMs to utilize what we call tools or skills to achieve some hierarchy tasks that a user, in this case a human, would provide, so the most canonical example of agents that I imagine are relatively widespread, that people can understand, are Claude code. For the sake of argument, Claude Code is endowing Claude, or any of these LLMs, with the tools, in this case, to read files on your computer, to write to files on your computer, to also internalize an understanding of how it should write what files, subject to you, the user, in this case, providing an overarching say command or goal, that is, please give me code that would otherwise achieve this function or this end goal, as an example, please give me an architecture that would replicate alpha zero on my laptop, and it should be able to internalize this information, read what it needs from files on its own computer or the internet, and then be able to write these scripts appropriately to service that. So, high-level agents are, in my opinion, LLMs that are equipped with tools such that they can use those tools to service user queries in pursuit of some goal that the user would otherwise care about.

Brian 9:57
So, I always feel like when you're talking to these. This LLM, when you're putting in a prompt or something, it really does give the same feeling that you'd get from talking to the computer on the Enterprise in Star Trek. You ask it a question in natural language, it interprets it, executes its own program to look for the information, and just returns the information. I don't have to speak computer language, it understands what I'm saying

Prithvi 10:19
effectively. LLMs, plus these tools have basically abstracted away the need for you to specifically understand how to code what it is you want to code, and simply explain in natural language, and, like I mentioned earlier, at the end of the day, what these systems have understood is based on the words that you have provided in natural language, what words or what tools now should come next, and what information should be fed to those tools to achieve what it is that your natural language query would otherwise want to achieve, predominantly on your computers.

Jason Wallace 10:45
In this space, I've seen some really interesting things in the context of computer games. So, a few years ago, there's a company called Altera, a startup company that made a bunch of agents in Minecraft, so they had a bunch of AI users all on a common server that were interacting with each other, and what was really interesting is that they started having emergent properties from interacting with each other, such that they put like 500 or 1000 agents all working together, and they would start to specialize careers. Apparently, some of them would start becoming farmers, others would start defending the city, others would be artists, or what have you. Reading their press release, I think they're over anthropomorphizing their agents, like they're giving them feelings and dreams, and everything is like no. At the bottom of it, it's just a bunch of math that is going on through the computer code, but it is still very interesting to see the complex architectures that come out. Or, Prithvi, you mentioned Starcraft a few times, which was another one of these areas, like along the pattern, there was AlphaGo and AlphaGo Zero. There was Alpha Star, which was DeepMind's attempt to create a program that would play Starcraft or Starcraft Two at a high level, and they achieved it as should be expected by this point. It played at Grandmaster level. It had a very interesting approach, though, in that they also created a sort of league for it to play against itself, because I think Brian, last time you mentioned the danger of an AI converging on a single strategy and getting kind of locked into it.

Brian 12:10
Yeah, we talked about this before in the context of biology, sort of a local fitness optimum, where it's like it's very good in that little space, but it can't really break out, because to leave that little hill makes it less fit.

Jason Wallace 12:23
Yeah, well, in making this little league of AIs playing against each other, DeepMind did something really interesting. They actually made two different types of agents. They made sort of the primary agents who were trying to win, they were trying to win against everyone else, and those are essentially like your top-tier Starcraft players. But then they made essentially friends of the agents, other AIs, whose goals were not to win, but they were to exploit weaknesses in the winner strategies. This is kind of like several of the top tier players will have their circle of friends, whose entire purpose is not to become top tier themselves, but to expose the flaws in their friend's strategy, so they can develop countermeasures, and thus that the friend, the winner, can go on to become a better player, and so by having all these agents interacting with the first tier, trying to beat each other, and then the second tier trying to expose the weaknesses, they ended up getting very high level advanced play out of this, just by having again a bunch of agents playing each other. I think this is still more in the reinforcement learning than the LLM area, but it's still using these individual AIs interacting with each other in unusual and interesting ways to try to develop emergent programming. I don't know what to call this, emergent behavior, or more complex behavior than you could get from anyone by itself.

Prithvi 13:33
It's actually quite interesting that Jason, that you mentioned that basically the way we architect these systems is to allow for this growth of emergent behavior, and also just a quick point. Modern LLMs are the LMS that we interact with on a daily basis, again, like the LLMs underlying ChatGPT, LLMs underlying Claude. They're actually not just trained models in the sense where I feed in a bunch of input output data of words and other words that come after it. They also have an RL (reinforcment learning) component at the final stage, in order for these models to resemble human speech or otherwise interact with humans in the way that we would otherwise like these systems to interact with us, and that is called preferential human reinforcement learning. If I'm not mistaken, and it's a term that our listeners can also go and try to look up if they would like, but without this step, actually the models that we otherwise would interact with wouldn't actually be these conversational models that we get to chat with, or otherwise interact with, on a daily basis. It would actually just be autocomplete. It's actually a wild step to go from just autocomplete, and then with this small amount of reinforcement learning, actually take these trained models, and instead of just doing autocomplete, now have these more flowing conversations, which have again allowed tool use and other things that allow for generative architectures as well. So, quick little side note: there is actually a marriage there between these generative architectures and reinforcement learning, as well.

Jason Wallace 14:43
Yeah, we mentioned previously how, although we're going through these episodes linearly, each subsequent step doesn't fully replace the one before, it tends to build on it instead,

Prithvi 14:52
for sure.

Brian 14:53
So, let me just bring the conversation just briefly back to Minecraft, because, again, like I said, Minecraft does have an end goal, there is technically an end of the game. Is that what some of these AI agents are trying to do, or are they simpler tasks like mine a diamond? What does it mean to have an agent play Minecraft?

Prithvi 15:12
I'll say that, as it regards wanting an agent to play a game, in this case Minecraft, and what you would want that agent to do, subject to the definitions that we've provided earlier, that is an agent is an LLM in this context, that's been endowed with tools. I guess it would depend on what you want that agent with those tools to be able to achieve. I'm going to take a step back in stating that for an agent to play Minecraft in this context, it probably wouldn't be using an LLM, it would use what's called a vision language model, or a vision model of some sort, which is a different type of transformer-enabled architecture, which I won't go into too much depth on, but the idea is very much now in the sense it's less a question of do you want the agent to complete the entire game, getting back to Jason's point from Altera, or is it, do you want an agent to just figure out how it should operate with potentially other agents in this game setting, updating its perhaps own internal rule set that operates by in order for to achieve in this case coexistence, and I realize the audience can't see that I'm putting air quotes around the words coexistence, because much like Brian, I am anthropomorphizing these systems, but it allows for us to talk conversationally about how these systems would interact both with others, which is an entire field of research called agent-agent communication, in this context, but also with human beings as well, which I think is important, as we want to interact with these systems to provide meanings to our lives.

Jason Wallace 16:25
and with the MineRL BASALT Challenge, specifically. So, BASALT actually is an acronym, it stands for Benchmark for Agents That Solve Almost Lifelike Tasks, and the idea is to actually give a fuzzy goal, something that doesn't have a clear end goal, they actually has four different tasks. This is the waterfall task, and the description of it is after spawning in a mountainous area with a water bucket and various tools, build a beautiful waterfall, and then reposition yourself to take a scenic picture of the same waterfall by pressing the escape key.

Brian 16:54
Okay,

Jason Wallace 16:55
that's what it's supposed to do. And I think the goal there is that this is not necessarily a yes-no thing, there's a lot of fuzziness to here. Other ones involve building a house that doesn't harm the village that it's being built in. There's one about building animal pens again in such a way that doesn't harm the rest of the village, which, again, harming a village is a very fuzzy goal. And so I think they're trying to explore more of the types of things we humans have to deal with, because a lot of our goals are not binary. Yes, you did this. No, it's not. It's like, okay, we need to solve this problem while also dealing with all these other things that may not be particularly well defined.

Brian 17:27
I'm excited to watch some AI Minecraft YouTubers.

Jason Wallace 17:32
You can definitely find some, because some of these companies have put out their results and they're out there on the internet. I'll see if I can find some for the show notes.

Brian 17:39
This is a whole subgenre of Minecraft YouTube is get a bunch of people together, give them a build challenge, and then have them do it with specific restrictions. It sounds like this is what we're trying to do.

Jason Wallace 17:49
Yeah. Now, Prithvi, I've got another question for you. Because when I originally outlined these episodes, this was sort of the frontier of what I knew, as far as challenging computers to play games, and then as I was researching this episode specifically, I noticed most of the things that I could find were all from like three or four or five years ago. It almost seems like this level of investigation of using games as a training ground for LLMs and agents, it almost seems like that too has been surpassed. And when I'm looking now, it seems like people are just training them in the real world, like there's this one area I found called OS World, which has a bunch of real-life tasks, like book me some tickets on Expedia, or balance this checkbook of mine, or here's a folder of recent transactions, go update my expenses, or other more real-world things. Have we basically surpassed the point at which we need games to train these algorithms, and that they're now at the point they can interact with the real world?

Prithvi 18:44
That's a phenomenal question, Jason. So, I have two answers to that. So, I'll state the first one. I think that, in part, yes, as we have moved away from games now, because of the capability that we have with these systems, to, as we were talking about Brian earlier, parse natural language and effectuate either tool use or something else to achieve goals that I set. This has allowed for us to move past games to balancing checkbooks, for the sake of argument, as you put it earlier, or these other more well-defined sure, because balancing checkbook is well defined, but still somewhat open-ended tasks, as we talked about earlier. While Minecraft does have effectively an end, at some point you have to defeat the ender dragon, and that is an end of the game, that is to state after balancing these checkbooks or doing other similar types of tasks. It's not that the task is over and I never have to do it again. I might balance a checkbook again, I might have to do other things, other similar tasks later on in life. And so, as we start getting to having the capability to have these systems parse our natural language to perform actions that are of value to us, we start to get more granular in the actions that have value to us, like the ones that we mentioned, and then try to build specific systems or general systems that can solve these specific tasks or certain subsets of tasks, so that on the front of games, but that being stated, I don't think we've moved past games entirely, because I find the notion that we otherwise used when we were using. Games to train these models or figure out ways to make better models or learning paradigms, so that these systems learn, function, understand better. I still think there's work to be done in that regime, maybe not specifically with the games that we were talking about earlier, in the context of, say, Go or Shoji or Starcraft or Minecraft, but potentially games in a generic, say, economics cons, like game theory games. How do I present actions to these systems where it's not just it interacting with a reward or not a reward? Because humans are social creatures, and as we interact with people around us, we oftentimes have to interact in multiplayer settings where we have to consider the information from the people around us to make optimal actions that affect not only ourselves but also the people around us as well. So, I still think there's, in this kind of multiplayer game setting, significant research that could still be done that would still allow for us to make better models, more interesting learning architectures, et cetera. But this very much is to your point, Jason, still down on the frontier.

Jason Wallace 20:54
Okay. And then I've got one final question, and this is the big one. The history over this series of minisodes, and over the past 80 years of computer design, has been a series of setting challenges for computers that humans can do and computers cannot, and then eventually the computer surpasses the human in that challenge, and the way things are going, it certainly looks like we are now at the point where the question is not if, but when general-purpose AIs will surpass humans in terms of their ability to carry out any meaningful task that we want. I could be wrong about that, but every previous benchmark we've set, computers have managed to pass, and they are doing it at a faster and faster pace. The question I have is, what happens when we achieve that? What happens when we achieve a level of AI sophistication that we humans simply cannot match on any general purpose task or even any specific task.

Prithvi 21:45
Great question. I'll also state a little bit, as a side note, as a roboticist who works with some of these models with robots in the lab, we might have a few years left, at the very least, before robots finally take over, given how much they fall or fail. But to your question, at the end of the day, I think the changes that these systems have wrought on our daily lives. If I can conceptualize them into a few sentences, I feel like the easiest way to describe it is, whereas previously we were the ones balancing the checkbook, doing these other menial tasks that were required on a daily basis for us to live the lives that we wanted to live. Now we can have these other systems, LLM enabled systems, agentic systems, do these tasks for us, which, what that has meant for me personally, at least in my life, interacting with these systems, is it gives me more time to do the things that I want to do, like not balancing in my checkbook, and in this case, reading and swimming, because that's what I particularly enjoy doing, as opposed to balancing my check, and so I find that as we get these systems, as they function more reliably, as they get to do these tasks, it frees up, and I imagine it should free up human time to do the things that we find most important, creativity, time with friends, socialization, and I think we should construct these systems towards that end goal to allow for this level of flexibility in our lives that is also just one man's opinion.

Jason Wallace 22:53
Yeah, this is a different conversation, I think, but I'm going to say, since you're a roboticist, you really need to get working on this, because my wife always says she wants an AI to do the dishes, so that she can do art, whereas currently we have an AI that does art and tells her to go do the dishes,

Jason Wallace 23:07
so that's still a problem I think we need to solve.

Prithvi 23:11
Very true. We'll see if I can try to make it so that they don't break plates as often, but we're getting there.

Jason Wallace 23:15
Okay. All right. Well, with that, I think we're going to wrap up this short series on teaching computers the game, and what we've learned. Hopefully, y'all have found it very entertaining and educational. Prithvi, thank you very much again for coming on and opening the box on deep learning and neural networks, and all this. If you want people to look you up, where should they find you?

Prithvi 23:35
Feel free to look me up on LinkedIn, I'm happy to chat.

Jason Wallace 23:38
All right, and with that, we're going to close this mini series, so thank you very much, listeners, for listening. Have a great month and great games,

Brian 23:45
and have fun playing games with computers. See ya.

Jason Wallace 23:50
This has been the Gaming with Science podcast. Copyright 2026 Listeners are free to reuse this recording for any noncommercial purpose, as long as credit is given to Game with Science. This podcast is produced with support from the University of Georgia. All opinions are those of the hosts, and do not imply endorsement by the sponsors. If you wish to purchase any of the games we talked about, we encourage you to do so through your friendly local game store. Thank you, and have fun playing dice with the universe.

Transcribed by https://otter.ai

Comment (0)

No comments yet. Be the first to say something!

S3E05.5 - Minecraft (Bonus - Teaching Computers to Game)

Summary

Timestamps

Links

Full Transcript