AI for Simulation

00:00:04: Hello and welcome to this new episode of Bosch Rexwood's Tech

00:00:07: Podcast.

00:00:09: My name is Robert Weber.

00:00:10: Before we start, a few notes.

00:00:12: We recorded this episode a few weeks ago, which is far too long ago in AI terms.

00:00:18: A lot has happened since then.

00:00:20: Johannes Brunschel has founded his own company, ME AI, raised a lot of money.

00:00:25: The neural dam mentioned in the episode is now available as an open source tool.

00:00:30: And the team has published a model for aerodynamic simulations, AB-UBT.

00:00:35: Congratulations.

00:00:37: And now let's get started.

00:00:40: And of course, thanks to our interviewer, Peter Sieberg.

00:00:44: Enjoy listening.

00:00:50: Today I'm going to be talking to, once more, Johannes Brandstetter.

00:00:54: He is professor at the Johannes Kepler University in Linz.

00:00:59: And Johannes and I are going to be talking about a neural DEM, which is all about real-time simulation of industrial particulate flows.

00:01:09: Hi Johannes.

00:01:11: Hi Peter.

00:01:12: Pleasure to be here again.

00:01:14: For those that did not hear our first talk, a couple of months back, please introduce yourself to our listeners, Johannes.

00:01:21: Yeah, hello, everyone.

00:01:22: My name is Johannes.

00:01:23: I'm here in Lenz.

00:01:24: I have a history with starting in physics, did my PhD in high energy physics, like so many other people, switched to machine learning then, spent a couple of years in the Netherlands, worked at Microsoft Research.

00:01:35: And approximately a year ago, moved back to Austria, back to academia, have the strong belief that AI is going to revolutionize the way we do engineering, the way we do manufacturing.

00:01:47: And in my take on this, this revolution happens deeply inside the simulation area.

00:01:53: And this is where we are now, I guess.

00:01:55: That

00:01:56: sounds great.

00:01:56: We're going to be talking about revolution today.

00:01:59: Yes, I do recall.

00:02:01: It's about fifteen weeks later, I believe, episode two hundred fifty five.

00:02:05: for those of you listeners who after this episode say, Oh, that's interesting.

00:02:10: I want to hear what Johannes had to say at that time as well.

00:02:13: You and I were at the Siemens AI for purpose summit.

00:02:16: We talked about weather forecasting.

00:02:18: You say Netherlands, Microsoft.

00:02:20: So for those of you, maybe we talk about it later.

00:02:22: You can make the link if you want to.

00:02:25: But I do recall you suggesting the same, the same like you already used the word, revolution would apply to industrial simulation.

00:02:34: Fifteen weeks later, here we are.

00:02:37: You and your team just published a paper in which you claim to have come to replace discrete element method, DEM, routines and coupled multi-physics simulations with deep learning surrogates.

00:02:54: That's a big claim.

00:02:56: So just we all understand how big it is.

00:02:59: Let's start from the beginning for those listeners not dealing with it on a daily basis.

00:03:05: What is the discrete element method?

00:03:08: Yeah, maybe let's start from the very beginning.

00:03:10: Thanks, Peter, for this introduction.

00:03:13: So that the problems we are looking at, I started to look at those problems approximately one and a half years ago.

00:03:19: And weather modeling at that point in time was pretty, pretty hyped already in the AI community.

00:03:24: They were the first neural network driven weather forecasts.

00:03:28: And those were like really large-scale simulations running on the cloud and basically starting to replace or augment numerics.

00:03:36: And for me, the first point was I was trying to figure out what is like a really tough problem to tackle because neural networks work nicely on like image data on regular data.

00:03:48: but we all know from engineering the data we have there or the problems we have.

00:03:52: they are not so nice.

00:03:53: they are on meshes they are non-regular they are manifolds and we have all these different interactions of multi-physics and so on.

00:04:00: so I put together my my head with some of smartest people I know from numerics, which actually happened to be here in Linz, which do this couple DMCFD simulations.

00:04:11: I will say a bit more about that later.

00:04:15: And we looked at some problems, and I like those problems a lot because they are high-dimensional, so they have many, many particles.

00:04:21: They have interactions of a fluid with particles.

00:04:24: They are, of course, three-dimensional, so the spatial dimension is three.

00:04:28: They have a time dimension, and they are chaotic in a way.

00:04:31: And I basically fall in love with these kind of systems because they are a challenge.

00:04:35: And those systems we can now tackle, those systems we can formulate, we found a way how to deal with that.

00:04:41: And for me, I think most importantly, a way how to scale this system.

00:04:45: So the largest systems we were simulating have half a million particles and about two hundred thousand CFD cells.

00:04:52: But if we go ten times bigger, it wouldn't be a problem.

00:04:55: We would just need more compute.

00:04:57: We would just need more data, but just fundamentally.

00:05:00: there is no barrier with which we're hitting now and that is important for me.

00:05:03: I wouldn't say that what we build now can be already used in production and can be already used in engineering because for that you actually need to have a concrete problem to tackle.

00:05:15: and we used rather famous and important settings from this community but not specifically tailored to a problem from an industrial partner.

00:05:23: But I would say that figuring out how to scale these networks at those problems is a very important step for us and for me it's personally a big breakthrough.

00:05:33: Now, today, engineers in simulations in exactly the area you talk about are using the discrete element method, DEM, integrated with, I understand, a grid-based computational fluid dynamics CFD, just two, three lines maybe on how that works today and maybe also what are the advantages or maybe also potentially disadvantages of the way we do things today.

00:06:02: So that's the million dollar question.

00:06:04: So DM is really works on a particle level.

00:06:07: So you model particle interactions and you can think, for example, you can model sand, you can model grain, you can model food in a silo, you can model any type of material and any type of particulate flows.

00:06:22: That's why it's called particle.

00:06:23: it flows.

00:06:24: And these methods do not only have the power to really model such systems very, very accurately, then it can also be relatively easily coupled to fluid dynamics.

00:06:35: And often when you have systems like in fluid as bed reactors, which we're modeling, when air is blown in from below and is basically messing around with the particles, which you use for coating, for example, then you have to couple these two types of simulations.

00:06:50: And this can be relatively easily done by DEM and therefore for this multi-physics there is this specific DEM-CFD coupling with which you can describe a lot of multi-physics phenomena.

00:07:01: and yes this is where we started.

00:07:04: and also this coupling of CFD and DEM is where the the Institute of Stefa Birka and my collaborators Tobias Kronlacher and Thomas Lichtnecker are really experts.

00:07:14: You already mentioned two three words so we all also those people not being themselves the simulation engineers but maybe their colleagues are.

00:07:22: you already mentioned a silo.

00:07:24: you mentioned grain sand so I can get a little bit of a feeling.

00:07:28: Nevertheless will be great if you could give us one or two typical use cases where finally.

00:07:35: this approach is being undertaken.

00:07:37: Yes, I didn't believe how important is silos for food industry, for example, but think of fluidized beds, which where particles are in, then the air is blown from below, as I said before, and this is extremely important in coating particles.

00:07:51: So all big chemistry companies, they are really interested in this coating and basically in mixing of particles and so on and so forth.

00:07:59: Okay, now... What is the disadvantage?

00:08:02: or moving into the new way of doing things with neural D?

00:08:08: What would you say is the typical D disadvantage?

00:08:11: Yeah, that's of course the next very tough question.

00:08:15: So when I talk about numerics, I don't like the term disadvantage because numerics has brought us very, very, very far.

00:08:22: I mean, it started with Gauss and there is so much going on and we can model so many things.

00:08:27: But I would rather say what my deep learning adds to this portfolio.

00:08:32: And there is a few things which are not possible with Merix right now.

00:08:37: And one of those things is definitely in the area of digital twins or real time simulation.

00:08:43: So think of you have something like a furnace or any other system which you need to control in real time.

00:08:50: And that is very hard because you just cannot simulate that quickly.

00:08:53: It takes you hours, days to simulate and even.

00:08:57: with the best GPU based accelerations.

00:08:59: At some point you hit the wall in simulation time.

00:09:03: And the more complex these processes are, the farther you get away from doing real time simulations.

00:09:08: And that is where deep learning has really the potential to shine.

00:09:12: It allows you to interact with these systems in real time, to observe when when systems get critical, but also to figure in how you as a human can interact with these systems and factor that in into the simulations.

00:09:25: And that is a huge potential towards real time engineering.

00:09:29: Okay, thank you for kind of correcting me there.

00:09:32: So I'm not going to be asking you to share with us how a neural DM tackles the disadvantages, but rather as I've learned.

00:09:43: How provides NeuroD and the add-ons the way forward on top of the numerics as we have it today?

00:09:54: I liked it, yes.

00:09:55: I like that.

00:09:57: Similarly, like the LLMs right there, they're not basically replacing our way we write, but they give us a new tool, how to get more gradients to come up with first drafts or whatnot.

00:10:08: And similarly, it is going to happen in engineering.

00:10:11: And the better you use those tools, the faster you can integrate them, the more advantage you have in the end.

00:10:18: So you want to share not too detailed because I have seen all kind of terms that at least the simulation engineers can of course should go to.

00:10:28: the paper will make the link available, which is maybe a couple of lines on how your solution works.

00:10:36: Yeah, so I think there's basically two fundamental ideas which we needed to make that work and first is the representation.

00:10:44: So think of these particle systems.

00:10:46: and we were really always, I mean, the largest system we were simulating was half a million particles, but we always had in our minds, what is happening if you go to ten millions to hundred millions?

00:10:55: How can we make sure that our method somehow still works and it's just a that the scale meaning that we just need more compute than bigger networks.

00:11:04: I'm similar to what is going on in language and in other domains.

00:11:08: And therefore we kind of were blocked by this particle representation because more particles means just more for the network to the chest and that is not scaling very well.

00:11:18: So we came to the conclusion that we have to get rid of this particle point of view.

00:11:23: And so we represent to our network everything as an underlying field.

00:11:27: You can think of as people did in the beginning of the twentieth century to think of the eta.

00:11:33: So some some field which perpetuates space and time.

00:11:37: And we were using this field-based approach, which we cannot really formalize, but which we can use the network to represent the data and to get a loss function on to model our problem.

00:11:47: This approach is much more scalable and it also turns out much more stable to train.

00:11:53: So really making a fundamental switch from the representation of particles to a representation of an underlying field, which might not even exist, but for the network.

00:12:04: it's fine to learn this underlying field and can do it extremely well as we've seen and has been surprised actually.

00:12:10: And the second one is using the power and might of deep learning by projecting the The problem to lower dimensional space and using different interaction mechanisms of this space, which type of physics, which type of modalities interact with each other, we can model very nicely in this down projected space.

00:12:30: And both of these approaches are really tailored towards scaling.

00:12:35: Okay, and the second one I understand is that related to or even based on the paper you did before on the universal physics transformer?

00:12:44: Exactly, it's basically the same idea extended to multi-physics.

00:12:48: Some very smart people, Benedict, Samuel, Tobias working on that and they really brought that to the next phase.

00:12:57: And actually

00:12:57: what is very exciting with this approach, we were now able I'm not spoiling, but to really go to other domains away from discrete element methods, other simulation domains.

00:13:09: And it seems that that we're kind of starting to converge towards a recipe which works for many different types of problems.

00:13:17: And that's actually what excites me the most.

00:13:20: Okay, so is it correct to say that neural DM is based on the universal physics transformer and that we may expect in the future maybe other solutions like neural DM in other simulation areas?

00:13:36: that is very likely to happen.

00:13:38: yes

00:13:38: okay i'm not going to ask you further But it's great to hear.

00:13:44: Okay, let me see.

00:13:46: You keep it at the correct level.

00:13:49: I'm certain that you could bring it to a lot more detailed level.

00:13:53: But as I said, that's for the specialist, for the simulation engineers.

00:13:58: I would like to understand from you.

00:14:01: then, we talked a little bit about the use cases.

00:14:03: What are the market settings in which neural DM will find usage?

00:14:08: or to put it differently?

00:14:10: For what type of companies, I think, do the simulation engineers that will be using NeuralDem, what do they work for?

00:14:18: The most interest is coming from chemistry, companies, using for coating, using for mixing.

00:14:27: there is a lot of interest from metal industries and yeah maybe what I should say is this is really at the early stages because there is to do DM with deep learning is what a few people tried.

00:14:41: but most approaches like they got stuck at five thousand particles.

00:14:46: so it's a bit surprising for people that one million, two million is actually not a limit anymore.

00:14:51: So I think it needs a bit of understanding.

00:14:54: Um, we have a lot of conversations and everyone is like surprised that we can model this.

00:14:58: I would say I've never been in a spot where people, where we show something and people are actually asking questions that you can model that really.

00:15:05: So I think it needs some time to see again and, and, and to develop the momentum for that.

00:15:11: But as I said, that's why we chose this, this really hard problems to have a good starting point when, when we finally are able to do them.

00:15:18: Okay, so I understand this is kind of, yeah, this is your initial announcement, as you say, you know, the market, the potential users need to get used to the idea that this is now possible.

00:15:32: So maybe we're gonna find and see further announcements in the near future.

00:15:37: So maybe you do not yet know in detail, but I'm gonna ask you what your thoughts are around.

00:15:43: Because I know from related solutions, algorithmic solutions in your company.

00:15:49: So how about NeuralDEM?

00:15:51: Is it going to be open source?

00:15:53: Is there a patent?

00:15:54: Are you thinking of eventually selling it as a product?

00:15:58: Maybe all kinds of questions.

00:15:59: you haven't decided, but nevertheless, I would appreciate your current thoughts on them.

00:16:05: Yes, and NeuralDEM, obviously we believe in open science for NeuralDEM.

00:16:10: Obviously, we've written a paper and we shared it with the community.

00:16:14: At some point, we will also share models and basically let people play around with them because people need to see what they can do and what the potential in them.

00:16:23: And this is how we will move forward.

00:16:25: So there is no way in holding these things back.

00:16:28: You have to share and you have to let people play with them that they get as excited as we do.

00:16:33: Okay.

00:16:34: When you say you're moving now beyond the five thousand, maybe two million or to a couple of million, no limit maybe to the size.

00:16:42: Does that mean when you're training, when you're modeling that you, where is that happening?

00:16:47: Is there a cloud partner that you're doing that?

00:16:50: Are you doing that within the area of the university in your company or?

00:16:55: Yeah, this is over cloud computing, why the company exactly.

00:16:59: And we are reaching the phase where we can tackle these problems, but now it's time to think about what is like a more generalist solution to these problems that you, in the end, what you don't want to have is some specific sort of problem for a specific.

00:17:17: sort of customer, specific sort of company, you want to have something more general, which is like more a general recipe.

00:17:25: That's my midterm goal, which can be applied to several processes, to several companies, to several problems and really recipe how to build that.

00:17:36: how to fine tune that, how to train that and so on and so forth.

00:17:39: So I would say along those lines we are making progress now.

00:17:43: Okay, tell us a little bit about your environment, your team, your base in Linz.

00:17:48: You mentioned two, three names, who are the people working in your team and are you maybe looking for new colleagues and if so, what should they bring?

00:17:59: Yeah, that's great.

00:18:00: So I can give a shout out again to my colleagues.

00:18:02: So one person who was really driving that was was Benedict Arkin, the first author.

00:18:08: He was also my first PhD student when I come back to Linn.

00:18:12: Basically, he wanted to work with me.

00:18:14: He was very eager to work with me.

00:18:16: Together with him, we did the Universal Physics Transformer.

00:18:20: He's a very smart person and extremely good at scaling deep learning.

00:18:23: That's where we really hit it off very well.

00:18:26: And then at this team, we added Tobias Thomas and Stefan from Stefan Birka's lab of particulate flow modeling.

00:18:35: They've been extremely helpful.

00:18:36: Tobias has basically led this project from the numerics perspective.

00:18:41: very fun for me to work with so many experts so that actually what we're doing was meticulously proofreader or cross-checked by numerics people.

00:18:51: I mean, this is super important because in the end, you want to know if you're impressing yourself or if you're impressing the community.

00:18:57: So it took us some time to impress them, but it was extremely nice working environment.

00:19:02: And then last but not least, maybe we have Samuela from Amsterdam.

00:19:08: I basically asked him to join us for an internship and he performed extremely well and was a very vital asset to the team and I knew somewhere since some time.

00:19:17: so I knew that this would work.

00:19:19: but I'm extremely proud to that.

00:19:21: now people from Amsterdam coming to Linz that they participate in this project and really their expertise is also shining.

00:19:28: Okay so you're moving forward I assume looking for other people to join and what should they bring if so?

00:19:36: Yeah, so there is a few domains I'm interested in.

00:19:39: I cannot share everything, but obviously, computational fluid dynamics is of big interest for me.

00:19:44: Basically, the game we play with fluid element methods, where you want to go for millions of particles.

00:19:50: In CFD you want for millions, tens of millions of grid cells or mesh cells where you run your simulations on.

00:19:57: Then this same game we play in these domains and therefore experts are needed both in numerics but also in scaling these deep learning systems.

00:20:08: So well then, coming to a close, how is neural DM by replacing?

00:20:14: Is it replacing?

00:20:15: Or maybe it's extending, I believe I've understood in the meantime, maybe extending replacing discrete element method.

00:20:22: How is it structurally going to change the world of

00:20:25: simulation?

00:20:26: Yeah, there is basically two changes which will happen.

00:20:29: The one time is real time engineering, so you can interact with the real world.

00:20:33: This is I mean, digital twins are floating around since a couple of years, but especially for this more complex and large processes like furnace, like furnace fluidized beds and then many more.

00:20:44: This is basically the way I imagine myself that this is going to happen.

00:20:50: And then that gives you better control, better process cycles and so on and so forth.

00:20:54: The second change is in real-time optimization, because if you have a fast surrogate, your engineer... progress is much faster.

00:21:03: Think of you want to design certain geometries or certain processes and if the simulation is extremely fast you can test a lot of different Parameters a lot of different materials and then pick what material basically fulfills all the properties you want And if so you can run the final set with standard numerics to just to make sure.

00:21:25: but these two two things are a bit opposite to real-time digital twins and optimized process engineering and in both ways approaches like neural DM will be will be the key to achieve that.

00:21:38: And we've seen that in the paper that we can seamlessly run the process for different angles, so for different geometries in that case, for different interactions, which means different particles.

00:21:47: And the model is very, very flexible in running those parameters and gives you the correct answer for a wide range of different parameter sets.

00:21:56: So these are the two things and at some point I was really amazed how well this deep learning model can actually capture this different physics because in the end it's really different physics which we're modeling without telling the model what type of physics it is.

00:22:09: To close I think we are pretty far in this process and lots more exciting stuff is going

00:22:15: to come.

00:22:16: We can sense your excitement, Johannes.

00:22:19: It's great to have you with us here.

00:22:23: You promised at the beginning a revolution.

00:22:26: You talked about it, and we're really looking forward to see a simulation become real time.

00:22:32: That's what I take away here.

00:22:34: I hope you agree with that one.

00:22:36: Yes.

00:22:36: Thank you very much.

00:22:38: Listeners that want to get in touch with you can best do so through LinkedIn.

00:22:42: Otherwise, if you'd a listener have any question.

00:22:45: comment.

00:22:45: As always, please send a short email to Peter at AIPod.de.

00:22:50: Happy you've stayed with us so far, looking forward to have you with us again.

00:22:54: And Johannes, thank you very much.

00:22:57: And please keep us updated on the revolution you started in simulation.

00:23:04: Thank you very much, Peter.

00:23:05: It was a pleasure.

00:23:06: Bye bye.