Hey, I am Antypodish and this is my first Sanctuary developer log.
Allow me to introduce myself; I have several years of both academic and industrial experience, in electrical engineering, process control & automation for steel works, water services industry and nuclear robotics. My journey with game development however reaches far earlier than that as something that has been a hobby of mine. My first contact with Unity dates back to 2014, when I was an active member of the “From The Depths” game community and I was one of few game modders back in those days, doing some projects in my spare time. A few years later I thought about creating something of my own, inspired by my modding efforts. Since then, I have created multiple Unity projects and become an active member of the Unity forum community.
From day one, my primary focus was on performance and how to scale to large extremes. I generally love experimenting, and have incorporated powerful techniques in my projects such as DOTS, machine learning and procedural generation. I also occasionally publish stuff on my Youtube channel. You may find some interesting stuff there, if you decide to poke around. Please do not hesitate to reach out to me. You can also speak to me on the official Sanctuary Discord. 😉
Besides playing From the Depths, I have also enjoyed games such as Zero-K, Factorio, Minecraft, Civilization and the Simcity series. I am not locked on these games only, and whenever time allows to, I like to explore other games too. Those are just some of my favorites.
With inevitable life changes, I pursued a path in game development. Eventually I was discovered by Sanctuary founders nine2 and tatsu. They were looking for a DOTS programmer. So here I am now, a happy member of our Sanctuary team. 😊
One of my initial tasks was in the domain of animations. While this is still a work in progress, our current implementation uses bones and rigged meshes. This allows us to use and aim our turrets and guns on our models. An earlier implementation was using multiple meshes per model. I adapted our models’ workflow, to use unified skinned meshes. This increased the performance, especially when considering there are thousands of units on the battlefield that need to be adjusted at a time. I also added support for future levels of detail. This allows our game to perform more fluidly at various zoom levels. This adds a lot of complexity when it comes to skinned meshes since it’s impossible to simply copy out all the possibilities as it’d take too much memory. On top of that we are using the DOTS framework, which means our code is designed to work with multiple threads. But all of this is solvable with a bit of commitment.
One of the experiments I conducted for our game logic and animations was to test if multi-gunned turrets can aim with different properties. For example, if the gun on the right of a tank can aim further than the gun on the left side. This would require different elevation of the unit cannons as the trajectory arcs would require different angles. This is technically moddable now, but won’t be a feature on all units. Of course, we want to support multi turreted units so I just did it. Future naval units such as ships may first come into mind. But also, it may become a sensible feature on flying units, with turrets underneath, attached to the belly, etc.
While I am talking about animation and bones, I looked into an animation solution of walking units as well, using inverse kinematics (FABRIC & CCD). Here I considered using procedural walking. On one hand, this adds complexity, but on the other it solves a number of issues for our modelling team, as they don’t need to worry about creating stock animations for walking units. This is also not a small feat, especially in conjunction with DOTS. But the payoff is huge. Consider multi-legged units like a spider walking around an uneven terrain; it would look extremely realistic and legs would not be hovering or clipping into ground! However, multi legged creatures walk differently than bipedal ones. At this point it is unclear at what level our final solution will be when we release. But it is these kinds of challenges that drive me.
As may be clear by now, Sanctuary RTS aims to utilize as much power from your CPU as possible. However, we do not only need to focus on performance, we also want to make game moddable with LUA support to allow it to be extensible and configurable. This is not a small challenge. We want to simulate 10,000 active units on the battlefield, with all of them performing potentially multiple actions such as shooting, construction, pathfinding, altering economy, etc. In an RTS, the situation on the battlefield is constantly changing. On top of that, we aim all this to be supported over the network for multiplayer as well.
Many games use one or two CPU threads, leaving the power of the CPU under-utilized, while offloading their pretty visual effects to the GPU. So why not maintain the existing model for Sanctuary? Surely we can keep doing the cool visual stuff on the GPU and drive the CPU to do all the simulation? Surely GPUs are powerful enough given they work well enough in other games? But in our project this is especially important, since it operates at such an enormous scale you want to reduce the sending of heavy data back and forth from CPU to GPU.
Our recent testing has shown stable 60fps for 10k units, on a mid class PC, with a modern eight core I7 processor. This is only possible at this scale since we created our game from the ground up to maximize the use of multiple cores wherever feasible through DOTS.
While multithreading is important, there are many other areas that are equally important from a performance standpoint. Consider garbage collection, the way code cleans up memory that is no longer being used. While on a job chasing performance, I was hunting for major GC occurrences and bottlenecks. I investigated how to mitigate these and why these would occur in live gameplay. It was important to address these to avoid unwanted game jitters or lag. This is a part of my job that is never over. There is always more optimization that can be done.
One method I used to optimize our game was to question whether we need to run every script on every game frame. Considering the time span of 0.1 second. If the game frame is running at 60FPS, that requires about 16ms per frames, to deliver a smooth experience. Real time strategy games are generally less demanding in terms of how fast certain things need to respond. For example in First Person Shooters (FPS) games, 20fps or 40fps delay may be the difference between hitting a head shot or not. You can see this very clearly when an FPS game lags in live multiplayer; although there is a lot of hiding done by the developers.
However, in simulated projectile real time strategy games, instantaneous effects are not often so critical, but the responsiveness of inputs made by the player are. Now we can consider which tasks don’t have to happen every frame. We can make units calculate aiming, or look for new paths less often, so long as they occur within a reasonable period. One of my tasks was to undertake the job of organizing and scheduling how our game frames operate.
I had to decide which tasks were happening on a single simulation frame, across a dedicated 6 frame model that I constructed. All of that with multi-threaded compatibility in mind. A more difficult constraint was that if a player is a game host; they would need to simulate both the host and client codes simultaneously. Spending half of the available frames for host simulation and other half for client simulation, would allow the reduction of stress of any singular frame. Hence, we could ensure a smoother gameplay experience for a host, and better CPU utilization. These are the types of challenges I have the most fun in! 😊
For now, I think this is now a good stopping time for me, writing this introductory dev log. I hope you enjoyed reading it.
Until the next one,
Anty