Learning from past successes and failures

[Vexus]

I would point out: they listened to players, and it's a single player game which is much easier to control for.

Definitely, if the Starbase devs see everyone is using rocket launchers and no one is using some other weapon, they can make adjustments, with no need for feedback. But a lot of the nuance of the game will come down to player experience. I can already see a lot of complaints that the devs should ignore: "It's too hard to fly!" "It's too hard to spot friends from enemies!" "It takes too long to rebuild my ship!" Many of these kinds of complaints are givens for a skilled PvP game. Others, like the ship building, will be solved by the evolution of the game with factions and factories. Some other poor suggestion like, "We need a ship weapon that disables the engines of another ship for 30 seconds!" also needs to be thrown out the window. Finding analytics for some of this stuff is unlikely. PvP games are much different than single player puzzles. All sides of the equation need to be considered, and that's a tough job.

Not sure where or if I mentioned it, but just recently, a new Atlas patch introduced extremely overpowered weapons into the game. A base ship cannon in the game does 100% damage. The best practical ship cannons you could find through normal gameplay prior to the new patch were upwards of 180% of base damage - still very overpowered. The devs seemed to have listened to some top groups, because the new patch introduced ship cannons upwards of 260% or more - doubling the bonus ships otherwise had previously. In addition, high durability ship planks were added at the same time - meaning a high end ship can take even more damage. Combined with the high damage output, this means there's no chance any new player has in defeating, or even damaging, a high end ship. Whereas a balanced game has a small range in effectiveness between the baseline and best 'stuff' in the game - 10-30%, Atlas has given the top end damage 160% or more, compounded by increased defense, leading to a high end ship being 500-600% more effective than the baseline.

I think the Atlas change was in response to devs listening to select groups complain that time to kill was too long. Instead of cutting back and making all ships more even in effectiveness, they buffed the high end to the extreme so high end ships could kill other high end ships faster. When if they just resorted towards baseline stats, ship fights would be balanced - and it would be just as likely to lose your own ship as it would be to kill another ship. Meaning skill and preparation would determine the outcome. Not who has the overpowered stuff.

Those devs are pretty birdbrained though. All over the place.

 

[Zistack]

@Vexus: Then lemme walk you though how the devs could have used analytics to figure out what the real problem was and come to the "correct" conclusion of simply rebalancing the ship stats to be more fair.

Let's say that they actually did what I would suggest and paid attention to the problem but not the solution. The problem statement is "Time to kill is too long.". What they should do is instrument the game with machinery that tracks some basic stats about ship engagements. How long do they last? What kind of ships do the different parties have? How long do the different vessels survive? After looking at these numbers, what they should see is that ships with high defense stats take a long time to die, regardless of what they're up against. They would have realized that even in fights where one side is clearly going to win, the battle ends up being drawn out because of arbitrarily large stat bonuses. They should also see that the highest-tier ships basically never lose to the lowest-tier ships, because the stat differences simply cannot be surmounted with any amount of skill or strategy. Assuming the devs were sane, they would have realized that people with overpowered stuff have an unfair advantage over the other players, ultimately making combat more of a tedious ritual than a fun activity. They would come to the conclusion that the stat differences between the best and worst stuff needs to be nerfed in order to give newer and poorer players a chance against the guys at the top.

I've never played Atlas, so I'm kind of guessing at what the game was like with the misfeatures you're describing, but this example should illustrate my point. If they measured the proper things, they could have at least determined what the true problem was, rather than just listening to some random players complaining that noobstomping takes too long. Analytics can't tell you what the solution to a problem will be, but it can give you perspectives on problems that the players face that aren't necessarily properly represented by what players are actually saying. There really isn't a meaningful difference between single-player and multi-player games in this regard. You're gathering data on the emergent behavior of the system either way. More agents just means more variables and more data.

 

[Vexus]

How long do they last?

This made me laugh.

Unfortunately, it's not like some single-player or 1v1 battle which can be tracked until one side definitively wins. In Atlas for example, a fight might occur; one ship runs out of repair materials or ammo, retreats to a nearby island, spends time doing a resupply on the island, then coming back to the fight area to re-engage, maybe finding another ship to fight, maybe finding the same ship to fight. It's not something you can put a number on. In addition, a high end ship destroying a low end ship dies extremely quickly - much like I imagine the month-old player-designed ships in Starbase will absolutely crush the dev-made faction fighters we've seen. What about multiple ships in an area, where one is the resupply ship for the other. Or a galleon - largest ship - as an area-of-denial component preventing smaller attackers from getting close to other defending ships. It's not a match-based game, not a single player game, and not easily quantifiable on many points.

Also, this doesn't require any underlying analytics of the game. If a brand new player, with their first ship, cannot hope to even put a dent into a high end ship, there is an obvious problem. It's like playing a game of dice, one player given 1d6, and the other 1d100 + 6, and having them roll dice against each other. The 1d100 + 6 is going to win every single time. No analytics required.

Back to Starbase - a game where it seems everyone will be on the same level playing field. This is great. I love this. What sort of analytics would you suggest? Considering the 1000's of different ship designs and so on, which you cannot really account for, you have material resistance, material weight, weapon weight, ammo weight, ammo type and function, weapon damage, weapon speed and so on - in terms of PvP balance. Sure, you could see how many generators or fuel cells were exploded by a certain type of ammo, but that would be a terrible way to balance things. "We see autocannon getting the bulk of critical hits, and the railgun rarely does, so we're going to buff railgun." No. They just serve different purposes, and as such the 'metrics' will come out completely different for their usage and player skill behind those weapons.

This somehow reminded me of an issue in Worlds Adrift, where at one point, the devs posted a "heat map" of player deaths, and showed how most player deaths occurred in the starter areas. They then used that information as justification for creating a PvE server. I pointed out - of course most player deaths are going to be in the starter areas! Players are learning the game there! There were also barriers-of-entry into the high-end zones, so there were few other players there. This is an example of how blindly following analytics is going to skew information. If that dev time was spent focusing on player concerns, and then getting data from those specific areas of concern, it would be much different.

Which is where analytics does work. You take a major concern in the community - as a whole - and then get data on that specific thing. If one weapon is determined to be too powerful from the community, you check that out to see if that is really the case, and then you will have good data to act upon for adjusting damage or some other aspect of that weapon. Weapons, thankfully, are very easy to adjust for and balance. It is the tricky stuff like time-to-kill and so on that gets out of the realm of something that can be solved with some math adjustments. For example, if ships in Starbase die very fast in combat, you'll likely hear players loudly asking for shields! All sorts of energy shields and capacitors and so on and so forth. If the devs capitulate to those demands - there are shockwaves of effects that will hit the rest of the game world. No longer are throw-away ships in demand. The demand for lots of basic resources drop. High-end resources that create shields goes up in demand, and so on. It's not an easy one-off fix and it's not one where you can just patch it later because it's a single player game that players can live with until you get around to it.

The bottom line is the first step is listening to players. That gives the starting point to then get to the root of the problem. In my UO example above, if they asked the players -why do you guys keep slaughtering every single animal in sight? They would have heard back the reasons behind it - because the skill system demands it. As such, instead of wasting time trying to come up with an eventually futile solution, they'd be able to tackle the problem from the perspective of the source of the problem - the players. Instead, they likely ran analytics, showing that, for example, 1000 animals die per hour, and decided, "We just need to increase the spawn rate to 2000 per hour, problem solved!" It was not known at the time, but that solution ended up being incorrect.

Players have concerns; analytics and data should be used to find out if those players' concerns are worth changing the game for. I think the issue for Atlas is they are listening to only a small group of players.

Extra credit example: Rend was pretty fun when it first launched. One of the big animals to kill near your home base, a mammoth-like creature, was hard to kill because it gave a lot of loot. Players would band together in a pack to kill the animal. But it was still hard to kill early on. One streamer suggested the devs add in a weak spot for critical damage on the animal's rear end. They implemented that change quickly, and it led to a lot of players working together to take down this animal, because their efforts were more effective. It began with player feedback, and since weak spots were present on many other mobs in the game, it worked out, and people cooperated more, and the game was more fun because of it.

There is a huge difference between a single player game, where you can easily track the output of one player against a set game encounter, and a multiplayer game, where there might be 1 or 100 people involved in the situation. In the later situation, there's plenty of metrics that are worth tracking and valuable to track, but overall it is the player experience you are trying to solve for, not a spreadsheet.

 

[Zistack]

Unfortunately, it's not like some single-player or 1v1 battle which can be tracked until one side definitively wins.

Actually, games like this automatically detect engagements all the time. There is no issue here. Even if a battle occurs over the course of several engagements, the data will still show clear patterns.

Also, this doesn't require any underlying analytics of the game. If a brand new player, with their first ship, cannot hope to even put a dent into a high end ship, there is an obvious problem.

True, but sometimes developers can be blindsided by their own assumptions. They might have some idea about what happens in the game based on the mechanics that they've implemented which is different from what actually happens. Unless they actually bother to check their assumptions, they might never realize this.

Back to Starbase - a game where it seems everyone will be on the same level playing field. This is great. I love this. What sort of analytics would you suggest? Considering the 1000's of different ship designs and so on, which you cannot really account for, you have material resistance, material weight, weapon weight, ammo weight, ammo type and function, weapon damage, weapon speed and so on - in terms of PvP balance.

don't underestimate the power of metadata. I would suggest basically the same as before, except that some extra care has to be taken when classifying ships themselves. I'd measure ship mass, breaking that down into the masses of the individual materials involved. I might even specifically look at the mass and materials that specifically make up the ship's hull (frame components and plates). I'd look at weapon loadout by taking inventory of all weapon parts attached. I'd track generator capacity and fuel reserves. I'd record the thrust capacity of the ship.

This somehow reminded me of an issue in Worlds Adrift, where at one point, the devs posted a "heat map" of player deaths, and showed how most player deaths occurred in the starter areas. They then used that information as justification for creating a PvE server. I pointed out - of course most player deaths are going to be in the starter areas! Players are learning the game there! There were also barriers-of-entry into the high-end zones, so there were few other players there. This is an example of how blindly following analytics is going to skew information. If that dev time was spent focusing on player concerns, and then getting data from those specific areas of concern, it would be much different.

This isn't a problem with analytics. This is a problem with the devs. They gathered data, but didn't stop to ask the question 'Why?'. This will be a problem no matter what mechanism people use to investigate problems. Gathering the data required to realize where players were dying was merely the first step. When scientists try to determine nature's laws, they don't just blindly gather data on everyday phenomena, they design experiments and refine their data gathering strategies to test specific hypotheses in order to truly understand what is happening.

It is the tricky stuff like time-to-kill and so on that gets out of the realm of something that can be solved with some math adjustments.

Now I have to call you out. This statement is in direct contradiction with the Atlas example that you had pointed out earlier, where you described the solution to their time-to-kill problem as precisely that.

Players have concerns; analytics and data should be used to find out if those players' concerns are worth changing the game for. I think the issue for Atlas is they are listening to only a small group of players.

I don't think anyone here disagrees with you on this point. In fact, this is precisely why I, Recatek, and Caddrel are advocating for gathering data and making a point of understanding the real problems that players are facing, rather than just implementing the suggestions of a noisy few.

There is a huge difference between a single player game, where you can easily track the output of one player against a set game encounter, and a multiplayer game, where there might be 1 or 100 people involved in the situation. In the later situation, there's plenty of metrics that are worth tracking and valuable to track, but overall it is the player experience you are trying to solve for, not a spreadsheet.

This statement makes literally no sense. In both multiplayer and singleplayer games, you're trying to learn about player experience. You're never just trying to solve a spreadsheet, and the number of players involved is irrelevant to the nature of the problem. I have no idea what makes you think that a game being multiplayer is so special, and I have no idea why you seem to think that we're making metrics out to be the end-all be-all feedback mechanism for game design. I'm beginning to think that you don't actually understand the nature of the problem we're discussing or the nature of the solution that we've been discussing. After watching you take a self-contradictory position that isn't even clearly-defined, I'm starting to find it difficult to take what you're saying seriously.

 

[Vexus]

This statement is in direct contradiction with the Atlas example that you had pointed out earlier, where you described the solution to their time-to-kill problem as precisely that.

Making a math adjustment to attempt to solve a problem does not actually solve the problem in this case. There is a distinct difference between solving a problem and attempting to solve a problem (along with creating many, many more problems in the process). The attempt made was not the solution. I think you can understand the difference; no call out required.

I don't think anyone here disagrees with you on this point.

Re: listening to players first.

One of the posts you agreed with on this thread was one claiming devs should never listen to players, though you clarified your position a bit better later. If the position is "listen to players but verify their motives with data" then we can agree.

I have no idea what makes you think that a game being multiplayer is so special

the number of players involved is irrelevant to the nature of the problem.

It's very simple; there are only a small number of successful MMO games to ever have existed, compared to the thousands of random single-player games and so on that do well on their own. There is a special formula for an MMO, and it's not easily pinned down as to what that formula is. Where if you want to make a single-player puzzle game, it's pretty simple overall. The fact no game has been able to be the "WoW killer" though many have tried is a good indication that there is something special to it. Which gets into the next point; WoW does a lot of DPS balancing in addition to considering the game experience. If you have some of the best gear but someone else of another class is just far outperforming you in damage 'just because', your game experience is not as good, so they do balance for the spreadsheet so to speak. Of course it's not just spreadsheet focus, however the indication by you and others is to gather data and that will help solve the problem - while the players are leaving the game due to some poor game mechanic. Sometimes the problem is one that would take longer to implement the right tools to track player interaction and do analytics than it would be to solve the problem. It is not the be-all end-all to just 'gather data' as if that will solve the problem. In the end, the final decision/solution may have nothing to do with the data, or the data could steer you in the wrong direction.

The number of players involved drastically changes the nature of many problems in a multiplayer game. If the game designers set a goal of 10 players mining out X number of asteroids per hour, and that is their baseline experience, the target they expect will occur, and then 1000 players come along and mine out 100X number of asteroids per hour, there is a huge, distinct impact of the larger number of players. You could claim the nature of the problem was the devs setting an experience expectation too low - but they have to set some expectation. Players come along in a multiplayer environment and mess everything up. If the expectation was 1000 players mining 100X per hour, and only 10 players come in mining X per hour, there are too many resources, crashing the economy as it floods with lots of cheap material relative to the number of players consuming that material. The large variance in the number of players has a huge impact on every problem in a multiplayer game. In a PvP setting, the ability to leverage numbers against an enemy does make for a tricky problem for devs, where if they could have only limited team sizes to 5v5 matches, they could handle expectations, but when 500 go and crush 5 other players, the nature of the problem has a lot to do with the number of players involved. I don't see how you can make the claim that the number of players is irrelevant.

I'll ask again because it was skimmed over: What sort of analytics would you suggest? Is there some abstract problem like time-to-kill in Starbase, which you'd want to track data on? What analytics would you suggest to help you solve this problem?

 

[Burnside]

Excellent points on both sides of the debate, data-gathering can only help implement a solution, it guarantees nothing, not even an accurate assessment of the problem, it also requires the variable quality of the dev team and the questions they ask, which I think Frozenbyte has nailed down so far, and the quality of the player-base. What makes each MMO good or bad is nebulous and we're kind of chasing each other's tails at this point.