Greedy Goblin

Thursday, January 23, 2014

Eliminating the soul-crushing lag

So, 6VTD happened again: in HED-GP, the side first on grid obliterated the side arriving later. -A- claims that it caught them by surprise, believing that server issues are fixed, but there were no such announcement since 6VDT, so I don't see any reason for their belief.

But the post isn't about their folly. It's about the fact that CCP can't fix this. I don't mean it's hard for them, I mean it's theoretically impossible to provide smooth gaming for everyone in large battles. Why? Because to play, I must have data of objects on grid. My overview must contain the name, position, relative speed of every ship, drone, missile, wreck, bubble and any other object. The number of objects is directly proportional to the number of players. Twice as many players have twice as many ships, drones, whatever. So the load created to serve me is proportional to the number of other players. However they must be served too, so the total server load is const*n*n, where n is the number of attending players. This means that to serve twice as many players, you need 4x more server resources. This quadratic scaling means that no matter how much resources CCP utilizes, it can't keep up with players, as both processor power and code efficiency is linear (2x stronger processor runs the battle 2x faster).

EVE is a unique game in the MMO landscape in allowing all players to take part of a fight and make such fights meaningful. You simply can't enter with more than 15v15 into a World of Tanks battle. Most MMOs are placing players on different shards and there is no real reason for the players of a shard to gather in some position and fight each other. In WoW with the introduction of Wintergrasp such reason was created, it lagged out the server horribly and was solved by maxing out participants to 80v80. Such solution is impossible in EVE as it would remove the politics and meta-gaming. If fleet fights are limited to 500v500, than a 500-man elite alliance could defeat anyone, take any staging system at will as the defenders can't outnumber them.

So we see that limiting n to a reasonable number is impossible, and getting server resources for n*n load is also impossible. How to solve it? First let's ask how it is solved now? Practically by burnout-limiting. While the server doesn't say that you can't join, by TiDi and lag it makes sure that you hate it enough to don't come back for another fight. This means that victory goes to not the stronger but to the nerd-er: N3 won HED-GP practically without a fight because they had more nerds who were ready to sit on the IHUB for hours to secure the on-grid-first advantage. CFC won 6VDT the same way. This is a horrible mechanic.

Now the question: how to solve the unsolvable, letting infinite number of players play the same battle while the server can reasonably handle only N players? I've written about it already, but now I refresh and simplify the idea. It requires the session-change problem to be solved, now loading a ship to grid needs lot of resources. CCP is working on it, and it's a linear-scaling problem (your session change has nothing to do with me). Assuming ships can jump in and out of a system with little load, let me introduce the "ghost-ship" mechanic.
  • When the server load is so high that TiDi is below 33% for more than a minute, a ghost factor (G) is calculated in order to return to no TiDi. Due to the n*n scaling, a ghost factor of 2 increases server speed 4x. The ghost factor is an integer.
  • The players in the system are separated into G (+1) groups. The selection is random (you have no control over which group you are placed), but it respects ship types, for example if there are 3 groups and 30 supercarriers in system, each group gets 10 supers.
  • Only one ghost group is present in the system. The others are instantly removed from the fight with all their drones. The players remain on local channel and can chat, marked by a ghost icon. Link bonuses stay if the squad commander is a ghost.
  • The active ships have their damage and repping multiplied by G. So a fleet has the same DPS, alpha and rep ability, despite only 1/G ships are present. Ewar, anti-ewar (like tracking links), neuting and energy transfers are not multiplied, since their effect is already multiplied by affecting a stronger target (if you ECM a ship, it can't apply its multiplied damage).
  • Ghosting cycles in regular periods, (like once in 3 minutes), the currently playing group is removed and the next group is placed on grid to play. Your modules are in the position as you left them.
  • Certain ships are immune to ghosting (they are the +1 group mentioned above), they are always on grid and receive no damage/rep bonus: titans, dreads, carriers in triage, fleet and wing boosters and commanders. Each fleet can also mark 5% of its members, they will to be immune too.
  • Ghosting doesn't break target lock. If you are on grid and your target is ghosted out, you keep targeting him, just can't interact. If you return from ghost state, your previous targets are still locked, unless they left grid or died while you were a ghost.
  • If you were orbiting, approaching or kept in range something or was flying in straight line, you return to the position where you'd be if you'd travel normally. Drones act the same way if they were orbiting/attacking something. The calculation is a one-step approximation, so it won't be accurate, but OK. I mean you were orbiting an anchor from 5km, you return 5km from him, but maybe you'll be above him, while in non-ghosted form you'd be below him.
  • If you are in warp when ghosting cycles, you aren't ghosted instantly, first you can land from warp and then you get 5 seconds to anchor up before you are removed from the grid (to make sure that fleets aren't split up by the ghosting). If you are fleetwarped while being a ghost you enter warp normally and after landing, your ship automatically anchors back if it was anchored to a fleetmate. Ghosts can't be fleetwarped if they were manually jammed in the moment they left as ghosts or if more than 50% of the non-ghost fleetmates are in a bubble (which means that they would be in a bubble too).
  • If you jump/undock/log into the system as newcomer, you are randomly assigned to one of the ghost groups. If that group is not active, you are instantly removed and will return with your groupmates.
  • Your timers don't expire while being ghosted. So if you log out while ghost, your ship returns to the field if it had PvP timer. If you didn't have any timer, it logs out.
  • As the total number of ships in the system is changing, the ghost factor is recalculated, and next cycle more or less players enter, keeping the rule that those who were out longest, should come in first. Let's say there was a ghost factor 3, group 1 active, group 2 played before and 3 before them. If the player number allows ghost factor 2, then next cycle the previous group 3 and half of group 2 can enter.
While this indeed means that only 1/G players can play, you still can play more than now. Currently if the server is having twice as many players as it should, it runs in 25% TiDi (because 1/(2*2)), so in an hour you can actually activate 15 minutes worth of modules. With ghost factor 2, which decreases server load to 1/4 too, you can play 30 minutes an hour, and can spend the other half an hour grabbing a sandwich or doing something on the other screen, instead of watching a veeeeery slow module to finish cycling so you could act. Don't forget that you are contributing to the battle while in ghost mode due to your teammates having increased damage and rep thanks to your presence.


Kontalaa said...

Typically you solve such problems by instancing.

In the case of eve i would simply instantiate more grinds to fight on. As every grind can (theoretically) run on any node you can serve m grids with m nodes.

As a consequence you still can't have all ppl on the same grid - BUT if the game-mechanics get changed so that you have to attend at multiple grids (Orbitting SBU + Shooting I-HUB + Shooting TCU + Shooting Station - all at the same time) you split fleets and have to reconsolidate on-the-fly if somethings someplace go wrong.

TiDi would also only affect the Grid instead of the whole System.

This wont get rid of the O(n²)-Problemetic but will reduce it to O((n/m)²) (m being the grids fought on; perfect load-balance in-between grids assumed).

To prevent cascading effects of players meeting at one point (escalation-spiral) you also must be able to claim the objective (conquering/defending system) without one of the grids held. This process will need more time, but in that case tidi is playing in your favour (as you have "more" time than the others on the slow grid).

but well... eve still has a pretty old codebase (pos-code comes to mind -.-), which needs to be changed. I'm looking forward to having farms&fields as objectives on such grids..

Anonymous said...

I'd imagine most people would prefer playing at 10% speed than being removed from the battle even for a moderate amount of the time.

We don't measure how many times our modules actually cycle, we measure how long we're on the battlefield.

Anonymous said...

It amuses me somewhat that the problems the two sides have identified are "Their side invites any nerd with a pulse", and "Drones assist!!!one!1"... and your approach is to implement Drone Assist for the ship itself, and decrease the level of involvement for the individual player.

It would have implications that I'm not entirely comfortable with (blobbing would become the tool of choice for the elite PvP (rich) coalitions, with massive amounts of PLEXed multiboxing), but it deserves some thought.

Gevlon said...

@Kontalaa: If you cut both fleets in half and let them fight on different nodes (and merge them later), you prevent both halfs shooting the same primary, halving their alpha. Half Maelstrom fleet can't break half Rokh fleet.

@Anonymous: TiDi is already exceeded. In both 6VDT and HED there was 10% AND lag. As a result, the players couldn't cycle their modules at all, being unable to play. I'm pretty sure that a CFC/RUS player who lost his dread while he was watching tunnel animation would pretty much prefer not to be on the battlefield.

Lucas Kell said...

They in fact can solve this issue, they just refuse.
Basically, the EVE server is single threaded, meaning that it can only run on a single processor core (because they chose to write it in stackless python). This is why we reach a cap so easily, because that processor core can only do one thing at a time and is restricted by clock speed, which the hardware industry no longer focusses on as a priority. Instead to get more processor power the industry focusses on getting more cores working together.

If CCP were to multithread the server, which they've stated themselves they need to do but don't really want to touch the old code base, they'd resolve their issues overnight. The load could be spread onto multiple processors with ease, and even other servers could lend their power. This is how modern day servers operate, effectively allowing infinite power, only limited by how many servers they have available.

A solution like you've suggested would be a pain to manage and be less fun than a huge fleet battle (when it works properly). The amount of time and effort it would take to write it, they could instead make a good headway into rewriting their server code in a threaded language.

Gevlon said...

Multithreading wouldn't solve the problem, just elevate the cap. The number of processors increase the speed less than linarly (10 processors < 10x speed), so if you have 10x more people, you still need 100 processors. Assuming EVE getting popularity (if not, there is no point spending resources on recoding), even 50K battles are possible. Do you seriously expect CCP to have a NASA-equal server farm to handle it?

Ghosting can handle INFINITE number of players, granted, they would spend most of the time ghosted out.

maxim said...

Don't fall in the trap of trying to reach "BEST SOLUTION EVER FOR ALL CASES".

If Eve has indeed reached the player cap, then you only need a solution that is able to support the player cap (with a bit of buffer for any unexpected growth).

Lucas Kell said...

"Do you seriously expect CCP to have a NASA-equal server farm to handle it?"
They currently manage to handle 4k people on a single core. Modern day retail servers will have at least 32 cores. They do in fact disable cores on their servers at the moment to spread heat from the extreme overclocking they have to do to push it.

Ghosting would take a lot of work to implement and honestly I don't think it would be effective. For starters you would have to session change everyone, and your +1 fleet wouldn't work, since to get that working, the servers would have to communicate in the same way multithreading allows. You'd also have the issue that while the fleets may split, the comms wouldn't, so it would be a nightmare for an FC to manage. Then you have the issue of ensuring that each split has the shiptypes it needs, and hasn't just been dumped with no support against an impossible fleet. Balancing it would be pretty much impossible.

Honestly though, server architecture is cheap these days. And bear in mind that to get the performance they have now, they actually do have a military grade CPU, which they had to get clearance to go see. Not to mention that since they have to disable cores to overclock, they'd simply need to lower the overclock and enable those cores to get more overall power.

Anonymous said...

"Multithreading wouldn't solve the problem, just elevate the cap. The number of processors increase the speed less than linarly (10 processors < 10x speed), so if you have 10x more people, you still need 100 processors. Assuming EVE getting popularity (if not, there is no point spending resources on recoding), even 50K battles are possible. Do you seriously expect CCP to have a NASA-equal server farm to handle it?"

It would solve the problem. First, only a part of the server load is O(n²). And if I think about what a CCP employee said about that the lag in Jita is mostly because of ships undocking and the resulting skill recalculation, I wouldn't underestimate this part.

Second, most of the servers of CCP don't really do all that much most of the time anyway. And with multi-threaded code the load could be much better balanced between the servers.

That wouldn't scale to infinity, however it could support a lot larger fights than we currently have in EVE.

Anonymous said...

The solution is algorithm and number based. As such, players - especially Eve players - will find a way to exploit the fuck out of it, one way or another.

The problem, and solution lies at design levels. Change the game and mechanics in a way that would encourage multi-front battles spread over larger areas instead of thousands of people trying to dogpile on single server. Ever read about the Battle of Leyte Gulf, where IJN bait carrier fleet led away sizable USN detachment away from the main battle, almost turning the tide despite IJN being outnumbered several times over, with USN battlegroups each larger than total IJN force in the region? Such a thing is completely impossible in EVE. The design needs more depth. Toss in multithreading on top of that and we'll be fine for anotehr couple hundred thousand subscribers.

Change SOV mechanic said...

Random events like removing a group and placing another group out of the "ghost pool" would make it impossible to win or lose a battle.

If you win your opponent will say: "CCP won the fight by giving us shitty groups!!! We would have had won otherwise!!!"
If you lose you will say:
"We had better fleet, more pilots, but the server f*cked up splitting our efficient groups into sitting ducks!!!"

How is that different from "We lost the fight because X was on grid before us!"?

It wouldn't solve the problem.

I really like how the SOV grinding works in factional warfare. A single frigate running a novice plex while the enemy is bashing the iHub can defend the whole system for some time. You don't have to wait for timers to happen, if you can't take system X you reorganize your fleet and jump over to system Y.
It doesn't matter if your enemy is "plexing" a novice plex with 50 frigates while you take 3 of his novice plex with one frigate each.
You don't focus that much on one system @ xx:xx UTC.

Sure, for nullsec it would require some tweaks and changes as the FW mechanic is focused on instant action and systems can be flipped too fast.

Oska Rus said...

I dont gnow about your IT background goblin but n*n problems are generally considered as easy.

problem of eves performance comes from its legacy of 2000s when noone even dreamt about dynamic load ballancing. Today it is common part of many applications. It allows to assign more resources to already running application which is under serious load or deassign resources from idling instance.

Eve server would require mayor overhaul to be able to do this because of its initial design and such overhaul is super costly and risky.

All other solutions are just temporrary patches and bandaids but couldnt possibly solve situation where you have to have assigned certain resources for every empty wormhole or nullsec system and cant concentrate them fast enough for such a large events.

Your sollution is one of that temporrary patches and is overly complicated thus prone to bugs, other performance problems and abuse.

Anonymous said...

EvE advertising is based on "Massive space battles" Cutting them up smaller teams would prevent that.

I'd fix it by grouping ships and treating them as a single unit. Aggregate similar ships into groups of 10. Individuals still get to active their modules (and F1 their weapons) but the surver aggregates this so all anyone else sees is the totals for the group.
The net result mostly unchanged from current. Leader still picks the target. Fleet still presses F1, but server load is reduced by a factor of 10 (or however the ships are grouped).

Anonymous said...

I dont know if threading would be the solution. If it was, and if it was easily done, CCP probably would have done it by now. They are no fools. Ghosting sounds interesting, +1 for out-of-the-box thinking. But it also sounds indeed complicated to implement, let alone understand.
What about limiting systems to 2,001 players in local? Granted, that would give victory to the first large group there - but that is the situation already. With everyone knowing the limit, everyone has the same chance to sit in the system as long before as possible. Of course, people awake during during down-time would have an advantage, but that could be adressed by giving a number of wildcards to people in system before downtime. Well details. Main thing is, reduce that numbers-issue for all on an equal, well-known and easy to understand basis! Make the game playable and enjoyable above "having the largest battles".

Anonymous said...

Gevlon, you are proposing an *immersion breaking* solution which quite probably will not work. To my knowledge you do not have a background in software or network engineering so you probably should leave the solutions up to the experts. Sufficed to say, any solution which breaks immersion is no solution at all. I mean they could write code which aggregated every single thing in a fleet, figured out its dps and auto faught the whole battle with a basic calculation - but the player base would not consider this acceptable.

Multithreading is throwing out there by commentators a lot whenever lag crops up, as some sort of holy grail solution. Firstly, number of CPUs does not equal number of threads. Threading is one method of concurrency management which generally involves timeslicing. Sure you can run more 'single' threads in an embarrassingly parallel system and it will go faster, but running multiple *threads* is not predicated on having multiple processors.

They already manage concurrency using Stackless Python. It uses microthreads as it's approach to concurrency. There are others, such as the event loop/reactor pattern which is popularised by node.js. All of these are valid means of trying to manage concurrency but none of them will really solve the issue.

Should eve be able to take advantage of additional processor cores? absolutely. It should. That it doesn't is a major major fuckup on CCPs part. Will it solve the lag problem? No, or at least not entirely. The speed up will not be the "instant problem solver" that people think it will be.

@Kontalaa - as far as I am aware, each "grid" cannot be run on a node. The nodes go down to a system level only. That is the lowest level of granularity that is available.

Anonymous said...

The problem cannot be solved by trying to cram more ppl inside one grid.

Ghost groups are a nice ideia but its over-convoluted, as a programmer I can tell you the amount of unpredictable variables running arround amok with such an implemtation would make not only gameplay erratic but also make the dev team lifes a living hell.

Grids and gates can be improved for sure, and there is no reason for ships that are "new on grid" to be held defenseless with a black screen on their client, CCP can and needs to fix that aspect of session change. But that will only get you so far and will give less gameplay not more.

What we need to do is to remove the incentives of current gameplay to gang so close together on 1 single grid. I'll give you a few examples:
- Take advantage of multiple SBU's, link them and require multiple coordinated strikes. Put them *outside* systems instead of inside.

- The game needs low SP artillery hulls, AOE artillery and not Alpha strike artillery. Dynamic battlefields breathe of of this. Bombers provide cirurgical strikes, artillery provides flak fire. This pushes weak subcaps to pounce instead of just sharing grid with caps.

- Remove drone assist, not because of TiDi but because of gameplay, make it a skill to be able to control up to 5 assisted drones, no more. Under Tidi, clump drones together into 1 single entity (overview, bracket, HP kills drones as it reaches 1/5, 2/5 ...).

- Make systems and grids have a max player restriction. The argument of the first guy who gets there wins is the real problem. 10 soldiers filling up a 1x1 meter square of ground provide a tactical advantage why??? Cramming a coalition into a single system is adavantageous to sov mechanics why???

Sov rethinking is urgent and will solve more problems than programmers or hardware.

Gevlon said...

@Oska Rus: the problem is that all kind of computer solution is linear. So if 10x more people want to play, you need 100x more cores or threads or whatever. If 100x more people play, you need 10000x more. If you assume some reasonable success to EVE, you soon reach the point where all the computers of the planet couldn't handle the load.

@Dobablo: if players are acting, they must get information. If they get information, they are producing load. The n*n problem remains.

@Last anonymous: because soul-crushing lag is not immersion breaking. Because dying without being able to turn on hardeners is not immersion breaking. Because camping the system from downtime is not immersion breaking. Our only option is seeking a less immersion breaking way.

Anonymous said...

> If you assume some reasonable success to EVE, you soon reach the point where all the computers of the planet couldn't handle the load.

EVE has been around for 10 years, has during all that time never grown fast and has been stagnant for the past 3 years or so, I think we have a pretty good idea how successful it can be.

As Jester says, everyone who is into Science-Fiction sandbox games has probably heard of EVE at this point and is either playing it or know why he isn't playing it. EVE is currently the only game filling this niche and is way too adapted to its niche to ever have much appeal beyond it.

The goal is to keep the game stable at its current playerbase and not to lose players to competitors that may be crowding into the same niche (Star Citizen).

> (if not, there is no point spending resources on recoding)

This is not true. EVE cannot handle the current player numbers and if this state of affairs continues long enough players will leave. Remember that EVE is the only source of income CCP has, CCP has no choice but to invest whatever it takes to keep it alive.

Players aren't just paying for what the game currently is, their decision to play (and pay) today is influenced by what they expect for the future. I can live with a year or two of massive lagfests (has happened before after the introduction of fleet finder) as long as I know that CCP is actually working to fix the problem. If I knew that CCP has no inclination to fix the problem I would quit right now.

You may say "but the player number of EVE will decline in the long-term" but total player numbers aren't what causes the current lag. You can have 4,000 players in a battle with 35,000 concurrent users just as you can have a 4,000 player battle with 20,000 concurrent users. What makes these battles big is the degree of organization of EVE alliances/coalitions, not the limit set by total player counts.
And while player numbers may drop the process of centralization will only continue - battles won't get any smaller for a long time.

Oska Rus said...

@Gevlon: n*n is not really a problem. You can easily assemble enough computation power to hndle HED-GP battle and much larger and cost of those servers is fairly low compared to developers team worktime. When servers are able to have about 4k players on grid with 10% tidi there must be at least 10+- other nodes assigned to several groups of systems. If 4 of them could be assigned to HED-GP on player spike, battle of 8k should be possible on 10% tidi comfortably.

Problem is to assign that power to that solarsystem. Eve is currently unable to add servershards to already running solar systems.

And having computation power to handle HED-GP battle ready on all solar systems at all times starts to be immensely costly.

So far eve doesnt grow faster than computation power. Even when we takes it as n*n problem coputation power should doble every two years ('s_law) and eve would have to double every four years at least which it is not (

Von Keigai said...

Here is my solution to soul-crushing lag:
this is EVE: if you want something removed, kill it. ... When a particular server is sufficiently overloaded -- let us say, below 20% TIDI -- then there is a chance of a very damaging area-effect explosion.

How does it compare to yours? First, yours is highly complicated. I have read it twice now and still am trying to work out the implications. By contract mine is simple. Second, yours is highly immersion breaking (and yes, so is soul-crushing lag, but in a different way). Mine is not.

Provi Miner said...

I see three main problems:

1: You are a ghost, so you go grab a bite to eat (in your example you take a 30 min food break). Now lets assume ghosting works. With ghosting working fights would be fast there would be minimal tidi. That means your fleet (remember you are gone for 30 min) is getting hammered flat so new members are pop'd in to replace the loss's. So you come back from sandwich making only to find yourself in a medical bay because you were unghosted and not at the key board.

2: A pod is a ship, as long as that pod is on grid no new members can come in. So teams that are winning won't pod people just because they can increase their effectiveness by keeping new ships off grid. Oh and pods do zero damage so your X dps times 0 effectively equals zero.

3: So take the fights in provi recently, as many as 5 to 8 different groups were fighting with and against each over for hours. Who deiceds who gets to load grid? My point is whats to stop one side from repeatedly loading grid with different corps thereby increasing their percentage of force in the system?

Just some things I thought of

Gevlon said...

@Von Keigai: your solution would necessarily change the outcome of the battle (compared to the imaginary ideal case where the server has infinite power). So the outcome of the battle would depend on server load. Assuming it's properly coded, my solution would give the exact same battle outcome.

@Provi miner: the time slices should be small enough to prevent "the battle was over while I was a ghost".

The ghosting would affect everyone randomly, so if there are 5 different groups, every group is on grid with half ships, other half is ghosted.

Anonymous said...

because soul-crushing lag is not immersion breaking. Because dying without being able to turn on hardeners is not immersion breaking. Because camping the system from downtime is not immersion breaking. Our only option is seeking a less immersion breaking way.

Yes it is immersion breaking. But there is a difference between immersion breaking because of a system exceeding its limitations and *designing* immersion breaking into a core mechanic.

Also your "only option is seeking a less immersion breaking way" only serves to reinforce the fact that you have no idea what you are talking about when it comes to software and network engineering, or game design. As has been pointed out before there are technical solutions which can be explored which are far cheaper and far more straight forward than introducing a convoluted immersion breaking mechanic.

Von Keigai said...

Of course my idea would change the battle relative to the same ships fighting with infinite server power. It works via removing ships from grid in an evenhanded, but still random, fashion. The entire point of the idea is to reduce lag by removing ships. High TIDI is solved in two ways: (a) by encouraging FCs to not blob as much, and (b) by blowing up ships from both sides when TIDI get high, thereby lowering server load.

Your idea is basically to take a statistical sample of the whole system population, compute their effect, then scale up their effect based on the weight of the whole before being applied to the enemy sample. Even assuming players aren't gaming it (EVE player here: I've already got a couple potential ways to game it), random fluctuations in the sampling will inevitably affect the outcome, since they are amplified.

Still, I'll agree with you that ghosting would keep the current meta unchanged. That is, it preserves "blobbing wins". I want to change that, because I think it has bad effects on the metagame: blobbing brings the blue donut.

Kate 'On said...

I don't like the assumption. Instead of encouraging a 6 year old soccer game style of play, make it so you don't want to put 4k people into a system. I don't know why players haven't figured this out on their own, but put a harassment fleet in when the other player loaded grid. so sent some other forces to reinforce a whole bunch of other stuff.

But if people cannot figure this out, then implement a mechanic. Hell, once the server reaches 4k people, log them all off for what do I care? Next time, someone loads the server, you send your forces elesewhere to take on other soft targets, bring enough into that system to take everyone out of the equasion for a while.

Anonymous said...

You can't have infinite players.
google can't have the whole internet. ok. well. they can, they trough much tech and brains at anything they encounter. but that's google.

We as players don't know the bottle necks.
Alot of us (read all) don't have the slightest clue about server client software on that scale in that particular EVE case plus 10 years codebase complexity. We jerkoff about the technews that for example they moved the permadb on fast SSDs. and we like the graph pr0n. but we don't know what it means.

in IT: optimisation. is. a. huge. expensive. mind wrecking. thing!
Alot of time and resources go into 0.001 % optimisation. It is expensive as hell and doesn't workout all the time. on top of it you need expensive brains to solve the problem(s).

The "cheap" solutions. like. run the DB on fast SSDs. Are pretty much taken. Otherwise why would they look at the player heads including all the skills and implants on loading time.

Your ghost idea breaks down complexity. this is one of the key things to do. If this helps at all, we do not know. maybe it is something else.

I don't know how 10% TiDi + 2FPS client feels like. I surely can't see meaningfulness or any enjoyment from it.
If the objectives can be split. meaningful. than there is the opportunity to split the masses and only connect the objectives.

Or introduce another game that takes over if the nodes can't handle it :)
My creativity limits me. I'm sure CCP can come up with something to make a scalable scyfi "minigame" that works well from 5k up to 1m players. why not.
let that game kick in after 30min of max TiDi and local is still spiking. System transforms into a whole different game that can handle allot of players. it's still better than hardwarecapping and watching 10% tidi + 2FPS. Does someone really enjoys that slow fight?

But that is all very expensive.
TiDi was already a expensive solution. no ghosts or new game will be made to help here.

The easiest solution comes from the community itself. People can cap them selfes to have a meaningful experience. you can have good 1500 blob fights 5 times in 5 systems.

Provi Miner said...

Time slices small enough the battle isn't over while you are ghost. Well again thats my poin if you get up you could be unghosted and never know only to find yourself in your home system med bay.

if 5 groups go then each get half their ships? My point, and I appologize if I was unclear, might be better explained by this example. Lets say CFC allaince loads grid, and then Nulli loads grid (not n3 just nulli) then NC loads grid (again not n3 but NC) now n3 effectivly has a 2-1 advantage even though they are seperate groups. while most people looking at this would lump nulli with nc will the computer? The computer will assign all groups within the CFC with CFC but if there is no "existing" N3. Then those who were "formally a part of N3" will gain an advantage. What happened in the Sound wars would be a night mare with your solution. CVA and allies (call them provi but they are not a part of a formal alliance) Stain wagon (again not a formal alliance) BL (which wasn't a part of cfc at the time) PL (which wasn't a part of n3) N3, Bombers bar (which has no formal alliance) were effectively shooting each other at one point then helping one another at another point. No way to appropreate a fair system of who gets unghosted. In fact as did happen in 9UY once one party started losing everyone gang piled them till they could extract only a few moments later everyone was gang piling another group with the former piley joining in for kills. Your system appears to me to require people to declare their alligence formally before the battle. Which in eve is pointless you always keep open target slots for turn coats and unexpected droppers and bombers.

Unknown said...

This is actually one of the better ideas I've heard for fixing things.

Anonymous said...

Multi-front battles would be interesting but realistically I theorize this is what would happen:

Attacking group A has to destroy 5 structures at around the same time, so has to split their fleet up into 5 groups. Defending group B doesn't have to split their group up at all, they just defend one structure and they win, so you have A/5 members of fleet going up against B members of fleet. So, A must bring 5x more numbers, and that's what'll happen.

If there were some way to force both attackers and defenders to split their fleets up then we'd have something to work with. Artificially making a ship limit is dangerous as well, but could be an option.

Satori Okanata said...

I've been reading about it, and everywhere they say that eve is running in a clustered environment, but the thing is that it behaves much like WoW where every domain is an individual server. There can be hideous TIDI in jita while everything is smooth in piekura, 4 jumps away, just like it was in it's own server.

I think they have problems with how they do their load balancing if jita is dying while other places looks like the servers are scratching their balls and having a beer.

If their cluster was working correctly, the load balancing would make that, instead of having jita (or wherever the massive pew pew is happening) running at 10% speed, all of the universe should be at 99% wich would be unnoticeable for anyone.

A good example of this is, have you ever noticed lag while searching google?

Gevlon said...

@Provi miner: ghost cycles are fixed. If you are a ghost, you know exactly when to return to your computer. There is a countdown.

The ghosting is random. So 3000 CFC sits on grid, they are already ghosted because they are a lot. So 1600 CFC sits on grid (200 ghost immune, 1400 active, other 1400 ghost)

Now 2000 N3 jumps in, 500 of them are ghost immune because they have several titans, blap dreads).

750 doesn't even materialize, they exit jump tunnel as a ghost. Other 750 will materialize.

So on field there will be:
1400 active CFC with 2x damage modifier
200 ghost immune CFC
750 active N3 with 2x damage modifier
500 ghost immune N3

Druur Monakh said...

Satori Okanata: "have you ever noticed lag while searching google"

Wrong comparison - Google only does search, whereas EVE does a simulation. Distributed searches don't have to communicate between the individual data hosts; and also, the searches of different users don't need to coordinate with each other. More importantly: the search results don't have to be absolutely correct all the time; eventual correctness is good enough.

It is true that EVE could make better use of their clustered environment, but that would take more programming effort than a simple map/reduce search. And one thing to keep in mind is that communication between processes is slow (compared to raw CPU speeds). If the processes sit on different servers, it's abysmal.

BleepBloop said...

@Gevlon: Sadly this simply wouldn't work without either limiting the ghost group immensely or increasing the length of each ghost cycle.

If you have a ghost group size of 500 then in a 4,000 strong fleet battle you'd on average spend 1/8 of your time on-grid, which as you said earlier is more than 1/10 so on-paper this would be better. However as I'll explain in a minute this simply wouldn't work.

What you're proposing would require a session change for each pilot rotated onto the grid. Session changes are a huge bottleneck in terms of lag. In a recent interview ( ) CCP Veritas explained that when you go through a session change all of your skills are recalculated, ship's loaded etc etc, which causes a lot of server load.

He goes on to explain how you can see this during bomb runs, each ship explodes and the pilot's put into a pod which is a session change and TiDi spikes.

You can see that in this vid, roughly 100 ships on-grid with no TiDi then bombs happen, which drops the server to 60% TiDi and takes a full minute for all pods to have loaded on-grid

Extending this to 500 pilots being dumped on-grid at the same time and it's easy to see that it'd cause sufficient lag to drop the node to 10% TiDi for well over 3 minutes, meaning that some of the active group wouldn't even be able to load grid before being ghosted out AND it'd still be 10% TiDi.

Two ways to mitigate this would be to reduce the size of the active group, which be worse than the current method. Since nodes only start breaking at around the 4,000 mark, reducing the group size would mean that on average a pilot would be on-grid for less than 1/10 of the time. The other solution would be to increase the cycle time of the ghosting process, which fails for the same reason.

This particular source of lag should actually be eliminated/mitigated when the Brain in a Box is completed. Basically that offloads the vast majority of the data loaded during session changes to a dedicated server, leading to a much smoother transition, which means your idea could become viable at that point, but I can't say for sure that it would.

LukeP said...

Nice idea, but you can take it further, especially that
1) big battles are selling points
2) valkyrie begs to be set IN such battles (and that means more small, fast objects to care about)

Ghosting as you propose is essentially compartmentalization, when part of people fights, the other supply passive bonus according to their actual number. Fine as fast stop-gap measure.

Way I would advance the measure is to set a grain to the battle, with reasonably-numbered cells where fighting happens., each cell evaluated as whole regarding to others. (pretty much as ghosting goes)

Grain is determined by number of big ships on the grid. Each titan == cell on his own etc. Now each cell need to have a minimal radius, so it would force spacing out big ships. If you have frigate blob, treat is as one entity too, centred on FC etc. Pretty much how table top games handle units.

From there, the cell can be in two states, ranged, when it engages other non-neighbouring cell, or "melee" when it engages objects in neighbouring cell, and there is child-object swap (drones, close range ships and munitions). Ranged - state allows only targeting other cell as entity, either focusing on cell-building entity or AOE whole cell. (state sums all effects per tick as one and apples that sum wholesale).
Melee-state shuts off all ranged states and treat cells involved as on-grid.

So, there might be many people involved in battle, but you need to throw computing power only on those actually engaging close-range, clumping other effects disregarding checking for particulars.

Also, do not bother in non-fighting entities like pods, just spirit them away after applying "chance of getting caught in blast" factor.

Then again, withoutnchanges to fight mechanics that will require more movement (not only moonwalking from bomber run), the fights would be just blobs slinging at each other.. after all after getting in range position does not matter that much, and stuff generals/admirals use to win actual battles (outflanking, eliminating c3 structures) do not apply. Then more sophisticated model == more computing power .. but only if done in crude manner)