Official state of skill lag and server optimizations

Official state of skill lag and server optimizations

in Guild Wars 2 Discussion

Posted by: BillFreist

BillFreist

Gameplay Programmer

I’d like to create a central place for you all to get the current state of where we stand on optimizing the servers and relieving what most of you have come to know as “skill lag”.

Over the past several months we’ve had a small team working on ways to relieve some of the load from the game server while there are large concentrations of players. Most of the work thus far has gone pretty much unnoticed from the player’s perspective, but there have been pretty large steps made towards the issue. This continues to be a large and high priority issue for us so we can continue to make improvements to make this a thing of the past.

The various teams, or as we call internally “Feature Teams”, are now subject to performance reviews both client and server before they can ship. This is one of the many things we’ve put in place to stop the bleeding. Preventing further issues has been the main focus up to this point to ensure we aren’t introducing new issues and constantly cleaning up behind them. Now that we’re to a point where we’re confident that we’ve plugged the holes, larger strides to actually make noticeable optimizations are in the works.

I mentioned sometime last month that I’d like to be more transparent about what steps we’re taking to address the issue. To give you more of an insight of where we stand, we’re starting with taking the bull by the horns. We’ll be going through almost every skill in the game over the coming months and optimizing them one-by-one. This in itself will make an extremely large difference, that when that patch note comes through, expect a very different WvW experience.

I’m going to leave this thread open for discussion, but I urge you to be respectful of one another and keep things civil, otherwise I’ll be forced to lock it.

-Bill

Official state of skill lag and server optimizations

in Guild Wars 2 Discussion

Posted by: BillFreist

BillFreist

Gameplay Programmer

(reserved)

Official state of skill lag and server optimizations

in Guild Wars 2 Discussion

Posted by: Yenn.9185

Yenn.9185

Thank you for your work on fixing the lag. I’m actually optimistic that this will make difference in WvW.

On the other hand, what about FPS issues? Apparently a lot of players, despite having top-of-the-line PC, are still having experiencing below 20fps on very crowded events.

Official state of skill lag and server optimizations

in Guild Wars 2 Discussion

Posted by: Kaiyanwan.8521

Kaiyanwan.8521

Skill lag can get pretty bad when the Wv3 areas are full, hope you can fix that sooner than later.
But don’t you feel like altering skills to optimize them might not cause more trouble than it may solve? I hope it works out, but sounds sort of risky.

Btw, something is broken with the Q: at least I didn’t intend to use this feature…

(edited by Kaiyanwan.8521)

Official state of skill lag and server optimizations

in Guild Wars 2 Discussion

Posted by: Luna.9640

Luna.9640

Hello Bill,thank you for taking time to speak with us and working on this very important and “pain in the kitten ” for WvW community problem.

  1. My 1st question is : Can you be more specific of what “optimization” means and will it affect the skills as we know it by any negative way ?
  2. My Second question is : Wasn’t all these problems caused because you have to broadcast every single condition/buff to everyone which is currently pushing your systems and servers/internet to its limits ?
  3. My 3rd question is :If you ever have a solution to this are there any plans to lift the AoE Limit for both WvW/PvE/PvP because thats why this AoE limit exist in 1st place
  • Isn’t skill delay in big battles (WvW or PvE like Taco) a direct result of those?

(edited by Luna.9640)

Official state of skill lag and server optimizations

in Guild Wars 2 Discussion

Posted by: Canidae Canis.2861

Canidae Canis.2861

Just wanted to say thank you for the update. These are the kind of posts that I would like to see happen more often, because it can be frustrating when we feel left in the dark. Keep up the great work!

Official state of skill lag and server optimizations

in Guild Wars 2 Discussion

Posted by: Wolfheart.1938

Wolfheart.1938

I have nothing constructive to offer, but wanted to congratulate you for your efforts and for being trasparent with us on the matter.

SFR chat spammer, forum Guardian

Do you like ascended weapons?

Official state of skill lag and server optimizations

in Guild Wars 2 Discussion

Posted by: Keysha.2815

Keysha.2815

I just have an observation I have noticed. My memory usage for GW2 skyrockets during lag. As in 3 million k plus. When it is less than a million, my game is smooth as silk.

Official state of skill lag and server optimizations

in Guild Wars 2 Discussion

Posted by: Robert.1086

Robert.1086

Thanks for the update. May I suggest that you start working through the skills in order of longest cast time first? Those seem to be the ones that are affected the most by lag.

Official state of skill lag and server optimizations

in Guild Wars 2 Discussion

Posted by: Nabuko Darayon.9645

Nabuko Darayon.9645

Thanks for informing us about the process. I do have one question. You said that when that patch note comes through, expect a very different WvW experience. should we expect to see these optimization on 15th October? or was that just writer’s freedom of speech?

Official state of skill lag and server optimizations

in Guild Wars 2 Discussion

Posted by: Kendra Nightwind.8734

Kendra Nightwind.8734

20131005 – Prior to the update on 03 September 2013 I had no lag issues what so ever. I was however having a lot of issues with a new sound card I had purchased (testing the third factory replacement) so when my FPS dropped from an average of 4(with all the eye candy turned on max) 5 in Lions Arch to 3 (in performance mode!) and lots of skill lag, I thought nothing the sound card driver was the cause of it. Since that time I have formatted the hard drive and did a full reinstall of Windows and reinstalled my old reliable sound card, my FPS went from 2 up to 12 (still in performance mode) with about the same number of people around. Since that time I have seen some improvement with recent optimizations, I can reach 18FPS (yes, still in performance mode) and most of the skill lag is gone. Almost everyone I know of that uses both an AMD processor on an x90 chip-set motherboard and an higher end AMD video card seems to have the same issues.
I want to make sure you understand, I don’t give a kitten about WvW or any other form of PvP, I am all about PvE. So here is my question; Will these upcoming optimizations improve the situation for us?

Edit: 20131006 – Today I had my sound card drivers crash when I logged into Guild Wars 2. My frame rates are almost back to normal and most of the skill lag is gone. I really dislike playing with that kind of disadvantage however.
After restarting my computer I expected the issue to return. It did not, which would indicate that the sound card driver was at fault, except for one thing. Before I had a single core always running about 50-75%, now I have five cores running between 30-75%. I think it may have been an issue with the processor driver. Either way, I am glad that it appears fixed.

GA 990FXA-UD3; AMD FX 8350; Antec 620; 16GB Corsair;
SoundBlaster X-FI Titanium Fatal1ty Pro; BigFootNetworks Xeno Pro NIC;
GigaByte Windforce GTX 770 4G; Samsung S24A300B (1920 × 1080 DVI-D);

(edited by Kendra Nightwind.8734)

Official state of skill lag and server optimizations

in Guild Wars 2 Discussion

Posted by: Ohoni.6057

Ohoni.6057

My main issue with lag is that when I have powers with casting times, I get used to a certain pacing, but then the casting times change under lag, so I’ll click on one ability, wait an appropriate time, click on another, and then the first will get skipped and never go off. Whatever you do, just ensure that every ability I click on will activate.

Official state of skill lag and server optimizations

in Guild Wars 2 Discussion

Posted by: Majic.4801

Majic.4801

I’m sorry I don’t have anything more concrete or helpful to say, but I really, really love it when you keep us in the loop like this.

Thanks for letting those of us who’ve been seeing this, checking our ping times, monitoring our client performance and occasionally shedding some hair that we’re not crazy (well, okay, not because of this, at least).

Fair winds and following seas, dear friend.

“Not the same, real and true. True you feel inside.
Always follow what is true.” — Sentry-skritt Bordekka

Official state of skill lag and server optimizations

in Guild Wars 2 Discussion

Posted by: TheBob.9863

TheBob.9863

Thank you but what about client side optimizations? We were promised some major engine/client side optimizations to improve our fps in WvW 6~ months ago, has this project been dropped because I haven’t seen anything regarding client performance since then

Official state of skill lag and server optimizations

in Guild Wars 2 Discussion

Posted by: Lopez.7369

Lopez.7369

I wish you kept us this in the loop for bleed cap: https://forum-en.guildwars2.com/forum/game/gw2/Condition-Catastrophe/

Official state of skill lag and server optimizations

in Guild Wars 2 Discussion

Posted by: Demosthene.2195

Demosthene.2195

In a vain attempt, I’ll try to keep this post away from corporate politics. Regardless, I believe that certain corporate-wide policies need to be re-examined in terms of game development & maintenance, patch/update presentation and community interaction for the future.

About transparency: not limited to recent WvW scandals and community outrage, I believe we could use some form of issue-tracker and update-scheduler, present on forums or a "sister" website to main guild wars 2 website (and definitely visible as a tab on main website, as well as a link from the forums).

Issue-tracker:

- a list of all current problems with the game, separated by tabs (graphics, sound, gameplay etc.)
- every problem defined: what, how, why, temporary workarounds, ETA (be it a week, month or unscheduled)
- every problem discussed: separate thread purposefully moved away from general technical/support forums

The purpose:

- community is informed about bugs, knows they’re being worked on: less duplicates and false-positives, feedback about singular issue concentrated in a single thread
- technical forum is still accessible to users requesting personal help, or bringing up new/unconfirmed issues (less forum clutter - identified issues are moved to a separate tab or place, to reduce the chaos)

Update-scheduler:

- less technical and more general in nature, outlines game going forward in new directions (be it fixing a core gameplay system/feature, or adding new systems/features, not limited to: maps, professions, skills, events, game modes, encounter types, conditions/boons/states)
- fosters community dialogue & feedback before features go live (including: philosophical, mechanical and social consequences for the game and it’s players)

Last Words

In varying intensity and form, we’ve already seen some of these steps taken by Anet: "creative design" blogs, feature blogs and posts, progress/update posts, feature videos. What is lacking, however, are two things: consistency of approach and organisation.

Organisation: central go-to place to study and discuss existing game problems in depth, as well as game-going-forward features which are in the works, but with more flexible schedule and final content/design.

Consistency: all current issues and future features are documented, catalogued and freely discussed with the community as a prime community relations policy, NOT "as exception" rule: when under duress of public outrage, trying to limit damage after scandal/leak or "almost everything is ready so now we talk about it" mentality.

Summarising, the irrational fear of: exploits becoming public knowledge, developers being publically lynched for missing a deadline/feature, needs to go.

Exploits: make a public list of bannable offenses, gather public feedback about them and fix them faster.

Missing a deadline/feature: players have to understand that not every feature will get implemented on time, at all or in the form outlined in early design sketch or philosophy - simultaneously, playerbase as a whole shouldn’t be punished for simple curiosity and interest in the ongoing health of game and it’s continuing success.

Purposefully brick-walling and withdrawing critical information about game issues and your policy towards fixing them until the very last moment causes permanent community damage - loss of players and declining morale. I believe we deserve better.

In short: please be more upfront about your game design, talk with us openly and accept constructive criticism where it’s due. Failing that, we will see more hostility towards developers and the company, more conspiracy theories and generally poor player feedback. Worst communication is silence.

Trivia:

- "Roadmap" http://www.planetside2.com/roadmap
- "the problem, what we’ll do to fix it and the philosophy behind it" post https://forums.station.sony.com/ps2/index.php?threads/operation-make-faster-game.150281/
- "update/issue" discussion thread https://forums.station.sony.com/ps2/index.php?threads/october-on-going-performance-improvements.152361/
- "when forum or blog post is not enough" https://www.youtube.com/watch?v=EaHgTR4yyuU

Official state of skill lag and server optimizations

in Guild Wars 2 Discussion

Posted by: Avatara.1042

Avatara.1042

Start with the super-lag skills please. Things like Ride the Lagning and Burning Retreat.

Official state of skill lag and server optimizations

in Guild Wars 2 Discussion

Posted by: KarateKid.5648

KarateKid.5648

I thank you and my poor keyboard (which I smash like mad when skills don’t fire on time) thanks you.

Official state of skill lag and server optimizations

in Guild Wars 2 Discussion

Posted by: Siphaed.9235

Siphaed.9235

When you say “optimization”, do you mean making your engine more compatible for multi-core systems? Right now your game’s engine runs primarily on single-thread, which Intel CPU’s seem to thrive on; while at the same time your game’s engine is poorly designed for multi-thread CPU’s, which AMD is best at.

Now, the problem is that a multi-core AMD doesn’t have good single-thread speeds, which on GW2 drives the performance down. Why is this? Because the video game -a graphical projection on a computer screen- is for some reason more prioritized for CPU usage instead of GPU usage. Why? I haven’t a clue. Maybe it’s because the high use of physics within the game, don’t really know. Either way, it’s a problem.

Official state of skill lag and server optimizations

in Guild Wars 2 Discussion

Posted by: Moka.9641

Moka.9641

Trivia:

- “Roadmap” http://www.planetside2.com/roadmap
- “the problem, what we’ll do to fix it and the philosophy behind it” post https://forums.station.sony.com/ps2/index.php?threads/operation-make-faster-game.150281/
- “update/issue” discussion thread https://forums.station.sony.com/ps2/index.php?threads/october-on-going-performance-improvements.152361/
- “when forum or blog post is not enough” https://www.youtube.com/watch?v=EaHgTR4yyuU

This is off-topic, but this is the level of transparency I would like to have. I think it would be beneficial for everyone if you give us insights on what are you doing with current game issues. We could understand what issues you’re dealing wih instead of going :

“Omgz skill lag, why not fix already, so easy, my grandma can do it, QQQQQQ, buy better server QQQQQ,”-Random GW2 player

[AW]- The Holy Engineer

Official state of skill lag and server optimizations

in Guild Wars 2 Discussion

Posted by: NeHoMaR.9812

NeHoMaR.9812

I imagine is not as easy as buying/renting better server hardware? or internet with more bandwidth, less latency? if that is not the solution, I wonder where the bottleneck causing this lag is?

Jade Quarry

Official state of skill lag and server optimizations

in Guild Wars 2 Discussion

Posted by: Alienmuppet.1942

Alienmuppet.1942

Great post Bill, I’m so pleased to see this! I was hoping the focus would shift a little to performance at some point because there are definitely issues that let the game-play down a bit here and there, mostly down to skill lag and FPS (on a NVidia GTX 670).

Besides these issues, I find the GW2 game-play some of the best I’ve ever come across, I’ve never felt so in control of a character (d/d ele in my case).

There is a graphical issue I often come across, and that is the character being half embedded in a rock or something else that is meant to be solid, such as standing on a wall in WvW to find my character’s feet are part of that wall, though that’s just a minor gripe. It just makes it feel a little unfinished..

I’m also finding a lot of “sound lag” when I first go into Lions Arch. I usually use RTL for instance to go straight to the trading post, and burning speed and lightening flash. The skills work almost immediately, but I’ll get all the sound FX bunched up together a good few seconds later (after that initial lag it is fine).

As a professional developer myself it would be great to have posts of a more technical nature such as shown in the link2 Moka.9641 provided, but I suppose it is understandable if you don’t want to go that far.

Anyway thanks again for the update!

Official state of skill lag and server optimizations

in Guild Wars 2 Discussion

Posted by: Zplus.4217

Zplus.4217

I don’t know, but ever since the teq patch I’m getting lag and DCs regularly now, specially during prime time. I guess DCing players during prime time is a way to relieve stress on the server.

Take a look at tech support and bug report forum, there are plenty of players who are having the same issues all over the world. Please bring back my old gaming experience where by I do not get DCs every 5 mins before you further “optimise” the game server, cheers.

Official state of skill lag and server optimizations

in Guild Wars 2 Discussion

Posted by: BillFreist

BillFreist

Gameplay Programmer

Next

Part 1:

To clarify some common questions:

  • What does optimization mean in terms of the original post?
    • I’m working primarily on server-side (as mentioned in the title) optimizations. Not to say that this doesn’t have some smaller impact on the client, which I mentioned in posts elsewhere that there will some client-side gains as the changes rollout. Client-side changes are being worked on completely seperate and will release seperately and, sadly, slower. I’m not going to get into the client-side portion of our optimization initiative, because I’m not directly involved with what we have going on there.
  • What exactly do these server-side optimizations do?
    • As pointed out above, they will mostly be noticeable with latency in skill execution in heavy battles. I mentioned WvW being a benefactor because that has one of the most consistently dense battles going on. That’s not to say that this won’t be noticeable in other parts of the game; it most definitely will.
    • Though server CPU usage is the biggest factor here, there are other optimizations as part of the changes going on. Bandwidth is another thing that plagues players with slower internet connections. That is something that we’re actively rolling out changes for, with the next set of changes coming at the end of October.
    • Removes the bottleneck in almost all of the game systems. Most of what goes on in the game is processed on the server, including combat, skills and movement. When this bottleneck is relieved, the changes will pretty much speak for themselves. A common example goes back to skill usage where in heavy battles, you press and skill key, but that skill never fires or your auto-attack skill fires instead.

Official state of skill lag and server optimizations

in Guild Wars 2 Discussion

Posted by: BillFreist

Previous

BillFreist

Gameplay Programmer

Next

Part 2:

Moving onto a few specific questions:
Q: My 1st question is : Can you be more specific of what “optimization” means and will it affect the skills as we know it by any negative way ?
A: These changes will go unnoticed functionality-wise. The only changes will be positive in that skill usage will be constantly more responsive.

Q: My Second question is : Wasn’t all these problems caused because you have to broadcast every single condition/buff to everyone which is currently pushing your systems and servers/internet to its limits ?
A: This is a good question. Buffs are a large part of Guild Wars 2 and at first glance you’d definitely think that telling everyone about them would be really expensive. In reality, this is only a hit with server->client traffic, which has been significantly improved with the handful of bandwidth optimizations that have already gone out. As far as server processing, its pretty minuscule (thankfully).

Q: My 3rd question is :If you ever have a solution to this are there any plans to lift the AoE Limit for both WvW/PvE/PvP because thats why this AoE limit exist in 1st place Isn’t skill delay in big battles (WvW or PvE like Taco) a direct result of those?
A: <<< The AoE hit limits exist for more than just performance reasons. We already have a lot of places in the game where AoE limits are extremely high (WvW siege) or uncapped (monster skills). The last thing we want to do is relieve one problem only to introduce it somewhere else.

Q: When you say “optimization”, do you mean making your engine more compatible for multi-core systems? Right now your game’s engine runs primarily on single-thread, which Intel CPU’s seem to thrive on; while at the same time your game’s engine is poorly designed for multi-thread CPU’s, which AMD is best at.
A: As I outlined above, this is mostly server changes that I’m involved with, but there are client optimizations in the works, but we don’t know when those will go out. Optimizing is extremely dangerous and requires a lot of testing. The server side changes are more controlled in the sense that we don’t have to worry about testing on different hardware configurations like with the client. Its unfortunate, but necessary to ensure that the changes don’t end up doing more harm than good.

The engine isn’t as single-threaded as you might think. It will only use as many threads as your machine can support. if you have a 4 core processor, you may only get 2 threads plus an additional 2 worker threads for processing things like animation. Though I we are definitely looking at other ways to improve other game systems to take better advantage of using multiple threads while they’re not in use.

Q: You said that when that patch note comes through, expect a very different WvW experience. should we expect to see these optimization on 15th October? or was that just writer’s freedom of speech?
A: I Apologize for any misunderstanding here. But the answer to this is no, there isn’t really anything of note coming next week. Though I did I have large hand in the skill and combat related changes coming in that release . We definitely do have a few things that’ll help, but most likely won’t be noticeable.

Q: Thank you but what about client side optimizations? We were promised some major engine/client side optimizations to improve our fps in WvW 6~ months ago, has this project been dropped because I haven’t seen anything regarding client performance since then
A: Again, like I mentioned above we are working on client changes, but they are on a much slower release cadence and have to go through a rigorous test cycle before ever being on deck for release. Patrick, the slayer of culling, is a large part of these changes that’re being cooked up.


To respond to the large response by Demosthene.2195:
The idea for having some sort of central location or “issue tracker” as you mentioned, I think there’s definitely something there. I think that central location would make sense to be on the Guild Wars 2 launcher itself, which from what I understand is going through a makeover. I think that’s definitely something for the Community team to investigate.

The rest of your post starting going off-topic. I will say that I want to make sure that people realize that we don’t always have the time to respond to every question, concern and issue that arises, even though we would love to. I constantly say how I try to be more involved and transparent with you guys, but obviously I don’t have the ability to spend time on the forums every day. We love our player-base and appreciate how passionate they are. Your feedback doesn’t go unnoticed, even if it seems like it does.

Also, thanks for the links, that’s really cool to see other developers be so open with their player-base about issues.

Official state of skill lag and server optimizations

in Guild Wars 2 Discussion

Posted by: BillFreist

Previous

BillFreist

Gameplay Programmer

Next

Part 3:

So what am I working on besides ensuring player skills are efficient? Well with our two week release cadence, I work along-side our shiny new Technical Design team and Living World teams to ensure their content is efficient. I’m also constantly exploring other ways to improve the game, which led me to something that I literally just finished wrapping up today. This change will go directly into testing early next week, which will then help us determine how significant it is and when we are comfortable enough to push it to the Live game. I’m really optimistic about this change in particular; seeing massive drops in server load .

In closing, I want to let you guys know what the release plans are for optimizing the game server. To put it simply, the plan is changing almost every day. We have the ability to ship changes whenever they’re ready. And because of our current release cadence, that means that we can literally get these changes out to you guys in a matter of 2 weeks. That being said, optimization is dangerous, especially to a live game. Ideally I will continue to sneak in changes every 6 weeks or so, and when there are changes that players will notice, by all means there will be a release note to call them out.

-Bill

Official state of skill lag and server optimizations

in Guild Wars 2 Discussion

Posted by: Bri.8354

Bri.8354

Thanks for the optimization and communication.

I have one question regarding this section if you don’t mind:

Q: My Second question is : Wasn’t all these problems caused because you have to broadcast every single condition/buff to everyone which is currently pushing your systems and servers/internet to its limits ?
A: This is a good question. Buffs are a large part of Guild Wars 2 and at first glance you’d definitely think that telling everyone about them would be really expensive. In reality, this is only a hit with server->client traffic, which has been significantly improved with the handful of bandwidth optimizations that have already gone out. As far as server processing, its pretty minuscule (thankfully).

With the optimizations that have already “significantly improved” bandwidth, and with those that are to come, do you think it would one day be possible to increase the condition cap in PvE? So far the answer given as to to why they can’t increase it is because of bandwidth concerns, so it stands to reason that if the game is better optimized, they shouldn’t be held back any longer.

(edited by Bri.8354)

Official state of skill lag and server optimizations

in Guild Wars 2 Discussion

Posted by: Luna.9640

Luna.9640

Thank you for sharing your work progress with us Bill.

I hope you keep us posted once in a while when you reach a milestone!

P.S. I just couldn’t resist to not upload this !

Attachments:

Official state of skill lag and server optimizations

in Guild Wars 2 Discussion

Posted by: Mif.3471

Mif.3471

Thanks for keeping us in the loop Bill

Tarnished Coast | Best cookies in all of Tyria

Official state of skill lag and server optimizations

in Guild Wars 2 Discussion

Posted by: Enektric.4527

Enektric.4527

Thanks for the update bill, I only have 1 question.

What about the Mac client?

Official state of skill lag and server optimizations

in Guild Wars 2 Discussion

Posted by: Uttar.2341

Uttar.2341

Hi Bill, thanks a lot for working on this, keeping us updated, and especially for responding to our questions! I think this is definitely what a lot of us want to see following Chris Whiteside’s “Collaborative Development” thread.

I’ve noticed that fights in Garrison and Stonemist tend to have more skill lag than comparable fights in other keeps or in the open field. My guess is that this is because the collision detection is more complex (multiple floors etc.) – if that’s correct, is this something you’re also looking at optimising?

More importantly, I was wondering since it seems you agree that WvW suffers the most from skill lag and that you said you’re going to optimise every skill one by one, would it make sense to prioritise the skills that are most widely used in large-scale WvW builds?

While there are obviously variations, the basic templates are by far the most common and you could probably get 90%+ of the effect by optimising only 25% of the skills.

This might change slightly with the patch (although I expect not much based on Jonathan Sharp’s teaser) but here’s a list that I hope will help you if you do decide to prioritise common WvW skills and don’t already have that data with your internal tools. I’ve listed the classes very roughly by their frequency in the meta.

Warrior
Weapons: Hammer, Sword/Warhorn (rarely Axe/Axe or Greatsword).
Skills: Healing Signet, Healing Surge, For Great Justice, Shake it Off, Balanced Stance, Doyak Signet, Endure Pain, Signet of Rage, Warbanner. (6xRune of the Soldier with Vigorous Shouts)

Guardian
Weapons: Staff, Hammer, Greatsword (rarely Mace/Focus).
Skills: Shelter, Signet of Resolve, Stand Your Ground, Hold the Line, Save Yourselves, Wall of Reflection, Sanctuary, Signet of Judgment, Renewed Focus, Tome of Courage.

Elementalist
Weapons: Staff by far most frequent (some Dagger/Dagger).
Skills: Ether Renewal, Lightning Flash, Mist Form, Armor of Earth, Arcane Wave, Tornado, Glyph of Elementals.

Necromancer
Weapons: Staff, heavy use of marks. Second set more open but often axe/dagger.
Skills: Consume Conditions, Well of Suffering, Well of Corruption, Well of Darkness, Spectral Wall, Plague.

Mesmer
Weapons: Staff, Scepter/Focus, Sword/Focus (rarely Greatsword).
Skills: Ether Feast, Veil, Null Field, Feedback, Blink, Timewarp.

Engineer: Pistol/Shield, Healing Turret, Bomb Kit, Throw Mine, Elixir Gun, Flame Thrower, Rocket Boots, Supply Crate (note: supply crate is one of the most lag-prone skills I know sadly).

Thief: Shortbow, Hide in Shadows, Smoke Screen, Shadow Step, Shadow Refuge, Blinding Powder, Caltrops, Basilisk Venom, Dagger Storm.

Ranger: Sorry, no clue.

I hope this helps and that you continue to keep us updated on your progress!

(edited by Uttar.2341)

Official state of skill lag and server optimizations

in Guild Wars 2 Discussion

Posted by: Phoebe Ascension.8437

Phoebe Ascension.8437

One thing I don’t get. In wvw PASSIVE effects NEVER LAG. They are always there, and always work. (Healing signet warrior for instance). Why is that so? Why do active skill lag while passive ones don’t. Honeslty at the moment this makes stonemist fights a bit unfair (passive heal so much strong then active heal, cause they won’t activate).

8x different lvl 80 profession.

Official state of skill lag and server optimizations

in Guild Wars 2 Discussion

Posted by: Amelia Knox.9362

Amelia Knox.9362

Thank for anwser some question, and Thank you for the effort.

[Dawn] Gandara
Guild Leader

Official state of skill lag and server optimizations

in Guild Wars 2 Discussion

Posted by: Draygo.9473

Draygo.9473

One thing I don’t get. In wvw PASSIVE effects NEVER LAG. They are always there, and always work. (Healing signet warrior for instance). Why is that so? Why do active skill lag while passive ones don’t. Honeslty at the moment this makes stonemist fights a bit unfair (passive heal so much strong then active heal, cause they won’t activate).

I’ll take a shot at this.

In very large battles when the server is lagging – time takes longer. The server only lets clients use skills after they are off cooldown. If the server is taking 2 seconds to process 1 second worth of data that means the real cooldown of your ability has doubled. The problem is the client itself has no way of knowing how slow the server is going so the timer on your skillbar only uses real time. So if your heal has a 20 second cooldown and you are in a laggy fight, the server might only think 10 seconds has passed when you try to hit the skill again, thus the skill wont activate because it is still on cooldown according to the server.

Passive effects i’m not absolutely sure about, but they might only be half effective in laggy situations as well, for example you only get 10 seconds of healing in a 20 second period. The major advantage of passive abilities is that you don’t need to know when its off cooldown and useable.

Delarme
Apathy Inc [Ai]

Official state of skill lag and server optimizations

in Guild Wars 2 Discussion

Posted by: Jamais vu.5284

Jamais vu.5284

Q: My 3rd question is :If you ever have a solution to this are there any plans to lift the AoE Limit for both WvW/PvE/PvP because thats why this AoE limit exist in 1st place Isn’t skill delay in big battles (WvW or PvE like Taco) a direct result of those?
A: <<< The AoE hit limits exist for more than just performance reasons.

But do you not find it problematic that one of the most broken gameplay tactics in the game is for a zerg to just ball into a little clump so that only ever 5 players take damage from AoEs?

Can you look at the suggestion I made here? https://forum-en.guildwars2.com/forum/wuv/wuv/Don-t-make-a-new-map-Solve-the-skill-lags/page/2#post2783543
Essentially, I’m asking whether it’s possible to ration object interactions so as to cut down on the runaway exponential growth of computing load.

ArenaNet, the company that removes vaguely nipple-like bumps from an armor within hours
but takes two years and counting to introduce colored commander tags for WvW.

Official state of skill lag and server optimizations

in Guild Wars 2 Discussion

Posted by: Kendra Nightwind.8734

Kendra Nightwind.8734

Part 2:

…snip…
Q: When you say “optimization”, do you mean making your engine more compatible for multi-core systems? Right now your game’s engine runs primarily on single-thread, which Intel CPU’s seem to thrive on; while at the same time your game’s engine is poorly designed for multi-thread CPU’s, which AMD is best at.
A: As I outlined above, this is mostly server changes that I’m involved with, but there are client optimizations in the works, but we don’t know when those will go out. Optimizing is extremely dangerous and requires a lot of testing. The server side changes are more controlled in the sense that we don’t have to worry about testing on different hardware configurations like with the client. Its unfortunate, but necessary to ensure that the changes don’t end up doing more harm than good.

The engine isn’t as single-threaded as you might think. It will only use as many threads as your machine can support. if you have a 4 core processor, you may only get 2 threads plus an additional 2 worker threads for processing things like animation. Though I we are definitely looking at other ways to improve other game systems to take better advantage of using multiple threads while they’re not in use.
…snip…

Thank you very much for your detailed answers to the many questions posed you and the rest of the development team.

As for the section quoted above and if you refer to my above post you will not that the lack of threads may not be an ArenaNet issue. Since then I have checked several of my friends computers, almost all of which have AMD processors. Of the six that are using FX class processors I found that only twice did the chipset/processor driver install properly. I can not speak to Intel processors, but I would as that you advise people experiencing low frame rates to uninstall their chipset and processor drivers, and then reinstall them starting with the oldest version first. This was the only way I found that consistently corrected the multi-threading issues in Guild Wars 2 and a few other games.

Thank you again for everything you do for us that we never get to hear about!

GA 990FXA-UD3; AMD FX 8350; Antec 620; 16GB Corsair;
SoundBlaster X-FI Titanium Fatal1ty Pro; BigFootNetworks Xeno Pro NIC;
GigaByte Windforce GTX 770 4G; Samsung S24A300B (1920 × 1080 DVI-D);

Official state of skill lag and server optimizations

in Guild Wars 2 Discussion

Posted by: Fenrina.2954

Fenrina.2954

One thing I don’t get. In wvw PASSIVE effects NEVER LAG. They are always there, and always work. (Healing signet warrior for instance). Why is that so? Why do active skill lag while passive ones don’t. Honeslty at the moment this makes stonemist fights a bit unfair (passive heal so much strong then active heal, cause they won’t activate).

Draygo is sorta right, but the explaination doesn’t really explain what you’re describing.

The closest explanation I have is that the thread, or daemon or code, that handles passives is not getting bogged down while the thread that handles cooldowns or skill activations is getting bogged down. Since the code for passives isn’t getting delayed or slowed, they always work.

Imagine it like a couple of people in an office. The person that handles active skills has a lot of work to do. The person that handles passives has little work to do. The passives person would naturally get done faster.

(edited by Fenrina.2954)

Official state of skill lag and server optimizations

in Guild Wars 2 Discussion

Posted by: SonicTHI.3217

SonicTHI.3217

Have you experimented/considered auto adjustable settings for the client? Say if FPS drops below certain levels it turns off selected features or scales them down and if it goes back above it turns them on/higher again (like downsampling).

If possible it would be nice to have them for each setting (that can be adjusted on the fly and dont cause too much of a spike) separately with a global desired FPS option at 30/45/60.

In particular this would be useful for shadows and reflections as they are the most demanding on the CPU.

It should help most people due to the variance in client performance due to forcing people into zerging certain events and it should be a relatively quick thing to implement.

Apart from that if you could maybe extrapolate on the things you are doing for client performance it would let people know that you are actually working on something significant. SOE has done some good reassuring with this:
https://www.youtube.com/watch?v=EaHgTR4yyuU
https://www.youtube.com/watch?v=bgOsrJaibjM

Finally IMHO you need a bigger leash on content creators. Most people who wanted a redesigned Tequatl fight didnt really ask for 100 people to run to single spot and fire off some 1000 skills in a matter of seconds. Not to mention all the other forced zergfest updates in the last few months.

“Otherwise, your MMO becomes all about grinding to get the best gear. We don’t make grindy games.”
-Mike O’Brien, President of Arenanet

Official state of skill lag and server optimizations

in Guild Wars 2 Discussion

Posted by: ViRuE.3612

ViRuE.3612

Awesome posts, very insightful. As a WvWvW player one thing that always surprises me in large skill lag battles is that my thief suffers a great deal less than my necro. Of course I can’t play both in the exact same battle but massive three ways in Stonemist I always wish I was on my thief who seems to suffer much less from skill lag on weapon skills (but probably same skill lag with primary heal).

Possibly something to do with animation or complexity of skills (less conditions?) or are thiefs just handled/prioritized first?

I wish you luck with this project, if you can get anywhere near the improvements made to culling you may become as legendary as Patrick.

(edited by ViRuE.3612)

Official state of skill lag and server optimizations

in Guild Wars 2 Discussion

Posted by: Valandil Dragonhart.2371

Valandil Dragonhart.2371

Thanks for keeping us updated on this.

Concerning skill lag, in my own experiences in PvE with ~50 players trying to down a dragon, I’ve noticed that the utility skills (7, 8 & 9) are the ones that are least likely to fire off in a timely manner under these conditions, but conversely, the heal skill will almost always work as desired when commanded.

Say on an ele, meteor shower is a bit 50/50, sometimes it will fire off on command, other times it just won’t, and my guess is the aforementioned AoE limit. Having said this, a utility skill such as Arcane Blast which is a simple single-projectile skill with a guaranteed crit, will almost never fire off until about 10 seconds after I’ve started mashing the key, and the number is worn off the keyboard.

Again this is just my observation and only really becomes an issue over weekends when the playerbase periodically peaks; normal times during the week there’s ~30 players at any given event and this isn’t as much of an issue, or if it is then it’s not as noticeable.

Kudos to you for keeping us in the loop.

The old-school Arrow-Key warrior.
“Obtaining a legendary should be done through legendary feats…
Not luck and credit cards.”

Official state of skill lag and server optimizations

in Guild Wars 2 Discussion

Posted by: Voland.5862

Voland.5862

I’d like to create a central place for you all to get the current state of where we stand on optimizing the servers and relieving what most of you have come to know as “skill lag”.

Does Arena-Net have any intention of investing in upgraded (Newer/Better) Server hardware and related solutions?

One of the main reasons World of Warcraft is losing people is because they have in so many words refused to put any money at all into better infrastructure. They have instead opted to “phase” things or “optimize” pre-existing bargain bin hardware.

Any input you could provide would be much appreciated.

Official state of skill lag and server optimizations

in Guild Wars 2 Discussion

Posted by: zamalek.2154

zamalek.2154

Sounds awesome!

One thing I noticed is that when the Scarlet events released I didn’t have any skill lag in the Scarlet events (WvW remained much the same). It looks like you’ve already finished with this thread; still I am interested to know whether WvW and PvE share the same server architecture and whether you have thought about capitalising on any of the improvements they have clearly been made in PvE.

Does Arena-Net have any intention of investing in upgraded (Newer/Better) Server hardware and related solutions?

Not all problems can be scaled vertically – quadratic problems like game servers are particularly poor candidates for vertical scaling. This is because, e.g. if you double your CPU speed you will only get a 50% speed improvement instead of the 100% you would expect. To compound the issue it may not even possible to get a system that is twice as fast as the current ones (because they might be close enough to the bleeding edge that only, say, a 0.3ghz jump is possible).

Not everything in WvW is quadratic, but “skill lag” is the symptom of one area that it is (everyone dropping AOEs) – so pulling out the wallet and improving the hardware would probably only make a marginal difference that us players wouldn’t really notice.

The best thing to do always is to improve the efficiency of your algorithms (or I would rather say, decrease the inefficiency). At the time you would hypothetically get more hardware you would have more bang-for-buck because of previous optimizations.

Auroraglade
Epistemic.8013: Guys this is bullkitten a sentient plant creature is hitting these
wooden doors with fireballs and it’s working
.

(edited by zamalek.2154)

Official state of skill lag and server optimizations

in Guild Wars 2 Discussion

Posted by: Voland.5862

Voland.5862

Sounds awesome!

One thing I noticed is that when the Scarlet events released I didn’t have any skill lag in the Scarlet events (WvW remained much the same). It looks like you’ve already finished with this thread; still I am interested to know whether WvW and PvE share the same server architecture and whether you have thought about capitalising on any of the improvements they have clearly been made in PvE.

Does Arena-Net have any intention of investing in upgraded (Newer/Better) Server hardware and related solutions?

Not all problems can be scaled vertically – quadratic problems like game servers are particularly poor candidates for vertical scaling. This is because, e.g. if you double your CPU speed you will only get a 50% speed improvement instead of the 100% you would expect. To compound the issue it may not even possible to get a system that is twice as fast as the current ones (because they might be close enough to the bleeding edge that only, say, a 0.3ghz jump is possible).

Not everything in WvW is quadratic, but “skill lag” is the symptom of one area that it is (everyone dropping AOEs) – so pulling out the wallet and improving the hardware would probably only make a marginal difference that us players wouldn’t really notice.

The best thing to do always is to improve the efficiency of your algorithms (or I would rather say, decrease the inefficiency). At the time you would hypothetically get more hardware you would have more bang-for-buck because of previous optimizations.

FYI, Arena-net hasen’t exactly been very transparent in terms of their infrastructure.

Worst Case Scenario: They are doing what blizzard did/still are doing. They snatched up a good deal of outdated hardware and put an MMO on it.
Best Case Scenario: The hardware is up to date enough that all they need is some serious effort into optimization and restructuring to resolve the issues full stop.

The most likely scenario
They have decent hardware but the poor choices made in terms of how the game as a whole was implemented both client and server side will be a time sink that will eat optimization efforts while only yielding minimal improvement.

Official state of skill lag and server optimizations

in Guild Wars 2 Discussion

Posted by: Omne.4603

Omne.4603

Awesome to see a thread like this exists. Much appreciated Bill.

I Cant Stop/ Ocularis
NSP | Os Guild Master
www.osguild.us | www.youtube.com/osthink

Official state of skill lag and server optimizations

in Guild Wars 2 Discussion

Posted by: morrolan.9608

morrolan.9608

This thread is more relevant than ever given the sharp increase in lag in the last 2 days. You guys have to come up with something fast because WvW has been verging on unplayable the last 2 days with the lag.

Jade Quarry [SoX]
Miranda Zero – Ele / Twitch Zero – Mes / Chargrin Soulboom – Engi
Aliera Zero – Guardian / Reaver Zero – Necro

Official state of skill lag and server optimizations

in Guild Wars 2 Discussion

Posted by: Sceinna.3561

Sceinna.3561

^

WvW is unplayable. We queue for 1-2 hours (mid tier server) and jump into 10 seconds of massive unplayable skill lag. All you can do is run forward and die, to the point even a sentry nearly killed me.

WvW imo is very broken right now, I don’t know if doing the seasons/archievements (which in my opinion are awesome ideas) are executed at the right time. There should have been some sort of fix in place first. Maybe adding the archievements shouldn’t have taken place at the start of the first WvW season either. Now you have “unfair” and “biased” results because a large amount of the PvE community is playing WvW at this point.

Just my two cents here.. I hope WvW becomes playable again soon because it’s what I love the most in GW2.. consumes all my time playing GW2 and now I can basically not play the game I like to be playing.

Official state of skill lag and server optimizations

in Guild Wars 2 Discussion

Posted by: zeromius.1604

zeromius.1604

FYI, Arena-net hasen’t exactly been very transparent in terms of their infrastructure.

Worst Case Scenario: They are doing what blizzard did/still are doing. They snatched up a good deal of outdated hardware and put an MMO on it.
Best Case Scenario: The hardware is up to date enough that all they need is some serious effort into optimization and restructuring to resolve the issues full stop.

The most likely scenario
They have decent hardware but the poor choices made in terms of how the game as a whole was implemented both client and server side will be a time sink that will eat optimization efforts while only yielding minimal improvement.

Assuming your most likely scenario, it could actually be an issue with design decisions not taking technical limitations into account. There is a good reason for why the combat in traditional MMORPGs were designed the way they were. Skill cooldowns, global cooldowns, client throttling were all systems or constraints put in place to alleviate bandwidth and server processing. The action heavy nature of Guild Wars 2’s combat translates to a heavier burden on the server side that wasn’t experienced in traditional MMORPGs.

There’s also a lot more overhead on server hardware for MMORPGs in general than other online games. You have more than just a single server for each world/realm. You have what are known as server clusters, which are a group of servers that pool their resources together to function as a single unit. With this setup, you don’t really get 100% of the resources because they are separate systems. They need to communicate with each other and there is always going to be latency. We’re still only looking at world servers.

All of the data, which includes skill inputs, needs to go through the world servers and into a database hosted on a database server which is running a database management system like Oracle, MySQL, etc. Every time you input a skill, that skill is sent by your client to the application (world) server, and the application server generates a query to the database residing on the database server. I think it would be safe to assume that the database and application servers all reside on the same network. Even on a 10GB Ethernet network there’s still going to be slight delay. That’s not to mention the I/O lag you get from moving data between memory and storage on the database server itself.

As for optimizations there are a few things that can be done if they aren’t already. I do think they have been done, are being considered already, or are technically infeasible. Good database design requires normalization of data, to eliminate insert, update, or delete anomalies, which means you’ll end up with a lot of heap (data) tables. Joining these tables in queries will produce a lot of overhead unfortunately so database designers will often try to do the minimum amount of normalization needed. They will also use indexes to make searches faster by narrowing down the search area of a given table. Yes, they could also improve their algorithms on the application server but these things could end up making things worse in other areas so thorough testing will be needed.

For the hardware side they could invest in NAND flash. I’ve never actually used it myself as it’s still fairly expensive right now. It’s really a question of whether it’s actually worth the investment. How much of a performance gain in percent is achieved relative to the dollar amount invested?

Another solution that ArenaNet could consider is to have both the application server and database server running on the same host as virtual machines. It’s what is referred to as a virtual application. This could significantly reduce the latency on server side processing but it would also require a significant investment of both time and resources into rebuilding the infrastructure to make it all happen. It all comes down to performance gain vs money invested.

Official state of skill lag and server optimizations

in Guild Wars 2 Discussion

Posted by: Syndrilious.9632

Syndrilious.9632

Sounds awesome!

One thing I noticed is that when the Scarlet events released I didn’t have any skill lag in the Scarlet events (WvW remained much the same). It looks like you’ve already finished with this thread; still I am interested to know whether WvW and PvE share the same server architecture and whether you have thought about capitalising on any of the improvements they have clearly been made in PvE.

Does Arena-Net have any intention of investing in upgraded (Newer/Better) Server hardware and related solutions?

Not all problems can be scaled vertically – quadratic problems like game servers are particularly poor candidates for vertical scaling. This is because, e.g. if you double your CPU speed you will only get a 50% speed improvement instead of the 100% you would expect. To compound the issue it may not even possible to get a system that is twice as fast as the current ones (because they might be close enough to the bleeding edge that only, say, a 0.3ghz jump is possible).

Not everything in WvW is quadratic, but “skill lag” is the symptom of one area that it is (everyone dropping AOEs) – so pulling out the wallet and improving the hardware would probably only make a marginal difference that us players wouldn’t really notice.

The best thing to do always is to improve the efficiency of your algorithms (or I would rather say, decrease the inefficiency). At the time you would hypothetically get more hardware you would have more bang-for-buck because of previous optimizations.

FYI, Arena-net hasen’t exactly been very transparent in terms of their infrastructure.

Worst Case Scenario: They are doing what blizzard did/still are doing. They snatched up a good deal of outdated hardware and put an MMO on it.
Best Case Scenario: The hardware is up to date enough that all they need is some serious effort into optimization and restructuring to resolve the issues full stop.

The most likely scenario
They have decent hardware but the poor choices made in terms of how the game as a whole was implemented both client and server side will be a time sink that will eat optimization efforts while only yielding minimal improvement.

I think most of the issue is poor optimization and a lack of virtualization. The combat system alone needs an absurd amount of number crunching per player. Take conditions for example. The condition system currently calculates and recalculates every single condition you apply in real time in order to account for stat changes from boons/food, additionally, it calculates condition damage on the server side, rather than the client side, and has to track the damage across stacks. It does that per active character, including for mobs that cause conditions. It also retroactively buffs already applied conditions to account for things like Might. This is one of the reasons the condition cap exists (that and the whole wvw/pvp balancing with pve issues). It also actively knocks the lowest damage condition stack from the total stack when in a group.

This means per 5-man party, the game could be accounting for a total of 5-15 conditions per attack (including sigil and skill condition adds) plus the already applied conditions, calculates and recalculates the whole group in case of stat change, re-applies and has to repeat that process every single tick of every single condition, and every single attempt to apply the conditions. Now think of the zergs we get occasionally get on champions, in wvw, or on world bosses. That probably accounts for 10-15% of the server side skill lag we get. Guild Wars 2 is the only game I’ve played that does all condition calculations server side, or updates already applied conditions based off new buffs, or makes players conditions over-write each other, most games use a player individualized stack to make balance of conditions easier.

Official state of skill lag and server optimizations

in Guild Wars 2 Discussion

Posted by: Herly.9107

Herly.9107

Hi Syndrilious,

I am completely ignorant as far as the background and what needs to be done as far as server or game client. I can only provide feedback from just an average user.

If I understood correctly GW2 has on the WvW feature one of its unique features (and quite frankly it is awesome). I do understand that when massive zergs join together and clash things tend to be a bit “slower”. As a player it is kind of a bummer… True, but manageable.

In the beggining there was some lag but I noticed that it went on getting better with the work invovled over time. However, since the WvW leagues and Bloodlust feature I noticed the lag and (something I had never experienced thus far) skill lag and NPC’s lag. One can say it has more people palying it (which is good – the more the merrier).

I understand that new features bring new issues and complication. I relate to that. However, it is hard for me to understand how can Arena Net have one of its core game features have more features which will obviously bring more people into it and not be fully ready for them (especially when they claim that they bring it out only when it is ready to launch).

Quite frankly I would rather not have WvW League (which in my server brought more conflict than benefits – but that is a totally different story not relevant for this thread) and have maps that do not lag with big zergs than have a league that you cannot play well enough because you simply cannot play.

I do not want to sound bitter or picky. I just completely love this feature (WvW) and obvisouly I may be sensitive when things do not go as they are planned. This needs to be addressed and I am happy to know that staff of Arena Net (BillFreist) officially made a statement on this. At least we have a confirmation that things (even though slowly) are being addressed. Let’s see how much time we will have to wait to see them implemented and detect them!

A proud ranger with an adorable pet from the European Underworld server.
“Heróis Lusitanos” Guild [LS]

Official state of skill lag and server optimizations

in Guild Wars 2 Discussion

Posted by: Adorable Kinkajou.4370

Adorable Kinkajou.4370

Bill, thank you very much for taking the time to create this thread, giving us such detailed information on what you are doing to save us from this horrible skill lag, and replying to our questions. Much appreciated, and I just have to add that you’re the nicest guy I personally have ever seen to interact with us on these forums. A very special thanks for that alone.

Some observations from my own gameplay experience in wvw which might perhaps be helpful to you and your team:

- Skill lag and weird fps drop issues significantly intensified for me with the Teq patch.
- I’m saying “weird” fps drops because normally I don’t experience it to that degree, and I never experience it in any other game I play.
- I get horrible skill lag and weird fps drops ONLY when our group (small or large) is outnumbered by the enemy. Never the other way around. So, it doesn’t seem to matter how many players are currently fighting on a given area, if we are, say, 25-30 and the enemy numbers are higher than that, I experience these two issues to such an extent that hardly any skill goes off.
- The skills that almost never go off are the utility skills in my bar (heal skill in particular), hardly ever my weapon skills.

I don’t know if other players have already mentioned similar observations or not, but these are my experiences in wvw fights. With the introduction of the Leagues, all of the above have gone from terrible to unplayable for me. Finally I had to exit the game last night because I simply wasn’t able to function in my role.

Wish you guys best of luck in identifying and “slaying” the culprits as soon as possible