HEADS UP: API temporarily down

HEADS UP: API temporarily down

in API Development

Posted by: Lawton Campbell

Lawton Campbell

Web Programmer

Next

Just disabled all of the APIs.

Not entirely sure of the reason, but the Javascript contexts for the API frontend servers are constantly crashing. Due to the holiday I won’t have the bandwidth to debug the problem until tomorrow evening, so I’ve temporarily taken the APIs down in the meanwhile. I should be able to figure out what’s going wrong tomorrow and probably get everything back up by Sunday.

I’ll post updates to this thread if something goes horribly wrong.

Sorry about the downtime

EDIT: Going to be down until Monday at least; going to add explicit rate-limiting to the API. I’ll have some more details later, still need to run some numbers.

(edited by Lawton Campbell.8517)

HEADS UP: API temporarily down

in API Development

Posted by: aandiarie.7195

aandiarie.7195

It happens. Thank you for giving us the info and etc.
It could be a lot worse. At least this time Jormag didn’t send a blizzard to cut off the power.

(edited by aandiarie.7195)

HEADS UP: API temporarily down

in API Development

Posted by: CJay.5013

CJay.5013

Thank you for letting us know! Looking forward to seeing it back up again.

HEADS UP: API temporarily down

in API Development

Posted by: Freezy.4952

Freezy.4952

Thank you for your diligence in this matter.
Looking forward to checking my bank ‘on the go’ again soon :P

HEADS UP: API temporarily down

in API Development

Posted by: Lenaja.5463

Lenaja.5463

Thanks for the info. I was wondering why my app suddenly did not show dailies anymore.

One small suggestion. If the service is off, maybe you could send a 503 header service unavailable so we can better handle this case in the future.

HEADS UP: API temporarily down

in API Development

Posted by: Reincarnated.1754

Reincarnated.1754

Thanks for the Heads Up.

HEADS UP: API temporarily down

in API Development

Posted by: Christian.1408

Christian.1408

Thanks for the heads up!
Is there any way to determine the API’s current status? Something like https://api.guildwars2.com/v2/status ? That would be really nice.

HEADS UP: API temporarily down

in API Development

Posted by: Yamagawa.5941

Yamagawa.5941

Dunno about official ways to determine if its up or not.
For your smoke-test pleasure try
https://api.guildwars2.com/v2/commerce/prices
or
https://api.guildwars2.com/v2/items

Both should return lists of numeric item IDs, and currently are returning ‘not-found’

HEADS UP: API temporarily down

in API Development

Posted by: Leablo.2651

Leablo.2651

Does this have any relation to the recent issues with the in-game TP where requests seem to stall and have to be repeated? That has been happening for some weeks now.

HEADS UP: API temporarily down

in API Development

Posted by: Lawton Campbell

Previous

Lawton Campbell

Web Programmer

Next

One small suggestion. If the service is off, maybe you could send a 503 header service unavailable so we can better handle this case in the future.

I should adjust that. Been turning off the APIs by disabling all endpoints via config; I should add an explicit off switch that sends the proper response.

Thanks for the heads up!
Is there any way to determine the API’s current status? Something like https://api.guildwars2.com/v2/status ? That would be really nice.

The root (https://api.guildwars2.com) kind of works like this? When the v1/v2 endpoints are disabled they’re removed from the list. But using an explicit status code to indicate “the world is on fire” is probably easier on the client side.

Does this have any relation to the recent issues with the in-game TP where requests seem to stall and have to be repeated? That has been happening for some weeks now.

AFAIK the API failure was totally self-contained and shouldn’t have affected other systems.

HEADS UP: API temporarily down

in API Development

Posted by: lynnae.4095

lynnae.4095

looks like, at least certain, endpoints have been up and down today. How big is the fire? ie would you appreciate it if people held off work on new features today that would hit the endpoints that seem to be up (or were up last time my services ran)

HEADS UP: API temporarily down

in API Development

Posted by: Lawton Campbell

Previous

Lawton Campbell

Web Programmer

Next

looks like, at least certain, endpoints have been up and down today. How big is the fire? ie would you appreciate it if people held off work on new features today that would hit the endpoints that seem to be up (or were up last time my services ran)

They should be entirely down, except around 24h ago when I briefly turned them on to get some metrics out.

Which endpoints seem to be up? I’m terrified.

HEADS UP: API temporarily down

in API Development

Posted by: lynnae.4095

lynnae.4095

The trading post prices endpoint is up and down, can we PM here? I can send you the times logs of when I’ve been able to get trading post data:
https://api.guildwars2.com/v2/commerce/prices
and the associated:
https://api.guildwars2.com/v2/commerce/prices?ids=

Most recently about 30 min ago 19:14 UTC

I’m checking the data that’s been returned now, take me a couple of minutes as today I’ve been moving databases.

Data looks legit from a cursory inspection, as in it hasn’t been the same record being returned each time

(edited by lynnae.4095)

HEADS UP: API temporarily down

in API Development

Posted by: Lawton Campbell

Previous

Lawton Campbell

Web Programmer

Next

Regarding the sporiadic up nature, queicherius provided a graph and that correlates strongly with a minor config change I made (to turn the APIs back on for the office IP addresses). I’m not sure why/????/how there’s a correlation there, but I’ve undid it (so the behavior should go back to the solid red bar of “down”.

This week is going to be a lot of fun.

(EDIT: boo the forums strip emoji).

HEADS UP: API temporarily down

in API Development

Posted by: lynnae.4095

lynnae.4095

hmm, here’s the list of times I’ve gotten a response since Saturday morning (hourly checks):

everything fine through 05:09 then nothing. Then sporadic successes:
2017/03/18 23:09:15
2017/03/19 06:09:15
2017/03/19 08:09:15
2017/03/19 10:09:15
2017/03/19 19:14:31 (new server, new time, one day I’ll fix that, but probably not)

I can’t see from here if those all correlate, with that much green I feel unlucky about the big break in hits today

(edited by lynnae.4095)

HEADS UP: API temporarily down

in API Development

Posted by: Lawton Campbell

Previous

Lawton Campbell

Web Programmer

Next

Which timezone? I made the config change at 3/18/2017 7:06 PM PDT (UTC-7); and it normally takes a few minutes to propagate out.

HEADS UP: API temporarily down

in API Development

Posted by: lynnae.4095

lynnae.4095

That’s all UTC

(and have some extra words so the forums let me post. Hi Mom!)

HEADS UP: API temporarily down

in API Development

Posted by: Bjyoric Orcstone.1728

Bjyoric Orcstone.1728

I found that Nginx is great as a rate limiting reverse proxy. Just lookup “rate limiting with nginx” to get started. Extra nice that you can partition the rate limiting. So >5reqs/sec I slowed a little for a couple seconds but >20reqs/sec got throttled to 4/s for a big chunk of time.

Good luck, I feel your pain. This stuff always happens at 2AM on a holiday.

HEADS UP: API temporarily down

in API Development

Posted by: Elayne.9514

Elayne.9514

Is the API still down or is the problem local for me?

HEADS UP: API temporarily down

in API Development

Posted by: Renske.6178

Renske.6178

Still down for me as well so I doubt it’s local!

HEADS UP: API temporarily down

in API Development

Posted by: Darkashara.4871

Darkashara.4871

Still down here too definitly not local

HEADS UP: API temporarily down

in API Development

Posted by: Leablo.2651

Leablo.2651

Going to be down until Monday at least

Key phrase being “at least”. It’s Monday now but as of this writing in PDT it’s roughly 6 AM. Give the man a chance to work.

HEADS UP: API temporarily down

in API Development

Posted by: palduno.9372

palduno.9372

I use Overlay timer for wvw, since the reset no more wvw maps, its out of work. I know that Overlay isnt an Anet product, my question is: is an existing link between the Api shut down and the unworking wvw maps?.
Sry for my poor english.

HEADS UP: API temporarily down

in API Development

Posted by: Itsjon.2869

Itsjon.2869

I use Overlay timer for wvw, since the reset no more wvw maps, its out of work. I know that Overlay isnt an Anet product, my question is: is an existing link between the Api shut down and the unworking wvw maps?.
Sry for my poor english.

Yes the existing links to api are shut down right now. That is why your overlay is not working.

HEADS UP: API temporarily down

in API Development

Posted by: morrolan.9608

morrolan.9608

3pm Monday and still down.

HEADS UP: API temporarily down

in API Development

Posted by: Chaba.5410

Chaba.5410

3pm Monday and still down.

:feelsbadman:

Chaba Tangnu
Founding member of [NERF] Fort Engineer and driver for [TLC] The Legion of Charrs
RIP [SIC] Strident Iconoclast

HEADS UP: API temporarily down

in API Development

Posted by: Lawton Campbell

Previous

Lawton Campbell

Web Programmer

Next

Distributed systems are hard

Tomorrow might be a better day.

HEADS UP: API temporarily down

in API Development

Posted by: Illconceived Was Na.9781

Illconceived Was Na.9781

Distributed systems are hard

Tomorrow might be a better day.

Thanks for the update.
(Get some rest; the API can wait, even if many of us are going through withdrawal.)

HEADS UP: API temporarily down

in API Development

Posted by: Bjyoric Orcstone.1728

Bjyoric Orcstone.1728

I bet the biggest number of hits on the API comes from the wiki – trading post prices.

Simple test if API is up: https://api.guildwars2.com/v2/build

Currently returns {"error":“not found”}

HEADS UP: API temporarily down

in API Development

Posted by: ShadowCatz.8437

ShadowCatz.8437

Distributed systems are hard

Tomorrow might be a better day.

How many days are “tomorrow”?

HEADS UP: API temporarily down

in API Development

Posted by: Hactarus.8563

Hactarus.8563

Still not enable on Quaggans. I’ll wait another day.

$ curl -i https://api.guildwars2.com
HTTP/1.1 200 OK
Content-Length: 2
Content-Type: application/json; charset=utf-8
Server: Quaggans
Access-Control-Allow-Origin: *
X-Content-Type-Options: nosniff
X-Powered-By: ARR/2.5
X-Frame-Options: SAMEORIGIN
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block
Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
Date: Tue, 21 Mar 2017 08:42:34 GMT

[]

HEADS UP: API temporarily down

in API Development

Posted by: Nekres.1038

Nekres.1038

Do some sort of caching, that, when API is down, can still dispense all the resources that have been frozen on take down?

HEADS UP: API temporarily down

in API Development

Posted by: gegula.8249

gegula.8249

any info when this will be up? i need my tradepost stuff ;(

HEADS UP: API temporarily down

in API Development

Posted by: Orstep.1532

Orstep.1532

Do some sort of caching, that, when API is down, can still dispense all the resources that have been frozen on take down?

Please consider this. Static endpoints such as /recipes, /recipes/search, /items, /achievements etc are all down, and they don’t have to be.

HEADS UP: API temporarily down

in API Development

Posted by: Ruggs.8420

Ruggs.8420

Do some sort of caching, that, when API is down, can still dispense all the resources that have been frozen on take down?

Beyond setting reasonable cache headers that’s really the client’s job. I know I maintain a local cache for my API consumers.

HEADS UP: API temporarily down

in API Development

Posted by: Orstep.1532

Orstep.1532

Do some sort of caching, that, when API is down, can still dispense all the resources that have been frozen on take down?

Beyond setting reasonable cache headers that’s really the client’s job. I know I maintain a local cache for my API consumers.

True.

It still bugs me that the API went down the morning after I implemented a caching bit into my script. All I would need right now to continue working on my school assignment is being able to run my script just once. (Fyi, I only need the static endpoints for my tool to function)

HEADS UP: API temporarily down

in API Development

Posted by: EAGLE.3429

EAGLE.3429

Could there please have a rough Eta as to how long the API site will be down for? I understand that if ya give an ETA and things dont turn out the way ya expect it may take longer. Any info would be appreciated.

HEADS UP: API temporarily down

in API Development

Posted by: Lawton Campbell

Previous

Lawton Campbell

Web Programmer

Next

Do some sort of caching, that, when API is down, can still dispense all the resources that have been frozen on take down?

Please consider this. Static endpoints such as /recipes, /recipes/search, /items, /achievements etc are all down, and they don’t have to be.

Those endpoints aren’t actually static. Greatly increased request volume to those endpoints is what caused these crashes, unfortunately.

any info when this will be up? i need my tradepost stuff ;(

I’m trying to get it back up today; I still need to get someone to review my changes to make sure I don’t cascade the failure to other systems if I turn everything back on.

Could there please have a rough Eta as to how long the API site will be down for? I understand that if ya give an ETA and things dont turn out the way ya expect it may take longer. Any info would be appreciated.

ETA was yesterday, except during testing there was a small bug in my implementation that caused it to use the wrong instance of the centralized rate-limiting service (i.e. it used the one that everything else uses instead of the API-specific one). So I punted on the deploy until we’re sure that the changes aren’t going to overwhelm anything.

HEADS UP: API temporarily down

in API Development

Posted by: EAGLE.3429

EAGLE.3429

Fair enough. Thank for the update. I fully understand that in programming things happen that are beyond your control. Id rather wait a bit longer for a good release then have a crappy release. Are you thinking tomorrow hopefully?

HEADS UP: API temporarily down

in API Development

Posted by: Lawton Campbell

Previous

Lawton Campbell

Web Programmer

Next

I’m really hoping for today; pretty sure I’ve got all the kinks worked out.

HEADS UP: API temporarily down

in API Development

Posted by: Behellagh.1468

Behellagh.1468

All the things crossed.

We are heroes. This is what we do!

RIP City of Heroes

HEADS UP: API temporarily down

in API Development

Posted by: EAGLE.3429

EAGLE.3429

If there is anything that we the community do to help ya out please ask. You may be surprised how many people would help out.

HEADS UP: API temporarily down

in API Development

Posted by: krethas.4287

krethas.4287

#hugops to you and a sympathetic nod re: distsys being hard.

(also, glad to see rate limiting coming. I saw the frequency of requests some sites were making just for my one account and the ops person in me went AAAAAAAAAA)

(edited by krethas.4287)

HEADS UP: API temporarily down

in API Development

Posted by: Lawton Campbell

Previous

Lawton Campbell

Web Programmer

Next

Okay, the API is tentatively back up with a limit of 60 reqs/minute while I double-check some things.

Will bump that up to the full 600 reqs/min in an hour or so when I’m confident that nothing has caught fire (which might happen, but doesn’t look likely at this point).

HEADS UP: API temporarily down

in API Development

Posted by: EAGLE.3429

EAGLE.3429

Thank you. My verify bot is already recieving data from it on my end.

HEADS UP: API temporarily down

in API Development

Posted by: Corax.7381

Corax.7381

Woo! You’re da real MVP.

“Quaggan will kick your tail so hard it slaps you in the face!” – Willoo

HEADS UP: API temporarily down

in API Development

Posted by: Lawton Campbell

Previous

Lawton Campbell

Web Programmer

And the ratelimit is bumped up to 600 reqs/minute.

As far as I can see, everything is mostly humming along okay.

HEADS UP: API temporarily down

in API Development

Posted by: Smoaking.6125

Smoaking.6125

You’re the best! Thanks for all your hard work, Lawton.

HEADS UP: API temporarily down

in API Development

Posted by: Tengauge.2165

Tengauge.2165

^This, very much this. I truly appreciate the hard work you put into the API, and even show it off to IT coworkers when I need to explain to them why the whole world needs well documented APIs for everything. Your efforts are strongly appreciated by all of us geeks in the community.

HEADS UP: API temporarily down

in API Development

Posted by: Renske.6178

Renske.6178

Awesome!! Thank you for you hard work. I was usng it for a school project so im very glad its back up. =D