HTTP vs UDP...is this the grey we have been seeing?

[Ayesha Askham]

Now let me be clear at the outset: I do not claim to know much about what Linden Lab are attempting to do in the increasing use of the http protocol to download data to our viewers.  I cannot be sure that what I say next is true, false or so much nonsense, it is just a thought.

Last week we in the UK and other significant parts of the World endured at best several hours and at worst several days of grey avatar textures, invisible objects and in some cases immutable bake-fail.

This was the second time in two weeks to my knowledge that the http pipeline had apparently failed.

When LL announced to no small fanfare that the http protocol was to be increasingly used to transmit data from SL's servers to our computers, it was questioned in some quarters as to whether this ultimately would be any improvement over tthe existing udp system.  Once http was in widespread use, some said, there would be as many and possibly more issues than had existed with the slower udp set-up, and indeed udp might turn out to be faster for many.

That struck a chord with one comment recently in this forum, namely that while all this grey chaos was reigning, the use of the Catznip viewer, which has http textures turned off as default, seemed to radically improve the texture-loading matter.  I also recall being told that some routers and modems are much better than others at handling http traffic.

I just wonder, in the uninformed way I sometime do, if this reliance now on http traffic vs udp, could be at the heart of our recent grey episodes?

If you know better I am glad to stand corrrected, but please be clear...I am not using this thread to bash LL for anything.  I just want to try to understand what has been happeneing since there appears to have been no "official" word about these problems.

[Dagger Faulkwing]

Ayesha I'm have texture stall problems this evening and inability to take off worn items. This lasts for up to five minutes at a time and then everything returns to normal. Earlier this evening (UK time) I was trying on skins. Suddenly the skins stopped changing. Ctrl+Alt+R stopped working, too. I was wearing one skin according to Inventory but what I saw on my av was the previous skin. I tried clicking several different skins and tried Ctrl+Alt+R several times then sat and waited. A few minutes later it was like watching a slide show as my av quickly went through wearing all the different skins I'd tried to wear. I did not go from the original skin to the last one I had tried, I saw all of the skins quickly rez and be replaced on my av. It seems as though we have a normal period of rezzing, then everything stalls, pauses, and returns to normal. I'm back on Firestorm at the moment because good though I found Catznip it is slow rezzing textures.

 

(Edit) Returned to Catznip. Texture stalling is bad even on the Main Linden viewer. One new pair of pants with 2 alpha layers in the folder. Neither alpha layer would rez, one of the 2 alpha layers refused to be taken off, and at one point I had both alpha layers showing as worn with neither rezzing. Finally logged in on Catznip and all was resolved. Will see if textures continue stalling on Catznip tomorrow.

[Theresa Tennyson]

---

Ayesha Askham wrote:

Now let me be clear at the outset: I do not claim to know much about what Linden Lab are attempting to do in the increasing use of the http protocol to download data to our viewers.  I cannot be sure that what I say next is true, false or so much nonsense, it is just a thought.

Last week we in the UK and other significant parts of the World endured at best several hours and at worst several days of grey avatar textures, invisible objects and in some cases immutable bake-fail.

This was the second time in two weeks

to my knowledge

that the http pipeline had apparently failed.

When LL announced to no small fanfare that the http protocol was to be increasingly used to transmit data from SL's servers to our computers, it was questioned in some quarters as to whether this ultimately would be any improvement over tthe existing udp system.  Once http was in widespread use, some said, there would be as many and possibly more issues than had existed with the slower udp set-up, and indeed udp might turn out to be faster for many.

That struck a chord with one comment recently in this forum, namely that while all this grey chaos was reigning, the use of the Catznip viewer, which has http textures turned off as default, seemed to radically improve the texture-loading matter.  I also recall being told that some routers and modems are much better than others at handling http traffic.

I just wonder, in the uninformed way I sometime do, if this reliance now on http traffic vs udp, could be at the heart of our recent grey episodes?

If you know better I am glad to stand corrrected, but please be clear...I am not using this thread to bash LL for anything.  I just want to try to understand what has been happeneing since there appears to have been no "official" word about these problems.

---

First of all, HTTP texture transfers are nothing new - they've been the default behavior for Linden Lab viewers since at least 2010 and probably for Firestorm for a considerable length of time. All avatar textures have been sent by HTTP since server-side baking rolled out. I believe meshes have always been sent by HTTP.

Two things HAVE changed. First of all, the Linden Lab viewer has been set up to use HTTP pipelining, which uses a small number of constantly open connections to get data rather than opening and closing a separate HTTP connection to get each file which was how the viewer was previously set up. However, if you use Firestorm this won't  mean vinegar to a rabbit because Firestorm doesn't use HTTP pipelining yet. However, it still uses HTTP texture transfers unless you specifically turn them off.

The other change does affect Firestorm users. Rather than having all textures and meshes directly from the Linden Lab server farm in Arizona, they're being cached on multiple regional servers provided by a "content delivery network" (CDN) named Highwinds that already was sending out baked avatar textures. Highwinds already provides similar services for companies like Valve, the owners of Steam. The CDN servers are closer to end users of SL and can send the data faster while simultaneously reducing load on the LL servers so they can spend more time calculating avatar positions, world physics interactions, etc.

The problems this weekend sound like they were in the Highwinds node in the UK. This would explain why Linden Lab support wouldn't have much information about it. I suspect that turning off HTTP textures bypasses the CDN which may be the reason it worked better for some people.

[SaraCarena]

Yep was http failure, hopefully it`s short term teething problems since CDN went gridwide. Like you I`ve only seen it happen twice for a short period of time.

There`s a few open jiras about this an I`m sure the Lindens are working on how/why it happened.

You can turn http textures off in any viewer and http inventory too, it`s on the develop menu. Then you`ll get what you need via udp, though it will be slower than http.

Personally I had http off permanently about 18 months ago mainly coz of texture blur on system clothing, there have been improvements in http over time and I`ve had it turned on for a year or so now.

I know from watching the TPV meetings on youtube (and in sl experience generally) that Monty Linden has made big improvements in http delivery compared to what they once were and TPV developers have commented as much.

TPV meetings are every other Friday, this weeks should be up on youtube on Saturday, It`ll be better than Eastenders :smileywink:

 

[Ayesha Askham]

Theresa and Sara

See?  I said that I'd happily stand corrected and I am so corrected.  Mind you Theresa, it would seem that it was more than just the UK node of Highwinds that was glitching. 

I had not appreciated the significance of http pipelining so that is a bit more info for me!

The "Highwinds" connection may well have had a pivotal role to play in these woes, so no doubt the CDN will be front and centre at future dev meetings.

Just when you think you are beginning to understand the mountains that are SL's functions along comes someone who knows enough to make you realise that you are still in the foothills! :smileysurprised:

 

Dagger: your experiences chime well with Theresa's explanation of the HTTP vs UDP texture issue.

[Whirly Fizzle]

As well as CDN having a couple of rather large hissy fits since release, which affects all viewers, there is a strange texture/mesh fetch problem for some people when using a pipelining enabled viewer on CDN regions.

Pipeline viewer on a non-CDN region behaves well and a non-pipeline viewer on CDN regions behaves well. The problem seems to happen when using both pipeline and CDN.

See BUG-7698 - Textures much slower to load on a CDN region then on a clone of the same region not running CDN

[Ohjiro Watanabe]

I'm in the UK and didn't see any grey the whole period. In fact my performance is great. The ping time to the CDN server is 7ms as compared to a ping to San Francisco which is about 150ms on a good day.

I am totally pro HTTP for textures. Well done Linden Lab. It is just dum to use UDP for to transfer a file over the internet. Use UDP to stream media, yes, use it for very short messages that can be sent on one packet, yes (there are loads of these between SL server and viewer).

But files, with no error checking, man o man! That's for the birds.

 

[Perrie Juran]

---

Ayesha Askham wrote:

<snip>

it would seem that it was more than just the UK node of Highwinds that was glitching. 

</snip>

---

I live in the States and it was the single worst night of bad rezzing I have ever experienced. 

And I know at least two others in the States who were having trouble.

[irihapeti]

last 2 days has been going pretty good for me. Like really good. No stalls. No grays. No never ending blurries. If it continues then is the best it has ever been (for me) since forever

fingers crossed for the weekend bc last weekend was not good. I has hope tho that will be like the last 2 days for evermore (:

[Nalates Urriah]

Ayesha, the HTTP pipeline did not fail. There was and is a problem with the CDN (Content Delivery Network). Both the CDN provider and Linden Lab are aware of the problem and changing things to solve the problem. For whatever exact reason, the system was not delivering textures. The problem persisted for certain regions for a day or two. Whirly points us to a JIRA where those still seeing a problem can add their information (POST).

It seems the new load from the Lab and another client created problems. The Lab’s side is being modified to improve performance. The CDN provider is closely monitoring SL performance according to Oz Linden in the Friday (11/7) third party developer’s meeting. Changes are being made by other parties.

While some may question the change from UDP to HTTP those that work with protocols and have some understanding of how SL works have no doubt that HTTP will be an improvement. With the addition of CDN the improvement will be substantial.

First CDN takes considerable load off the region servers freeing them up to handle other tasks. The UDP channel has no error correction. HTTP has error correction built in at low levels allowing network cards to handle resends and freeing the CPU to work on other things. Plus HTTP Pipeline allows a channel to be opened for multiple file transfers thus eliminating the overhead of opening a channel for each file and then closing it. The load on the router/gateways we use is greatly reduced as they have less channels to deal with.

UDP as a protocol can be faster, unless you have a connection with lost packets. But, when we factor in the servers on the Lab’s side, UDP will always be slower. This is a case of it does not matter how fast the jet is if you can’t get it on deck or launched from the carrier.

It is understandable that you would wonder if HTTP is causing the grey, that is a recent change and associating recent new problems to recent changes is rational. But, in this case the grey is an artifact of the HUGE system change to CDN. As Oz Linden said, any change this big to a large system is an adventure. I think meaning there are many things that just cannot be foreseen until it is tried.

And of course, after your post the Lindens have now posted about the problems. (See forum tech section)

[irihapeti]

just add on here. For feedback since

+

has been going pretty good for me last week now. 3.7.21 (296734) currently

was only one day I was at this event and I had a big consistent FPS lag (like down to 6fps) when my cam was on the crowd. We was all jammed into this quite small space. About 30 avatars. Dunno what was the actual reason. Some avatar wearing something I think. bc when I cam away then it ran normal ok. About 40fps. Cam back and downhill again

went to TFT. and Cupcake. Both big builds. Lots of stuff. Lots of new textures, mesh and sculpt for me to dl. No probs. Fast dl, load and render. Same other places like these

so woohoo! (:

 

[Sean Heying]

---

Ayesha Askham wrote:

I just wonder, in the uninformed way I sometime do, if this reliance now on http traffic vs udp, could be at the heart of our recent grey episodes?

If you know better I am glad to stand corrrected, but please be clear...I am not using this thread to bash LL for anything.  I just want to try to understand what has been happeneing since there appears to have been no "official" word about these problems.

---

Not so much http faulty over UDP, but pipelining. In particular a few bugs in libcurl made worse by recent CDN issues - BUT bugs never-the-less.

I have been pre-alpha testing a bug fix to libcurl that addresses the issue. This bug fix makes a substantial difference to the loading in Malls, to the point that I can go into BlackRose and see the textures steaming in without pause at 8,000+ network speed.

Very technical discussion for our more au fait forum goers http://thread.gmane.org/gmane.comp.web.curl.library/44070