Deploys for the week of 2013-09-23

[Maestro Linden]

Second Life Server (main channel)

The main channel is updating to the server project that was on Magnum last week.  This project fixes a few crash modes, addresses some LSL bugs, prevents a technique used to bypass parcel autoreturn rules, and fixes a bug in the parcel access logic, along with other miscellaneous fixes.

https://wiki.secondlife.com/wiki/Release_Notes/Second_Life_Server/13#13.09.21.281328

Scheduled Tuesday 2013-09-24 05:00-11:00 PDT

  

Second Life RC BlueSteel, RC Magnum, and RC LeTigre:

This week, all three RC channels are getting the same version as the main channel.  This version includes the same simulator code that Magnum had last week - see the main channel section above for a summary of changes.

https://wiki.secondlife.com/wiki/Release_Notes/Second_Life_RC_BlueSteel/13#13.09.21.281328

https://wiki.secondlife.com/wiki/Release_Notes/Second_Life_RC_LeTigre/13#13.09.21.281328

https://wiki.secondlife.com/wiki/Release_Notes/Second_Life_RC_Magnum/13#13.09.21.281328

Scheduled Wednesday 2013-09-25 07:00-10:30 PDT

 

We will be monitoring this thread as the code gets released, so feel free to note any observations you have about the server updates.  If you have a specific bug you'd like to report, please file a Jira. 

[Maestro Linden]

Second Life Server (main channel)

The main channel is updating to the server project that was on Magnum last week.  This project fixes a few crash modes, addresses some LSL bugs, prevents a technique used to bypass parcel autoreturn rules, and fixes a bug in the parcel access logic, along with other miscellaneous fixes.

https://wiki.secondlife.com/wiki/Release_Notes/Second_Life_Server/13#13.09.21.281328

Scheduled Tuesday 2013-09-24 05:00-11:00 PDT

  

Second Life RC BlueSteel, RC Magnum, and RC LeTigre:

This week, all three RC channels are getting the same version as the main channel.  This version includes the same simulator code that Magnum had last week - see the main channel section above for a summary of changes.

https://wiki.secondlife.com/wiki/Release_Notes/Second_Life_RC_BlueSteel/13#13.09.21.281328

https://wiki.secondlife.com/wiki/Release_Notes/Second_Life_RC_LeTigre/13#13.09.21.281328

https://wiki.secondlife.com/wiki/Release_Notes/Second_Life_RC_Magnum/13#13.09.21.281328

Scheduled Wednesday 2013-09-25 07:00-10:30 PDT

 

We will be monitoring this thread as the code gets released, so feel free to note any observations you have about the server updates.  If you have a specific bug you'd like to report, please file a Jira. 

[WolfBaginski Bearsfoot]

This looks very unusual, all the sims on the same code.

This does conjure images of the programming equivalent of dioxygen difluoride synthesis and I really hope nothing that bad has happened.

Though I don't think that it's a bad thing to have this pause, from the point-of-view of the customers.

[Cincia Singh]

Or ... other server updates in the pipeline simply aren't ready for release yet.

[Ayesha Askham]

OK Maestro

This is ridiculous.  My sim just went down with absolutely NO warning. None. Now OK, I know better than to be doing anything at this time on a Tuesday, but your servers ARE supposed to give warnings, not just boot folk off unceremoniously.

No, I did not log in less than one minute before the restart, though it might have been less than 5 minutes.  This was supposed to get fixed.  It clearly is not fixed. :smileymad:

Or was this another ham-fisted shutdown by the server team?

[Maestro Linden]

Hi Ayesha, which region were you in?

[Ayesha Askham]

Maestro

It is Woods of Heaven, now sim no. 9786.

There are issues on restarted sims all over Main Server, though: I have just submitted a ticket regarding Lost Island, currently sim no. 9038 which is showing 100ms more sim ping than Woods and packet loss spikes of over 5%.

About 50% of TPs from there (Lost Island) are resulting in crashes which result in me being returned home, so I am assuming that it must be a networking issue within SL for that to occur.

ETA:

Upon crashing during TP out of Lost Island I relogged, to find myself at home. No great surprise there.  I spent a few minutes dealing with messages and then logged off.  (In order to forestall comments, it was a clean log out - no signs of any sort of viewer crash).  I was rather surprised to be logged in, when I returned, to Lost Island, where I had crashed earlier.  This, Linden Lab, is NOT amusing.  To be honest I did not even think it was possible!

Maestro, something is very wrong.  Please get it sorted. :smileysurprised:

[Maestro Linden]

Thanks Ayesha, I looked into your case, but unfortunately couldn't find any error messages.  The shutdown sequence did detect agents in the region and waited 5 minutes, but it's unclear whether notifications were delivered to your or not.  My logged-in avatar did receive the rolling restart notifications while in a different sim, so at least the issue isn't universal.

There are plans to have rolling restart tool use the 'faster shutdowns' codepath that was introduced in 13.08.23.280213 for non-rolling restarts; this new codepath has better server-side logging, so it should be easier to identify the source of restart notification issues in the future.

[Hitomi Tiponi]

I logged in to Kolamaafushi and a minute later it did a restart with no message, and I was logged out. It would be good if logging in to a sim about to go offline elicited a message.

[Ayesha Askham]

Maestro

This issue becomes stranger the more I look into it.  You say the region detected agents and DID give the appropriate warnings - this is not likely since prior to my arrival there were NO OTHER agents on the sim.

I can categorically state that I received NO warning at all, not even the usual "You have been logged out of SecondLife" one, so something in the sim-client communication was decidedly not functioning.

I do not say the warning was not issued by the server, but I DO say that it was never received by the client.

[Maestro Linden]

Hm Ayesha, if you didn't receive the "You have been logged out of SecondLife" message, was it a generic timeout sort of message, as though the simulator crashed or you lost connectivity?

[Perrie Juran]

---

Ayesha Askham wrote:

Maestro

This issue becomes stranger the more I look into it.  You say the region detected agents and DID give the appropriate warnings - this is not likely since prior to my arrival there were NO OTHER agents on the sim.

I can categorically state that I received NO warning at all, not even the usual "You have been logged out of SecondLife" one, so something in the sim-client communication was decidedly not functioning.

I do not say the warning was not issued by the server, but I DO say that it was never received by the client.

---

Is it posible that the Client failed to display it rather than 'never recieved' it?

 

[Paul Hexem]

Whatever was changed this week broke group access land.

My estate uses a chain of 14 sims. Many are group access, specific for motorcycles.

All day, people have been unable to access them, despite being in the required group, and despite having been able to access them in the past.

[Ayesha Askham]

Yes, Perrie, it IS possible but since it has never failed to display the restart message before, and displays other server-sourced messages, it is less likely in my opinion.  Also there was nothing in the logs to suggest a client crash.  It was the usual log-out greyscale image, without the message.

[Maestro Linden]

Hi Gadget, could you file a Jira with details about which parcels are having the issue?

[Maestro Linden]

Hitomi, Perrie, Ayesha: my hope is that once the rolling restart tool is updated to have better logging about the restarts (including about when sims send notifications to present avatars), we'll have a much better idea about the cause of a failed restart notification.

[Paul Hexem]

---

Maestro Linden wrote:

Hi Gadget, could you file a Jira with details about which parcels are having the issue?

---

It's a whole chain of sims.

 

https://jira.secondlife.com/browse/BUG-3992

[WolfBaginski Bearsfoot]

I may well have missed problems when the current code was at the RC stage, just by blind chance, but I cannot rule out a combination of Linden Lab server vulnerabilities triggered by delays on the wider internet. I am not in the USA.

Having said that, sim crossing sucks.

Even with a "light" Avatar, crossing a sim boundary often sees rubber-banding lasting for several seconds. Vehicles have sometimes gone haywire.

 

[Ayesha Askham]

Adding to Wolf's comment:

Last night (UK time) on Blake was just about the worst sim-crossing experience I have EVER had in SL.  Like Wolf I must have failed to recognise the problem when this software was on Magnum.  I was stuck ghosted on nearly every sim crossing on Blake, camera view was corrupted several times and ultimately my viewer crashed.  Upon attempting a relog to the same point I got the message "Login Failed.  Agent not on sim.  Please try again in a few minutes".  I eventually managed to log in at home, but the whole process was appalling.

Coupled with the currently atrocious performance of the transatlantic/USA servers of abovenet and some undefined networking issues between LL datacentres, in the USA business day, anything that involves more than sitting around chatting is almost impossible for users in Western Europe.

Linden Lab this must stop now.

[Aeon Voom]

Hey Maestro.

I'd like to point you towards BUG-3990

My Helicopters and some Fixed wing aircraft got affected. I also get reports from the Sailors that things are getting wonky on their end. Generally i tend to say that the problem looks like the buoyancy setting and/or the mouselook steering gets badly affected.

[Sky Linnaeus]

Issues on Grand Pacific (Magnum) after yesterdays restarts. Friends randomly appear to go offline even though they are still online. IM's to most people get the standard "[11:07:53] Second Life: User not online - message will be stored and delivered later." and they don't get the IM even though they are still online. Restarting the region helped for a bit last night but today even another restart hasn't helped. Today I'm also getting "Unable to create item that has caused problems on this region". This error is coming from scripted items and from huds. Moving to a region on a different server stops the IM issues although a relog is needed to correct my friends list to show who is really online and as soon as I go back to Magnum it starts again.

[WolfBaginski Bearsfoot]

I'm inclining to the view that the problems are more to do with the network than with the SL servers, since I have been seeing a pattern related to time of day. I'm not discounting the internal LL network as a factor, but since I've had crossing failures between Blake Sea regions, with no other AVs in the locality, it doesn't look like a heavily loaded sim server is the problem.

[WolfBaginski Bearsfoot]

Your stuff is usually pretty good. I am seeing a lot of the vanishing HUD problem, plus camera position and sit position errors.

[Uggo Vieria]

Hi Maestro,

I'm seeing regions that are online but won't let any object in.

Happened saturday 28/09 in "Blake Sea - Indian" (https://support.secondlife.com/case-details/01771896/)

and today in "Treasure Cove Passage" (https://secondlife.com/my/support/?caseID=01772358)

 

Crossing with a vehicle was imposible but entering w/o the vehicle was possible. Once in the region, rezzing a boat failed with "Unable to create requested object. The region is full" and trying to create a prim failed with the same kind of message. The region had prims available :

[02:34] SIM prims (bonus factor version): .Prims : Total = 3750 - Used = 2910 - Available = 840

In Blake Sea - Indian yesterday, a restart fixed the problem.

 

Treasure cove was finally restarted by estate owner and it fixed the problem, her comment was " Hi Uggo... I just reset Treasure Cove.  I am familiar with this sort of problem in the last week.  A reset fixes it."

[Uggo Vieria]

Here are some reports of the problems I've been encoutering recently at sim crossings around Blake Sea and the USS. I hope it will give more details about the hazardous crossings and help resolving the problems. Some seems to be related to autoreturn and started after the last server deploy.

Equipage : Me (Uggo Vieria) and my alt UggoV in a boat. Not sure it's important to have 2 avatar on the vehicle but I had the boat returned at sim

crossing several times since the last server update and each time it has happened, we were 2 or 3 avatars on the boat.

Boat : YachtLine 420 Key : cfdb25b9-bdbb-8802-04c5-e16a6f2b2764

 'YachtLine 420' [10/10] running scripts. 640Kb consumed for 0.026845ms of cpu time.

 'Uggo Vieria' [9/9] running scripts. 246Kb consumed for 0.039028ms of cpu time.

 'UggoV Resident' [0/0] running scripts. 0Kb consumed for 0.000000ms of cpu time.

on 2013-09-29 2:04 AM SLT secondlife://Mumford%20Cove/254/125/20

    At the crossing between "Watch Hill Passage" and "Mumford Cove" I saw my alt unseated from the boat and the boat with me flying away.
    I right-clicked the boat and fond myself at 0/0/0 in the SW Corner of Wacth Hill passage.
    After a few seconds, I was standing at http://maps.secondlife.com/secondlife/Mumford%20Cove/254/125/1

    On the second screen my alt is still seeing us on the boat but the boat is stuck at the W crossing between Little Gull Islands and Treasure

Cove Passage.

    I could see the boat and TP'd there. I landed for the TP seated on the boat but w/o animation and the boat was still stuck at the border

(another boat arrived and has hit the border too)

    I stood up and found myself in the sky, the position given by my viewer :  Treasure Cove Passage/59/-87/29



At 02:45 AM  - secondlife://Union%20Passage/20/1/20

    Boat returned at the crossing from  Little Gull Islands to Union passage.

    Boat was returned "due to parcel owner return" with 2 avatrs sitting on it. (not sure if this was owner return or auto return), happened later in Hay Harbor (see below) and there I'm sure it was "auto return")

    From the 2 avatars view, the viewer keeps interpolating and flying up in the sky but w/o the boat. Had to TP from world map to get back.

Rezzed the boat from Lost & Found (new UUID 9ae8be87-fd3b-9fb7-52b1-bb72598e0806) and crossed into Union Passage w/o any problem.


At 03:04 AM - secondlife://Blake%20Sea%20%2D%20Turnbuckle/16/1/19

    Entering Blake Sea - Turnbuckle from Blake Sea - Windlass.

    UggoV disappeared from my screen during a few seconds. On the second screen, he was at the sea bottom at the region border.
    a few seconds later, back in the boat and seated. Only dammage was animation stopped.

At 03:25 AM in Blake Sea - Crows Nest, my friend Dex Moleno joins me in the boat.

At 03:30 AM - secondlife://Blake%20Sea%20%2D%20Norwegian/1/116/20

    Dex is unseated from the boat at the crossing into Blake Sea - Norwegian from Blake Sea Japan, I see his dot around 100 m ahead in minimap

and he seems to be moving up and down. Apaprently a crash on his side.


At 03:52 AM - secondlife://Blake%20Sea%20%2D%20Caribbean/255/71/19

    At the crossing into Blake Sea - Carribean from Blake Sea - China, I lost my camera parameters (the boat uses llSetCameraParams)
    Had to stand up and sit again to have my cam back to its position

At 03:58 AM - secondlife://Blake%20Sea%20%2D%20Kendra/83/2/19

    At the crossing into Blake Sea - Kendra coming from Blake Sea - Beaufort, lost UggoV.
    On the second screen I see water flickering and reported position is Blake Sea - Beaufort (124, 384, 0) but I see the dot on world map in the

center of Blake Sea - Kendra (copy SLURL from world map gives http://maps.secondlife.com/secondlife/Blake%20Sea%20-%20Kendra/128/128/5)

    From my main account screen, I see UggoV in the center of BS - Kendra moving up and down fast above the ground.

    I got UggoV back to the boat with a TP offer but couldn't sit him on the boat again because the sit targets had not been freed for the

numerous crashes

    I rezzed a new boat, new UUID : 8193fa90-ffa4-8f65-b963-12b65ecef665


Another weird thing. I noticed that a hud that I made for reporting collisions was no longer working. Resetting the scripts didn't work and I had to

open the script to see it had been disabled. Re-checking "Running" in the script editor worked but if it happens in a nomod object, the object is

probably broken. I searched my logs to see when it stopped working.
    The last collision report was after crossing into Blake Sea - Yellow at 03:39 AM
[2013/09/29 03:39]  USB HUD Pro 1.2 Beta6: secondlife://Blake%20Sea%20%2D%20Yellow/4/142/21 (%RSYC 56 Time trial Waypoints%2%)
[2013/09/29 03:39]  Collision detector: Key : 9ae8be87-fd3b-9fb7-52b1-bb72598e0806 / Name : YachtLine 420 #BS86f0 / OwnerKey : 86f0ee10-938e-4737-

b989-ad4e6fa9948f / Position : <1.69571, 142.32930, 20.13313>
    Afetr that, I no longer had any collision report until I re-enabled the script at 04:26. I have regular collision with my own boat or a

passenger at crossing but that's not systematic so I can't say when the script has stopped working, I can only tell the last time it worked.




At 04:50 AM - secondlife://Fishers%20Island/194/255/18

    At the crossing into Fisher Island coming from Race Rock, the boat started flying away, even after my hud reported I had entered Fisher

Island. After a few seconds, the boat vanished and I was stuck above water. Position given by the viewer : Fisher Island (314, 271, 23) but world map

is showing me offworld in the north of Hay Harbor, can't get SLURL

    Dex is seeing himself in the sky too, his viewer is reporting Sailors Cove, Race Rock (155, -174, 40) and his world map gives an SLURL :

Fishers Island (156)-175/41

    UggoV's viewer is reporting Fishers Island (198, 255, 6) and world map is accurate with that (http://maps.secondlife.com/secondlife/Fishers

%20Island/199/255/6)

    By mogin my cam, I've been able to TP in Hay Harbor but after a few seconds, I was under the ground in Hay Harbor

(http://maps.secondlife.com/secondlife/Hay%20Harbor/166/203/5)

    In the meanwhile, Dex who was still floating in the air had his cam back to the boat that had vanished. he right-clicked it and found himself

standing next to UggoV at the sea bottom, no boat for him either.

    I used world map to TP back to Fisher Island. The boat is still missing, I have one in my L&F but I don't know if it's from now or earlier.
    I had no return message but when I re-rezzed the boat using Firestorm "Restore at last position" feature, the boat rezzed at 0/0/0 :

secondlife:///app/objectim/8bb6f865-b0e1-9282-7b74-18d1e6047008/?name=YachtLine 420 #BS86f0&owner=86f0ee10-938e-4737-b989-ad4e6fa9948f&slurl=Fishers

%20Island%2F0%2F0%2F0


    I sit me and UggoV on the new Boat, start it and enters Hay Harbor. UggoV is unseated from the boat. Position in the viewer title bar :

Fishers Island (258, 202, 3). World map gives http://maps.secondlife.com/secondlife/Hay%20Harbor/3/202/4 that seems to be the real position reported

by the server. The problems seems to be due to bad connection wih the new region, it's all blue on minimap and no land or object are showing on UggoV

screen.

    UggoV is still stuck at the sea bottom in Hay Harbor, used double-click TP to go back into Fishers Island and it worked but Hay Harbor is

still not visible.
    UggoV back into Fishers Island could hardly move but managed to cross into Race Rock again after a crossing that took around 10 seconds. I

was hoping it would establish the connection with Hay Harbor but it didn't work. A TP offer sent from my main account to UggoV worked.


at 05:27 - secondlife://Cats%20Paw/20/255/20

    Entering Cats Paw coming from Hay Harbor. Again the boat keeps flying after my hud was saying I had entered the region. Boat returned to

inventory at Hay Harbor 20, 254 due to parcel auto return.
    On my main screen the viewer is reporting position Cats Paw (118/128/173 and moving along the Y axis) i see UggoV moving with me but his dot

on mimimap is offworld in the south. My world map gives http://maps.secondlife.com/secondlife/Cats%20Paw/55/190/-142

    On UggoV's screen, viewer is giving Cats Paw (20, 255, 2) world map as well. http://maps.secondlife.com/secondlife/Cats%20Paw/20/255/3. my

(Uggo Vieria) dot was showing there too on the second screen but my avatar was not visible.

    At 05:35 my main screen was still showing both avatars flying above Cats Paw. i send a TP to Uggo Vieria from UggoV's screen. The avatar

landed but on my main screen, i was still flying. Position on the title bar : Cats Paw 137/102/221 and moving world map was giving

http://maps.secondlife.com/secondlife/Cats%20Paw/160/83/218. On UggoV's screen, I was stuck in the boat sitting pose on the sea bottom next to UggoV.

    I TP'd into Hay Harbor from world map and after a few seconds in the sky above Anchor Cove region, I landed at the desired position but with

CONTROL_ROT_LEFT and CONTROL_ROT_RIGHT not responding. I rezzed a new boat in Hay Harbor, sat on it and stood up, controls have been released.

    I TP'd UggoV back and sat him on the boat again. Tried to cross into Cats Paw again but the exact same problem happened, boat returned due to

parcel autoretun. UggoV standing at the sea bottom in Hay Harbor (Hay Harbor (49, 6, 3)) and able to walk. Me in the sky again. Viewer title bar

saying Cats Paw and moving fast underground, world map and minimap showing my avatar doing large circles in the sky above Cats Paw, Schooner Run and

going offworld. A TP from world map ([05:49] Second Life: Teleport completed from Cats Paw-13/-73/-110) got me back into Hay Harbor after a few

seconds.
    Called the estate owner who restarted Cats Paw. at 05:54 New boat and new attempt to enter Cats Paw => boat returned due to parcel

autoreturn. Both avatars standing at the sea bottom in Cats Paw. My ROT_LEFT/RIGHT controls still blocked.
    
    Calling estate owner again for restarting Hay Harbor.

    06:04, both regions have been restarted. Rezzing the boat in Hay Harbor again. Crossing from Hay Harbor at the same location was Okay.

Last think, I have encoutered a problem with ghosted prims that are still blocking the way in Blake Sea - Half Hitch and Blake Sea - Windlass, I filled a Jira yesterday about this problem. https://jira.secondlife.com/browse/BUG-4024.