Jump to content

Deploys for the week of 2011-06-27

Oskar Linden

By Oskar Linden,
in Second Life Server

 Share
You are about to reply to a thread that has been inactive for 4670 days.

Please take a moment to consider if this thread is worth bumping.

Recommended Posts

Oskar Linden

Oskar Linden

Posted
Posted
This week we have Kelly's "mono2 performance" branch from LeTigre being promoted to the full grid. This has been a long time coming. We thank you for your patience and for filing bugs when you found them.

Second Life Server (main channel)

  • Script performance and robustness
    This release includes many features that improve the performance and robustness of scripts compiled to Mono. This includes a lot of back end infrastructure work with unit tests and script life cycle to make it easier and safer to work with the script system in the future.
  • New Script Profiling LSL Functions
  • Bug Fixes
    • SVC-3895 Rezzing Mono scripted object cripples sim FPS
    • Improved 'Top Scripts' report
      Each object now reports an average script time per frame since the object was created or the last 30 minutes, whichever is shorter. This fixes a lot of odd behavior and misinformation in the previous results as well as makes it easier to determine recent lag sources instead of only immediate lag sources.
      It is still not expected for this result to exactly match the 'Script ms' reported in the simulator stats summary as these measure different things on different time scales.
    • (Fixed a) Sim crash in mono memory monitor.
    • Includes the fix that caused the zero day rollback.
    • Improved simulator performance when removing objects with scripts from a region.
    • Fixed a bug that could cause scripts to crash with NullReferenceException errors.
    • Fix for "Periods of very low (or nonexistent) script performance"
    • Fix for SCR-55 "script reset"
    • Fix for SCR-76 "llRegionSayTo() silently fails in LSL2 scripts"
    • Fix for SCR-66 "llRegionSayTo() Fails to send messages to attachments on sitting avatars"
    • Fix for SCR-61 "llOwnersay Truncates multibyte characters at the end of it's range"
    • Fix for SVC-6573 "Execution of script is not blocked on parcel for single user if scripts are not blocked for groups on parcel"
    • Fix for regions that would fail to start.
    • Fixed an issue that would create long pauses when avatars logged out or TP'd to other regions.
 
2011-06-28, 5:00am: Rolling Restart - Release Notes: https://wiki.secondlife.com/wiki/Release_Notes/Second_Life_Server/11
 

Second Life RC BlueSteel
This will be the same maint-server branch form last week.
Features:
  • Added new parameter to llHTTPRequest - HTTP_VERBOSE_THROTTLE. Value is '''4''', type is '''integer'''. If specified in the parameters list as a non-zero value (true), the script will emit errors when the throttle for outgoing requests is hit. If zero (false), the script will remain silent on errors. Example: 
    key http_request_id = llHTTPRequest("http://example.com/", [HTTP_VERBOSE_THROTTLE, 0], "");
  • SCR-67: Added new constant PRIM_OMEGA for use with llGetPrimitiveParams, llSetPrimitiveParams, llGetLinkPrimitiveParams, llSetLinkPrimitiveParams, llSetLinkPrimitiveParamsFast. This gives access to the "target omega" property of primitives which can be set via llTargetOmega, but allows getting the current target omega for a prim (previously not supported) and setting the target omega on a child prim (previously only doable with a script on each prim). The list has the signature [vector axis, float spinrate, float gain] like the parameters for llTargetOmegaExamples: 
    // Set this prim rotating
llTargetOmega(<1.0,3.0,0.5>, TWO_PI, 1.0);

// Read back this prim's current target omega
list current_omega = llListToList( llGetPrimitiveParams([PRIM_OMEGA]); // should be [PRIM_OMEGA, axis, spinrate, gain]
vector axis = llList2Vector(current_omega, 1); // Should be <1.0,3.0,0.5>
float spinrate = llList2Float(current_omega, 2); // Should be TWO_PI
float gain = llList2Float(current_omega, 3); // Should be 1.0

// Set all child prims rotating
llSetLinkPrimitiveParamsFast(LINK_ALL_CHILDREN, [PRIM_OMEGA, <1.0,3.0,0.5>, TWO_PI, 1.0]);
Bug Fixes:
  • Fixed issue with "Buy" option remaining set on purchased child prims of no-mod objects (Originally VWR-9796)
  • Addressed issue with teleport and other service failures affecting simulator hosts.
  • Server-side support for STORM-1261 - send message IDs on login failure to allow localization
  • SVC-6905: Fixed issue with objects resisting return, delete, take and autoreturn
  • WEB-3096: Fix unsubscribe link in IM-to-Email messages"
  • SVC-6953: Fixed issue where region restart notification appears to be from the avatar being notified.
  • SVC-5780: More specific error message when deeding land to group fails because transfers are blocked on the region

2011-06-29, 8:00am: Rolling Restart - Release Notes: https://wiki.secondlife.com/wiki/Release_Notes/Second_Life_RC_BlueSteel/11
 

Second Life RC LeTigre
LeTigre and Magnum are both getting an updated version of "mesh-prep". We're calling it "mesh-prep 2". It takes us a step closer to getting mesh on AGNI. We have taken the latest development changes from the mesh development branch and merged them in with this code. Included are infrastructure and framework changes to support the release of mesh. This release does not include any mesh functionality.
  • Havok physics engine updated.
    • 10 cm gap between colliding objects has been removed for some objects
      • For convex prims (with or without physics) and for concave prims (without physics), the 10cm gap has been removed wherever possible.
      • For small prims ( <~0.2m on a side ) it can't be completely removed so those objects may still exhibit a gap, but it will typically be smaller than it was before the change.
    • Complete overhaul of physics shape construction and management
  • Parcel prim accounting code has been overhauled.
  • llCastRay() function included but disabled.
    • Scripts will compile but will not generate results.

2011-06-29, 9:00am: Rolling Restart - Release Notes: https://wiki.secondlife.com/wiki/Release_Notes/Second_Life_RC_LeTigre/11
 

Second Life RC Magnum
This branch is identical to LeTigre. We have put "mesh-prep 2" on both channels to increase the exposure.

2011-06-29, 10:00am: Rolling Restart - Release Notes: https://wiki.secondlife.com/wiki/Release_Notes/Second_Life_RC_Magnum/11
 

We will be monitoring this thread during the next week so please feel free to post issues that you feel have been introduced by the new code. Please file a JIRA for issues you find and post the JIRA link into this thread. It really helps us out. When determining if issues are relevant or not research is key. Tracking down exactly the right situation where an issue is occurring greatly speeds up the development process to get fixes in place.
I appreciate your help. Have a good week!
 
__Oskar
 
p.s. If you are interested in helping test SecondLife in beta please join the group "Second Life Beta" in-world. We also have an email list where we communicate upcoming projects and how you can help. ( https://lists.secondlife.com/cgi-bin/mailman/listinfo/server-beta ) Once a week we meet on ADITI to discuss new features, new bugs, new fixes, and other fun stuff. You are more than welcome. Information is here: https://wiki.secondlife.com/wiki/Server_Beta_User_Group

 

Link to comment
Share on other sites
  • Replies 50
  • Created
  • Last Reply

Top Posters In This Topic

Popular Days

Top Posters In This Topic

Popular Days

Ayesha Askham

Ayesha Askham

Posted
Posted

@Oskar

OK, I suspect you have a problem with the new Main Server software, Oskar.  TD on my sim has dropped from a perfectly respectable 0.95-0.98, to 0.65-0.80 after this roll.  That is not good at all, we are "rubber-banding" all over the place, and vehicles are behaving very badly in sim, never mind on sim crossings (this is an island so that doesn't apply anyway).

We only have about 1200 scripts running, and about 2500 prims out, so we aren't taxing the server much.

You are at 257371.9, 321841.9, 901.8 in Woods of Heaven located at sim2517.agni.lindenlab.com (216.82.19.231:13002)
Second Life Server 11.06.17.233176.

I'd be grateful for an acknowledgement and a suggested solution please.

 

Additional info:

Script time has significantly increased and sim spare time, formerly 12-15ms is now down to 4 or less.  Something in your software is seriously impeding sim/script interaction.  Please remedy this, Oskar.

I suspect only a certain class of server is being impacted by this, but I would expect some sim crashes soon if this is not addressed.

 

I know you've had two crappy weeks in development, but that doesn't mean you can roll this sort of rubbish to the main server!

 

Additional info 2:

I'm going to try a restart anyway, 'cos some homestead servers seem unaffected as yet by whatever script-bug you have introduced.  

An acknowledgement that there was an issue would have been civil, Oskar

 

Update 3:

Tried 2 sim restarts, one was markedly worse, with a TD of 0.5-0.6, second was marginally better with the sim running btween 0.7 and 0.85 TD and spare time between 4 and 8 ms.  Now we have zip response, Oskar.  This is simply NOT GOOD ENOUGH. A lot of LL's income comes from tier on Homesteads.  OK so vehicle movement between mainland sims is better...oh Whoopee, most Homesteads are islands so that isn't exactly useful (no dis to those who DO use vehicles across sim borders), but COME ON, let's see some response, either from you or Maestro over this...or do I need to raise a JIRA??

Looking at what was actually changed in this roll, it looks as though by attempting a fix on SVC-3895, you've borked script -sim interaction to the point that some scripts are now taking up more simulator time than they used to.  This would surely not be "expected behaviour"?  Or is this the thin end of the script-limit wedge on Homestead sims?

 

Update 4:

I notice now that all references to the Main Server roll yesterday has vanished from the Grid Status page...Why on Earth do that?  The conspiracy theorists are going to have a field-day on this.  And you were all doing so well...<sigh>.

Link to comment
Share on other sites
Zena Juran

Zena Juran

Posted
Posted

Afetr 5 sim resets i can not find a stable server.

 

Started at 9552...

8532 borked

8780 borked

8541 borked

9648 borked.

 

These are all Homestead servers.

 

I TP'd to an Open Space sim which was running fine. I TP'd to a diff Homestead @ 2759 and it was running fine.

 

What gives Oscar?

Link to comment
Share on other sites
Blaze Nielsen

Blaze Nielsen

Posted
Posted

serious performance issues occurring with the new Second Life Server 11.06.17.233176. script times dramatically increasing on six sims of mine.

 

Virgins landing should be 8 ms is now 12

Virgins Reef should be at 4.5 - 5.0ms and now is at 7.7

Virgins bay should be at 6-8ms and now is 14.8

Virgins cove should be roughly at 6ms and is now 10.5

Virgins lagoon should be 7ish ms and is at 12

 

for example a bed  script that was running at .050 now runs at .200

can you advise regarding this problem? We monitor script time twice a day, so know what the norm is for each sim.

 

 

Link to comment
Share on other sites
Rian Jayaram

Rian Jayaram

Posted
Posted

Something seems off... i monitor all 10 of my sim (1 full and 9 homesteads) the statistics for FPS and TD are average over a 60 second period... 7 of the homesteads are running worse since the roll..  there are 0 to 2 avatars on each of the homesteads...

 

Rian

Link to comment
Share on other sites
Sassy Romano

Sassy Romano

Posted
Posted

Just to add to the "me too" of seeing much worse TD.  2 Avatars in a homestead TD around 0.7 to 0.9.  It's never great but it's not usually this bad and TP'ing there with my SVC-3895 testing platform (A pair of stupidly scripted Stiletto Moody shoes) and when I landed, TD dropped to rubbish and rubber banding.

This isn't fixed.

Link to comment
Share on other sites
Blaze Nielsen

Blaze Nielsen

Posted
Posted

I beg to differ, I have 6 FULL regions and ALL of them are experiencing severe performance hits. We monitor sim performance twice a day on all six sims, so I know this immediate change after the new server install was a direct result of the new version. This is a SIGNIFICANT performance hit and totally unacceptable.

Link to comment
Share on other sites
Ayesha Askham

Ayesha Askham

Posted
Posted

@Blaze

Latif is partly correct, Blaze.  Full-Region sims do seem better able to cope with this script time penalty that the Roll imposed, but then, some Homesteads also seem to be running OK too!  I think this finally proves that not all full-region servers are current generation, and that some Homestead servers are better than others.  I just wish LL would Pony up and tell us.

The bottom line is still that this is a totally borked upgrade.  I suspect that Kelly Linden's move to introduce Mono2 has caused this, and the deafening silence from LL is a sure sign that:

1)They introduced some of this as a prelude to Homestead and avatar script limits, and

2) They haven't a clue how to fix it.

Silence from LL is nearly always a sign that they know they've screwwed up but don't yet know what is wrong or how to fix it.

Link to comment
Share on other sites
Zena Juran

Zena Juran

Posted
Posted

SL Grid Status Page

We are aware that a recent update has caused the timing of scripts on all homesteads to appear higher than normal. Our devs are investigating this issue. Homesteads should continue to function normally. Please watch this blog for updates.

It's not only higher than normal... it's erratic! The Time Dilation ia bouncing all over the place.

There is no functioning normally under these conditions.

Let's get this taken care of toot sweet!

Link to comment
Share on other sites
Blaze Nielsen

Blaze Nielsen

Posted
Posted

I agree roll back this crappy server! What makes me the most angry is why did they roll it out in the first place with problems like this? I resent being a Beta tester for Linden lab. I got this response to my ticket from Dee:

 

Dee

We are aware that a recent update to Second Life Server regions has caused the timing of scripts on homesteads and full regions to appear higher than normal. Our developers are investigating this issue. Please watch this blog for updates: http://status.secondlifegrid.net/2011/06/29/post1328/

also:  https://jira.secondlife.com/browse/SVC-7064

Please note that the jira itself was created prior to any reports such as yours coming from full prim islands.  I have confirmed with developers that the higher script times are on full islands as well as homesteads.  I even see it on myself with regards to my scripted attachments.  This issue is getting a lot of attention.


Best wishes,

Link to comment
Share on other sites
Oskar Linden

Oskar Linden

Posted
  • Author
Posted

I apologize for the "deafening silence". This became a suspected issue around noon yesterday. It became a known issue a few hours later. Recognizing the issue is the first step. The second step is reproducing the issues in a dev environment. The third step is discovering the root cause for the issue. The fourth step is determining man grid impact. The fifth step is finding a fix. The sixth step is determining if the fix solves the issue. The seventh step is creating a deploy plan. The eight step is implementing the deploy plan. We are doing steps six and seven right now. Multiple members of the team have been working on this full time since it was discovered. That was only about 24 hours ago.

Please accept our apologies for this bug reaching the main grid. We work hard to make sure bugs like this don't get to the main server channel. In this case we failed.

__Oskar

Link to comment
Share on other sites
Sassy Romano

Sassy Romano

Posted
Posted

Oskar Linden wrote:

I apologize for the "deafening silence". This became a suspected issue around noon yesterday. It became a known issue a few hours later. Recognizing the issue is the first step. The second step is reproducing the issues in a dev environment. The third step is discovering the root cause for the issue. The fourth step is determining man grid impact. The fifth step is finding a fix. The sixth step is determining if the fix solves the issue. The seventh step is creating a deploy plan. The eight step is implementing the deploy plan. We are doing steps six and seven right now. Multiple members of the team have been working on this full time since it was discovered. That was only about 24 hours ago.

I very rarely post critical items about LL but I agree with Blaze.  Do you actually have a CAB sit and review any of the planned deployments BEFORE deploying?

If you're CAB reviews your Change Requests, any good change request will have as part of it a roll back plan if it goes wrong.

None of the above is relevant to the production grid anymore that is all for your dev and QA.  

You have identified that it is broken, the immediate remedy is revert the change, as any good CAB would mandate.

This is a massive negative impact on peoples businesses and the pleasure of others.

Link to comment
Share on other sites
Ayesha Askham

Ayesha Askham

Posted
Posted

@ Oskar

24 hours?? I knew this roll was borked at 3pm BST on Tuesday.  Stop giving us corporate crap and DO SOMETHING Oskar, Maestro and Kelly.  You guys created this abomination, You knew there would be impact last week on LeTigre.

This is the worst rubbish to hit the Main Server in 12 months.  SORT IT, I've run out of patience and politeness.

Just look at these posts and who is posting, that tells you all you need to know.

Link to comment
Share on other sites
WolfBaginski Bearsfoot

WolfBaginski Bearsfoot

Posted
Posted

The JIRA system is part of how they manage how they fix problems, so I don't see a problem with that. Some companies, we wouldn't even see that much information.

It does seem strange that, with both pre-release testing and the RC stage, this wasn't spotted until it went out on main channel. But we don't know about the things which don't get that far.

I reckon they're working pretty fast on this, but their communications could be better. Look at the notice on Status.

 

Thinking back, I might have spotted something was wrong last week, but I was in a place where the script-load varies a lot anyway. But I was on the right RC and seeing high script times. 

Link to comment
Share on other sites
You are about to reply to a thread that has been inactive for 4670 days.

Please take a moment to consider if this thread is worth bumping.

Please sign in to comment

You will be able to leave a comment after signing in



Sign In Now
 Share
Go to topic listing
×
×
  • Create New...