Jump to content

Oskar Linden

Retired Linden
 View Profile  See their activity

  • Posts

    461

  • Joined

 Content Type 

Posts posted by Oskar Linden

    Deploys for the week of 2012-02-20

    in Second Life Server

    Posted

    Lydia Craig wrote:

    The restart went smoothly, it did not seal the massive memory leak that is debilitating all viewers, including V 3, indeed V3 seems the worst hit by this thing which is not a reflection so much of a defect in the viewer but a massively overloaded database that is not sending out responses properly and causing massive procedural loops to build up in memory cache and eventually lead to inevitable crashes.  The more immediate issue though is that this update seems to once again have broken the inventory database and loading inventory is now taking hours to accomplish as commands time out and the loading process freezes.  This was not a serious problem for a couple of weeks but is back now worse than ever and along with that huge memory leak which you can see literally growing as you watch has rendered sl almost unplayable.  Here is hoping that eventually LL makes fixing the database a major priority and does not clutter it further with more burdensome operations such as web profiles and desinger names or more mesh.   All of these potentiall can be very nice but not at the expense of the basic integrity of performance and play.  Hopefully, someone will be listening, and if not welcome to the next wild and crazy misadventure which we will likely see rolled out next week on the RC channels in all its glory.

    This is a forum for Seond Life Server. You can see it in the title for the forum: "Seond Life Server". Issues that the viewer has with memory bloating will not be resolved with a server restart. That's not how it works. In this forum I communicate server changes and discuss issues found in the RC channels. If you would like to discuss viewer related bugs please use the appropriate forum.

     

    If you are seeing inventory issues it is not tied to the RC's. I don't know of any new code recently that would modify inventory server behaviour.

    __Oskar

    Deploys for the week of 2012-02-20

    in Second Life Server

    Posted

    Yesterday was a holiday at the Lab. I did not get a chance to update this forum since I was not working.
     
    Second Life Server (main channel)
    There was no promotion of code to main channel this week. This code is the same.
    Certain regions were restarted for health reasons.


    Second Life RC BlueSteel
    This channel has some voice changes. There was some unnecessary info being passed on region crossings that we removed. This should make region crossings easier. This code is also on LeTigre.
     
    2012-02-22, 7-11:00am: Release Notes:https://wiki.secondlife.com/wiki/Release_Notes/Second_Life_RC_BlueSteel/12
     

    Second Life RC LeTigre
    This is the same project as BlueSteel.

     

    2012-02-22, 7-11:00am: Release Notes:https://wiki.secondlife.com/wiki/Release_Notes/Second_Life_RC_LeTigre/12
     

    Second Life RC Magnum
    This is the Threaded Region Crossing Stage 1 Project. We have hopefully fixed all the ghosting issues.

     

     
    We will be monitoring this thread during the next week so please feel free to post issues that you feel have been introduced by the new code. Please file a JIRA for issues you find and post the JIRA link into this thread. It really helps us out. When determining if issues are relevant or not research is key. Tracking down exactly the right situation where an issue is occurring greatly speeds up the development process to get fixes in place.
    I appreciate your help. Have a good week!
     
    __Oskar
     
    p.s. If you are interested in helping test SecondLife in beta please join the group "Second Life Beta" in-world. We also have an email list where we communicate upcoming projects and how you can help. (https://lists.secondlife.com/cgi-bin/mailman/listinfo/server-beta ) Once a week we meet on ADITI to discuss new features, new bugs, new fixes, and other fun stuff. You are more than welcome. Information is here:https://wiki.secondlife.com/wiki/Server_Beta_User_Group

    Deploys for the week of 2012-02-13

    in Second Life Server

    Posted

    We are promoting the code that was on the RC's this week.
     
    Second Life Server (main channel)
    This maint-server has huge improvements to certain LSL functions. We attempted to roll this code out a few weeks ago but noticed a script crashing bug that necessitated a rollback. This version is fixed. This code was on all three RC channels.


    Second Life RC BlueSteel
    This channel has some voice changes. This code is also on LeTigre.

    2012-02-15, 7-11:00am: Release Notes: https://wiki.secondlife.com/wiki/Release_Notes/Second_Life_RC_BlueSteel/12
     

    Second Life RC LeTigre
    This is the same project as BlueSteel.

     

    2012-02-15, 7-11:00am: Release Notes: https://wiki.secondlife.com/wiki/Release_Notes/Second_Life_RC_LeTigre/12
     

    Second Life RC Magnum
    This is the Threaded Region Crossing Stage 1 Project. We have hopefull fixed all the ghosting issues.


     

    We will be monitoring this thread during the next week so please feel free to post issues that you feel have been introduced by the new code. Please file a JIRA for issues you find and post the JIRA link into this thread. It really helps us out. When determining if issues are relevant or not research is key. Tracking down exactly the right situation where an issue is occurring greatly speeds up the development process to get fixes in place.
    I appreciate your help. Have a good week!
     
    __Oskar
     
    p.s. If you are interested in helping test SecondLife in beta please join the group "Second Life Beta" in-world. We also have an email list where we communicate upcoming projects and how you can help. ( https://lists.secondlife.com/cgi-bin/mailman/listinfo/server-beta ) Once a week we meet on ADITI to discuss new features, new bugs, new fixes, and other fun stuff. You are more than welcome. Information is here: https://wiki.secondlife.com/wiki/Server_Beta_User_Group

    Deploys for the week of 2012-02-06

    in Second Life Server

    Posted

    Cincia Singh wrote:

    A larger sample size improves the chance of finding any complications before it goes to the main channel.

    Exactly. This is an opportunity to make certain that we've uncovered as many issues as possible. If anyone finds something that might hold up promotion next week file a JIRA and then link it here.

    Thanks!

    __Oskar

    Deploys for the week of 2012-02-06

    in Second Life Server

    Posted

    There will be no main channel promotion this week. There was a bug in the RC channel code that we determined was not worthy of promoting.
     
    Second Life Server (main channel)
    No promotion this week. Regions might still be restarted during the Tuesday release window.

    Second Life RC BlueSteel
    This maint-server has huge improvements to certain LSL functions. We attempted to roll this code out a few weeks ago but noticed a script crashing bug that necessitated a rollback. This version is fixed. This code is on all three RC channels.

    2012-02-08, 7-11:00am: Release Notes: https://wiki.secondlife.com/wiki/Release_Notes/Second_Life_RC_BlueSteel/12
     

    Second Life RC LeTigre
    This is the same maint-server that is on BlueSteel.

     

    2012-02-08, 7-11:00am: Release Notes: https://wiki.secondlife.com/wiki/Release_Notes/Second_Life_RC_LeTigre/12
     

    Second Life RC Magnum
    This is the same maint-server that is on BlueSteel.


     

    We will be monitoring this thread during the next week so please feel free to post issues that you feel have been introduced by the new code. Please file a JIRA for issues you find and post the JIRA link into this thread. It really helps us out. When determining if issues are relevant or not research is key. Tracking down exactly the right situation where an issue is occurring greatly speeds up the development process to get fixes in place.
    I appreciate your help. Have a good week!
     
    __Oskar
     
    p.s. If you are interested in helping test SecondLife in beta please join the group "Second Life Beta" in-world. We also have an email list where we communicate upcoming projects and how you can help. ( https://lists.secondlife.com/cgi-bin/mailman/listinfo/server-beta ) Once a week we meet on ADITI to discuss new features, new bugs, new fixes, and other fun stuff. You are more than welcome. Information is here: https://wiki.secondlife.com/wiki/Server_Beta_User_Group

    Deploys for the week of 2012-01-30

    in Second Life Server

    Posted

    Ayesha Askham wrote:

    Oskar

    Forgive my obtuseness.  This issue with a new bug...does it mean that the ghosting issue will not now be fixed this week or did I mis-read?

    Also, thankyou once again for your painstaking explanation of this process such that numbskulls like me can understand what has to be done each week.

    I do hope this ghosting issue can be got rid of because it is not simply affecting region-crossers.  If an av that has previously been ghosted on a sim TPs back into that sim, apparently it can never get out again, even if it cleaned up its doppelganger!:smileysurprised:

    The ghosting issue was created by the code on BlueSteel and LeTigre. That code will not be on any region after tomorrow morning. As for the claim that some avatars are permanently stuck I am unaware of that situation. If you find anyone who for certain cannot log in after tomorrow and support cannot help them let me know.

    __Oskar

    Deploys for the week of 2012-01-30

    in Second Life Server

    Posted

    I have updated the notes for tomorrow's deploy. The code on BlueSteel and LeTigre caught a crashing bug during the merge phase. We had a fix in place for the stuck presence issue and verified that it worked. However during our QA phase we recognized a new crashing bug and did not have the time to implement a fix and QA it before release tomorrow morning. We decided to pull this project until we can work out these new issues. The maint-server that was going to only be on Magnum is now going to be deployed to each of the three RC's. 

    If you are curious about our Dev/QA process it would be of interest to you to know that there is a lot that goes on between Friday and Wednesday. Friday morning is when we decide which RC channel we should promote to the main channel ("trunk"). After this any existing RC channels and any new code needs to merge with the code that will be the main channel the next Tuesday. The merge process can take many hours. It is common for there to be new issues that need to be worked out on the fly since you are basically combining two entirely separate code branches. Then you have to hope that it builds properly. After that it requires a deploy to a development grid. Each of these builds then needs a QA verification pass. Time is very short for us in this process. Issues need to be found quickly or there isn't time to fix the found bugs. This is a case where our process worked as expected. We found an issue before release. Sadly we didn't have enough time to get a fix out before release. Sometimes we do.

    I would encourage you to keep the scope of this entire process in mind when critiquing QA. In the same span of time this process is often done in triplicate if there is a busy backlog of RC candidates ready to go. 

    __Oskar

    Deploys for the week of 2012-01-30

    in Second Life Server

    Posted

    WolfBaginski Bearsfoot wrote:

    But you have, in the past, been able to quickly revert to an earlier, safe, version of the code, which bypasses the whole QA element. Why was that choice ruled out?

    Somebody chose to keep the system running with broken code. 

    It only seems quick from your perspective but it takes a minimum of two people working nonstop for multiple hours. It is a labor intensive process. It is also highly disruptive to the grid, commerce, stability, and user perception. We try very hard to stick to the release schedules we publish so people can plan downtime around them. Rollbacks are always highly disruptive. More people are upset at rollbacks that are unnecessary than those who were affected by this issue. We had a support level fix in place for this issue. It didn't escalate to the level of an emergency. 

    __Oskar

    Deploys for the week of 2012-01-30

    in Second Life Server

    Posted

    Ayesha Askham wrote:

    @Oskar

    You know you didn't really address one of the main tenets of Wolf's post; namely, why wasn't the faulty code (which you soon KNEW was faulty), rolled back?

    And you wouldn't want TPVs to keep pace with the LL viewer so why give them the function IDs? *coughs*

    Yes, things are better than they were two or three years ago in some ways, but definitely not in others.  I doubt your words would comfort someone ghosted on Thursday.

    I suppose it is unrealistic to expect LL to have staff working at weekends, after all when would they get their recreation?

    "It's just a game" *coughs again* :smileywink:

    Rollbacks are very costly for a number of reasons. We don't roll back lightly. Every release to the grid is a very complicted process. Rollbacks are reserved for content loss issues or server crashes above a certain threshold. This issue, while frustrating, was easily handled by support. Content was not lost, there were no griefing exploits, and regions weren't crashing. It wasn't an emergency.

    Yes it is unrealistice to expect staff to work on the weekends. However most still do. They don't get paid extra, they are just passionate about making Second Life better. 

    I misread the LSL function ID request when I worded my answer. Even so, there isn't anything I can do other than tell the viewer team and they already know.

    __Oskar

    Deploys for the week of 2012-01-30

    in Second Life Server

    Posted

    WolfBaginski Bearsfoot wrote:

    So you have fixed the ghosting issue. Great.

    But you're not deploying the fix for another 30+ hours. Not Great.

    If you do have an explanation for continuing to run server code which you know is broken, I would like to hear it.

    The best answer would be that you are running the fix through QA processes. That still doesn't explain why, when some regions are inaccessible because of the number of ghosted AVs, you have not reverted them to an earlier, known-safe, version of the code.

     It's Tuesday now, there has been time for a decision to be made at a senior level. There has been some good work done since the problem became obvious of Saturday. But the current situation leaves a distasteful impression that senior management at Linden Labs don't give a **bleep**.

    I do have an explanation. Time. There was a time when fixes like this would take 3-6 months or longer. Ask around about how it used to be. Getting a fix out in a week or less is a massive improvement. I wish everyone could realize that instead of jumping to judgement. Every step in the QA/Dev process takes time. This bug was only introduced on Wednesday. We knew immediately it was an issue. Step 1 was to isolate the issue and determine the cause. This takes time. It takes time away from QA/dev for new projects. Step 2 was isolating the faulty code. Step 3 is finding a fix. Step 4 is implementing the fix. Step 5 is building the code. Step 6 is deploying the code. Step 7 is testing the fix. Each step takes multiple hours. When you overlay that onto our standard west coast work schedule you realize how strapped for time we are to get fixes out so fast. In the best case situation a developer has about a day and a half to find the bug and make a fix. If they don't have a fix by Friday we have to pull their code for the next release cycle. QA doesn't get their hands on the code until Monday at the earliest. This gives us a very small window to attempt to verify the bug and the fix. If the fix doesn't work we have even less time. Realistically, if the code is not complete and bug free by Monday night it won't make the Wednesday morning release window.

    I hope you can understand how much gets done in such a short period of time. I also would appreciate your understanding of the stress most here are under and the passion and diligence they put into creating a better Second Life. I am sorry that your experience hasn't been good.

    __Oskar

     

    Deploys for the week of 2012-01-30

    in Second Life Server

    Posted

    Ansariel Hiller wrote:

    Hmmm well... yeah nice for adding new LSL functions, but could you or anybody else please update the 
    so we can properly include them for the LSL syntax highlighting at least in TPVs?

    I agree and the viewer team is aware of the issue. Since they operate on a different release schedule from the server team there can sometimes be a disconnect. We've been investigating other options for keeping the function list up to date. As for TPV's we have no control over syntax highlighting in their code.

    __Oskar

    Deploys for the week of 2012-01-30

    in Second Life Server

    Posted

    The code from Magnum is getting promoted this week.  llSetRegionPos() is be enabled grid wide! The region crossing code on BlueSteel and LeTigre last week had the presence issue as you were well aware of. We fixed that issue, but unearthed a serious crashing bug during our QA phase. We will not be releasing it tomorrow and will be putting a maint-server on all three RC's.
     
    Second Life Server (main channel)
    • Features
    • New LSL function integer llSetRegionPos(vector position)
      The object with the script will move the root prim position to the given location. The position is any position within the region. If the position is below ground, it will be set to the ground level at that X,Y spot. The function has no delay or throttle.
      • Returns 1 if the object is successfully placed within 0.1 m of position.
      • Returns 0 and does not move the object if position is more than 10m off region or above 4096m.
      • Returns 0 and does not move the object if the object is dynamic (has physics enabled).
      • Returns 0 and does not move the object if the object can not move to position due to object entry rules, prim limits, bans, etc.
    • "frame_number" option added to llGetEnv()
      Returns an integer that represents the current 'frame' of the simulator. Generally only useful for specific debugging cases.
    • Bug Fixes
    • SVC-7466 A notecard holds more data than a script can read
    • SVC-7520 Keyframe motion doesn't move towards the correct position when specified time is not an exact multiple of 1/45 seconds
    • SVC-7485 llSetKeyframedMotion cannot stop animation if none is running ... sounds less important than it is ...
    • SVC-7493 Weird mesh land impact issue
    • Fixed several simulator crash bugs and potential memory leaks.
    • Fixed a notecard crashing bug.
    • SVC-7613 Greatly increased network activity since the 1/18/12 RC channel rolling restart
    • SVC-7608 Sims are not visible when diagonally opposed. (this was not a simulator-side fix, but the bug was originally visible only in the previous iteration of this project)

    Second Life RC BlueSteel
    This maint-server has huge improvements to certain LSL functions. We attempted to roll this code out a few weeks ago but noticed a script crashing bug that necessitated a rollback. This version is fixed. This code is on all three RC channels.

    2012-02-01, 7-11:00am: Release Notes: https://wiki.secondlife.com/wiki/Release_Notes/Second_Life_RC_BlueSteel/12
     

    Second Life RC LeTigre
    This is the same maint-server that is on BlueSteel.

     

    2012-02-01, 7-11:00am: Release Notes: https://wiki.secondlife.com/wiki/Release_Notes/Second_Life_RC_LeTigre/12
     

    Second Life RC Magnum
    This is the same maint-server that is on BlueSteel.


     

    We will be monitoring this thread during the next week so please feel free to post issues that you feel have been introduced by the new code. Please file a JIRA for issues you find and post the JIRA link into this thread. It really helps us out. When determining if issues are relevant or not research is key. Tracking down exactly the right situation where an issue is occurring greatly speeds up the development process to get fixes in place.
    I appreciate your help. Have a good week!
     
    __Oskar
     
    p.s. If you are interested in helping test SecondLife in beta please join the group "Second Life Beta" in-world. We also have an email list where we communicate upcoming projects and how you can help. ( https://lists.secondlife.com/cgi-bin/mailman/listinfo/server-beta ) Once a week we meet on ADITI to discuss new features, new bugs, new fixes, and other fun stuff. You are more than welcome. Information is here: https://wiki.secondlife.com/wiki/Server_Beta_User_Group

    Deploys for the week of 2012-01-23

    in Second Life Server

    Posted

    Cerise Sorbet wrote:

     seems to have started in anger at the same time the threaded crossing code went to RC. Some resident closed it out as "contact support" .... but this is a resurgence of the old "system is loggin you out, come back in 5 minutes" forever loop and really really looks like a new bug.

    It is a new bug. If you contact support they will fix your avatar state for you until we get the server patched. That is why it was marked in that state.

    __Oskar

    Deploys for the week of 2012-01-23

    in Second Life Server

    Posted

    Dora Gustafson wrote:

    We had llSetRegionPos() working on
    RC Le Tigre
    during the last week but not anymore.

    It looks like it is rolled back:smileysad:

    This code is still in RC. It is just not on LeTigre this week. This code is only in the Magnum RC channels. Once it gets promoted it will work in all regions again.

    __Oskar

    Deploys for the week of 2012-01-23

    in Second Life Server

    Posted

    Chrissy Ambrose wrote:

    Hi a question on this
    "Sims are not visible when diagonally opposed" from somone who knows not a great deal about technical issues.

     3 sims in a group in an L shape, 2 on magnum Rc servers and one on a sl server version. One sim is removed (the middle one), leaving 2 in the group, but diagonally opposed no longer can see one another. (left is one on magnum and one sl server version) . Is this a natural occurance, or should diagonally opposed sims still be visible as before.

     

    Diagonnaly adjecent regions can only see each other if there is a directly adjacent simulator. That is by design. This is regardless of server version. It has to do with how simulators talk to each other. 

    __Oskar

    Deploys for the week of 2012-01-23

    in Second Life Server

    Posted

    Alexi Raynier wrote:

    Oskar is the Blue Steel servers have been fixed check again, as of a few minutes ago they were all but unusable with the worst lag and unexpected crashes I have seen yet in sl.  Moreover they have been almost this bad for most of the last five weeks with a brief respite two weeks ago when they were apparently fixed, only to be quickly destroyed by more improvements.  The people that are paying tier for land on these servers deserve better at this point, and since both my home and the place where I make my living are on blue steel servers I have had very little income for the last several weeks and spent most of my time elsewhere or recovering from crashs and lag when I am there.  This is hardly the service I and the other paying customers deserve, and if it does not improve I see no reason to continue to pay for otherwise useless land in these area or to pay for services I am not hand have not been getting.  This is a serious problem and, if LL wishes to maintain creditability needs not only to be addressed but fixed and fixed so that it does not happen again. 

    The BlueSteel regions have had different code on them every week for the last 5 weeks. If there is an issue there it wouldn't be unique to the BlueSteel regions and would exhibit the same behaviour in the main channel. When it comes to tracking down bugs in an RC channel what I need are bugs that are new and specific only to that RC. Those are the ones that we are trying to 'nip in the bud'. If you are noticing regular trends in decreased performance you can file a regular jira. That is an issue within itself. As far as the BlueSteel regions go nothing is different for them except they have the chance to get the code the main channel gets one week early.

    __Oskar

    Deploys for the week of 2012-01-23

    in Second Life Server

    Posted

    Ayesha Askham wrote:

    @Dilbert & Tymus

    Oh Lor that sounds very much as though the dreaded "Timewarp" is back.  I had thought LL had exorcised that before the end of 2011.

    In my limited understanding of things I had thought that the OS upgrades to the newest Linux had been the cure.  Seems not.

    The TIMEWARP is 100% for certain NOT back. That doesn't mean that a similar bug does not exist. The bug known as the TIMEWARP has been eradicated. Since it was a huge issue for so long people have latched onto it and claim that it is still around. It isn't. :-)

    __Oskar

    Deploys for the week of 2012-01-23

    in Second Life Server

    Posted

    Ayesha Askham wrote:
    So we had issues on LeTigre...were they fixed...were we told?
     
    PS Oskar
     
    PLEASE get someone at your end to fix this forum for IE users, it is a Right Royal Pain in the Ass.  On one occasion I managed to post without even logging in...this is not my browser.  The Fault is at your (LL) end.
     

    The issues on RC have been fixed. The code that was on all channels last week will only be on Magnum this week. 

    As for the forums I have no control over them. I'm sorry it's annoying. I would recommend having another browser handy until it does get fixed. I usually run Safari, Chrome, and FireFox at the same time because of similar reasons. One of the three will view what the other won't.

    __Oskar

×
×
  • Create New...