Jump to content

Delayed and failed deliveries during the early hours...


Sybil Hallison
 Share

You are about to reply to a thread that has been inactive for 4398 days.

Please take a moment to consider if this thread is worth bumping.

Recommended Posts

  • Replies 74
  • Created
  • Last Reply

Top Posters In This Topic

Well then here is a theory on why sales deliveries are slowest between 1am and 5am every day....

With LL's revenue base receding faster than the bay of fundy's tide, maybe LL Sr. Management instituted major cost saving measures, including turning half their Data Center off along with all the lights in the unused offices and bathrooms.  Or for efficiency sake they got their clustered servers on one of those manual timers and it shuts them off at 1am and turns them back on at 5am.

Its a thought

Link to comment
Share on other sites

i happened to be awake earlier .. during the Twilight Zone time ... and witnessed this set of events:

November 15, 2011 12:33 am PST : Initial Order Placed (from the Reports > Orders report)
November 15, 2011 03:20 am PST : Initial Order Placed (from the Reports > Orders report)
November 15, 2011 05:03 am PST : Initial Order Placed (from the Reports > Orders report)
November 15, 2011 05:23 am PST : Initial Order Placed (from the Reports > Orders report) 

November 15, 2011 12:53 am PST : Magic Box delivered the purchased item (from IM-to-Email)
November 15, 2011 04:30 am PST : Magic Box delivered the purchased item (from IM-to-Email)
November 15, 2011 05:05 am PST : Magic Box delivered the purchased item (from IM-to-Email)
November 15, 2011 05:25 am PST : Magic Box delivered the purchased item (from IM-to-Email)

November 15, 2011 01:04 am PST : Purchase Credited (from SL Transaction History)
November 15, 2011 05:28 am PST : Purchase Credited (from SL Transaction History)
November 15, 2011 05:05 am PST : Purchase Credited (from SL Transaction History)
November 15, 2011 05:25 am PST : Purchase Credited (from SL Transaction History)

To make it easier to see, I'll arrange it like this:

Purchase / Deliver / Paid / Total

12:33 am / 12:53 am / 01:04 am / Total = 31 minutes
03:20 am / 04:30 am / 05:28 am / Total = 128 minutes
05:03 am / 05:05 am / 05:05 am / Total = 2 minutes
05:23 am / 05:25 am / 05:25 am / Total = 2 minutes

Notice how the second transaction took over an hour to deliver and then almost another hour to credit to my account. The backlog began earlier though .. at the first transaction. It was also the first transaction of the new day.

Anyone wanna put money on the cause of the backlog as ... Daily Processing Load?

Link to comment
Share on other sites

I saw a few purchases in the past hour from my store.

The 3 that I saw processed were 1 minute, 22 minutes, 2 minutes respectively.  The 22 minute TX was my one free Demo Pack.  My exepensive items were the 1 and 2 minute TXs.

So not consistent by far but also and more importantly... NOT FAILED.

I was not the one buying the items but I can say that the 1 and 22 minute TXs were from the same customer who first bought my Demo pack and while it took 22 minutes for it to be delivered she bought one of my products from the demo pack which was delivered in 1 minute. :)  So the delay had to be from the system side and not because she logged out.

It would be nice if all transactions were instant from SLM but in my opinion most seasoned SLM Customers have got used to the fact that buying on SLM always runs the risk of a very slow delivery.  What I dont like is the FAILED delivery.

Link to comment
Share on other sites

lol 37 min and still waiting...

guess i was pushing it to think i'd ever get a love heart from TT ;p

______________________________________________________

Order number: 1284235836
Purchased using: L$

*************************

 Item: Heart Love accessory - Overhead Gusher VALENTINE SPECIAL
 Quantity: 1
 Price: L$10

Order total: L$10

Link to comment
Share on other sites

I saw your TX from my offline IM notification :

Date: Tue, 15 Nov 2011 11:15 AM

Sales ConfirmationDear Toysoldier Thor,Order ID: 1284235836Purchased By: Zanara ZenovkaLine Item 1: Heart Love accessory - Overhead Gusher VALENTINE SPECIALDelivered to: Zanara ZenovkaListing URL:
Delivered Quantity: 1 of 1Purchased on: November 15, 2011 11:15 am PSTBuyer's Price (for delivered items): L$10Distributions Total (for delivered items): L$0Marketplace Commission (for delivered items): L$1Your Earnings (for delivered items): L$9Your Total Earnings (for delivered items): L$9

Delivery Confirmation:
Date:
Tue, 15 Nov 2011 11:15 AM


The object 'Xstreet SL Magic Box v3.0.11' has sent you a message from Second Life:Xstreet SL - Delivered item Heart-Overhead-Gusher.
Link to comment
Share on other sites


Darrius Gothly wrote:

 

ETA: Order was placed November 15, 2011 11:57 am PST ... about 24 minutes ago. What the heck is Processing Payment on a FREEBIE?!?

Extortion algorithm. If the routine can't get 5% from the freebie, it eventually gives up and the freebie is delivered in a bruised and battered state.

It's like divide by zero, with baseball bats.

 

 

Link to comment
Share on other sites

I wonder if this has any bearing, found under llHttpRequest API

"Requests made at approx 0625 SLT may fail with a 503 status code, with "ERROR: The requested URL could not be retrieved", and "(111) Connection refused" in the body of the response.  This has been confirmed as expected behaviour by Kelly, due to the nightly maint & log rotation.  It does reliably impact object to object HTTP at that time, and quite probably may impact object to/from web around the same time.  The interruption in service is fairly brief, and the precise timing may vary as LL adjust their nightly maint processes, or due to server load."

 

Link to comment
Share on other sites


Dartagan Shepherd wrote:


Darrius Gothly wrote:

 

ETA: Order was placed November 15, 2011 11:57 am PST ... about 24 minutes ago. What the heck is Processing Payment on a FREEBIE?!?

Extortion algorithm. If the routine can't get 5% from the freebie, it eventually gives up and the freebie is delivered in a bruised and battered state.

It's like divide by zero, with baseball bats. 

 

ROFL!! Two Points!

If your theory is correct, they must need some more effective baseball bats because .. it's still in "Order Status: Processing payment" ... and it's now almost four hours later. Additionally I have not received a Delivery IM from any of my Magic Boxes.

Link to comment
Share on other sites

I know that for any of the products I've created that accessed a web service, I have always had to put the llHTTPRequest on a retry loop .. or yes, you get weird and spurious failures that go away immediately. I'm pretty sure the Magic Boxes do their absolute best to avoid things like the 0625 lockout ... if it actually occurs in the middle of the Magic Box doing something important. Chances are good though that the outage is temporary and brief if noticed at all.

Link to comment
Share on other sites

OK - just awake. I'll keep doing regular purchases overnight. I'll be interested to see how the numbers go.

So far -

Order number / Purchase / Deliver / Paid / Total

Order number: 1284395735   09:10 pm / 09:11 pm  / 09:11 pm  / Total = 1 minute

(Oh and I found the problem with the previous one TT - I had you on mute - must have been having a particularly windy rant on the Commerce Merchants group when I was working or something , lol).

btw Darrius - did you notice the linear pattern in delays if you take midnight as hour zero? ie  0030 purchase = 30 min delay; 0200 = 60 min; 0330 = 120 min - roughly doubling every 90 min. I'll aim to plot this tonight too.

 

Link to comment
Share on other sites

Yep, despite being a global service in multiple languages, LL still thinks in California time, and so there is one guy, probably Geek Linden, managing the entire system, balancing loads etc, plus talking to his gf on the company phone or having cyber sex himself under an alt.

In fact it's probably worse now, judging by the amount of abandoned land and hence revenue loss for LL, they have probably made job cuts and a bot is probably on night duty...

Link to comment
Share on other sites

My data from this morning:


Order number     / Purchase / Deliver / Paid (Conf email) / Total

1284395735    09:10 pm / 09:11 pm  / 09:11 pm  / Total = 1 minute
1284397710    10:04 pm / 10:05 pm  / 10:05 pm  / Total = 1 minute
1284399661    11:04 pm / 11:05 pm  / 11:06 pm  / Total = 2 minutes
1284401622    12:13 am / 12:24 am  / 12:28 am  / Total = 15 minutes
1284403905    01:46 am / 02:24 am  / 02:49 am  / Total = 63 minutes
1284403566    02:24 am / 03:02 am  / 03:28 am  / Total = 64 minutes
1284405308     02:42 am / 03:28 am  / 03:45 am  / Total = 63 minutes
1284407032    03:47 am / 04:44 am  / 04:50 am  / Total = 63 minutes
1284408679    04:43 am / 05:02 am  / 05:15 am  / Total = 32 minutes
1284410120    05:30 am / 05:31 am  / 05:31 am  / Total = 1 minute
1284411290    06:04 am / 06:05 am  / 06:05 am  / Total = 1 minute

Link to comment
Share on other sites

I'm just tickled to see a lag curve exactly like I predicted. Starting at midnight SLT, something is running on the Marketplace that takes ALL its resources and then some. The main website doesn't slow down, so the delivery system is what's lagged. They need to move the midnight chores OFF that machine and put them on one of the empty Sim servers they have .. unless they aren't powerful enough to handle the job. LOL

Link to comment
Share on other sites


Darrius Gothly wrote:

I'm just tickled to see a lag curve exactly like I predicted. Starting at midnight SLT, something is running on the Marketplace that takes ALL its resources and then some. The main website doesn't slow down, so the delivery system is what's lagged. They need to move the midnight chores OFF that machine and put them on one of the empty Sim servers they have .. unless they aren't powerful enough to handle the job. LOL

Seems more regular than curved, as if SLM transactions are being held aside for fixed blocks of time while the SLM database itself does its own chores - ?

Link to comment
Share on other sites

Aaaah!

*lightbulb*

I just remembered something that was nagging at me last night while watching these transactions - i'd notice that when things did happen, they were in 2s or 3s - eg I'd get paid for a sale at the same time that something I bought was delivered, as if the database was suddenly clearing a backlog. When it happened it was remarkable enough that I even asked in the Commerce chat at one point if anyone else just had a whole lot of transactions go through.

Link to comment
Share on other sites

A couple things to note on all these orders to my magicbox all night:

ONE:  With all these orders to my box entering and throughout the period that is supposed to be so bad that as the OP mentioned they get constant delivery failures every night,  Zanara did not get ONE delivery failure to my magicbox the entire time.  Yes there was a clearly noticed lag from ORder start to delivery but they all delivered.  100%

TWO:  An annoying observation which I never really took notice of until Zanara did all these test, LL could not be bothered to put a basic piece of information in the email Delivery Confirmations sent out..... THE ORDER # that was successfully delivered to the customer!  Look at my emailed order delivery confirmation below.  See something missing?  BTW, this is the only place in LL SLM where the delivery details are provided since the SLM transation & order logs do not report the time the order was actually delivered to the customer.

 

The object 'Xstreet SL Magic Box v3.0.11' has sent you a message from Second Life:Xstreet SL - Delivered item Heart-Overhead-Gusher. = Xstreet SL Magic Box v3.0.11 is owned by Toysoldier Thor = http://slurl.com/secondlife/Grojnowski/64/3/91
Link to comment
Share on other sites

You are about to reply to a thread that has been inactive for 4398 days.

Please take a moment to consider if this thread is worth bumping.

Please sign in to comment

You will be able to leave a comment after signing in



Sign In Now
 Share


×
×
  • Create New...