A New Theory Why The Shopping Cart Causes "Failed Deliveries"

[Darrius Gothly]

Lots of times it has been posted on the Forums that when a customer uses the Shopping Cart, that increases the probability that the delivery will fail. There is a lot of anecdotal evidence to support this claim ... however, it's all wrong. (Please don't start screaming or hit the Reply button until you read this post in its entirety. I promise I'll make sense before it's over.)

The Way It Used To Be

Back in the old days of XStreet, when a customer made a purchase, the website sent a command to a Magic Box instructing it to deliver the product to the customer. Some Merchants had horrible troubles, some had no trouble at all. As a result, a series of recommendations were created that improved the odds of a successful delivery. These included multiple backup boxes, boxes on separate Sims, limiting the number of items in any one box, etc. Over time most people managed to resolve their issues with delivery and it stopped being a "Big Issue".

But the one most important factor of the old XStreet method was ... you NEVER saw the results of anyone else's delivery problems in your records. If someone else was having a problem with their deliveries, they were the only people to ever see it. If you never had any delivery problems, you only saw perfect delivery records. This "only see my stuff" attribute is crucial, so keep that firmly in mind.

How Marketplace Uses The Cart

The SL Marketplace added a Shopping Cart feature. Not only does this allow a customer to purchase more than one item at a time, it allows them to purchase items from more than one Merchant at a time too. Shoppers are now putting up to 10 items at a time in their shopping cart and purchasing them all at once. As a result, most of us are now getting sales orders where the number of items purchased from our store is less than the total number of items purchased on that order ... in other words we are seeing delivery records and problems from our Magic Boxes AND from the Magic Boxes of other Merchants.

This is a very important change because it changes how "far" we can see with each order. Rather than only seeing the results of deliveries from one Merchant .. ourselves .. we are now seeing the results of deliveries for 2, 3 or even 10 Merchants By increasing the number of Merchants included in a single sales order, we also increase the odds that one of them will have problems with their Magic Boxes. If we assume that the average number of Merchants per sales order is five then we are five times more likely to see a failed delivery among them.

It's like Russian Roulette. If you only pull the trigger once, your odds of blowing your brains out are pretty low (1 in 8 or thereabouts). But if you pull the trigger five times, your odds are much higher, on the order of 5 in 8.

What makes it even worse is that because shopping habits have changed and shoppers are putting a LOT of Freebies and Cheapies in their shopping carts, those type of items are getting much higher delivery rates and thus putting even more stress on the Magic Boxes of those Merchants selling a lot of those type items. (My Freebie delivery rate has more than doubled since the introduction of the Shopping Cart, and I'd bet that most everyone is seeing the same increase in their "sales" of Freebies and Cheapies.)

The Theory Goes Like This

There are two parts to the theory. The first part is that because we now have several "Failure Messages" possible, many people are looking at "Delivery Partially Failed" errors and mentally counting that as a Failed Delivery .. even when their stuff didn't fail to deliver. If just one Merchant in the whole sales order had a delivery problem, that is five times as many people thinking "crap another failed delivery". This makes the problem "feel" much more prevalent than it really is.

This also leads to problems for the Merchants that are having bonafide errors with their Magic Boxes. Because the number of items delivered has increased, and because of the general expectation that "the whole thing is borked", those Merchants with actual errors are basically ignoring the errors and not doing anything with their Magic Boxes to fix the problem. This means that the error doesn't get fixed and continues to "contribute failures" to more and more sales orders ... orders that include Merchants that are NOT having any problems at all.

So far this year (2011) I have had two actual delivery errors that caused my items to fail to deliver. However I have 100's of sales orders in which at least one other item in the order failed to deliver. The problem is, I can't see the name of the Merchant with the failure, so I can't contact them and say "Hey, you got a Magic Box problem .. here's how to fix it."

What Linden Lab Should Do

They have a complete database that shows the Merchants that are experiencing actual failed deliveries. A simple database query should be run over the last three months sales records to identify those Merchants with the highest ratio of orders to failures. Those Merchants should then be contacted and given explicit help and instructions to remedy their Magic Box issues. Whether those issues are due to out of date Magic Boxes, or communication errors, or Sim Lag issues .. or whatever other reasons there might be .. LL needs to identify the top 100 Problem Sources and get them fixed.

What I Personally Will Do

If LL is unwilling to devote the employee time to make this happen, I will happily volunteer my time to contact each one and personally hand-hold them through to a successful remedy. I only need the names of those that need some help.

Conclusion

It is my strong belief that if we are able to remedy the top 100 "troublemakers", we will ALL see a drastic reduction in perceived delivery problems. I also believe that a small group of "Authorized Residents" can be assembled into a support group that can receive delivery problem reports and work directly with Merchants to resolve the issues .. and that such an effort will have massive rewards for all.

But this has to start with LL and the Marketplace Dev Team. I know they are "this close" to releasing Direct Delivery, but we honestly won't see the end of Magic Boxes for a long time after DD is released .. just because people hate change and there will be a long period of resistance to DD that is a result of the overall fear that it's just one more "bad thing" to avoid.

It really is time to stop throwing blame around and just "Git 'er done!" I'm ready and willing to invest some time and effort. Is anyone else? Linden Lab, Brooke, Customer Support? Are you folks willing to participate too?

[Josh Susanto]

In principle, I think you are correct that the shopping cart will show people "delivery partially failed", even if all the items personally relevant were delivered.

But as that has also occurred to me, and as I have scrrutinized probably more than 1000 examples by now, this simply has not been the pattern in my case, or, at least not a significant part of the prevailing pattern.

The prevailing pattern in my case is that if the order includes more than one item from my store and it shows "partially failed", then at least one of the items turns out, on further examination, to have failed delivery.

My own theory about why the shopping carts fail (and DO fail) is essentially so that Direct Delivery will be "necessary", but, also, so that a large enough pattern of failure can be produced to mask other problems in which money will be sent somewhere other than to the merchant or back to the customer.

The massive borking that coincided with the mesh release process was probably a test of what can be passed off as simple shopping cart failure during a period where everyone is distracted with some new feature such as mesh.

What to expect next is more than a test; that is: many of the same problems as we began to see at the beginning of August, repeated either during or immediately before the deployment of Direct Delivery... except that there will be some more substantial total money involved.

[Darrius Gothly]

A major weakness in any theory put forward by anyone is the fact that we are lacking real solid data. Dakota recently posted some very specific details about the cause of one of your failures, proving beyond all doubt that LL has very complete and specific data regarding each sale, each delivery ... and most importantly, each failed delivery.

What totally flabbergasts me is that proving any theory is a simple matter of some very basic database analysis and error condition analysis. Given access to the database, I am very confident that I'm one of several thousand SL Residents that could quickly identify the primary causes, design a resolution, and (with a little bit of adminstrative backing in the form of "official" contact to people) implement that solution. As a sideline output from that process would be a document and trouble-shooting flowchart that would allow any new people to quickly find their own resolutions too.

This really is not rocket science .. and even if it was, we've got a lot of folks in SL that make most Rocket Scientists look like cavemen.

So even though I resist strongly the tin foil hat theories you've been expounding lately, and even put forth this theory today that not only has a very logical and rational flow but offers a pretty simple resolution .. I keep coming back to the thought that they are not actually fixing the problems because of some other, less honorable and more nefarious reason.

But then I listen to Rodvik and he has this rather undeniable ring of decency and honesty to him that shines brightly. My built-in BS Detector stays firmly pegged at "This is the straight dope" whenever he talks, so I'm at a loss why we can't just get someone to be open and up-front and just tell us exactly what is going on .. why .. and how to fix it.

@Rodvik .. please? We're not looking for reasons to derail DD or cast foul smelling stuff all over your staff. We're just looking for someone to be decent enough to explain it clearly .. in geek terms even. I promise, we'll "get it".

[Josh Susanto]

Rodvik's speech is the main reason that I didn't just quit.

It's pretty facepalmy, though, that with access to all that data, as you have said, they can't do any better than to tell me that their own synch button isn't fast enough for my customers by telling me "you didn't synch", and completely ignoring the fact that the pertinent system and the accompanying instructions are both set up to create failures which their indicator system is also set up to ignore.

[Marcus Hancroft]

I just had my very first failed delivery on the 23rd.  It was on my best selling item too.  Of course the system showed "Delivery Partially Failed" and when I looked at the actual order, it was supposed to go to another avatar as a gift but did indeed show "Delivery Failed."  I don't know what caused it to fail but since it was a partial fail, I'm assuming from your information, Darrius, that I was not the only Merchant the customer was trying to purchase from.  Is it best to reset the magic boxes once a month, a week, per day, or what do you suggest we do to try to make sure these fails don't start occuring even more often?

[Dartagan Shepherd]

The problem with theories in business is that they don't get anything done, and don't help sell goods.

Your theory is testable. Whip up a crude delivery system, pummel it with various types of requests and see how that goes.

[Chelsea Malibu]

I agree however, you do bring up the issue of Database.  SL has always had terrible DB engineers.  This is why our inventory system has always been an issue and why our search is always borked.  DB is a science and most companies who hire DBA's will have one or two on staff with P.H.D.s. 

If anyone had the chance to see Life 2.0 last night, you could clearly see the direction of thinking that Philip has long held in that we, not them will figure this all out in time.  Basically stating that "we made a platform, you make the best of it."

Oddly though, they do everything in their power to sequester innovation that takes money away from LL.  The best answer I can see for all of us is another site where we can sell our goods like we used to have when we had SL Boutique (built by electric sheep, sold to LL then shut down) and SLExchange (Owned by Anche Chung then bought by SL and borked into the Marketplace we have now).

More and more I am seeing the need to open a competitive market but I know that if I or anyone else does this, LL will do everything in their power to get us shut down.

[Darrius Gothly]

The only way to tell if other Merchants' items were included in your order is to compare two numbers from the order. The first number is called "Number of items in order", the second is called "Number of items from your store". Both of these values can be found in the lower left corner when you inspect the order from the Orders > Transaction history report. If the number of items in the whole order is greater than the number of items from your store, then the order contained items from other Merchants.

As far as resetting on a schedule or the actual cause for the errors .. only LL can tell us that. But I seldom (if ever) reset mine.

[Dartagan Shepherd]

Agree that there may be database issues, couple of gotcha's ... the info isn't out there any longer. I remember when people used to make statements about storage, not realizing large chunks of that were off-base, such as assets not being stored in the database, but as XML flat file, with the database being a glorified pointer system.

In this case, the theory seems to be that the point of failure isn't the Magic Box, or not entirely the Magic Box but the shopping cart. Not sure why we're going there, as scripted delivery systems all have the same points of failure that Dakota was good enough to provide. These are clearly identified.

Have seen these points of failure from writing a delivery system a few years ago, to SL bank software when banks were allowed to current day from a networked vendor system we use (support incidents we have to deal with every week for delivery failure, with a non SL system.).

We need to write another delivery system very soon and I know I'm going to hit those same conditions and points of failure, that Dakota described, those same issues have been with us for years.

Was it you that had software already written for a 3rd party Marketplace? I'd say go for it if you can manage the time, there's a flux here in that many "competitors" are gone and there might be enough of a niche to gain traction. Larger problem is that you can't verify a transaction that a Magic Box won't communicate, regardless of poor database design.

User revenue should be in the hands of users anyway, that's the kind of opportunity that built SL in the first place.

[Darrius Gothly]

Please reread my OP Mr. Shepherd. If you had read it, you would see that I'm not blaming the Shopping Cart .. in fact I'm pretty clearly stating the system isn't broken at all. It's just the perception that it's broken that we are dealing with, and the reason for that perception is because of side-effects of how the Shopping Cart works.

[Chelsea Malibu]

What we actually created was a 3rd party billing system known as an IPSP (Internet Payment Services Provider) in credit card processing terms.  The design was originally so that our guests on Club Jenna (AKA Jenna Jameson's Ecstasy Islands) was to bring new users in with our own registration API to our own orientation island.  Since most had no lindens and lindens purchases are capped we met with Visa to set up a system for purchasing virtual goods in a virtual world called CardBiller.

It had many features though it still required the inventory be boxed in a scripted object however, that object could also be used as your store vendor.  When a user clicked to it, it asked if they wanted to pay with Lindens or $$.  It then added it to a shopping cart.  They could continue to fill this cart for up to 24 hours then make one final purchase. 

We built an API to allow our merchants to open their own online stores if they wanted and it had very granular reporting which I use in part today.

It also had a feature where a merchant could toggle an item for Self Re-Delivery which was very productive as we still had occasional problems due to the same things we see today, bad server responses.

And the last piece which we did for Visa Compliance was to offer a script that could be added to the delivery box.  When it was rezzed for the first time, it notified us if the object had been opened, by who and where.  This was our confirmation piece to protect us from charge backs.

We no long have our merchant account active for this as an IPSP however, we can now tie this into CCBill or other IPSP's but have not done so as we don't see the market for this.  If there is, we would launch it along with all the merchant tools for both direct funds or lindens.

Since it would now be CCBill and not our merchant account, CCBill would send you a check weekly or wire the funds into your account.

I would love to launch this again and my programmer has the time to work on this but we just aren't sure it's worth the risk financially as it would require a significant re-investment by me.

[Dartagan Shepherd]

Right, got it the first time, commenting on the post about database. Babbling in a round about way that the issues are as stated, pretty much on target with the failure rate that we've been told. So in that sense, yes it's always been about perception.

If the point is that perception is finally lining up with the information given, great. Took that at face value from the beginning, so no leap of perception for me. Dakota was pretty conclusive explaining how the shopping cart orders affect delivery issues.

Not seeing where "theory" comes into play, unless you're just catching up with how the whole thing does actually mesh.

[Dartagan Shepherd]

Beautiful, and I remember CCBill from adult days, up until they started charging $700 or so for covering chargeback ratio's. Sounds like a fantastic system.

Not to get too far off topic, but often wondered what would happen if someone were to release a merchant store as a product, meaning that one merchant could set up, host and run their own store. Might be more of a venture there and certainly less overhead, but again ... possibly high risk in terms of time.

[Chelsea Malibu]

Actually that $700 registration fee is only for adult content sites in the US and it's not just CCBill that has to charge this since it's a mandate for high risk covering the US region. SL merchants would not fall into that category.

But on your point, this type of empowerment could be widely successful but the growth we expected to see in SL has not been anywhere near what the analyst predicted. 

Sorry about getting off topic a bit as Darius has started a good thread and I don't want to throw it off.

[Darrius Gothly]

"Theory" comes into play because the details of what causes the majority of failures, and which merchants are experiencing the most failures, is not being used or disclosed. Yet based on Dakota's discussion of and detailed explanation of several other Merchants' failures, the data is there .. it's just not being used.

Now, if you'd "gotten it" from the start, you wouldn't have wandered off into that tangential and totally unrelated bit about XML files and the Asset Database. Instead you would have recognized that data is being stored on every failed transaction .. in intimate detail no less .. and totally ignored. You would have opined about how unless used, data is just useless numbers .. and clucked your tongue at the waste of all that "science", instead allowing the growth and rampant proliferation of endless opinions.

After all, knowing is not the same as doing. So it doesn't really matter what LL knows, until they DO something with it, it's just all wasted effort anyway.

[Dartagan Shepherd]

Well, tried to agree on some of your points more than you think, but again with the pissing contest. Hinted that you could verify batches of problem areas with your own tests. As for notifying merchants, can't say I agree there, just keep getting the job done on DD, that's the important part that I could have agreed with.

Will try the redneck approach next time and see if that's less threatening.

[Josh Susanto]

@Dart

I have tested my theory as having predictive value, and, finding that it does, I have used it to reduce the number of products that don't get sold.

That's pretty useful, really.

[Josh Susanto]

@Darrius

The shopping cart doesn't need to be broken in order to justify getting rid of it at least until DD is working.

The shopping cart has been vaguely explained by Dakota as simply being too fast for the magic boxes, which means they may even work perfectly, other than being incompatible.

But as long as the magic boxes remain necessary, it is fair to consider the shopping cart to be a problem because it's both unnecessary and detracts from the utility of another vital part of the system for which there is currenly no alternative.

[Dartagan Shepherd]

There you go, good for you.

[Darrius Gothly]

---

Dartagan Shepherd wrote:

Well, tried to agree on some of your points more than you think, but again with the pissing contest. Hinted that you could verify batches of problem areas with your own tests. As for notifying merchants, can't say I agree there, just keep getting the job done on DD, that's the important part that I could have agreed with.

Will try the redneck approach next time and see if that's less threatening.

---

Your style of "agreeing" comes off challenging and demeaning. Telling me to go build a delivery system myself is a thinly disguised way of saying "you're so naive, you just don't get it". Then you pick on one word of my title (Theory) and make it out to be a burden to read my post because "here we go again, more lame theories". The whole attitude of your post didn't say "I agree", it said "you are worthless, stupid and annoying". The demeanor of my reply was in line with that attitude.

Direct Delivery is a big project .. agreed. But no way can I believe that every employee of every skill level in LL is 100% tasked because of it. Obviously Dakota knows how to access the delivery faults database .. she gave a report to another merchant using it. So is the development of DD so consuming everyone that understands how to run a SQL query that we need to give them a pass just because they're working on "The Next Big Thing"? Oh c'mon .. gimme a break.

This is an ongoing issue for LL and Marketplace .. failed deliveries. It existed before there was a Marketplace and it continues to exist today. It's not just a technical issue either. It has become a PR issue and a financial issue too.

But it's also not a systemic issue either. Some Merchants have so few failures as to be effectively zero. In fact a lot of Merchants have no problems. But those that do have problems have them bad. But no one in Support is trying to fix those with problems. They just write it off as "it's broken .. but DD will be here any day now!" as if that's supposed to somehow make it okay to let customers continue to suffer problems. It's not.

It's a very good thing that DD is coming along .. I'm just as anxious for it to arrive as anyone. But I'm not happy that they are allowing existing customers to suffer continuing problems when it appears fixing those problems .. or at least fixing them enough to make it tolerable .. is not that tough to do. And if manpower is the reason they don't try, I've tossed my own hat into the ring to help out (at the ridiculously high pay rate of free!)

So .. you see Mr. Shepherd .. I'm as fed up with "theorizing" as you are. Except I'm trying to get someone in LL to point out a direction, give us the tools and info necessary to go there .. and let's get it done. If you actually knew anything about me and my professional career, you'd know that I've never been one to sit around and theorize once a direction is established .. I'm the guy with a wrench in my hand, tightening bolts and making things work again too.

To put it in "Redneck Lingo" ... "We done figgered this thing out. Now hand me the duct tape and let's get to work!"

[Dartagan Shepherd]

On the contrary, giving your skills the benefit of the doubt, I said a "crude" delivery system. A little LSL, a little PHP a  little bit of work and there you go, a Marketplace simulator to test out the same thing the Marketplace experiences and you can reproduce various situations with little effort.

It's less than I would give a programmer as an on the fly test before hiring them to vet a potential employee, so not much more than an excercise there.

Rather have Lindens do the official word and contacting to Merchants though on guidelines, etc. We're just here for peer to peer help and feedback.

Sourcing help here is best for things like mentor programs, etc. Which may be coming back, time will tell on that one.

[Darrius Gothly]

I'm with you on the desire to have LL do the support work. I still don't understand why they've totally abandoned any attempts at fixing the issues. Especially when it's obvious they have sufficient data to be proactive about it, it would make sense to go fix the issues before the Merchants lose gobs of money or start screaming in public about it.

As for building a "simulator" .. I did that while working at MVX. Except it wasn't an offline test system, it was a live production system complete with full data tracking and diagnostics. Siann Beck did the programming to build the site and system (and she's sharp). When I joined up, one of the tasks I took on was handling the support for people that were having issues.

About once a week, I went on customer site to handle a problem. (Which in SL is a whole heck of a lot easier than in RL .. trust me. LOL) One particular problem we encountered had to do with how LSL handles special characters .. specifically the angle brackets. Once I identified the issue, I changed the coding in the website support code to undo what LSL did. It was an "individual" problem that led to a systemic fix.

The deeper point is, I don't need to "test bench" a delivery system to understand what's going on. Even if I did, it would suffer its own particular failures and quirks and wouldn't accurately represent those existing in the current Magic Box system. It would just add more theory and conjecture when what is really needed is action.

It's "boots on the ground" time, not "blue sky" time.