Jump to content
animats

HTTP rate limit

Recommended Posts

Recently, I got script errors indicating a script of mine had hit the HTTP rate limit. The wiki says the limits are "25 requests in 20 seconds for each object, and 1000 in 20 seconds for each owner." But that may change, says the wiki.

All the requests that got to my server are logged and timestamped. I'm not seeing those limits being hit in the log:

mysql> select time, eventtype, msg, auxval, region_name from events where tripid = "d5d33cf9a943dd86ee5c62a87539bdb180d62959" order by serial;
+------------+-------------+---------------------------------------------------------------------+----------+-------------+
| time       | eventtype   | msg                                                                 | auxval   | region_name |
+------------+-------------+---------------------------------------------------------------------+----------+-------------+
| 1556346420 | STARTUP     | animats Resident/Joe Magarac                                        |        0 | Beirne      |
| 1556346420 | DRIVERKEY   | dadec334-539a-4875-ad0e-d9654705f437                                |        0 | Beirne      |
| 1556346421 | SITTER      | on prim #1 :animats Resident distance to seat                       |   1.4103 | Beirne      |
| 1556346421 | SITTER      | on prim #2 :animats Resident distance to seat                       |   1.4257 | Beirne      |
| 1556346421 | RIDERCOUNT  |                                                                     |        2 | Beirne      |
| 1556346421 | PERMS       | Got permissions                                                     |        0 | Beirne      |
| 1556346424 | CROSSSPEED  |                                                                     | 0.992935 | Dunlevie    |
| 1556346424 | CROSSEND    | Region crossing complete in                                         | 0.066895 | Dunlevie    |
| 1556346472 | SLOW        | Slowing for double region cross to                                  |   4.6399 | Dunlevie    |
| 1556346473 | SLOW        | Slowing for double region cross to                                  |  3.83697 | Dunlevie    |
| 1556346473 | SLOW        | Slowing for double region cross to                                  |  3.15458 | Dunlevie    |
| 1556346473 | SLOW        | Slowing for double region cross to                                  |  3.36189 | Dunlevie    |
| 1556346474 | SLOW        | Slowing for double region cross to                                  |  1.97424 | Dunlevie    |
| 1556346479 | CROSSSPEED  |                                                                     |  4.27327 | Eldridge    |
| 1556346479 | CROSSEND    | Region crossing complete in                                         | 0.200195 | Eldridge    |
| 1556346480 | SLOW        | Slowing for double region cross to                                  |  2.93807 | Eldridge    |
| 1556346486 | CROSSSPEED  |                                                                     | 0.334506 | Clyde       |
| 1556346486 | CROSSEND    | Region crossing complete in                                         | 0.266113 | Clyde       |
| 1556346488 | SLOW        | Slowing for double region cross to                                  |  14.1767 | Clyde       |
| 1556346490 | CROSSSPEED  |                                                                     |  13.7874 | Welsh       |
| 1556346490 | CROSSEND    | Region crossing complete in                                         |  0.26709 | Welsh       |
| 1556346521 | CROSSSPEED  |                                                                     |  17.0014 | Rachleff    |
| 1556346521 | CROSSEND    | Region crossing complete in                                         |   0.2229 | Rachleff    |
| 1556346532 | CROSSSPEED  |                                                                     |   11.634 | Kremer      |
| 1556346533 | CROSSEND    | Region crossing complete in                                         | 0.200928 | Kremer      |
| 1556346540 | CROSSSPEED  |                                                                     |   14.845 | Misso       |
| 1556346540 | CROSSEND    | Region crossing complete in                                         | 0.220459 | Misso       |
| 1556346582 | CROSSSPEED  |                                                                     |  17.6306 | Paroo       |
| 1556346582 | CROSSEND    | Region crossing complete in                                         | 0.200928 | Paroo       |
| 1556346595 | CROSSSPEED  |                                                                     |  14.4689 | Bolinas     |
| 1556346595 | CROSSEND    | Region crossing complete in                                         | 0.177979 | Bolinas     |
| 1556346601 | CROSSSPEED  |                                                                     |  9.20502 | Hooper      |
| 1556346601 | CROSSEND    | Region crossing complete in                                         | 0.181641 | Hooper      |
| 1556346617 | CROSSSPEED  |                                                                     |  16.0241 | Bolinas     |
| 1556346617 | CROSSEND    | Region crossing complete in                                         | 0.043213 | Bolinas     |
| 1556346634 | CROSSSPEED  |                                                                     |  13.5913 | Hooper      |
| 1556346634 | CROSSEND    | Region crossing complete in                                         | 0.066895 | Hooper      |
| 1556346636 | CROSSSPEED  |                                                                     |  12.3418 | Bolinas     |
| 1556346636 | CROSSEND    | Region crossing complete in                                         | 0.068115 | Bolinas     |
| 1556346692 | CROSSSPEED  |                                                                     |  9.87885 | Bodega      |
| 1556346692 | CROSSEND    | Region crossing complete in                                         | 0.243652 | Bodega      |
| 1556346703 | CROSSSPEED  |                                                                     |  8.94701 | Bolinas     |
| 1556346703 | CROSSEND    | Region crossing complete in                                         | 0.069092 | Bolinas     |
| 1556346722 | CROSSSPEED  |                                                                     |  10.2868 | Hooper      |
| 1556346722 | CROSSEND    | Region crossing complete in                                         | 0.086426 | Hooper      |
| 1556346741 | CROSSSPEED  |                                                                     |  8.93777 | Montara     |
| 1556346741 | CROSSEND    | Region crossing complete in                                         |  0.20166 | Montara     |
| 1556346785 | CROSSSPEED  |                                                                     |  10.5262 | Phobos      |
| 1556346785 | CROSSEND    | Region crossing complete in                                         | 0.222412 | Phobos      |
| 1556346804 | CROSSSPEED  |                                                                     |  7.10207 | Europa      |
| 1556346804 | CROSSEND    | Region crossing complete in                                         |  0.18042 | Europa      |
| 1556346819 | CROSSSPEED  |                                                                     |  6.43058 | Leda        |
| 1556346820 | CROSSEND    | Region crossing complete in                                         | 0.158203 | Leda        |
| 1556346859 | CROSSSPEED  |                                                                     |  7.73833 | Epimetheus  |
| 1556346860 | CROSSEND    | Region crossing complete in                                         | 0.244141 | Epimetheus  |
| 1556346871 | CROSSSPEED  |                                                                     |  10.4674 | Janus       |
| 1556346872 | CROSSEND    | Region crossing complete in                                         | 0.269287 | Janus       |
| 1556346890 | CROSSSPEED  |                                                                     |  13.6329 | Mimas       |
| 1556346891 | CROSSEND    | Region crossing complete in                                         | 0.221924 | Mimas       |
| 1556346944 | CROSSSPEED  |                                                                     |   9.7282 | Tethys      |
| 1556346944 | CROSSEND    | Region crossing complete in                                         |  0.31543 | Tethys      |
| 1556346964 | CROSSSPEED  |                                                                     |  9.54587 | Arlberg     |
| 1556346964 | CROSSEND    | Region crossing complete in                                         | 0.262207 | Arlberg     |
| 1556347015 | CROSSSPEED  |                                                                     |  8.14114 | Garmisch    |
| 1556347015 | CROSSEND    | Region crossing complete in                                         | 0.266357 | Garmisch    |
| 1556347036 | CROSSSPEED  |                                                                     |  6.93713 | Kitzbuhel   |
| 1556347037 | CROSSEND    | Region crossing complete in                                         | 0.265869 | Kitzbuhel   |
| 1556347054 | SLOW        | Slowing for double region cross to                                  |  9.62612 | Kitzbuhel   |
| 1556347056 | CROSSSPEED  |                                                                     |  3.14528 | Anton       |
| 1556347056 | CROSSEND    | Region crossing complete in                                         | 0.244385 | Anton       |
| 1556347070 | CROSSSPEED  |                                                                     |  4.36055 | Zermatt     |
| 1556347070 | CROSSEND    | Region crossing complete in                                         | 0.199951 | Zermatt     |
| 1556347087 | CROSSSPEED  |                                                                     |   14.729 | Moritz      |
| 1556347087 | CROSSEND    | Region crossing complete in                                         |  0.17749 | Moritz      |
| 1556347097 | CROSSSPEED  |                                                                     |  14.9217 | Wengen      |
| 1556347097 | CROSSEND    | Region crossing complete in                                         |   0.2229 | Wengen      |
| 1556347116 | CROSSSPEED  |                                                                     |  9.02714 | Oberstdorf  |
| 1556347116 | CROSSEND    | Region crossing complete in                                         | 0.249756 | Oberstdorf  |
| 1556347127 | CROSSSPEED  |                                                                     |  8.69436 | Valmorel    |
| 1556347127 | CROSSEND    | Region crossing complete in                                         | 0.266113 | Valmorel    |
| 1556347143 | CROSSSPEED  |                                                                     |  7.91297 | Eagan       |
| 1556347143 | CROSSEND    | Region crossing complete in                                         | 0.269531 | Eagan       |
| 1556347158 | SLOW        | Slowing for double region cross to                                  | 0.487929 | Eagan       |
| 1556347158 | SLOW        | Slowing for double region cross to                                  | 0.493334 | Eagan       |
| 1556347158 | SLOW        | Slowing for double region cross to                                  | 0.493756 | Eagan       |
| 1556347159 | SLOW        | Slowing for double region cross to                                  | 0.494134 | Eagan       |
| 1556347164 | CROSSSPEED  |                                                                     | 0.486514 | Jaffee      |
| 1556347164 | CROSSEND    | Region crossing complete in                                         | 0.241455 | Jaffee      |
| 1556347165 | CROSSSPEED  |                                                                     |  4.68891 | Heaton      |
| 1556347168 | CROSSSLOW   | Waiting for avatar(s) to cross regions.                             |        0 | Heaton      |
| 1556347171 | CROSSSLOW   | Waiting for avatar(s) to cross regions.                             |        0 | Heaton      |
| 1556347174 | CROSSSLOW   | Waiting for avatar(s) to cross regions.                             |        0 | Heaton      |
| 1556347177 | CROSSSLOW   | Waiting for avatar(s) to cross regions.                             |        0 | Heaton      |
| 1556347180 | CROSSSLOW   | Waiting for avatar(s) to cross regions.                             |        0 | Heaton      |
| 1556347183 | CROSSSLOW   | Waiting for avatar(s) to cross regions.                             |        0 | Heaton      |
| 1556347186 | CROSSSLOW   | Waiting for avatar(s) to cross regions.                             |        0 | Heaton      |
| 1556347189 | CROSSSLOW   | Waiting for avatar(s) to cross regions.                             |        0 | Heaton      |
| 1556347192 | CROSSSLOW   | Waiting for avatar(s) to cross regions.                             |        0 | Heaton      |
| 1556347195 | CROSSSLOW   | Waiting for avatar(s) to cross regions.                             |        0 | Heaton      |
| 1556347196 | CROSSFAIL   | Crossing is taking too long. Probably stuck. Try teleporting out.   |  30.0549 | Heaton      |
| 1556347196 | SHUTDOWN    | Distance traveled (km)                                              |  7.33224 | Heaton      |
| 1556347207 | CROSSSPEED  |                                                                     |        0 | Jaffee      |
| 1556347207 | SECONDCROSS | second region cross started before first one completed. Cross time: |  40.5789 | Jaffee      |
+------------+-------------+---------------------------------------------------------------------+----------+-------------+

Got closest to the limit around "Slowing for double region cross".The bike is headed for a region corner and the control system is slamming on the brakes automatically. 13 messages in 20 seconds. Was the limit lowered?

(At the end, the log shows is one type of failed region crossing, where the vehicle makes it across but the avatar does not. Editing the vehicle to move it back to the avatar will often recover the situation.)

(At the beginning, the log shows the avatar sitting on both seats of the bike. The sim is really in that state, due to a previous failed region crossing. Yet another uncaught sim out of sync condition. When a vehicle gets into that state, only a sim restart will recover it. Can't unsit the nonexistent avatar from LSL; I tried that. Deleting the vehicle erases the problem along with the vehicle.)

 

 

Share this post


Link to post
Share on other sites
Posted (edited)

The more you go over the lower the limit gets. You should halt all for at least twenty seconds to clear. If not it will keep adding to the rejection.

Edited by steph Arnott
  • Like 1

Share this post


Link to post
Share on other sites
Posted (edited)
11 minutes ago, steph Arnott said:

The more you go over the lower the limit gets. You should halt all for at least twenty seconds to clear. If not it will keep adding to the rejection.

Hm. I don't think the threshold was ever exceeded. The "time" column is UNIX time, seconds since 1970. There are multiple log entries at a region crossing, but otherwise, nothing.

Edited by animats

Share this post


Link to post
Share on other sites
Just now, animats said:

Hm. I don't think the threshold was ever exceeded. The "time" column is UNIX time, seconds since 1970.

Look LL implemented a safe guard to stop bad people crashing the sims. It is crude but it works.

Share this post


Link to post
Share on other sites

It's a little more complicated than you think, @animats.  Take a look a bit farther down in the Caveats to

"Object requests are throttled at approximately 25 requests per 20 seconds. This is to support a sustained rate of 1 per second or a max burst of up to 25 every 40 seconds (2x the interval for max burst), smaller bursts are recommended."

and then follow the link to Kelly Linden's original note:

llHTTPRequest is throttled in the following manner:
More than 25 requests per 20 seconds across all scripts on an object will trigger the throttle.

The objects will become unthrottled quicker if ALL requests stop, and take a little longer if requests continue.

If a request is throttled, llHTTPRequest will return a NULL key. This means you can know immediately if a request happened or not.

Examples:
* sending 1 request every 1 seconds (or longer) will never get throttled.
* sending bursts of 25 requests (or fewer) every 20 seconds (or longer) will never get throttled.

Share this post


Link to post
Share on other sites
Posted (edited)

I went over the logs and calculated the intervals, and these were the absolute highest amount of requests within 20 seconds:

| 1556346472 | SLOW        | Slowing for double region cross to                                  |   4.6399 | Dunlevie    |
| 1556346473 | SLOW        | Slowing for double region cross to                                  |  3.83697 | Dunlevie    |
| 1556346473 | SLOW        | Slowing for double region cross to                                  |  3.15458 | Dunlevie    |
| 1556346473 | SLOW        | Slowing for double region cross to                                  |  3.36189 | Dunlevie    |
| 1556346474 | SLOW        | Slowing for double region cross to                                  |  1.97424 | Dunlevie    |
| 1556346479 | CROSSSPEED  |                                                                     |  4.27327 | Eldridge    |
| 1556346479 | CROSSEND    | Region crossing complete in                                         | 0.200195 | Eldridge    |
| 1556346480 | SLOW        | Slowing for double region cross to                                  |  2.93807 | Eldridge    |
| 1556346486 | CROSSSPEED  |                                                                     | 0.334506 | Clyde       |
| 1556346486 | CROSSEND    | Region crossing complete in                                         | 0.266113 | Clyde       |
| 1556346488 | SLOW        | Slowing for double region cross to                                  |  14.1767 | Clyde       |
| 1556346490 | CROSSSPEED  |                                                                     |  13.7874 | Welsh       |
| 1556346490 | CROSSEND    | Region crossing complete in                                         |  0.26709 | Welsh       |

The above shows 13 requests within 20 seconds, not nearly enough to trigger the supposed limit of 25.

| 1556347158 | SLOW        | Slowing for double region cross to                                  | 0.487929 | Eagan       |
| 1556347158 | SLOW        | Slowing for double region cross to                                  | 0.493334 | Eagan       |
| 1556347158 | SLOW        | Slowing for double region cross to                                  | 0.493756 | Eagan       |
| 1556347159 | SLOW        | Slowing for double region cross to                                  | 0.494134 | Eagan       |
| 1556347164 | CROSSSPEED  |                                                                     | 0.486514 | Jaffee      |
| 1556347164 | CROSSEND    | Region crossing complete in                                         | 0.241455 | Jaffee      |
| 1556347165 | CROSSSPEED  |                                                                     |  4.68891 | Heaton      |
| 1556347168 | CROSSSLOW   | Waiting for avatar(s) to cross regions.                             |        0 | Heaton      |
| 1556347171 | CROSSSLOW   | Waiting for avatar(s) to cross regions.                             |        0 | Heaton      |
| 1556347174 | CROSSSLOW   | Waiting for avatar(s) to cross regions.                             |        0 | Heaton      |
| 1556347177 | CROSSSLOW   | Waiting for avatar(s) to cross regions.                             |        0 | Heaton      |

And the above is only 11 requests within 20 seconds. Other areas of the log are showing 2-8 requests within 20 seconds.

But this assumes that

  1. All of the requests are pointed at this server
  2. All of the requests arrive successfully
  3. No other scripts are making requests (that would total 1000 requests per 20 seconds)

Edit 1:

I don't know if I understand what is classified as a "burst" though. (This is from the bottom of the caveats..)

Quote

"Object requests are throttled at approximately 25 requests per 20 seconds."

"...a max burst of up to 25 every 40 seconds (2x the interval for max burst)..."

Is anything above one-per-second a "burst" that would lower the throttle limit for any follow-up requests? The wiki is unclear.
Either way, there were lots of times where two requests were logged within the second, sometimes 3 and rarely 4, but never more than that.

Edit 2, I'm leaving the first edit in to bridge the gap:

It says "25 per 40" because of this from earlier:

Quote

Once reached, the throttles will remain in effect as long as requests continue, but will clear if there is a silent period with no requests at least twice the throttle interval, in this case 2 * 20 or 40 seconds.

So as long as you never reach 25 total requests within a 20 second period (even if you send 12 requests in a second, wait 5, and send 12 more and stop), you will not be throttled.

Edited by Wulfie Reanimator

Share this post


Link to post
Share on other sites
8 minutes ago, Wulfie Reanimator said:

 

Bursting is when you accumilate multiple and send it in one. Rather than individually. The script should be timer based so that data never exceeds the penalty limit. In other words do not send every single piece everytime the script recives it. Stack it and then send it when the timer triggers.

  • Like 1

Share this post


Link to post
Share on other sites

FWIW, back before I switched my heavy comms scripts to Experience KVP messaging, I used HTTP a lot and ended up buffering and combining messages to keep the rates far below the documented throttle trigger. Others claimed they only needed to stay within the documented limits but it sure seemed that I could never get anywhere near them or the throttle would kick in.

Share this post


Link to post
Share on other sites
Posted (edited)
27 minutes ago, Qie Niangao said:

FWIW, back before I switched my heavy comms scripts to Experience KVP messaging, I used HTTP a lot and ended up buffering and combining messages to keep the rates far below the documented throttle trigger. Others claimed they only needed to stay within the documented limits but it sure seemed that I could never get anywhere near them or the throttle would kick in.

It is server subjective. That means the server triggers capping by server load. The stated limit is the max alowed.

Edited by steph Arnott

Share this post


Link to post
Share on other sites
2 hours ago, Qie Niangao said:

FWIW, back before I switched my heavy comms scripts to Experience KVP messaging, I used HTTP a lot and ended up buffering and combining messages to keep the rates far below the documented throttle trigger. Others claimed they only needed to stay within the documented limits but it sure seemed that I could never get anywhere near them or the throttle would kick in.

OK, thanks. Also, the posting below indicates that server load may matter. That makes sense; I send more log messages for region crossing problems when having problems. When things are going well, few messages are sent.

Share this post


Link to post
Share on other sites

The documented limits are correct, but as others have pointed out even getting close to them can cause throttling events, and once throttled it will take quite a while for the condition to completely clear (twice the period that triggered the throttle initially, typically).

Yesterday I posted an example script to show how to employ a fuzzy exponential backoff to deal with remote server failures; it would work equally well for throttling.

http://wiki.secondlife.com/wiki/HTTP_Server_URL_Registration

  • Like 1
  • Thanks 2

Share this post


Link to post
Share on other sites
2 hours ago, Oz Linden said:

The documented limits are correct, but as others have pointed out even getting close to them can cause throttling events, and once throttled it will take quite a while for the condition to completely clear (twice the period that triggered the throttle initially, typically).

OK. I was just surprised to see that message for the first time ever, after a year of using that logging system.

Share this post


Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.


×
×
  • Create New...