Technical question about material fetch.

[animats]

Obscure technical question for viewer devs. This one has had me stuck for days. In viewer development, I have to talk to the SL servers, and I'm doing that from all new code, in Rust. So I get different bugs. I'm doing something wrong or incompatible here.

If, inside a connected viewer, I query the "RenderMaterials" capability for a connected region, and do a GET, I get all the materials in that region within the view distance. That bulk request works fine.

The LL viewer and Firestorm can also request individual materials, or a list of them, by UUID, using a POST request. That's not working for me. The sim server accepts the request without reporting an error but always replies with an empty array of materials.

The request format I'm sending, which tries to follow what llmaterialmgr.cpp, around lines 660-700 in the LL viewer, generates, looks like this:

Query:

    <?xml version="1.0" encoding="UTF-8"?>
    <llsd>
    <map>
    <key>Zipped</key>
    <binary>eJyLZmBgYCw9aveN5/L2br8jjtGMKfbzUmIBXkYInQ==</binary>
    </map>
    </llsd>

POST "Content-Type" is application/llsd+xml.

All that, except for the contents of <binary>, appears to be OK. Any variation from that format returns an HTTP error 400. So that's being checked.

The <binary> section is just a list of UUIDs, packed for transmission. First, the UUIDs are accumulated in to an LLSD array. That array is converted to LLSD binary format. Then it's compressed with ZIP compression. That byte string becomes the "binary" section of the XML above, encoded in Base64 encoding. This format is used for both the request and the reply. Reply decoding works fine. I've been doing that for months on the big block of materials obtained by doing a GET with no arguments.

The XML part of this gets checked by the sim server. If I don't have the "Zipped" field, or the content type is wrong, I get an HTTP error 400. However, I can put garbage in the <binary> field, and still get an empty array back. No error status.  So if I botch that part, I get no useful diagnostics, just a correctly formatted but empty list of materials.

The "binary" is Zip-compressed LLSD. That content, which I re-decompress as a check after compressing it, dumps, edited, as:

Array([UUID(••••••••-••••-••••-••••-••••••••••••)])

(UUID has been censored).

The binary is compressed without the LLSD binary header, but with the header the ZIP process adds,, which seems to be what the C++ code does. I've tried putting on the LLSD binary header

<? LLSD/Binary ?>\n

 but it doesn't make a difference. Both ways I get an empty list of materials back, with no errors. It's a valid material in the region; I get that UUID from an object update, a valid one that causes content to be displayed correctly without materials. So it's not a legit no-find situation.

I'm reasonably confident the zipping and unzipping part works, because I do that on the reply side, and it works there. The code used is the standard Rust crate "flate2".

Incidentally, this is on the beta grid (Morris region), not one of the new "gLTF materials" regions.

So, I can do bulk requests and get all the material data for everything present at login/region connection, but if a new object appears while the viewer is connected to the region, I can't get its material data.

I'm doing something subtle wrong, but what? Any ideas?

[Love Zhaoying]

Do you have an option to send the request "not zipped" (possibly "not binary", clear text UUIDs)? If that worked, it would imply something in your (Rust) zip/binary format vs. the one LL is using in their C++ libraries.

I ask this specifically because you wrote that you get a similar response if you send "garbage" in the binary field.

* Not a viewer dev, but I'm a C++ developer.

Edited December 10, 2022 by Love Zhaoying

[animats]

6 hours ago, Love Zhaoying said:

Do you have an option to send the request "not zipped" (possibly "not binary", clear text UUIDs)?

Nope. If the word "Zipped" isn't in that key/value pair, I get an HTTP error 400. The XML part is strictly validated.

But you're right to suspect the zipping process. ZLib, as a format, turns out to have five compression levels and at least four implementations. Picking the right implementation is a known problem in building the viewer. Time to dig into low-level libraries. Thanks.

[animats]

18 minutes ago, animats said:

But you're right to suspect the zipping process.

Good idea, but changing the compression level from default (6) to none (0) and max (9) didn't change the results. The compressed data does change, but the server behaves the same, returning an empty array of materials. Onward.

[Monty Linden]

[Edit:  deleted original.  Thought there was a missing layer of encoding but it appears correct.]

Okay, original answer was almost correct.  The above looks like:  binary_object(base64(zcompact(binary_format(array(uuid))))).  What the service requires is:  binary_object(base64(zcompact(binary_format(array(binary_object(uuid)))))).  I.e. the array must contain binary objects formed by casting the UUIDs as is done in LLMaterialID::asLLSD().  Non-binary (e.g. LLSD UUIDs) are silently ignored.

The documentation is very clear on this.  *cough*

[Love Zhaoying]

3 hours ago, Monty Linden said:

[Edit:  deleted original.  Thought there was a missing layer of encoding but it appears correct.]

Okay, original answer was almost correct.  The above looks like:  binary_object(base64(zcompact(binary_format(array(uuid))))).  What the service requires is:  binary_object(base64(zcompact(binary_format(array(binary_object(uuid)))))).  I.e. the array must contain binary objects formed by casting the UUIDs as is done in LLMaterialID::asLLSD().  Non-binary (e.g. LLSD UUIDs) are silently ignored.

The documentation is very clear on this.  *cough*

Oh, so a typecast issue!

[animats]

Ah. It wants the UUID represented as LLSD::Binary, not LLSD::UUID.

Tried that, and it worked. Thanks.

(Details, for the small number of people interested.)

My previous code, in Rust, was:

let uuids_llsd = LLSDValue::Array(uuids.iter().map(|uuid| LLSDValue::UUID(uuid.clone())).collect()); // convert to LLSD array

(That's Rust. What it means is, take an array of UUIDs and iterate over them. For each uuid, create an LLSDValue::UUID from a clone copy of the original. Then collect up all of those new items into a new array.)

Resulting data, compressed and then de-compressed as a check.

[INFO  libscene::assets::assetload] Material fetch UUID list, re-decompressed: Array([UUID(c53ef60c-d3b7-8b4e-CENSORED)])

That's "(array(uuid))" in LLSD speak. Server doesn't like that.
    
New code. One line change:

let uuids_llsd = LLSDValue::Array(uuids.iter().map(|uuid| LLSDValue::Binary(uuid.as_bytes().to_vec())).collect()); // convert to LLSD array of binary UUID

(That's Rust. What it means is, take the array of uuids and iterate over them. For each uuid, create an LLSDValue::Binary value from the UUID retrieved as a byte array.. Then collect up all of those new items into a new array.)

New data: UUID is an array of bytes, not an LLSD UUID.
   

[INFO  libscene::assets::assetload] Material fetch UUID list, re-decompressed: 
    Array([Binary([197, 62, 246, 12, 211, 183, 139, 78, 196, 65, 91, 1, 100, 63, 158, 100])])

That's (array(binary_object(uuid)) in LLSD speak. which is what the server wants.
   
Now the server no longer treats the request as empty, and replies:

    [INFO  libscene::assets::queuerequestmaterial] Got single material from server:
        MaterialRequest
            { uuid: c53ef60c-d3b7-8b4e-c441-CENSORED } :
        Material asset: 
            UUID: c53ef60c-d3b7-8b4e-c441-CENSORED,
            normals: Some(76de205d-a0d5-1c1e-93cf-CENSORED),
            speculars: Some(59593639-5fb3-6ff6-10ff-CENSORED)

That's a summary of the material; there are other fields not listed. All those data items you enter in the texture tab are in there. New gLTF materials have more layers. (Fortunately, I'm already using a renderer, Rend3, which knows about all the gLTF layers, so upgrading wont' be too bad.)

Off to dinner and a workout. Thanks, all.