terrain import/export - issues

[Rachael Orsini]

Hello there,

I was starting to play around with terrains and ran straight into issues, wondering if they are known issues and if there are solutions ...

Since I wanted to setup the whole sim- scene in Blender (latest version) I thought it was a good idea to download the terrain to work on it. I got the primstar-plugin for blender which allows .raw import and importing the .raw works fine, but exporting leads to a complete flat terrain (on the sim, as well when reimported in blender), so something goes wrong here. 

The primstar latest version is rather old so I tried again using an older version of Blender (2.59) and here the export works.... well ... sortof.

When looking closer at the imported terrain on the sim and comparing it to the reference in blender, it appears rather rough, even the parts that are absolutely smooth in blender. But thats not the only issue: the general accuracy appears to be lacking. Some points differ a whole meter of height from the reference in blender. Since I wanted to align offsim-sculpties perfectly to the terrain this is a big problem for me ... 

I wonder if these accuracy issues are bugs or if the fileformat used for export limits it? if the latter is the case then I can sortof bin the whole project ...

Maybe someone could shed some light on this? Or maybe someone knows other tools for working with terrain .raws?

Many thanks!

Edited July 8, 2018 by Rachael Orsini

[OptimoMaximo]

A great tool to work with terrains is The World Machine. The free version allows export for up to 513 square pixels, which is what SL needs.

However, i wonder if you did respect the limitations of number of vertices you can work with for a SL terraina and, moreover, did you sculpt the terrain locking the X and Y brush displacement? Gaia has a couple of tutorials about this plug-in use, anyway. I'd suggest reviewing them if the answer to the previous two questions is "no"  

Edited July 8, 2018 by OptimoMaximo

[Chic Aeon]

Just wanted to note in a "not Blender" part of the equation that the last terrain that I downloaded via the viewer and then TRIED to have someone install at LEA6 -- failed miserably.  I was told that this was happening often (this was six months ago).  So I edited the flat sim by hand which was just fine. 

So there MAY be issues aside from the Blender ones.  You might want to test a regular install of raw file if that is possible. I had done this before and it was "close" if not perfect. There too I had areas pop up as bumps that were not on the land inworld. 

 

 

[Rachael Orsini]

Hello Optimo, thanks for the hint, will take a look at this tool. 

Yes I respected the limitation and only touched the z-axis. I did some further tests now and it seems that the uploaded raw data gets processed in some way, hence the terrain differs somewhat from the reference. Guess thats not a problem in most cases but I hoped i can rely 100% on the reference in blender which is not the case

I created a bit of an extrem example to illustrate:

 

Screenshot of terrain in Blender

The exported raw in PS for checking, looks fine

Screenshot of the terrain in SL .... yuck, look at the ringing!

 

Wonder if I should file a Jira ticket ... uploaded raws should not be cooked (processed)

[Callum Meriman]

To "fix" the rings a little

• Open landscape...
• Without selecting any land at all choose smooth, set the strength to about 1/3 of the way
• Press apply.

But also be aware that with the limitations of landscaping anything more than a few meters terrain cliff will cause huge jaggy problems due to rounding errors and the way terrain actually works. You are never going to get this landscape (as your dropbox screenshot shows) correct.

Edited July 12, 2018 by Callum Meriman

[Rachael Orsini]

Thanks Chip and Callum. I wonder if theres a way to make your own terrain. I'm willing to sacrifice lots of prims for it but. A 64x64m mesh-tile with the same number of vertices as terrain  would eat around 250 prims and I'd need 16 to cover a sim, so that is too much. A 12x12m sculpty would cover the same texture resolution as the terrain, so I'd need ~450 sculpties to cover the sim. Still insane even if I'd happily spend so many prims. Guess I have to try and experiment more .... 

 

[Chic Aeon]

You CAN make reasonably good mesh terrains with low land impact.  I have both "walkable hills" and "landforms" in my store and MP.  Not going to link to the page since this isn't an add, but the secret of course is to KEEP IT SIMPLE so that the PHYSICS can be simple. 

I just looked up the walkable hills (not what you are making but are both 32 and 64 meters square and they are 20 land impact). The waklable plateaus are 15 to 20 land impact up to 64 meters. So you COULD do this. BUT getting those puppies aligned and STAYING aligned will be tricky.

 

 

 

[Callum Meriman]

7 hours ago, Rachael Orsini said:

I wonder if theres a way to make your own terrain. I'm willing to sacrifice lots of prims for it but. A 64x64m mesh-tile with the same number of vertices as terrain  would eat around 250 prims and I'd need 16 to cover a sim, so that is too much.

Generally you won't make it with the same number of vertices, but it will still eat into your prims, especially on a homestead.

As you show in your photo with the huge cliffs, it's the sort of thing I would use mesh landforms for, either my own or cliffs from Skye or Fanatik (and many others) for.

Other tricks are to do it mostly in terrain, using sloped land rather then hard drops and just build the paths in mesh/prim.

[IvyTechEngineer]

Sorry, I am late to the post but is there a Blender addon for editting the SL terrain.raw file? I am using Blender 2.82

Thanks

[ChinRey]

2 hours ago, IvyTechEngineer said:

Sorry, I am late to the post but is there a Blender addon for editting the SL terrain.raw file? I am using Blender 2.82

No there isn't as far as I know. If there is, I'd love to know about it too.

I think the best solution if you want to convert a SL terrain to mesh is to use JVTEK. It can only do 32x32 m patches and makes them in 1/10 size but merging and resizing the meshes is no big deal. Just remember to do it in that order, first merge, then resize.

Alternatively, install Sim-on-a-Stick (SOAS) and L3DT. Both are freeware and may be useful for other things to. Import the RAW file into SOAS, export as a .ter or .r32 file (I recommend .ter since .32 files may end up rotated), import into L3DT and export as mesh (.dae).

I'm afraid getting the terrain back into Second Life is even more cumbersome. What you have to do is create a 32 bit 255x255 (not 256x256!!!) .r32 heightmap in Blender (how to do that is more than enough for a whole post here alone), import to L3DT, export as .ter, import to SOAS and export as .raw.

Please note that the terrain you end up with will not be exactly the same as the one you started with. Converting aan SL terrain to mesh works fine with either of the two methods I described. The JVTEK way is probably the most precise since L3DT will try to simplify the mesh. But I've yet to find a way to convert a mesh to SL .raw terrain with any kind of precision. The method I described is the most reliable one of those I've tried but you still have to expect up to half a meter margin of error.

Edit:

One solution is of course to do it the way Chic suggested, simply upload the mesh and use it as the ground. The problem is that the raw ground mesh of a sim has 131,072 tris and that means more than 5,000 LI. So you have to simplify the mesh without loosing essential details. This is not hard to do for a skilled mesh maker and usually you can reduce it down to 300-400 LI - sometimes even less than 200 - but it's a lot of tedious work since much of the simplification would have to be done manually. You'd also have to make custom textures for the mesh ground of course (oh and also custom normals to hide the edges between the elements but that's not a big job). I once had the idea to offer to do this on comission but even though I have the routines down pat now and can work very efficiently, it's still a week or so of full time work and I can't imagine there'll be many people in SL who can really pay me for all of those working hours.

Edited February 23, 2022 by ChinRey

[IvyTechEngineer]

Thank you for your post but I have found this topic very weird. Last night I couldn't sleep so thought I would download software that another user posted in another thread - Lindal Kidd say "There are a number of terrain generator software packages out there, and some of them are even free. See: https://listoffreeware.com/free-terrain-generator-software-windows/" so I I tried to download them. to include the L3DT software. I think only one would import a RAW file and I just got tired, nothing seemed to work...

I was surprised at some posts that talked about Opensim and Unity. (When so far as downloading Opensim but not much further) Did find this link but couldn't figure out how to get the Blender addon for 2.8.

https://blender.stackexchange.com/questions/53534/how-to-import-a-raw-model-into-blender 

I am a visual learner so I tired YouTube and also found these two videos I want to watch. May not answer my question but 

BLENDER 2.8x - TRUE TERRAIN ADDON! 

How to Create 3D Terrain with Google Maps and Blender!

Also, I believe that I can open it in Photoshop which I have on my School computer...

[ChinRey]

36 minutes ago, IvyTechEngineer said:

https://blender.stackexchange.com/questions/53534/how-to-import-a-raw-model-into-blender

Reading that thread there's no mention of Second Life there. Keep in mind that .raw is not a standardized file format, it's a generic file suffix for several widely different uncompressed graphics formats. The discussion at stackexchange seems to be about Visual Studio's 16 bit raw files which is very different from SL's 104 bit ones and although I don't know, I doubt it can handle both.

36 minutes ago, IvyTechEngineer said:

BLENDER 2.8x - TRUE TERRAIN ADDON!

Oh yes, Blender's terrain add-on has come a long way since that two-bumps-and-a-sloping-field generator 2.6 had and since 2.79 at least it has definitely been one to consider if you want to generate a decent ground mesh from scratch.

But you were asking about exporting an existing lalandscape from SL and that's a completely different matter.

 

36 minutes ago, IvyTechEngineer said:

See: https://listoffreeware.com/free-terrain-generator-software-windows/

That's a great list but it's a bit old so two updates:

• The full version of L3DT is free and has been so for a few years. You need an activation code you receive by email for the full version but that's all.
• Two interesting programs that are too new to be included on the list are Terresculptor and World Machine. Terresculptor is a rather barebones app with limited fucntionality but it can generate some really nice fractal mountain ranges and also has a few other unique tricks up its sleeve. World Machine on the other hand, is an everything-but-the-kitchen-sink style program. It has a rather daunting and poorly documented interface but if you can spend the time needed to figure it out, it can do most anything you want it to.

Edited February 23, 2022 by ChinRey

[IvyTechEngineer]

I was able to convert the RAW terrain file into a jpg file using Photoshop using

 

Is there a way to convert an image into a mesh object? I am still looking for something I can use...

Thanks

[ChinRey]

1 hour ago, IvyTechEngineer said:

Is there a way to convert an image into a mesh object? I am still looking for something I can use...

Yes. L3DT can do that and I assume at least some other reasonably advanced terrain programs can do as well.

But terrain maps are never in colour, they are greyscale, usually 32 bit grey scale but 16 bit or even regular 8 bit works too if you don't need to much details.

SL's proprietary RAW format is sort'a a 16 bit terrain map but the bits for each pixel are split into two channels of a colour image, the ones that determine the big height differences are in the green channel, the ones for the finer details in red channel. What you need to do, is remove the blue channel from the image, then merge the green and red into a single 16 bit grayscale channel. I'm sure Photoshop can do this but I have no idea how. Simply merging the two is not enough since the valus from the two channels does not have the same weight and if I understand the little documentation LL has provided right, the values from the red channel aren't even linear.

Edited February 28, 2022 by ChinRey

[IvyTechEngineer]

22 hours ago, ChinRey said:

Yes. L3DT can do that and I assume at least some other reasonably advanced terrain programs can do as well.

But terrain maps are never in colour, they are greyscale, usually 32 bit grey scale but 16 bit or even regular 8 bit works too if you don't need to much details.

SL's proprietary RAW format is sort'a a 16 bit terrain map but the bits for each pixel are split into two channels of a colour image, the ones that determine the big height differences are in the green channel, the ones for the finer details in red channel. What you need to do, is remove the blue channel from the image, then merge the green and red into a single 16 bit grayscale channel. I'm sure Photoshop can do this but I have no idea how. Simply merging the two is not enough since the valus from the two channels does not have the same weight and if I understand the little documentation LL has provided right, the values from the red channel aren't even linear.

Thanks for your advice. I will explore further. I was thinking that based on some of the videos I was watching that maybe the terrain should be a greyscale image. Do not have admin rights on my school PC but I did download L3DT to my personal laptop. In Photoshop it looks like the RGB levels but the G channel is a constant 46 across the image. only the R channel changes. I have the 

If the G channel is just a constant 46 then it can be ignored and you should only need to scale the mesh in Blender because the multiplier would be a constant? 

Thanks

[ChinRey]

21 hours ago, IvyTechEngineer said:

If the G channel is just a constant 46 then it can be ignored and you should only need to scale the mesh in Blender because the multiplier would be a constant?

I managed to locate the official documentation so I could recheck and it's actually more detailed than I remember it to be. Also I mis-remembered one important part: it's the R channel that is the main heightmap and the G channel the modifier for finer details, not the other way round. (Nominally that is, in reality it turns out the two values have equal significance. Whoever wrote the docs must have missed the elementary school lesson where they taught that the order of the factor doesn't affect the prouct. )

The height (in meters) each pixel represents is: R*G/128

Source: https://wiki.secondlife.com/wiki/Tips_for_Creating_Heightfields_and_Details_on_Terrain_RAW_Files#Notes_for_Creating_Height_Field_Maps_for_Second_Life

On a sidenote, once cupon a time those ten extra channels in the file actually had some purpose. Does anybody have any clue what it might have been? Even considering that the entire SL terrain system was a horribly botched rush job, I can only think of three parameters it would have made even the slightest hint of sense to encode that way and that leaves at least seven unexplained ones.

Edited March 2, 2022 by ChinRey

[Fluffy Sharkfin]

33 minutes ago, ChinRey said:

On a sidenote, once cupon a time those ten extra channels in the file actually had some purpose. Does anybody have any clue what it might have been? Even considering that the entire SL terrain system was a horribly bothced rush job, I can only think of three parameters it would have made even the slightest hint of sense to encode that way and that leaves at least seven unexplained ones.

https://web.archive.org/web/20070117013715/http://secondlife.com/app/help/islands/pi_rawfiles.php

 

[IvyTechEngineer]

7 hours ago, ChinRey said:

I managed to locate the official documentation so I could recheck and it's actually more detailed than I remember it to be. Also I mis-remembered one important part: it's the R channel that is the main heightmap and the G channel the modifier for finer details, not the other way round. (Nominally that is, in reality it turns out the two values have equal significance. Whoever wrote the docs must have missed the elementary school lesson where they taught that the order of the factor doesn't affect the prouct. )

The height (in meters) each pixel represents is: R*G/128

Source: https://wiki.secondlife.com/wiki/Tips_for_Creating_Heightfields_and_Details_on_Terrain_RAW_Files#Notes_for_Creating_Height_Field_Maps_for_Second_Life

On a sidenote, once cupon a time those ten extra channels in the file actually had some purpose. Does anybody have any clue what it might have been? Even considering that the entire SL terrain system was a horribly botched rush job, I can only think of three parameters it would have made even the slightest hint of sense to encode that way and that leaves at least seven unexplained ones.

This almost makes no sense because in my terrain.raw file the G channel is just 46 for all pixels or the scale factor is 0.36 times the R channel value for every pixel in the image. Seems somewhat silly and superfluous since we can easily scale objects in SL. I still am puzzled by the Photoshop file for the R channel. It looks like I have a variation of 0 to 128 or so. This would mean more variation than I am seeing in the BW png file. I have asked a photoshop friend if she could help. So to import a terrain.raw file it is Widthe = 256, Height = 256, Channels Count = 13 and the Interleave needs to be on and the Depth is 8 Bits with a Header Size of 0. OBTW, I cannot increase the Depth to 16. I also attempted to review the ASCII file of the BW png file and it doesn't show much variation. 

[IvyTechEngineer]

https://share.text-image.com/33a0f53bfce4ea70

[IvyTechEngineer]

[ChinRey]

1 hour ago, IvyTechEngineer said:

This almost makes no sense because in my terrain.raw.

You have to really look hard to find anything in SL's terrain system that makes sense. Just look at the old docs Fluffy Sharkfin gave us a link to. Which part of that makes sense to you?

It's a bit strange because nearly everything else in the original software (unlike all the shinies they've bolted onto it later) is very well thought out, making the most out of the possibilities that existed back then. But when it came to the terrain they really, really messed it up.

This is why it's so rare to see elaborate/efficient terraforming in SL. It's a shame because there's so much you can do with terrain to reduce the need for laggy landscaping objects.

It's also the reason why I recommend everybody who want to work seriously on terrain to install SoaS (or some other opensim fork if you, unlike me, can get any of them to work for a local grid). The opensim developers tried their best to emulate SL as closely as possible but when it came to the terrain, they said F it and made their own far superior system, reasonably compatible with proper terrain software and still, amazingly, fully compatible with the SL mess, making an ideal bridge between two completely different worlds.

Edited March 2, 2022 by ChinRey

[IvyTechEngineer]

2 hours ago, ChinRey said:

You have to really look hard to find anything in SL's terrain system that makes sense. Just look at the old docs Fluffy Sharkfin gave us a link to. Which part of that makes sense to you?

It's a bit strange because nearly everything else in the original software (unlike all the shinies they've bolted onto it later) is very well thought out, making the most out of the possibilities that existed back then. But when it came to the terrain they really, really messed it up.

This is why it's so rare to see elaborate/efficient terraforming in SL. It's a shame because there's so much you can do with terrain to reduce the need for laggy landscaping objects.

It's also the reason why I recommend everybody who want to work seriously on terrain to install SoaS (or some other opensim fork if you, unlike me, can get any of them to work for a local grid). The opensim developers tried their best to emulate SL as closely as possible but when it came to the terrain, they said F it and made their own far superior system, reasonably compatible with proper terrain software and still, amazingly, fully compatible with the SL mess, making an ideal bridge between two completely different worlds.

Thanks for the response. I have been messing around and decided to recreate it in Blender so I was able to clear items from above the island and take a picture looking down. Then I imported the image into Blender and used it as a reference to create a terrain I could extrude. This is what I have so far. 

https://www.ivytechengineering.com/SecondLife/files/island4.blend

Watched this YouTube video to build what I have...

 

[Aquila Kytori]

18 hours ago, IvyTechEngineer said:

This is what I have so far. 

https://www.ivytechengineering.com/SecondLife/files/island4.blend

For very large "terrain" type models you need to keep the geometry as low as possible otherwise when rezzed the LI cost will be in the 100's !

The model you shared uses 162 vertices along the outer edge of the island. This is to many !

 

Try redoing it but this time aim for around 60 or so vertices for the outer edge ring.

Example :

On 2/23/2022 at 5:27 PM, IvyTechEngineer said:

I am a visual learner so

................ so no more text :

 

 

 

 

 

 

 

 

 

 

Ready to do a test export and upload to check Download and Physics costs :

 

 

 

 

 

I was "lucky" with this example, when using the high LoD model for Physics the physics cost was well below the Download cost so avoided having to make a separate physics model.

What to do if you need the model to be larger than the 64m max. ? ...................... you will just have to slice it up into dimensions that are permitted and assemble inworld.

 

Edited March 3, 2022 by Aquila Kytori

[IvyTechEngineer]

1 hour ago, Aquila Kytori said:

For very large "terrain" type models you need to keep the geometry as low as possible otherwise when rezzed the LI cost will be in the 100's !

The model you shared uses 162 vertices along the outer edge of the island. This is to many !

 

Try redoing it but this time aim for around 60 or so vertices for the outer edge ring.

Example :

................ so no more text :

 

 

 

 

 

 

 

 

 

 

Ready to do a test export and upload to check Download and Physics costs :

 

 

 

 

 

I was "lucky" with this example, when using the high LoD model for Physics the physics cost was well below the Download cost so avoided having to make a separate physics model.

What to do if you need the model to be larger than the 64m max. ? ...................... you will just have to slice it up into dimensions that are permitted and assemble inworld.

 

Holly cow Batman, that is totally awesome! I know I have too many vertices and some are not really very useful. This was my first attempt at trying to make a mesh model from a terrain. Trying to decipher the terrain.raw file has been a nightmare so I thought I might be able to do something in Blender. Initially, I wanted a model that was a close approximation of what I currently have. Along that line I think I could take some take pictures of the terrain with the reference plane adjusted in 5 or 10 meter changes in elevation. This would give me a series of pictures that could be used to build the mesh. I really appreciate your help and advice. 

[ChinRey]

1 hour ago, IvyTechEngineer said:

I know I have too many vertices and some are not really very useful.

I think I've already said this earlier in the thread but just in case:

• It's not usually make LOD models for meshes larger than - say about 30-40 m along any axis, you use the full model all the way. The reason is that the LOD swap distances for objects that big are so far out the LOD models have little or no effect on the land impact and are so rarely actually displayed (and are low enough in tri count even at full LOD) they aren't going to have any significant effect on anybody's fps. You might as well let people enjoy the full view even on those rare occasions. You will save a few L$ in upload fee if you do it Aquila's way - that is scaled down with simplified LOD models though so you can justify it if those few RL cents matter to you.
• The land impact you get for such "full LOD" models is somewhere between one per 20 and one per 35 tris, depending on the geometry, the sorting order of the vertices and possibly the star constellations at the time you upload. As you see from the stats Aquila's version is right at the lower end of this even though she used simplified LOD models -  a good illustration of the point above.