macOS arm64 build for running natively on Apple Silicon Macs

[Erik Kundiman]

Greetings.

Please allow me to continue a topic that was discussed in a previous thread entitled "Viewer on Apple M3, M3 Pro, M3 Max CPUs".

I just thought maybe even at this quick-and-dirty stage of the viewer development, some people might already find the arm64 build useful. I know, some users already have their Apple Silicon Macs performing well for SL through Rosetta 2. And I know, in most (if not all) cases, one wouldn't use this viewer as their main like they do with other viewers out there that are established, as this one is really still vanilla but with less features and more bugs 😜

But just maybe, there's such a situation where your Apple Silicon Mac is not performing too well with any viewer, and that you have no other computer to log in to SL with. Then maybe by having it native, it helps. Or maybe not. I don't know. Cause I can't test this build myself, as I don't have any Apple Silicon Mac. I cross-compiled it from my Intel Mac (hence the prohibition sign shown on the image I attached, meaning I can't run it on my own system).

I'm also assuming that some people are still looking for such a build, and that it's not available yet (I could be wrong, yes). It's just something I conclude from reading other threads in this forum.

With that assumption, the new thing with this build is just that. The arm64 architecture. But, it's still using OpenGL like the SL viewer it is forked from, so it's not using Metal yet, neither directly nor indirectly. And as pointed out by Henri Beauchamp and Gavin Hird (both who have much more experience in viewer developments than I do), and also concluded from a test conducted by a friend (though he's not really an SL player), this build may not give you much (if any) performance boost because of it. But then at least we realise, having it native is still not enough, that we still need to work on using a newer graphics API. And at least you have one more viewer for comparisons or reference, or to dissect, or to borrow the dynamic libraries.. Up to you.

I realise, I don't have any reputation here (I'm an old resident, but new to this forum and new to viewer developments), that you might not trust my build enough. But, you could try to find my name on the Kokua project (only added this year), so Kokua users should already trust me in a way 😉

I started this one as a personal project, after I was finished with Kokua's Linux port big task, so it's only been a few months. I have applied for the viewer to be listed on the TPV directory, but it's still being reviewed. And, I have my macOS builds notarised by Apple (it's an automatic process, not a manual review), so your macOS Gatekeeper should say that the app contains no malware.

Anyway, you can find the build on the website https://megapahit.net

There's also a link there to the source code main repository (while its mirror is one of SL GitHub forks), just in case you still don't trust the binary and want to build it yourself 😄

Thank you 🙂

 

Edited November 27, 2023 by Erik Kundiman

[Ardy Lay]

Does somebody have a lead on an M1 for Erik here?  They have the will, but not the way.  Mac users are in need of someone with the will.

[Gavin Hird]

The viewer does not run on M-series machines. It is does not manage to initialize the application start.

If I understood you correct, you took the Kokua Intel version and cross compiled it, but did not build it in Xcode on a machine with M-series processor?

If that was the way you did it (I may have completely misunderstood), it will never work as the library references that it needs to link with to run on native arm64 will be wrong and it will miss the references to the arm64e libraries needed to initialize and run OpenGL on those machines in addition to a bunch of other system libraries.

You will need to do a full build including linking with arm64 versions of all the libraries needed for the viewer in an Xcode project (xcode_build or full Xcode).

The Dayturn viewer preview is a fork of Kokua (originally from 2014), but there are lots of changes and removals in it for it to run better on macOS. 

[Gavin Hird]

The problem guys, with building a native Mac and iOS viewer, is the renderer.

Since the renderer is very specific to the needs of the dynamic environment in SL, you can't just stick a different (game) renderer into the viewer and think it will work. It will have to be a purpose built renderer for the dynamic content that exist in SL. 

Since Apple in the future only will support Metal, the renderer for a Mac viewer will have to be made for Metal, and the cost of building and maintaining it can only be offset on Mac and potentially iOS (paying) users. We are talking 10-20 man years of effort.

Add to that, many of Apple's libraries for use with Metal only works with Swift, and not a single line of the viewer code is written in Swift. It might be possible to take the C++ code and make the changes to it for it to compile properly as Objective-C++ that also can be used with Metal, but you might have to write bridging code for Metal functionality that is only available via Swift. 

Sure it can be done, but it takes more than two week-ends on the backroom tweaking some code. 

[Gavin Hird]

15 hours ago, Erik Kundiman said:

I cross-compiled it from my Intel Mac

I'd be curious how you did this?

If you look in the folder Megapahit.app/Contents/Resources you have a large number of dylibs that must be linked into the Megapahit binary in Megapahit.app/Contents/MacOS for it even have a hope of working

[Erik Kundiman]

On 11/28/2023 at 4:10 PM, Gavin Hird said:

The viewer does not run on M-series machines. It is does not manage to initialize the application start.

It does, as shown on this video.
https://megapahit.net/downloads/AppleSilicon.mp4
I set the minimum macOS requirement to macOS 12 Monterey.
I'm sorry to hear it doesn't run on your computer. Maybe you tried to run it on macOS 11 Big Sur.. I don't know

 

On 11/28/2023 at 4:10 PM, Gavin Hird said:

If I understood you correct, you took the Kokua Intel version and cross compiled it, but did not build it in Xcode on a machine with M-series processor?

I didn't fork Kokua, I forked the official Second Life Viewer.

 

On 11/28/2023 at 4:10 PM, Gavin Hird said:

You will need to do a full build including linking with arm64 versions of all the libraries needed for the viewer in an Xcode project (xcode_build or full Xcode).

I do a full build including linking with arm64 versions of all the libraries needed for the viewer to run.

 

On 11/28/2023 at 4:10 PM, Gavin Hird said:

The Dayturn viewer preview is a fork of Kokua (originally from 2014), but there are lots of changes and removals in it for it to run better on macOS. 

Yes, I know about the Dayturn viewer, that's why I mentioned how you have much more experience in viewer developments than I do.

 

On 11/28/2023 at 6:56 PM, Gavin Hird said:

I'd be curious how you did this?

Just like you cross-compile an x86-64 viewer from your Apple Silicon Mac.

[Gavin Hird]

2 hours ago, Erik Kundiman said:

It does, as shown on this video.
https://megapahit.net/downloads/AppleSilicon.mp4
I set the minimum macOS requirement to macOS 12 Monterey.
I'm sorry to hear it doesn't run on your computer. Maybe you tried to run it on macOS 11 Big Sur.. I don't know

 

I didn't fork Kokua, I forked the official Second Life Viewer.

 

I do a full build including linking with arm64 versions of all the libraries needed for the viewer to run.

 

Yes, I know about the Dayturn viewer, that's why I mentioned how you have much more experience in viewer developments than I do.

 

Just like you cross-compile an x86-64 viewer from your Apple Silicon Mac.

No, I did not run it on Big Sur? – Do you really think I build the viewer on that? 😉

Sorry about the confusion, but that is not what we call cross compile. It is called (in Apple speak) building for an architecture. 

[Gavin Hird]

2 hours ago, Erik Kundiman said:

I'm sorry to hear it doesn't run on your computer.

I see why it did not run because it use the SecondLife folder in ~Library/Application Support, which is an exceptional bad idea for a viewer. There was tons on log spew and crashdumps where it tried to use SL Viewer Settings files in addition to the SL Viewer caches.

So my initial suggestion to you is to get that sorted, so you don't overwrite SL viewer settings and make sure you have full control of your own including the caches.

[Sprout Evergarden]

I gave it a try it worked but feels very feature incomplete (and i got a message about not having hardware in spec in spite of having a M2 Max.) Lots of warnings about medic codecs and the graphics felt a little fuzzy compared to firestorm but i confirmed it is running native as a ARM application so that was cool

[Gavin Hird]

16 minutes ago, Linnea Evergarden said:

I gave it a try it worked but feels very feature incomplete (and i got a message about not having hardware in spec in spite of having a M2 Max.) Lots of warnings about medic codecs and the graphics felt a little fuzzy compared to firestorm but i confirmed it is running native as a ARM application so that was cool

I also got the messages about not having minimum requirements on an M2, and it set all graphics to minimum.

I suspect it might have to do with using SL viewer settings (I upgraded it to version 7 today) and there is also the settings.xml file in user_settings that might interfere with this viewer.

[SilkMaiden]

Hi Erik!, I'm a big fan of you! 

[Kris Spade]

Thank you for your contributions!  Hopefully this will pave the way for all viewers to have a Apple Silicon build in the future.

[Gaxtrope]

On 11/28/2023 at 9:48 AM, Gavin Hird said:

Since the renderer is very specific to the needs of the dynamic environment in SL, you can't just stick a different (game) renderer into the viewer and think it will work. It will have to be a purpose built renderer for the dynamic content that exist in SL. 

Since Apple in the future only will support Metal, the renderer for a Mac viewer will have to be made for Metal, and the cost of building and maintaining it can only be offset on Mac and potentially iOS (paying) users. We are talking 10-20 man years of effort.

Add to that, many of Apple's libraries for use with Metal only works with Swift, and not a single line of the viewer code is written in Swift. It might be possible to take the C++ code and make the changes to it for it to compile properly as Objective-C++ that also can be used with Metal, but you might have to write bridging code for Metal functionality that is only available via Swift.

So Gavin, do you think there could be a point for us macOS users to take on the following tasks in order?
1) To prepare things, convert the Objective-C++ base to Swift, taking into account that a Swift/C++ bridge must work. Merge this into the SL official GitHub - I have had a few macOS patches accepted recently.
2) Test a method for mapping the SSE optimizations to Neon. There is an sse2neon.h out there, don't know if it's any good. This should be the biggest obstacle in the viewer itself - the rest seems to be easy in XCode.
3) Push for automatic (at first unsupported) builds of Apple Silicon (but for now still OpenGL) Viewer. This will not only require building the viewer itself, but also the various third-party libraries. But the last holdout as I see it, Vivox/SLVoice, now exists in an Apple Silicon version.

Later: Engage with Linden Lab on the effort to move from OpenGL to Vulkan, and in that process use MoltenVK for the macOS viewer. Don't know if their current effort to make Unity-based mobile viewers might supercede/surpass this.

Edited February 1 by Gaxtrope

[Kathrine Jansma]

1 hour ago, Gaxtrope said:

There is an sse2neon.h out there, don't know if it's any good.

It does work and is used for the ARM (Linux) Port of the Cool VL Viewer, so should be okay.

[Gavin Hird]

3 hours ago, Gaxtrope said:

So Gavin, do you think there could be a point for us macOS users to take on the following tasks in order?
1) To prepare things, convert the Objective-C++ base to Swift, taking into account that a Swift/C++ base must work. Merge this into the SL official GitHub - I have had a few macOS patches accepted recently.
2) Test a method for mapping the SSE optimizations to Neon. There is an sse2neon.h out there, don't know if it's any good. This should be the biggest obstacle in the viewer itself - the rest seems to be easy in XCode.
3) Push for automatic (at first unsupported) builds of Apple Silicon (but for now still OpenGL) Viewer. This will not only require building the viewer itself, but also the various third-party libraries. But the last holdout as I see it, Vivox/SLVoice, now exists in an Apple Silicon version.

Later: Engage with Linden Lab on the effort to move from OpenGL to Vulkan, and in that process use MoltenVK for the macOS viewer. Don't know if their current effort to make Unity-based mobile viewers might supercede/surpass this.

I am sorry, no

From my standpoint rewriting in Swift is pointless as it is both a (very) major undertaking, but also makes the codebase for Apple platforms totally incompatible with the Windows, Xbox or whatever other platforms LL wants to pursue. I don't see Apple dropping C++ and ObjectiveC support anytime soon. 

The very first objective is to clean up the C++ code base to the point you can start using Apple libraries everywhere; being getting rid of all the BOOL so the codebase effectively becomes a ObjectiveC++ base (other cleanup needed). Boost also has to go.

The above means you can use a good portion of the C++ code as is, and start replacing functionality with Apple libraries for audio, video, get rid of the monstrosity that is the embedded browser and replace it with WKWebview, use native system threading and scheduling, network stack, preference management, add iCloud support, Apple SIMD, Apple compression libraries etc, etc. 

Separate the OpenGL view to purely render in-world, overlaid by a 2D view for the UI. Move all menus to standard Apple menus, all UI widgets to be based on standard Apple UI widgets (with necessary modifications), essentially stopping to render any UI in OpenGL. 

Finally rewrite the world rendering in Metal. Vulkan is a dead end on Apple platforms. If you want to be in business in Apple platforms, you cannot make your product dependent on a 3rd party for the very core functionality that will and can break at any time. Apple is never going to design for or test that Vulkan will work.

 

I know it is a tall charge, but the reward is a viewer that can run on all Apple platforms with a, depending on how you count, 1-2 billion user base. 

Edited February 1 by Gavin Hird

[Gaxtrope]

Oh sorry, I didn't mean rewriting all of the Viewers C++ code in Swift. I meant rewriting the 8 .mm and .h files in Swift, including bridging code. And then, through sse2neon simply make the code build and run natively on Mx processors. That was the scope of the original post here, as I see it. I wanted to take that - admittedly small scoped - idea, and submit it to the official viewer.

I agree completely on the need to separate the OpenGL to an in-world render so as to make it easier to replace engines, and to offload a lot of work to the more optimized OS. That way, the usual time and resources constraints can determine if there will be a dedicated Metal or MoltenVK or whatever renderer first.

As for Apple never accepting Vulkan - maybe. But they've kept the shamefully ancient OpenGL implementation around for far longer than anyone thought they would. The only way I can see they can nix OpenGL is to make sure small developers who still want to make cross-platform titles have a viable alternative. Metal only runs on Apple devices.
Of course, the announcement of a mobile Unity-based viewer might imply that the open source desktop viewer as we know might be replaced with a closed Unity Viewer 😕

[Gavin Hird]

2 hours ago, Gaxtrope said:

Oh sorry, I didn't mean rewriting all of the Viewers C++ code in Swift. I meant rewriting the 8 .mm and .h files in Swift, including bridging code. And then, through sse2neon simply make the code build and run natively on Mx processors. That was the scope of the original post here, as I see it. I wanted to take that - admittedly small scoped - idea, and submit it to the official viewer.

I agree completely on the need to separate the OpenGL to an in-world render so as to make it easier to replace engines, and to offload a lot of work to the more optimized OS. That way, the usual time and resources constraints can determine if there will be a dedicated Metal or MoltenVK or whatever renderer first.

As for Apple never accepting Vulkan - maybe. But they've kept the shamefully ancient OpenGL implementation around for far longer than anyone thought they would. The only way I can see they can nix OpenGL is to make sure small developers who still want to make cross-platform titles have a viable alternative. Metal only runs on Apple devices.
Of course, the announcement of a mobile Unity-based viewer might imply that the open source desktop viewer as we know might be replaced with a closed Unity Viewer 😕

OpenGL support will be removed once the OS support for the last Intel based Mac Pro is over. I'll give it 1 1/2 more years toppers. 

I really don't see much point in rewriting the ObjC code that exist in the current viewer in Swift.

If you look at my repository I have used bridging code to get rid of autorelease pools (did this many years ago) and a few other changes. I have also converted in the order of 12500 BOOL (still ongoing). That alone makes portions of the code faster as the compiler can optimize on native bools and not on random ints.

Edited February 1 by Gavin Hird

[Henri Beauchamp]

16 minutes ago, Gavin Hird said:

I have also converted in the order of 12500 BOOL (still ongoing).

I claim the first move on BOOL-eradication (almost 10 years already I removed them all from my viewer sources) ! 😛

[Gaxtrope]

Yeah, those BOOL definitions puzzled me too. Why would you represent boolean values as integers???

By the way, @Erik Kundiman - your Megapahit build works here on my M1 🙂

[Gabriele Graves]

2 minutes ago, Gaxtrope said:

Yeah, those BOOL definitions puzzled me too. Why would you represent boolean values as integers???

By the way, @Erik Kundiman - your Megapahit build works here on my M1 🙂

It's because the language at one point didn't have a dedicated bool type and so integers were the next best thing.

[JeromFranzic]

Just here to say thanks for tackling the return of Kokua in Linux. I run that, plus the current Megapahit build in Ubuntu Studio LTS 22.04.3.

 

I do like that Megapahit is forked from the LL viewer... I sometimes mess with the Windows versions under wine, so it's nice to see a project that forks it and runs natively in Linux and MacOS.

[Erik Kundiman]

On 11/30/2023 at 1:52 AM, Linnea Evergarden said:

I gave it a try it worked but feels very feature incomplete (and i got a message about not having hardware in spec in spite of having a M2 Max.) Lots of warnings about medic codecs and the graphics felt a little fuzzy compared to firestorm but i confirmed it is running native as a ARM application so that was cool

 

On 1/28/2024 at 1:23 AM, Kris Spade said:

Thank you for your contributions!  Hopefully this will pave the way for all viewers to have a Apple Silicon build in the future.

 

On 2/6/2024 at 3:32 AM, Gaxtrope said:

By the way, @Erik Kundiman - your Megapahit build works here on my M1 🙂

 

21 hours ago, JeromFranzic said:

Just here to say thanks for tackling the return of Kokua in Linux. I run that, plus the current Megapahit build in Ubuntu Studio LTS 22.04.3.

 

I do like that Megapahit is forked from the LL viewer... I sometimes mess with the Windows versions under wine, so it's nice to see a project that forks it and runs natively in Linux and MacOS.

 

And thank you, all for giving it a try, and for the feedback! I really appreciate it. 😊
Things are getting fixed when there is time.
It helps to know if it works there on system configurations I haven't got access to in order to test them myself.

About Kokua, I'm glad if I could be of any help there too. 😇

About LL viewer, Megapahit isn't planned to stay vanilla on the outside forever, but yeah it's going to take a while until it's starting to not look like the SL viewer. 😃
There are just so many tasks with higher priorities on the list. 🥲

[Gaxtrope]

On 2/5/2024 at 9:35 PM, Gabriele Graves said:

It's because the language at one point didn't have a dedicated bool type and so integers were the next best thing.

Then it just baffles me,
1) Why LL have not seized this obvious optimization opportunity a long time ago themselves. But then again I found Rez files in the repo, which harkens back to the time when we had resource forks in our filesystems, so priorities/resources must be somewhere else.
2) Why none of the many forks that have gone through this arduous process have been successful in having this obvious optimization merged to the main base a long time ago, massively reducing the amount of retro-merging when pulling from the main base.

I think I'll have a go at replacing them once I have the time 🙂

Edited February 8 by Gaxtrope

[Kathrine Jansma]

1 hour ago, Gaxtrope said:

Why none of the many forks that have gone through this arduous process have been successful in having this obvious optimization merged to the main base

Because the main base diverged quite a bit from the typical TPV code base. Not worth the trouble at the time it was not on github i guess.

[Gaxtrope]

55 minutes ago, Kathrine Jansma said:

Because the main base diverged quite a bit from the typical TPV code base. Not worth the trouble at the time it was not on github i guess.

That makes so much sense - thank you 🙂