Testing report – Case ID: 102254267403 – 25/4/2024
PCIe Card Model: SON-FUS-SSD-4X4-E3 Sonnet M.2 4x4 PCIe 3.0 x16 Card Silent Version
NVMe Drives: 4 x Samsing 2TB V-NAND SSD 970 EVO PlUS NVME M.2
Background and issue summary:
Issue present across multiple OS updates from late 2024.Background: An Issue arose late 2023, we now assume after an OSX update, where drives on this particular PCIe card would not initialise, appear in finder or disk utility, usually after a restart.
On a small number of occasions, the drives have disappeared one by one over around 5 seconds when the mac and OS were running, but this is unusual. Potentially indexing or time machine was running when this happened, but I can’t be sure and I can’t reproduce that behaviour reliably.
A reliable step to reproduce is a restart. Usually after one or two restarts, the drives will not be present when MacOS starts.
Sonnet technical support worked with me and early this year, said that they had replicated the issue and it was a problem that was introduced with a MacOS update in November 2023.
Sonnet confirmed that apple were aware and the fix would come in the form of a MacOS update. They also confirmed that this issue did not just affect Sonnet cards, but also competitors equivalents. I have replicated this issue with an OWC card so I can confirm.
Sonnet were expecting a fix early 2024, but as of the time of writing this, no fix has been offered.
I’m engaging with apple support to confirm that this is indeed the issue, and to try to ascertain what can be done to remedy it if a MacOS fix is not forthcoming.
Troubleshooting:
- Multiple OS updates tested from November 2023 when issue first occurred, up to 14.5 public beta.
Problem persists.
- New PCIe card purchased and installed.
I believed this would solved the problem as we thought it to be a bridging chipset incompatibility with MacOS since November 2023.
Problem persists.
Note - now the drives and card have been changed over to the OWC card, for the rest of the troubleshooting after this point, the 4 drives listed above are installed on an OWC Accelsior 4M2 PCIe card since the issue persists and behaviour is exactly the same.
OWC states in a text chat that there was no issue with this card on the 2019 mac pro running Sonoma.
I also confirmed with another user that his similar OWC card, the OWC Accelsior 8M2 PCIe 4.0 NVME M.2 SSD adapter card with M.2 NVME SSD was working with no problems on his 2019 Mac Pro on Sonoma 14.4
I’d also confirmed that the card I was testing had the same bridge chipset as the user who was experiencing no issues
Unfortunately, the problem persisted, exhibiting the exact same behaviour.
After I reported this to OWC, they said that they were, after all, aware of an issue and, as Sonnet had stated, it was an Apple MacOS issue that needed a fix via a software update.•
Problem persists.
- Moved card to an 8 lane PCIe slot (from the 16 lane one)
Problem persists.
- Create partition and perform fresh MacOS installation.
1. Catalina 10.15.7
Performed multiple restarts and power cycles.
Drives behave as expected, problem is NOT present.
2. Sonoma Update performed, test partition is now MacOS 14.4.1
Mac shut down for the night.
First start up from power off:
Drives initialise and show as expected.
Restart mac, first restart:
Drives did not initialise or show.
Full shutdown of mac – wait for 30 seconds – Power up
Drives initialise and appear in finder and Disk Utility
Restart
Drives do not initialise / appear
2nd Restart
Drives DO initialise / appear
3rd Restart
Drives DO NOT initialise / appear
Other Behaviour
When the drives do mount and are working, intermittently, while using the drives to stream music sample library data by a DAW, the machine has frozen, showing a sudden CPU overload on every core.
Sometimes this is accompanied by corrupt colourful graphics on all monitors, looking like a GPU fail, then shutdown and restart with serious error / panic report. I believe this to be linked to the issue, perhaps a wider PCIe handling problem, but this is only anecdotal at this stage, I have no testing or evidence to robustly link this problem with the one discussed above.
Conclusion
It is a safe assumption from behaviour and testing that the issue affects both the Sonnet and the OWC card in the same way, therefore, troubleshooting carried out on the clean test partition with fresh installs of MacOS 10.15.7 And MacOS 14.4.1 is assumed to produce the same results with both cards, even though only the OWC was tested.
The results of the tests support information from Sonnet engineers, that the issue arose with a MacOS update around November 2023 which affects multiple NVMe PCIe cards across multiple manufacturers.
The early assumption that the problem concerned a specific bridge chipset on some of these cards appears to be wrong, since the two cards tested here have different chipsets.
Tests conclude that the drives and card work as expected with no issues on a new MacOS 10.15.7 installation but fail to work properly and reliably when an update to MacOS 14.4.1 is performed.
The test partition installs both had no additional software or driver installs.
Update from apple since this report was submitted
Apple conducted thorough troubleshooting and data collection with me including data captures, screen shots and this report. They subbmitted it to their engineering team who, in turn said they had to exculate it above their level. I'm now told that it is a case of waiting and there is no time line.
This machine represents a significant business investment and the ability to run fast storage on a 16 lane PCIe IO was a major factor in the purchasing decision.
The fact that this functionality is now broken indefinitely is, in my view unacceptable. Apple sell expensive machines and don't support functions that people but these machines to perform.
I've never heard of a PC manufactorer talking about the difficulty of using third party PCIe cards when issues like this occure. They've made and sold a machine with PCIe slots - it should work and continue to work.
I've spoken to the store manager of the apple store this machine was purchased from. His stance seemed indifferent and unsympathetic to me. He mentioned that the machine had worked well for a few years as if that was something I should be happy about. I wasn't sure what he meant by this, maybe that I shouldn't expect these machines to last more than 3 years?
This is not an acceptable or appropriate attitude in my opinion, but it is worrying that this seems to confirm Apple's lack of interest in solving this problem. I hope I'm proved wrong, but it's understandable if I put myself in their position that they don't like supporting the Intel Mac Pro.
The conspiracist in me hypothesises that Apple may even be delaying fixes like this on purpose to force us to upgrade machines. If anything it will push me to the dark side (PC). That's an alarming thought.
Apple: if by some small chance you're reading this, please reach out and help! This issue has caused delays and hardship, and loss of opportunity.
Update from apple engineering after going through the above and collecting data captures.
Engineering have found evidence of a drive issue in the logging
The drive is going into controller fatal state which in turn is causing the drive to be terminated
Apple engineering referring customer back to the third party manufacturer
The issue is with the drive itself, not Mac OS or the hardware
I have replied asking the following:
- My testing and evidence shows that the drives and card work perfectly in Catalina and Ventura, but the problem reoccurs as soon as MacOS is updated to Sonoma. How, with this evidence, has engineering made a pronouncement that this is not an OS issue.
- The drive consists of 4 M.2 drives and a host PCIE card. The card has been replaced with a different model, both models approved by apple. The drives have been tested on the Mac Pro in question running and older OS and the problem is not present. The drives have been tested and gone through Samsungs testing process on a windows machine and passed. They also work perfectly on the Mac Pro in question as long as it’s a previous OS. With this evidence, how is this not a macOS issue.
- Engineering state "Engineering have found evidence of a drive issue in the logging”. There are 4 drives and a host card. It’s unclear which drive they found an issue with, can you clarify as all 4 are affected intermittently, only under specific versions of MacOS.
- I have now been through troubleshooting with Samsung. They are surprised by the statement from engineering and have deemed all four of my drives functional and fault free after we ran their diagnostics again. They have also mentioned that one or two drives being faulty is possible, but all 4 would be a statistical impossibility. This is why we seek further clarification.
All drives pass all diagnostics and work perfectly under any Mac OS apart from Sonoma. They also work perfectly under Windows11.
Two cards have been tested, different brands, different chipsets.
Apple appear to have closed this case and will not be investigating any further. I hope they will reconsider this when they read the above.
FURTHER UPDATE
Engineering have reopened my case and I'm told I'll be updated very soon. I dare not hope for any great news at this point, after all, if there was a fix, I'd have seen an OS update pop up.
Update - 27th June 2024
Apple engineering passed a message to me via Apple UK's Executive Relations.
They changed their stance from previous conclusions that my drives were at fault. Now the message is that they are actively working on a solution but the issue is difficult and will take some time. No indication of weather this means weeks, months or years (if it's many months or years, the Intel Mac Pro may be obsolete before there is a solution)
When asked for clarification, ER stated:
"Engineering are working on your case and as advised please keep your software up to date as the solution may come in the form of a software update."
Update 2 - 27th June 2024
During the whole testing process above, there was a Sonnet TEMPO SSD Pro Plus PCIe card present in the machine with 2 x Samsung 1TB SATA 2.5" SSD drives mounted in a raid 0 array (setup in disk utility).
This card and the drives performed as expected for the many months that the above observations were made.
With the above as an assurance, I purchased and installed a second card, the tempo was discontinued and succeeded by the Sonnet FUSION duel 2.5-inch SSD Raid PCIe card, but since this card is similar, I went ahead. I installed 2 x Crucial MX500 4TB SSD drives on this card.
I removed the M.2 card for the time being.
On 4 occasions so far, the volume that's the raid away on the original Sonnet TEMPO card has not initiated on boot up, and intermittently disappeared during operation. That comes with a drive unplugged without being ejected warning.
The new Sonnet FUSION card has also not initialised on boot up.
I placed the cards in different PCIE slots (7 and 3) and this improved but the original Tempo card has still vanished once during operation this week.
I have not conducted full testing on this so it's just observations so far. Unclear if this is related to the issue above, a PCI resource allocation issue (although pool A is currently at 12% and pool B is 75% so there should be no problem), or a power issue as has been suggested early on by Sonnet. This is speculation however.
The only storage solution fast enough (and the reason I purchased a Mac Pro) is the M.2 drives on a fast PCIe card. Everything else is a compromise, especially since there still seems to be intermittent issues with every PCIe based storage solution now.
Is this related?
The 2019 Mac Pro has two SATA connections on the logic board. You can buy a power connector kit from Apple to enable you to add SATA drives to these ports.
I added 2 x 2.5" Samsung Eco 4TB SSD drives to these ports.
I noted that when one was wired to port 2 on its own - it would not initialise, but a 1TB drive would. The drives would only initialise when both 4TB drives were wired.
I went through this with Apple support at the time and the conclusion was that if I wanted both, and they reliably worked when both were plugged in, then there was no need to try to solve the issue. I thought it was worth noting however.
Update 3 - 19th August 2024
A solution?
My case is being managed by a named contact at Apple now. He has promised to keep me informed as to when Apple enginieering come back with an update, and at this point, I have heard nothing. However, I've tested Mac OS Sequoia public beta 1 and 3 and the results are encouraging.
Sequoia Public Beta 1
10+ standard restarts
All drives initialised as expected every time
5 full shut downs and power ups
All drives initialised as expected every time
2 Shut downs, power disconnects, power reconnects and start ups.
All drives initialised as expected every time.
Sequoia public beta 3
Multiple restarts and power ups (5+) All drives initialised as expected every time.
All drives initialised as expected every time
The results above are encouraging, but only being in beta, with no official work or mention of this issue in the changelogs means that this fix may not be permanent.
Update 4 - 20th August 2024
More Signs of a solution?
In 14.6.1 (23G93) which is a sonoma public beta which I've been using as my main OS, mainly in the hope of getting a fix for this, all drives have been initialising and operating as they should. I've not tested properly, so this is just under normal use.
Update 5 - 21st August 2024 Nope - Forget I said that.
Booted up in 14.6.1 today and two out of four NVME drives on the one sonnet card did not initialise and one of the 2.5" SATA raid arrays on another pcie card did not initialise. So no fix yet on Sonoma.
Thank you for your persistent work getting Apple to acknowledge and address this bug, and for making your effort and Apple’s response visible to the public. In my opinion Apple introducing a serious bug into System software, then not fixing it until release of the next major system version, is not acceptable. But I doubt Apple cares much about my opinion.
Thank you also for documenting when the bug was introduced into macOS 14 Sonoma. For a long time I stayed on Big Sur, fearing newer System versions would lead to more new bugs than bug fixes. Then, about the time Sonoma was released, I was surprise-updated from Big Sur to Ventura due to a bug in the macOS Updater. After a short time running Ventura I was again surprise-updated to Sonoma. With such unexpected and unwelcome changes from a System with one set of bugs to another System with a new set of bugs, I have found it difficult to figure out what failures began or ended where.
I suspect the problem of PCIe-mounted memory not being mounted is one manifestation of a bigger problem of macOS not properly reliably recognizing “external” memory. When a Mac has User Spaces relocated off the Startup Disk this is easily seen in System Settings / General. The first item in this window should show the extent of use of the Startup disk by the System as well as various categories of user files. In my case it correctly says 307.64 GB of 1 TB used. However, what uses this space is my System, Library and Applications folders. My 10.75 TB Documents and 1.58 TB Photos System Library are located on other disks. Clicking the button opens another window which mistakenly implies the User files are still located on the Startup Disk. This window also shows only three of my four PCIe-mounted SSDs, and does not show the RAID array created from the SSDs. What I am describing are errors shown by the User Interface. I suspect these are visible artifacts reflecting underlying problems in how macOS deals with Apple File System. Other users can look here to easily see what System Settings tells them about their own computer. Similarly, Apple system engineers can easily look here to see what System Settings tells them about the hardware they use for development. I do not doubt that the solution may be complicated. Seeing that there is a problem is clear and simple.
In case it is of use, I have filed the following Feedback Reports which include a number of System Diagnostic, etc., reports. These reports were submitted through the period of beta testing 14.6.
FB13692130. March 19, 2024. PCIe mounted disks not available on restart
FB14198514. July 4 2024. PCIe-mounted RAID array lost from recognition while in use
FB14666166. August 5 2024. PCIe-attached SSD RAID disconnects after hot restart
Thanks Tom, it looks like there’s an end to this but the issue is still present in the latest beta of Sonoma, but seems to occur far less frequently. The whole thing has been very disruptive, and, as usual has disproportionately affected those same creative people, without whom, Apple may not exist today. Not bitter at all. lol.
THANK-YOU so much! Many people who have invested (So Much $) in this Mac Pro Intel continue to have this exact Problem! You are the best for your continuing persistence for a fix!
BTW-The last I have heard(from Sonnet) regarding a fix is their hope for Sequoia as well having the FIX!
Hi Paul, yes, it does look like sequoia will be our only fix, great if you just use logic and a couple of plugins but for most of us, an early upgrade presents a plethora of compatibility and performance issues with various third party software. I have the latest Sequoia on a test partition with my core software so I’ll be able to tell when things are stable enough to make the switch.
Thank you Stuart for your ongoing documentation on this issue. There is a new Sonoma 14.7 out today along with the other option for MacOS Sequoia 15.0. I wonder if the Sonoma 14.7 update has some of the same promising behavior you were seeing in the Sequoia Beta versions? I have been running Sonoma 14.6.1 now for many weeks and somehow everything on my Sonnet card (Not using anything in RAID) is being seen on repeated boots just fine. So I am reluctant to touch it with any MacOS updates at this point until we hear some official word.
Hi Brian, (I was on the MOTU zoom last week btw, great stuff). I found in 14.7.1 to be a lot more stable, I restarted 10 times and the drives initialised as expected, BUT under normal use yesterday, I was restarting to install software and the sonnet card containing the 4 Samsung NVME drives did not initialise and would not until a full power down adn restart, so the issue is still there, and with no official word from Apple, it continues to be quite frustrating.
This is the most thoroughly documented troubleshooting experience I’ve ever read. Fellow Mac Pro 2019 owner here. I’ve been experiencing the same issue with my OWC PCIE 4M2 drives. My workaround has been to never power off the computer–just put it into sleep mode. I’m glad to hear Sequoia shows some promise. Unfortunately I won’t be able to upgrade to it until my 3D modeling software (Maya) officially becomes compatible with the new OS. Thank you for publishing this!
Thanks, software compatibility is an issue for me too so I’ve been testing on a partition. Mac partitions are flexible these days, so I can just have the test partition take over as my main OS installation when I feel it’s ready.