Discussion:
Can not remove the new MacPorts Mojave package
Add Reply
S. L. Garwood via macports-users
2018-10-03 12:16:33 UTC
Reply
Permalink
I installed the new Macports bundle mentioned in Josha Root’s email.
Install went fine, but now it will not uninstall — doing the “rm” commands in /opt/local gets a series of errors

rm: /opt/local/var/macports/home/Library/Preferences: Operation not permitted
rm: /opt/local/var/macports/home/Library: Operation not permitted
rm: /opt/local/var/macports/home: Operation not permitted
rm: /opt/local/var/macports: Directory not empty
rm: /opt/local/var: Directory not empty
rm: /opt/local: Directory not empty
mymbp:~ slgarwood$
Chris Jones
2018-10-03 12:25:28 UTC
Reply
Permalink
*Exactly* what command are you running ? Are you running it through sudo
? (your regular user account will not be owning /opt/local)

Chris
I installed the new Macports bundle mentioned in Josha Root’s email.
Install went fine, but now it will not uninstall — doing the “rm”
commands in /opt/local gets a series of errors
rm: /opt/local/var/macports/home/Library/Preferences: Operation not permitted
rm: /opt/local/var/macports/home/Library: Operation not permitted
rm: /opt/local/var/macports/home: Operation not permitted
rm: /opt/local/var/macports: Directory not empty
rm: /opt/local/var: Directory not empty
rm: /opt/local: Directory not empty
mymbp:~ slgarwood$
Dr M J Carter
2018-11-09 10:31:39 UTC
Reply
Permalink
I installed the new Macports bundle mentioned in Josha Root’s email.
Install went fine, but now it will not uninstall — doing the “rm”
commands in /opt/local gets a series of errors
rm: /opt/local/var/macports/home/Library/Preferences: Operation not permitted
rm: /opt/local/var/macports/home/Library: Operation not permitted
rm: /opt/local/var/macports/home: Operation not permitted
rm: /opt/local/var/macports: Directory not empty
rm: /opt/local/var: Directory not empty
rm: /opt/local: Directory not empty
mymbp:~ slgarwood$
*Exactly* what command are you running ? Are you running it through sudo ?
(your regular user account will not be owning /opt/local)
I can reproduce this exactly and repeatedly, and I've been banging my
head on it for a fortnight. Our nightly build scripts succeed once,
but the next attempt fails to remove the above directory, and the
build dies aborning. The actions in the Makefiles, shorn of
interlocks and safety traps, boil down to:

# .... disable any MacPorts daemons
port -u uninstall # succeeds
port -q uninstall all # succeeds
killall -TERM -u macports # succeeds
rm -rf /Applications/MacPorts/* # succeeds
for strike in 1 2 3 4 5 out ; do \
if [ -z "`echo /opt/local/*`" ] ; then \
echo "+++ Info: dir empty" ; \
exit 0 ; \
elif rm -rf /opt/local/* ; then \
echo "+++ Info: rm succeeded" ; \
exit 0 ; \
else \
echo "+++ Info: Strike $strike" ; \
[ "$strike" != "out" ] || exit 1 ; \
sleep 5 ; \
fi ; \
done
# fails

This is all done by a sargasso of Makefiles and shell scripts, kicked
off in the background using either sudo or root's crontab. (I'll
leave my adventures with atrun for another day.) Please find enclosed
the logfile for this morning's attempt, which shows a failure to
uninstall font-misc-ethiopic (which may or may not be relevant).

My current theory stems from noting that the offending directory is
~macports/Library/Preferences, hence the killall above; but by the
time I get to see the result, whatever process is holding down that
directory is long gone (so a reattempt would succeed). Frustratingly,
an empty build fails to fail, and a full build takes twelve hours or
more; as Mr Micawber would say, "result Misery". I'm doing a reduced
test build as we speak, after which I'll try running the offending
Make target by hand using sudo.

All suggestions welcomed: we can't roll out Mojave in production till
this is fixed or worked around. Thanks in advance.
--
Dr Martin J Carter
Computer System Administrator
Astrophysics, University of Oxford
Dr M J Carter
2018-11-09 10:34:29 UTC
Reply
Permalink
Please find enclosed the logfile for this morning's attempt
Sigh: said logfile.
--
Dr Martin J Carter
Computer System Administrator
Astrophysics, University of Oxford
Vincent Habchi
2018-11-09 15:05:34 UTC
Reply
Permalink
I installed the new Macports bundle mentioned in Josha Root’s email.
Install went fine, but now it will not uninstall — doing the “rm”
commands in /opt/local gets a series of errors
rm: /opt/local/var/macports/home/Library/Preferences: Operation not permitted
rm: /opt/local/var/macports/home/Library: Operation not permitted
rm: /opt/local/var/macports/home: Operation not permitted
rm: /opt/local/var/macports: Directory not empty
rm: /opt/local/var: Directory not empty
rm: /opt/local: Directory not empty
Did you check the script was executed with root privilege? Also what files are left in /opt/local/var/macports/home/Library/Preferences? Any hard link to any other file that could be protected by a sandbox mechanism?

Vincent
Dr M J Carter
2018-11-09 16:36:04 UTC
Reply
Permalink
Post by Vincent Habchi
I installed the new Macports bundle mentioned in Josha Root’s email.
Install went fine, but now it will not uninstall — doing the “rm”
commands in /opt/local gets a series of errors
rm: /opt/local/var/macports/home/Library/Preferences: Operation not permitted
rm: /opt/local/var/macports/home/Library: Operation not permitted
rm: /opt/local/var/macports/home: Operation not permitted
rm: /opt/local/var/macports: Directory not empty
rm: /opt/local/var: Directory not empty
rm: /opt/local: Directory not empty
Did you check the script was executed with root privilege?
Info from current tests (not yet completed): "whoami" just before the
removal yields "root" under all conditions.
Post by Vincent Habchi
Also what
files are left in /opt/local/var/macports/home/Library/Preferences?
Completely empty; no fancy permissions either afaict. But by the time
I get to check them, the rm would succeed, so that probably doesn't
tell us much.
Post by Vincent Habchi
Any hard link to any other file that could be protected by a sandbox
mechanism?
No hard links that I spotted (but would that not fail under 10.13? the
same code runs fine). The only differences between what fails and
what works are 10.14 vs 10.13 and older, and [sound effect: penny
dropping] APFS vs HFS+ journaled. If there's some protection going
on, it seems to be of macports's home dir, as the errors change to
"directory not empty" for that dir's parent.

Hope this helps. More information next week, once the build system
has tried to do a build unattended twice in succession, this time with
exponential backoff between removal attempts. Suggestions for extra
checks to add into the runtime scripting welcomed .... as you might
gather, this one really has got under my fingernails. Apologies for
any fallout.
--
Dr Martin J Carter
Computer System Administrator
Astrophysics, University of Oxford
Vincent Habchi
2018-11-09 19:56:43 UTC
Reply
Permalink
Hi again,
Post by Dr M J Carter
Post by S. L. Garwood via macports-users
rm: /opt/local/var/macports/home/Library: Operation not permitted
rm: /opt/local/var/macports/home: Operation not permitted
Both lines are very suspect. When I ls what’s inside the first directory
Air > ll /opt/local/var/macports/home/Library/Preferences/
total 616
-rw-r--r-- 1 macports admin 314139 7 Nov 18:33 com.apple.dt.Xcode.plist

This looks to me very much like a hardlink or a protected file.

Air > ls -lORa /opt/local/var/macports/home/Library/Preferences/
total 616
drwxr-xr-x 3 root admin - 96 8 Nov 14:53 .
drwxr-xr-x 3 root admin - 96 25 Oct 2016 ..
-rw-r--r-- 1 macports admin - 314139 7 Nov 18:33 com.apple.dt.Xcode.plist

But that doesn’t mean anything in my case, since I have disabled SIP for files (yes, I got rid of several dumb apps such as Chess or Stocks, I only have a 128 MB SSD).
Post by Dr M J Carter
No hard links that I spotted (but would that not fail under 10.13? the
same code runs fine). The only differences between what fails and
what works are 10.14 vs 10.13 and older, and [sound effect: penny
dropping] APFS vs HFS+ journaled. If there's some protection going
SIP protection might have been extended to newer files between 10.13 and 10.14.

Vincent
Ryan Schmidt
2018-11-10 21:47:26 UTC
Reply
Permalink
Post by Dr M J Carter
Post by Vincent Habchi
I installed the new Macports bundle mentioned in Josha Root’s email.
Install went fine, but now it will not uninstall — doing the “rm”
commands in /opt/local gets a series of errors
rm: /opt/local/var/macports/home/Library/Preferences: Operation not permitted
rm: /opt/local/var/macports/home/Library: Operation not permitted
rm: /opt/local/var/macports/home: Operation not permitted
rm: /opt/local/var/macports: Directory not empty
rm: /opt/local/var: Directory not empty
rm: /opt/local: Directory not empty
Did you check the script was executed with root privilege?
Info from current tests (not yet completed): "whoami" just before the
removal yields "root" under all conditions.
Post by Vincent Habchi
Also what
files are left in /opt/local/var/macports/home/Library/Preferences?
Completely empty; no fancy permissions either afaict. But by the time
I get to check them, the rm would succeed, so that probably doesn't
tell us much.
Post by Vincent Habchi
Any hard link to any other file that could be protected by a sandbox
mechanism?
No hard links that I spotted (but would that not fail under 10.13? the
same code runs fine). The only differences between what fails and
what works are 10.14 vs 10.13 and older, and [sound effect: penny
dropping] APFS vs HFS+ journaled. If there's some protection going
on, it seems to be of macports's home dir, as the errors change to
"directory not empty" for that dir's parent.
Hope this helps. More information next week, once the build system
has tried to do a build unattended twice in succession, this time with
exponential backoff between removal attempts. Suggestions for extra
checks to add into the runtime scripting welcomed .... as you might
gather, this one really has got under my fingernails. Apologies for
any fallout.
Since it sounds like you have some kind of Heisenbug that disappears under later analysis, I'd suggest adding any analysis you can to the script you're running when it fails.

For example, have the script use `find` to show all the files remaining in /opt/local after the script fails to delete it. If you're root, you should be able to delete anything, so it's not permissions. It's not System Integrity Protection (SIP) either; that applies to files Apple provides with the OS, which does not include MacPorts.

You could also have the script run `lsof | grep /opt/local` to see what files in /opt/local/are still open at the time of the failure. The only file MacPorts always copies into its home directory is the Xcode preferences plist. On Mojave you're using Xcode 10, which is of course different from Xcode 9 and earlier. One big change is that the "new build system" is used by default. Since it is not as well tested as the old build system, it may still have bugs. Maybe it is holding the preferences file open where the old build system did not.

Xcode also loves to spawn its simulator service, even though, as far as I know, MacPorts doesn't make use of it. Maybe the simulator service in Xcode 10 is doing things with the preferences directory that it did not on previous versions. You might even investigate having your build script kill the simulator service (com.apple.CoreSimulator.CoreSimulatorService) to see if that clears up the problem.

I'm not aware of any changes in MacPorts specific to Mojave that would relate to the handling of the home directory. So I think the bug you're investigating is a bug in macOS or Xcode, not MacPorts.
Dr M J Carter
2018-11-12 10:15:47 UTC
Reply
Permalink
Post by Ryan Schmidt
Since it sounds like you have some kind of Heisenbug that disappears
under later analysis, I'd suggest adding any analysis you can to the
script you're running when it fails.
I'm attempting to do that, but with thin results so far. I'll expand
the in-build test set from merely lsof on the Prefs directory (which
comes up empty) Thanks for the suggestions.
Post by Ryan Schmidt
I'm not aware of any changes in MacPorts specific to Mojave that
would relate to the handling of the home directory. So I think the
bug you're investigating is a bug in macOS or Xcode, not MacPorts.
.... Possibly APFS? that test's pending an upcoming changearound of
hardware.
--
Dr Martin J Carter
Computer System Administrator
Astrophysics, University of Oxford
Ryan Schmidt
2018-11-12 19:08:15 UTC
Reply
Permalink
Post by Dr M J Carter
Post by Ryan Schmidt
I'm not aware of any changes in MacPorts specific to Mojave that
would relate to the handling of the home directory. So I think the
bug you're investigating is a bug in macOS or Xcode, not MacPorts.
.... Possibly APFS? that test's pending an upcoming changearound of
hardware.
APFS certainly has caused some weird errors already. It caused parallel builds of gcc to fail. We had a fix in place for that for 10.13, but with 10.14, some users have reported the problem resurfacing, despite the fix.
Dr M J Carter
2018-11-12 10:52:37 UTC
Reply
Permalink
Post by Ryan Schmidt
The only file MacPorts always copies into its home directory is the
Xcode preferences plist.
Quick note: on our 10.13 build systems, I find that that preferences
file has been saved in the clone of successful nightly builds, but is
missing from the /opt/local that's been cloned. Might the underlying
problem be that I'm using Xcode's make to run the builds? in which
case, mebbe the kludgearound would be to do the purge from one
cronjob, and the full build in another an hour later.

Thanks for that observation. I'll report results once I've results to
report.
--
Dr Martin J Carter
Computer System Administrator
Astrophysics, University of Oxford
Ryan Schmidt
2018-11-12 19:09:29 UTC
Reply
Permalink
Post by Dr M J Carter
Post by Ryan Schmidt
The only file MacPorts always copies into its home directory is the
Xcode preferences plist.
Quick note: on our 10.13 build systems, I find that that preferences
file has been saved in the clone of successful nightly builds, but is
missing from the /opt/local that's been cloned. Might the underlying
problem be that I'm using Xcode's make to run the builds? in which
case, mebbe the kludgearound would be to do the purge from one
cronjob, and the full build in another an hour later.
Most ports use Xcode make (instead of the one from the gmake port), so if there were a general problem with Xcode make we probably would have seen it already.
Dr M J Carter
2018-11-13 09:58:31 UTC
Reply
Permalink
Post by Ryan Schmidt
Most ports use Xcode make (instead of the one from the gmake port),
so if there were a general problem with Xcode make we probably would
have seen it already.
EDESPERATION: See straw, *grasp*.
--
Dr Martin J Carter
Computer System Administrator
Astrophysics, University of Oxford
Dr M J Carter
2018-11-16 12:05:30 UTC
Reply
Permalink
Post by Ryan Schmidt
You could also have the script run `lsof | grep /opt/local` to see
what files in /opt/local/are still open at the time of the
failure.
[snip]
Post by Ryan Schmidt
Xcode also loves to spawn its simulator service, even though, as far
as I know, MacPorts doesn't make use of it. Maybe the simulator
service in Xcode 10 is doing things with the preferences directory
that it did not on previous versions.
Well suggested. Once I removed the "killall -u macports" in the
Makefiles and broadened the diagnostics, I found the following, which
survived into an interactive session:

$ sudo lsof | grep /opt/local
[snip]
com.apple 85084 macports 7r DIR 1,4 96 26913415 /opt/local/var/macports/home/Library

$ ps auxww | grep -i com.apple
[snipissimo]
macports 85084 0.0 0.1 4854824 14384 ?? S 2:27pm 0:01.08 /Library/Developer/PrivateFrameworks/CoreSimulator.framework/Versions/A/XPCServices/com.apple.CoreSimulator.CoreSimulatorService.xpc/Contents/MacOS/com.apple.CoreSimulator.CoreSimulatorService

.... Bingo, for sufficiently provisional values of Bing. Much thought
needed on which of several approaches to take next, and I'll need to
guard against premature rejoicing, but it's the first substative Clue
I've seen yet. Wish me luck, folks.

PS: There were also odd icon files under /opt/local/libexec/qt5 which
iconserviceagent was holding open (from a console login by sysadmin),
but /opt/local/libexec no longer existed. I'll worry about whether
that matters later; one problem at a time.
--
Dr Martin J Carter
Computer System Administrator
Astrophysics, University of Oxford
Vincent Habchi
2018-11-16 12:26:18 UTC
Reply
Permalink
Post by Dr M J Carter
.... Bingo, for sufficiently provisional values of Bing. Much thought
needed on which of several approaches to take next, and I'll need to
guard against premature rejoicing, but it's the first substative Clue
I've seen yet. Wish me luck, folks.
Good luck! :)
Don’t forget to tell us if Oumuamua is an extraterrestrial probe or not ;)

Vincent
Dr M J Carter
2018-11-23 17:18:44 UTC
Reply
Permalink
Post by Ryan Schmidt
It's not System Integrity Protection (SIP) either; that applies to
files Apple provides with the OS, which does not include MacPorts.
SIP may well care to some extent about users' home directories, as
logging in with a nonexistent homedir causes more fundamental grief
than if it's empty. We get that sometimes with our systems; from
memory, an attempted macOS login with a missing homedir can hang the
OS hard enough to need a reboot, while an empty one can be populated.
Post by Ryan Schmidt
Xcode also loves to spawn its simulator service
SIGHUPping the simulator causes it to exit, but that didn't solve the
problem.
Post by Ryan Schmidt
I'm not aware of any changes in MacPorts specific to Mojave that
would relate to the handling of the home directory. So I think the
bug you're investigating is a bug in macOS or Xcode, not MacPorts.
Following some heavy net searching today, I found the Full Disk Access
pane in Preferences, and discovered sshd has been whitelisted. I've
come to the tentative conlusion that this is transitive, in that
software run from whitelisted apps (eg stuff launched from an SSH
login) will also be whitelisted, while those run via cron will not,
hence the Heisenbug nature of the problem .... but I've been wrong
before.

After a month of headbanging, I've realised it probably doesn't matter
for my purposes if these directories are deleted --- they're empty,
and are about to be recreated in any case. Our builds will now
complain (in case it turns out to matter later), but continue under
protest. Thanks for the feedback, folks.

And there, you see, my head no longer throbs. -- G'Kar
--
Dr Martin J Carter
Computer System Administrator
Astrophysics, University of Oxford
Ryan Schmidt
2018-11-25 09:30:28 UTC
Reply
Permalink
Post by Dr M J Carter
Post by Ryan Schmidt
It's not System Integrity Protection (SIP) either; that applies to
files Apple provides with the OS, which does not include MacPorts.
SIP may well care to some extent about users' home directories, as
logging in with a nonexistent homedir causes more fundamental grief
than if it's empty. We get that sometimes with our systems; from
memory, an attempted macOS login with a missing homedir can hang the
OS hard enough to need a reboot, while an empty one can be populated.
As far as I know, SIP has nothing to do with your home directory. SIP prevents you from modifying programs, libraries and other files that Apple installed as part of macOS in system directories like /usr and /System.

Completely unrelated to that, I'm not surprised that macOS assumes your home directory exists, and that things blow up if it does not, so don't remove your home directory.
Bill Cole
2018-11-09 18:01:42 UTC
Reply
Permalink
I installed the new Macports bundle mentioned in Josha Root’s email.
Install went fine, but now it will not uninstall — doing the “rm”
commands in /opt/local gets a series of errors
rm: /opt/local/var/macports/home/Library/Preferences: Operation not permitted
rm: /opt/local/var/macports/home/Library: Operation not permitted
rm: /opt/local/var/macports/home: Operation not permitted
Those imply protection by Apple's SIP *or* BSD file flags.

What does 'ls -lORa /opt/local/var/macports/home/' tell you about flags?
(the field after the group, will be '-' if no flags are set.)
Dr M J Carter
2018-11-12 09:53:19 UTC
Reply
Permalink
Post by Bill Cole
Those imply protection by Apple's SIP *or* BSD file flags.
That's my suspicion too.
Post by Bill Cole
What does 'ls -lORa /opt/local/var/macports/home/' tell you about flags?
(the field after the group, will be '-' if no flags are set.)
Absolutely nothing (for completeness, no hard links). Adding the '@'
flag shows ~macports's parent (/opt/local/var/macports) has extended
attribute info "com.apple.FinderInfo 32" fwiw, but that's set in 10.13
as well, so that's probably an infrared herring.
--
Dr Martin J Carter
Computer System Administrator
Astrophysics, University of Oxford
Loading...