mte/2025_06_21_macos-sequoia-include-path.json at main · anil.recoil.org/thicket-eeg

Thicket data repository for the EEG
thicket-eeg / mte / 2025_06_21_macos-sequoia-include-path.json
at main 17 kB view raw
 1{
 2  "id": "https://www.tunbury.org/2025/06/21/macos-sequoia-include-path",
 3  "title": "Surprising C++ failures in the macOS workers",
 4  "link": "https://www.tunbury.org/2025/06/21/macos-sequoia-include-path/",
 5  "updated": "2025-06-21T00:00:00",
 6  "published": "2025-06-21T00:00:00",
 7  "summary": "@mseri raised issue #175 as the macOS workers cannot find the most basic C++ headers. I easily eliminated Obuilder, as opam install mccs.1.1+19 didn’t work on the macOS workers natively.",
 8  "content": "<p>@mseri raised <a href=\"https://github.com/ocaml/infrastructure/issues/175\">issue #175</a> as the macOS workers cannot find the most basic C++ headers. I easily eliminated <a href=\"https://github.com/ocurrent/obuilder\">Obuilder</a>, as <code>opam install mccs.1.1+19</code> didn’t work on the macOS workers natively.</p>\n\n<p>On face value, the problem appears pretty common, and there are numerous threads on <a href=\"https://stackoverflow.com\">Stack Overflow</a> such as this <a href=\"https://stackoverflow.com/questions/77250743/mac-xcode-g-cannot-compile-even-a-basic-c-program-issues-with-standard-libr\">one</a>, however, the resolutions I tried didn’t work. I was reluctant to try some of the more intrusive changes like creating a symlink of every header from <code>/usr/include/</code> to <code>/Library/Developer/CommandLineTools/usr/include/c++/v1</code> as this doesn’t seem to be what Apple intends.</p>\n\n<p>For the record, a program such as this:</p>\n\n<div><div><pre><code><span>#include</span> <span>&lt;iostream&gt;</span><span>\n</span>\n<span>using</span> <span>namespace</span> <span>std</span><span>;</span>\n\n<span>int</span> <span>main</span><span>()</span> <span>{</span>\n    <span>cout</span> <span>&lt;&lt;</span> <span>\"Hello World!\"</span> <span>&lt;&lt;</span> <span>endl</span><span>;</span>\n    <span>return</span> <span>0</span><span>;</span>\n<span>}</span>\n</code></pre></div></div>\n\n<p>Fails like this:</p>\n\n<div><div><pre><code>% c++ hello.cpp <span>-o</span> hello <span>-v</span>\nApple clang version 17.0.0 <span>(</span>clang-1700.0.13.3<span>)</span>\nTarget: x86_64-apple-darwin24.5.0\nThread model: posix\nInstalledDir: /Library/Developer/CommandLineTools/usr/bin\n <span>\"/Library/Developer/CommandLineTools/usr/bin/clang\"</span> <span>-cc1</span> <span>-triple</span> x86_64-apple-macosx15.0.0 <span>-Wundef-prefix</span><span>=</span>TARGET_OS_ <span>-Wdeprecated-objc-isa-usage</span> <span>-Werror</span><span>=</span>deprecated-objc-isa-usage <span>-Werror</span><span>=</span>implicit-function-declaration <span>-emit-obj</span> <span>-dumpdir</span> hello- <span>-disable-free</span> <span>-clear-ast-before-backend</span> <span>-disable-llvm-verifier</span> <span>-discard-value-names</span> <span>-main-file-name</span> hello.cpp <span>-mrelocation-model</span> pic <span>-pic-level</span> 2 <span>-mframe-pointer</span><span>=</span>all <span>-fno-strict-return</span> <span>-ffp-contract</span><span>=</span>on <span>-fno-rounding-math</span> <span>-funwind-tables</span><span>=</span>2 <span>-target-sdk-version</span><span>=</span>15.4 <span>-fvisibility-inlines-hidden-static-local-var</span> <span>-fdefine-target-os-macros</span> <span>-fno-assume-unique-vtables</span> <span>-fno-modulemap-allow-subdirectory-search</span> <span>-target-cpu</span> penryn <span>-tune-cpu</span> generic <span>-debugger-tuning</span><span>=</span>lldb <span>-fdebug-compilation-dir</span><span>=</span>/Users/administrator/x <span>-target-linker-version</span> 1167.4.1 <span>-v</span> <span>-fcoverage-compilation-dir</span><span>=</span>/Users/administrator/x <span>-resource-dir</span> /Library/Developer/CommandLineTools/usr/lib/clang/17 <span>-isysroot</span> /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk <span>-internal-isystem</span> /Library/Developer/CommandLineTools/usr/bin/../include/c++/v1 <span>-internal-isystem</span> /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/local/include <span>-internal-isystem</span> /Library/Developer/CommandLineTools/usr/lib/clang/17/include <span>-internal-externc-isystem</span> /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include <span>-internal-externc-isystem</span> /Library/Developer/CommandLineTools/usr/include <span>-Wno-reorder-init-list</span> <span>-Wno-implicit-int-float-conversion</span> <span>-Wno-c99-designator</span> <span>-Wno-final-dtor-non-final-class</span> <span>-Wno-extra-semi-stmt</span> <span>-Wno-misleading-indentation</span> <span>-Wno-quoted-include-in-framework-header</span> <span>-Wno-implicit-fallthrough</span> <span>-Wno-enum-enum-conversion</span> <span>-Wno-enum-float-conversion</span> <span>-Wno-elaborated-enum-base</span> <span>-Wno-reserved-identifier</span> <span>-Wno-gnu-folding-constant</span> <span>-fdeprecated-macro</span> <span>-ferror-limit</span> 19 <span>-stack-protector</span> 1 <span>-fstack-check</span> <span>-mdarwin-stkchk-strong-link</span> <span>-fblocks</span> <span>-fencode-extended-block-signature</span> <span>-fregister-global-dtors-with-atexit</span> <span>-fgnuc-version</span><span>=</span>4.2.1 <span>-fno-cxx-modules</span> <span>-fskip-odr-check-in-gmf</span> <span>-fcxx-exceptions</span> <span>-fexceptions</span> <span>-fmax-type-align</span><span>=</span>16 <span>-fcommon</span> <span>-fcolor-diagnostics</span> <span>-clang-vendor-feature</span><span>=</span>+disableNonDependentMemberExprInCurrentInstantiation <span>-fno-odr-hash-protocols</span> <span>-clang-vendor-feature</span><span>=</span>+enableAggressiveVLAFolding <span>-clang-vendor-feature</span><span>=</span>+revert09abecef7bbf <span>-clang-vendor-feature</span><span>=</span>+thisNoAlignAttr <span>-clang-vendor-feature</span><span>=</span>+thisNoNullAttr <span>-clang-vendor-feature</span><span>=</span>+disableAtImportPrivateFrameworkInImplementationError <span>-D__GCC_HAVE_DWARF2_CFI_ASM</span><span>=</span>1 <span>-o</span> /var/folders/sh/9c8b7hzd2wb1g2_ky78vqw5r0000gn/T/hello-a268ab.o <span>-x</span> c++ hello.cpp\nclang <span>-cc1</span> version 17.0.0 <span>(</span>clang-1700.0.13.3<span>)</span> default target x86_64-apple-darwin24.5.0\nignoring nonexistent directory <span>\"/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/local/include\"</span>\nignoring nonexistent directory <span>\"/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/System/Library/SubFrameworks\"</span>\nignoring nonexistent directory <span>\"/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/Library/Frameworks\"</span>\n<span>#include \"...\" search starts here:</span>\n<span>#include &lt;...&gt; search starts here:</span>\n /Library/Developer/CommandLineTools/usr/bin/../include/c++/v1\n /Library/Developer/CommandLineTools/usr/lib/clang/17/include\n /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include\n /Library/Developer/CommandLineTools/usr/include\n /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/System/Library/Frameworks <span>(</span>framework directory<span>)</span>\nEnd of search list.\nhello.cpp:1:10: fatal error: <span>'iostream'</span> file not found\n    1 | <span>#include &lt;iostream&gt;</span>\n      |          ^~~~~~~~~~\n1 error generated.\n</code></pre></div></div>\n\n<p>That first folder looked strange: <code>bin/../include/c++/v1</code>. Really? What’s in there? Not much:</p>\n\n<div><div><pre><code>% <span>ls</span> <span>-l</span> /Library/Developer/CommandLineTools/usr/bin/../include/c++/v1\ntotal 40\n<span>-rw-r--r--</span>  1 root  wheel  44544  7 Apr  2022 __functional_03\n<span>-rw-r--r--</span>  1 root  wheel   6532  7 Apr  2022 __functional_base_03\n<span>-rw-r--r--</span>  1 root  wheel   2552  7 Apr  2022 __sso_allocator\n</code></pre></div></div>\n\n<p>I definitely have <code>iostream</code> on the machine:</p>\n\n<div><div><pre><code>% <span>ls</span> <span>-l</span> /Library/Developer/CommandLineTools/SDKs/MacOSX<span>*</span>.sdk/usr/include/c++/v1/iostream\n<span>-rw-r--r--</span>  1 root  wheel  1507  8 Mar 03:36 /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/c++/v1/iostream\n<span>-rw-r--r--</span>  1 root  wheel  1391 13 Nov  2021 /Library/Developer/CommandLineTools/SDKs/MacOSX12.1.sdk/usr/include/c++/v1/iostream\n<span>-rw-r--r--</span>  1 root  wheel  1583 13 Apr  2024 /Library/Developer/CommandLineTools/SDKs/MacOSX14.5.sdk/usr/include/c++/v1/iostream\n<span>-rw-r--r--</span>  1 root  wheel  1583 13 Apr  2024 /Library/Developer/CommandLineTools/SDKs/MacOSX14.sdk/usr/include/c++/v1/iostream\n<span>-rw-r--r--</span>  1 root  wheel  1583 10 Nov  2024 /Library/Developer/CommandLineTools/SDKs/MacOSX15.2.sdk/usr/include/c++/v1/iostream\n<span>-rw-r--r--</span>  1 root  wheel  1507  8 Mar 03:36 /Library/Developer/CommandLineTools/SDKs/MacOSX15.4.sdk/usr/include/c++/v1/iostream\n<span>-rw-r--r--</span>  1 root  wheel  1507  8 Mar 03:36 /Library/Developer/CommandLineTools/SDKs/MacOSX15.sdk/usr/include/c++/v1/iostream\n</code></pre></div></div>\n\n<p>I tried on my MacBook, which compiled the test program without issue. However, that was running Monterey, where the workers are running Sequoia. The <em>include</em> paths on my laptop look much better. Where are they configured?</p>\n\n<div><div><pre><code>% c++ <span>-v</span> <span>-o</span> <span>test </span>test.cpp\nApple clang version 15.0.0 <span>(</span>clang-1500.3.9.4<span>)</span>\nTarget: x86_64-apple-darwin23.5.0\nThread model: posix\nInstalledDir: /Library/Developer/CommandLineTools/usr/bin\n <span>\"/Library/Developer/CommandLineTools/usr/bin/clang\"</span> <span>-cc1</span> <span>-triple</span> x86_64-apple-macosx14.0.0 <span>-Wundef-prefix</span><span>=</span>TARGET_OS_ <span>-Wdeprecated-objc-isa-usage</span> <span>-Werror</span><span>=</span>deprecated-objc-isa-usage <span>-Werror</span><span>=</span>implicit-function-declaration <span>-emit-obj</span> <span>-mrelax-all</span> <span>--mrelax-relocations</span> <span>-disable-free</span> <span>-clear-ast-before-backend</span> <span>-disable-llvm-verifier</span> <span>-discard-value-names</span> <span>-main-file-name</span> test.cpp <span>-mrelocation-model</span> pic <span>-pic-level</span> 2 <span>-mframe-pointer</span><span>=</span>all <span>-fno-strict-return</span> <span>-ffp-contract</span><span>=</span>on <span>-fno-rounding-math</span> <span>-funwind-tables</span><span>=</span>2 <span>-target-sdk-version</span><span>=</span>14.4 <span>-fvisibility-inlines-hidden-static-local-var</span> <span>-target-cpu</span> penryn <span>-tune-cpu</span> generic <span>-debugger-tuning</span><span>=</span>lldb <span>-target-linker-version</span> 1053.12 <span>-v</span> <span>-fcoverage-compilation-dir</span><span>=</span>/Users/mtelvers/x <span>-resource-dir</span> /Library/Developer/CommandLineTools/usr/lib/clang/15.0.0 <span>-isysroot</span> /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk <span>-I</span>/usr/local/include <span>-internal-isystem</span> /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/c++/v1 <span>-internal-isystem</span> /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/local/include <span>-internal-isystem</span> /Library/Developer/CommandLineTools/usr/lib/clang/15.0.0/include <span>-internal-externc-isystem</span> /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include <span>-internal-externc-isystem</span> /Library/Developer/CommandLineTools/usr/include <span>-Wno-reorder-init-list</span> <span>-Wno-implicit-int-float-conversion</span> <span>-Wno-c99-designator</span> <span>-Wno-final-dtor-non-final-class</span> <span>-Wno-extra-semi-stmt</span> <span>-Wno-misleading-indentation</span> <span>-Wno-quoted-include-in-framework-header</span> <span>-Wno-implicit-fallthrough</span> <span>-Wno-enum-enum-conversion</span> <span>-Wno-enum-float-conversion</span> <span>-Wno-elaborated-enum-base</span> <span>-Wno-reserved-identifier</span> <span>-Wno-gnu-folding-constant</span> <span>-fdeprecated-macro</span> <span>-fdebug-compilation-dir</span><span>=</span>/Users/mtelvers/x <span>-ferror-limit</span> 19 <span>-stack-protector</span> 1 <span>-fstack-check</span> <span>-mdarwin-stkchk-strong-link</span> <span>-fblocks</span> <span>-fencode-extended-block-signature</span> <span>-fregister-global-dtors-with-atexit</span> <span>-fgnuc-version</span><span>=</span>4.2.1 <span>-fno-cxx-modules</span> <span>-fcxx-exceptions</span> <span>-fexceptions</span> <span>-fmax-type-align</span><span>=</span>16 <span>-fcommon</span> <span>-fcolor-diagnostics</span> <span>-clang-vendor-feature</span><span>=</span>+disableNonDependentMemberExprInCurrentInstantiation <span>-fno-odr-hash-protocols</span> <span>-clang-vendor-feature</span><span>=</span>+enableAggressiveVLAFolding <span>-clang-vendor-feature</span><span>=</span>+revert09abecef7bbf <span>-clang-vendor-feature</span><span>=</span>+thisNoAlignAttr <span>-clang-vendor-feature</span><span>=</span>+thisNoNullAttr <span>-mllvm</span> <span>-disable-aligned-alloc-awareness</span><span>=</span>1 <span>-D__GCC_HAVE_DWARF2_CFI_ASM</span><span>=</span>1 <span>-o</span> /var/folders/15/4zw4hb9s40b8cmff3z5bdszc0000gp/T/test-71e229.o <span>-x</span> c++ test.cpp\nclang <span>-cc1</span> version 15.0.0 <span>(</span>clang-1500.3.9.4<span>)</span> default target x86_64-apple-darwin23.5.0\nignoring nonexistent directory <span>\"/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/local/include\"</span>\nignoring nonexistent directory <span>\"/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/Library/Frameworks\"</span>\n<span>#include \"...\" search starts here:</span>\n<span>#include &lt;...&gt; search starts here:</span>\n /usr/local/include\n /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/c++/v1\n /Library/Developer/CommandLineTools/usr/lib/clang/15.0.0/include\n /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include\n /Library/Developer/CommandLineTools/usr/include\n /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/System/Library/Frameworks <span>(</span>framework directory<span>)</span>\nEnd of search list.\n <span>\"/Library/Developer/CommandLineTools/usr/bin/ld\"</span> <span>-demangle</span> <span>-lto_library</span> /Library/Developer/CommandLineTools/usr/lib/libLTO.dylib <span>-no_deduplicate</span> <span>-dynamic</span> <span>-arch</span> x86_64 <span>-platform_version</span> macos 14.0.0 14.4 <span>-syslibroot</span> /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk <span>-o</span> <span>test</span> <span>-L</span>/usr/local/lib /var/folders/15/4zw4hb9s40b8cmff3z5bdszc0000gp/T/test-71e229.o <span>-lc</span>++ <span>-lSystem</span> /Library/Developer/CommandLineTools/usr/lib/clang/15.0.0/lib/darwin/libclang_rt.osx.a\n</code></pre></div></div>\n\n<p>I’ve been meaning to upgrade my MacBook, and this looked like the perfect excuse. I updated to Sequoia and then updated the Xcode command-line tools. The test compilation worked, the paths looked good, but I had clang 1700.0.13.5, where the workers had 1700.0.13.3.</p>\n\n<div><div><pre><code>% c++ <span>-v</span> <span>-o</span> <span>test </span>test.cpp\nApple clang version 17.0.0 <span>(</span>clang-1700.0.13.5<span>)</span>\nTarget: x86_64-apple-darwin24.5.0\nThread model: posix\nInstalledDir: /Library/Developer/CommandLineTools/usr/bin\n</code></pre></div></div>\n\n<p>I updated the workers to 1700.0.13.5, which didn’t make any difference. The workers still had that funny <code>/../</code> path, which wasn’t present anywhere else. I searched <code>/Library/Developer/CommandLineTools/usr/bin/../include/c++/v1 site:stackoverflow.com</code> and the answer is the top <a href=\"https://stackoverflow.com/a/79606435\">match</a>.</p>\n\n<blockquote>\n  <p>Rename or if you’re confident enough, delete /Library/Developer/CommandLineTools/usr/include/c++, then clang++ will automatically search headers under /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/c++/v1 and find your  header. That directory is very likely an artifact of OS upgrade and by deleting it clang++ will realise that it should search in the header paths of new SDKs.</p>\n</blockquote>\n\n<p>I wasn’t confident, so I moved it, <code>sudo mv c++ ~</code>. With that done, the test program builds correctly! Have a read of the <a href=\"https://stackoverflow.com/a/79606435\">answer</a> on Stack Overflow.</p>\n\n<p>Now, rather more cavalierly, I removed the folder on all the i7 and m1 workers:</p>\n\n<div><div><pre><code><span>$ </span><span>for </span>a <span>in</span> <span>{</span>01..04<span>}</span> <span>;</span> <span>do </span>ssh m1-worker-<span>$a</span>.macos.ci.dev <span>sudo rm</span> <span>-r</span> /Library/Developer/CommandLineTools/usr/include/c++ <span>;</span> <span>done</span>\n</code></pre></div></div>",
 9  "content_type": "html",
10  "author": {
11    "name": "Mark Elvers",
12    "email": "mark.elvers@tunbury.org",
13    "uri": null
14  },
15  "categories": [
16    "macos,clang",
17    "tunbury.org"
18  ],
19  "source": "https://www.tunbury.org/atom.xml"
20}