Skip to main content

Improving build times

Setup

Use a clang stdenv:

nix develop .#native-clangStdenvPackages

Then delete build/ if you were using gcc before. Enable build-time profiling with:

just setup; meson configure build -Dprofile-build=enabled

Then run the build: just compile.

Enabling build-time profiling itself costs about 10% of compile time but has no other disadvantage.

Build time reports

Use maintainers/buildtime_report.sh build/ into generate a clangbuild stdenvtime when building with Meson.report. This will simply tell you where all our build time went.went by looking at the trace files and producing a badness summary.

Sample report

Build-time report sample 
lix/lix2 » ClangBuildAnalyzer --analyze buildtimeold.bin
Analyzing build trace from 'buildtimeold.bin'...
**** Time summary:
Compilation (551 times):
  Parsing (frontend):         1465.3 s
  Codegen & opts (backend):   1110.9 s

**** Files that took longest to parse (compiler frontend):
 10478 ms: build/src/libstore/liblixstore.so.p/build_local-derivation-goal.cc.o
 10319 ms: build/src/libexpr/liblixexpr.so.p/primops.cc.o
  9947 ms: build/src/nix/nix.p/flake.cc.o
  9850 ms: build/src/libexpr/liblixexpr.so.p/eval.cc.o
  9751 ms: build/src/nix/nix.p/profile.cc.o
  9643 ms: build/src/nix/nix.p/develop.cc.o
  9296 ms: build/src/libcmd/liblixcmd.so.p/installable-attr-path.cc.o
  9286 ms: build/src/libstore/liblixstore.so.p/build_derivation-goal.cc.o
  9208 ms: build/src/libcmd/liblixcmd.so.p/installables.cc.o
  9007 ms: build/src/nix/nix.p/.._nix-env_nix-env.cc.o

**** Files that took longest to codegen (compiler backend):
 24226 ms: build/src/libexpr/liblixexpr.so.p/primops_fromTOML.cc.o
 24019 ms: build/src/libexpr/liblixexpr.so.p/primops.cc.o
 21102 ms: build/src/libstore/liblixstore.so.p/build_local-derivation-goal.cc.o
 16246 ms: build/src/libstore/liblixstore.so.p/store-api.cc.o
 14586 ms: build/src/nix/nix.p/.._nix-build_nix-build.cc.o
 13746 ms: build/src/libexpr/liblixexpr.so.p/eval.cc.o
 13287 ms: build/src/libstore/liblixstore.so.p/binary-cache-store.cc.o
 13263 ms: build/src/nix/nix.p/profile.cc.o
 12970 ms: build/src/nix/nix.p/develop.cc.o
 12621 ms: build/src/libfetchers/liblixfetchers.so.p/github.cc.o

**** Templates that took longest to instantiate:
 42922 ms: nlohmann::basic_json<>::parse<const char *> (69 times, avg 622 ms)
 32180 ms: nlohmann::detail::parser<nlohmann::basic_json<>, nlohmann::detail::i... (69 times, avg 466 ms)
 27337 ms: nix::HintFmt::HintFmt<nix::Uncolored<std::basic_string<char>>> (246 times, avg 111 ms)
 25338 ms: nlohmann::basic_json<>::basic_json (293 times, avg 86 ms)
 23641 ms: nlohmann::detail::parser<nlohmann::basic_json<>, nlohmann::detail::i... (69 times, avg 342 ms)
 20203 ms: boost::basic_format<char>::basic_format (492 times, avg 41 ms)
 17174 ms: nlohmann::basic_json<>::json_value::json_value (368 times, avg 46 ms)
 15603 ms: boost::basic_format<char>::parse (246 times, avg 63 ms)
 13268 ms: std::basic_regex<char>::_M_compile (28 times, avg 473 ms)
 12757 ms: std::__detail::_Compiler<std::regex_traits<char>>::_Compiler (28 times, avg 455 ms)
 10813 ms: std::__detail::_Compiler<std::regex_traits<char>>::_M_disjunction (28 times, avg 386 ms)
 10719 ms: std::__detail::_Compiler<std::regex_traits<char>>::_M_alternative (28 times, avg 382 ms)
 10508 ms: std::__detail::_Compiler<std::regex_traits<char>>::_M_term (28 times, avg 375 ms)
  9516 ms: nlohmann::detail::json_sax_dom_callback_parser<nlohmann::basic_json<... (69 times, avg 137 ms)
  9112 ms: std::__detail::_Compiler<std::regex_traits<char>>::_M_atom (28 times, avg 325 ms)
  8683 ms: std::basic_regex<char>::basic_regex (18 times, avg 482 ms)
  8241 ms: std::operator<=> (438 times, avg 18 ms)
  7561 ms: std::vector<boost::io::detail::format_item<char, std::char_traits<ch... (246 times, avg 30 ms)
  7475 ms: std::vector<boost::io::detail::format_item<char, std::char_traits<ch... (246 times, avg 30 ms)
  7309 ms: std::reverse_iterator<std::_Bit_iterator> (268 times, avg 27 ms)
  7131 ms: boost::stacktrace::basic_stacktrace<>::basic_stacktrace (246 times, avg 28 ms)
  6868 ms: boost::stacktrace::basic_stacktrace<>::init (246 times, avg 27 ms)
  6518 ms: std::reverse_iterator<std::_Bit_const_iterator> (268 times, avg 24 ms)
  5716 ms: std::__detail::_Synth3way::operator()<std::variant<nix::OutputsSpec:... (182 times, avg 31 ms)
  5303 ms: nix::make_ref<nix::SingleDerivedPath, nix::DerivedPathOpaque> (178 times, avg 29 ms)
  5244 ms: std::__uninitialized_move_a<boost::io::detail::format_item<char, std... (246 times, avg 21 ms)
  4857 ms: std::make_shared<nix::SingleDerivedPath, nix::DerivedPathOpaque> (178 times, avg 27 ms)
  4813 ms: std::__detail::_Synth3way::operator()<std::variant<nix::TextIngestio... (158 times, avg 30 ms)
  4648 ms: nlohmann::detail::json_sax_dom_callback_parser<nlohmann::basic_json<... (69 times, avg 67 ms)
  4597 ms: std::basic_regex<char>::basic_regex<std::char_traits<char>, std::all... (10 times, avg 459 ms)

**** Template sets that took longest to instantiate:
 55715 ms: std::__do_visit<$> (3603 times, avg 15 ms)
 47739 ms: std::__detail::__variant::__gen_vtable_impl<$>::__visit_invoke (11132 times, avg 4 ms)
 43338 ms: nlohmann::basic_json<$>::parse<$> (85 times, avg 509 ms)
 43097 ms: std::__detail::__variant::__raw_idx_visit<$> (2435 times, avg 17 ms)
 32390 ms: nlohmann::detail::parser<$>::parse (83 times, avg 390 ms)
 30986 ms: nix::HintFmt::HintFmt<$> (1261 times, avg 24 ms)
 30255 ms: std::__and_<$> (25661 times, avg 1 ms)
 29762 ms: std::unique_ptr<$> (2116 times, avg 14 ms)
 28609 ms: std::__tuple_compare<$>::__eq (2978 times, avg 9 ms)
 27560 ms: nlohmann::detail::parser<$>::sax_parse_internal<$> (167 times, avg 165 ms)
 27239 ms: std::variant<$> (1959 times, avg 13 ms)
 26837 ms: std::__invoke_result<$> (10782 times, avg 2 ms)
 25972 ms: std::tuple<$> (5714 times, avg 4 ms)
 24247 ms: std::__uniq_ptr_data<$> (2116 times, avg 11 ms)
 24061 ms: std::__result_of_impl<$> (9029 times, avg 2 ms)
 23949 ms: std::__uniq_ptr_impl<$> (2116 times, avg 11 ms)
 21185 ms: std::optional<$> (2502 times, avg 8 ms)
 21044 ms: std::pair<$> (4989 times, avg 4 ms)
 20852 ms: std::__or_<$> (24005 times, avg 0 ms)
 20203 ms: boost::basic_format<$>::basic_format (492 times, avg 41 ms)
 20184 ms: std::tie<$> (2895 times, avg 6 ms)
 19938 ms: nlohmann::basic_json<$>::create<$> (668 times, avg 29 ms)
 19798 ms: std::allocator_traits<$>::construct<$> (5720 times, avg 3 ms)
 19182 ms: std::__detail::__variant::_Variant_base<$> (1959 times, avg 9 ms)
 19151 ms: std::_Rb_tree<$>::_M_erase (2320 times, avg 8 ms)
 19094 ms: std::_Rb_tree<$>::~_Rb_tree (2022 times, avg 9 ms)
 18735 ms: nlohmann::basic_json<$>::basic_json (243 times, avg 77 ms)
 18546 ms: std::__detail::_Synth3way::_S_noexcept<$> (2542 times, avg 7 ms)
 17174 ms: nlohmann::basic_json<$>::json_value::json_value (368 times, avg 46 ms)
 17111 ms: nlohmann::detail::conjunction<$> (907 times, avg 18 ms)

**** Functions that took longest to compile:
  2091 ms: _GLOBAL__sub_I_primops.cc (../src/libexpr/primops.cc)
  1799 ms: nix::fetchers::GitInputScheme::fetch(nix::ref<nix::Store>, nix::fetc... (../src/libfetchers/git.cc)
  1388 ms: nix::Settings::Settings() (../src/libstore/globals.cc)
  1244 ms: main_nix_build(int, char**) (../src/nix-build/nix-build.cc)
  1021 ms: nix::LocalDerivationGoal::startBuilder() (../src/libstore/build/local-derivation-goal.cc)
   918 ms: nix::LocalStore::LocalStore(std::map<std::__cxx11::basic_string<char... (../src/libstore/local-store.cc)
   835 ms: opQuery(Globals&, std::__cxx11::list<std::__cxx11::basic_string<char... (../src/nix-env/nix-env.cc)
   733 ms: nix::daemon::performOp(nix::daemon::TunnelLogger*, nix::ref<nix::Sto... (../src/libstore/daemon.cc)
   589 ms: _GLOBAL__sub_I_tests.cc (../tests/unit/libutil/tests.cc)
   578 ms: main_build_remote(int, char**) (../src/build-remote/build-remote.cc)
   522 ms: nix::fetchers::MercurialInputScheme::fetch(nix::ref<nix::Store>, nix... (../src/libfetchers/mercurial.cc)
   521 ms: nix::LocalDerivationGoal::registerOutputs[abi:cxx11]() (../src/libstore/build/local-derivation-goal.cc)
   461 ms: nix::getNameFromURL_getNameFromURL_Test::TestBody() (../tests/unit/libutil/url-name.cc)
   440 ms: nix::Installable::build2(nix::ref<nix::Store>, nix::ref<nix::Store>,... (../src/libcmd/installables.cc)
   392 ms: nix::prim_fetchClosure(nix::EvalState&, nix::PosIdx, nix::Value**, n... (../src/libexpr/primops/fetchClosure.cc)
   390 ms: nix::NixArgs::NixArgs() (../src/nix/main.cc)
   388 ms: update(std::set<std::__cxx11::basic_string<char, std::char_traits<ch... (../src/nix-channel/nix-channel.cc)
   340 ms: _GLOBAL__sub_I_primops.cc (../tests/unit/libexpr/primops.cc)
   332 ms: nix::flake::lockFlake(nix::EvalState&, nix::FlakeRef const&, nix::fl... (../src/libexpr/flake/flake.cc)
   305 ms: _GLOBAL__sub_I_lockfile.cc (../src/libexpr/flake/lockfile.cc)
   300 ms: nix_store::opQuery(std::__cxx11::list<std::__cxx11::basic_string<cha... (../src/nix-store/nix-store.cc)
   296 ms: nix::parseFlakeRefWithFragment(std::__cxx11::basic_string<char, std:... (../src/libexpr/flake/flakeref.cc)
   289 ms: _GLOBAL__sub_I_error_traces.cc (../tests/unit/libexpr/error_traces.cc)
   278 ms: nix::ErrorTraceTest_genericClosure_Test::TestBody() (../tests/unit/libexpr/error_traces.cc)
   274 ms: CmdDevelop::run(nix::ref<nix::Store>, nix::ref<nix::Installable>) (../src/nix/develop.cc)
   269 ms: nix::flake::lockFlake(nix::EvalState&, nix::FlakeRef const&, nix::fl... (../src/libexpr/flake/flake.cc)
   257 ms: nix::NixRepl::processLine(std::__cxx11::basic_string<char, std::char... (../src/libcmd/repl.cc)
   251 ms: nix::derivationStrictInternal(nix::EvalState&, std::__cxx11::basic_s... (../src/libexpr/primops.cc)
   249 ms: toml::result<toml::basic_value<toml::discard_comments, std::unordere... (../src/libexpr/primops/fromTOML.cc)
   238 ms: nix::LocalDerivationGoal::runChild() (../src/libstore/build/local-derivation-goal.cc)

**** Function sets that took longest to compile / optimize:
 10243 ms: std::vector<$>::_M_fill_insert(__gnu_cxx::__normal_iterator<$>, unsi... (190 times, avg 53 ms)
  9752 ms: bool boost::io::detail::parse_printf_directive<$>(__gnu_cxx::__norma... (190 times, avg 51 ms)
  8377 ms: void boost::io::detail::put<$>(boost::io::detail::put_holder<$> cons... (191 times, avg 43 ms)
  5863 ms: boost::basic_format<$>::parse(std::__cxx11::basic_string<$> const&) (190 times, avg 30 ms)
  5660 ms: std::vector<$>::_M_fill_insert(std::_Bit_iterator, unsigned long, bo... (190 times, avg 29 ms)
  4264 ms: non-virtual thunk to boost::wrapexcept<$>::~wrapexcept() (549 times, avg 7 ms)
  4023 ms: std::_Rb_tree<$>::_M_erase(std::_Rb_tree_node<$>*) (1238 times, avg 3 ms)
  3715 ms: boost::stacktrace::detail::to_string_impl_base<boost::stacktrace::de... (166 times, avg 22 ms)
  3705 ms: std::vector<$>::_M_fill_assign(unsigned long, boost::io::detail::for... (190 times, avg 19 ms)
  3326 ms: boost::basic_format<$>::str[abi:cxx11]() const (144 times, avg 23 ms)
  3070 ms: void boost::io::detail::mk_str<$>(std::__cxx11::basic_string<$>&, ch... (191 times, avg 16 ms)
  2839 ms: boost::basic_format<$>::make_or_reuse_data(unsigned long) (190 times, avg 14 ms)
  2321 ms: std::__cxx11::basic_string<$>::_M_replace(unsigned long, unsigned lo... (239 times, avg 9 ms)
  2213 ms: std::_Rb_tree<$>::_M_get_insert_hint_unique_pos(std::_Rb_tree_const_... (203 times, avg 10 ms)
  2200 ms: boost::wrapexcept<$>::~wrapexcept() (549 times, avg 4 ms)
  2093 ms: std::vector<$>::~vector() (574 times, avg 3 ms)
  1894 ms: bool std::__detail::_Compiler<$>::_M_expression_term<$>(std::__detai... (112 times, avg 16 ms)
  1871 ms: int boost::io::detail::upper_bound_from_fstring<$>(std::__cxx11::bas... (190 times, avg 9 ms)
  1867 ms: boost::wrapexcept<$>::clone() const (549 times, avg 3 ms)
  1824 ms: std::_Rb_tree_iterator<$> std::_Rb_tree<$>::_M_emplace_hint_unique<$... (244 times, avg 7 ms)
  1821 ms: toml::result<$> toml::detail::sequence<$>::invoke<$>(toml::detail::l... (93 times, avg 19 ms)
  1814 ms: nlohmann::json_abi_v3_11_2::detail::serializer<$>::dump(nlohmann::js... (39 times, avg 46 ms)
  1799 ms: nix::fetchers::GitInputScheme::fetch(nix::ref<$>, nix::fetchers::Inp... (1 times, avg 1799 ms)
  1771 ms: boost::io::detail::format_item<char, std::char_traits<char>, std::al... (190 times, avg 9 ms)
  1762 ms: std::__detail::_BracketMatcher<$>::_BracketMatcher(std::__detail::_B... (112 times, avg 15 ms)
  1760 ms: std::_Function_handler<$>::_M_manager(std::_Any_data&, std::_Any_dat... (981 times, avg 1 ms)
  1733 ms: std::__detail::_Compiler<$>::_M_quantifier() (28 times, avg 61 ms)
  1694 ms: std::__cxx11::basic_string<$>::_M_mutate(unsigned long, unsigned lon... (251 times, avg 6 ms)
  1650 ms: std::vector<$>::vector(std::vector<$> const&) (210 times, avg 7 ms)
  1650 ms: boost::io::basic_altstringbuf<$>::overflow(int) (190 times, avg 8 ms)

**** Expensive headers:
178153 ms: ../src/libcmd/installable-value.hh (included 52 times, avg 3426 ms), included via:
  40x: command.hh 
  5x: command-installable-value.hh 
  3x: installable-flake.hh 
  2x: <direct include>
  2x: installable-attr-path.hh 

176217 ms: ../src/libutil/error.hh (included 246 times, avg 716 ms), included via:
  36x: command.hh installable-value.hh installables.hh derived-path.hh config.hh experimental-features.hh 
  12x: globals.hh config.hh experimental-features.hh 
  11x: file-system.hh file-descriptor.hh 
  6x: serialise.hh strings.hh 
  6x: <direct include>
  6x: archive.hh serialise.hh strings.hh 
  ...

173243 ms: ../src/libstore/store-api.hh (included 152 times, avg 1139 ms), included via:
  55x: <direct include>
  39x: command.hh installable-value.hh installables.hh 
  7x: libexpr.hh 
  4x: local-store.hh 
  4x: command-installable-value.hh installable-value.hh installables.hh 
  3x: binary-cache-store.hh 
  ...

170482 ms: ../src/libutil/serialise.hh (included 201 times, avg 848 ms), included via:
  37x: command.hh installable-value.hh installables.hh built-path.hh realisation.hh hash.hh 
  14x: store-api.hh nar-info.hh hash.hh 
  11x: <direct include>
  7x: primops.hh eval.hh attr-set.hh nixexpr.hh value.hh source-path.hh archive.hh 
  7x: libexpr.hh value.hh source-path.hh archive.hh 
  6x: fetchers.hh hash.hh 
  ...

169397 ms: ../src/libcmd/installables.hh (included 53 times, avg 3196 ms), included via:
  40x: command.hh installable-value.hh 
  5x: command-installable-value.hh installable-value.hh 
  3x: installable-flake.hh installable-value.hh 
  2x: <direct include>
  1x: installable-derived-path.hh 
  1x: installable-value.hh 
  ...

159740 ms: ../src/libutil/strings.hh (included 221 times, avg 722 ms), included via:
  37x: command.hh installable-value.hh installables.hh built-path.hh realisation.hh hash.hh serialise.hh 
  19x: <direct include>
  14x: store-api.hh nar-info.hh hash.hh serialise.hh 
  11x: serialise.hh 
  7x: primops.hh eval.hh attr-set.hh nixexpr.hh value.hh source-path.hh archive.hh serialise.hh 
  7x: libexpr.hh value.hh source-path.hh archive.hh serialise.hh 
  ...

156796 ms: ../src/libcmd/command.hh (included 51 times, avg 3074 ms), included via:
  42x: <direct include>
  7x: command-installable-value.hh 
  2x: installable-attr-path.hh 

150392 ms: ../src/libutil/types.hh (included 251 times, avg 599 ms), included via:
  36x: command.hh installable-value.hh installables.hh path.hh 
  11x: file-system.hh 
  10x: globals.hh 
  6x: fetchers.hh 
  6x: serialise.hh strings.hh error.hh 
  5x: archive.hh 
  ...

133101 ms: /nix/store/644b90j1vms44nr18yw3520pzkrg4dd1-boost-1.81.0-dev/include/boost/lexical_cast.hpp (included 226 times, avg 588 ms), included via
:
  37x: command.hh installable-value.hh installables.hh built-path.hh realisation.hh hash.hh serialise.hh strings.hh 
  19x: file-system.hh 
  11x: store-api.hh nar-info.hh hash.hh serialise.hh strings.hh 
  7x: primops.hh eval.hh attr-set.hh nixexpr.hh value.hh source-path.hh archive.hh serialise.hh strings.hh 
  7x: libexpr.hh value.hh source-path.hh archive.hh serialise.hh strings.hh 
  6x: eval.hh attr-set.hh nixexpr.hh value.hh source-path.hh archive.hh serialise.hh strings.hh 
  ...

132887 ms: /nix/store/h2abv2l8irqj942i5rq9wbrj42kbsh5y-gcc-12.3.0/include/c++/12.3.0/memory (included 262 times, avg 507 ms), included via:
  36x: command.hh installable-value.hh installables.hh path.hh types.hh ref.hh 
  16x: gtest.h 
  11x: file-system.hh types.hh ref.hh 
  10x: globals.hh types.hh ref.hh 
  10x: json.hpp 
  6x: serialise.hh 
  ...

  done in 0.6s.

Manually looking at traces

Note that the summary in the report can miss details like why one particular header is bad; to find that out, use a trace viewer to inspect the JSON trace file; we suggest rg -t json -uu error\.hh build/ | less to find some .cc trace that the bad header (in this example, error.hh) appears in.

BuildsYou withcan clanglook andat mesonindividual arefile profiledtraces by default,opening because we want to encourage improving the build time.

If you want to stare down a specificsome file harder, look inlike build/src/libcmd/liblixcmd.so.p/command.cc.json for the relevant json file and import it toin https://ui.perfetto.dev or another Chrome-trace-json compatible trace viewer like Speedscope.

This will produce a flamegraph of the trace (screenshot shows Perfetto):

The most general spans of compile time are at the top, and the constituent spans are shown as you go down.

Successful build time reduction CLs

Back to top