Wikipedia:Reference desk/Archives/Computing/2019 March 6

Welcome to the Wikipedia Computing Reference Desk Archives
The page you are currently viewing is a transcluded archive page. While you can leave answers for any questions shown below, please ask new questions on one of the current reference desk pages.

March 6

UNIX/MacOS make and timestamps

I have a reasonably complex project with multiple directories each providing a sub-library by linking a couple of object files with ar rcs NAME.a file1.o file2.o [...]. I recently found that I get unnecessary partial rebuilds for some of these libraries at least on my development computers (which run macOS Mojave - 10.14.3). My suspicion is that with a modern CPU and on a fast SSD, the build process is so fast that the .a file has the same time stamp as the .o files it is build from - so to be on the safe side, make decides to rebuild. This suspicion seems to be confirmed, because when I added a sleep 1 in front of each ar call, the unnecessary rebuilds stopped. But this is a quite inelegant solution (it adds 10 seconds to a full rebuild - not crippling, but annoying). Does anyone have an idea that might be useful? --Stephan Schulz (talk) 08:16, 6 March 2019 (UTC)

Compile everything, touch the filestamps on all the compiled object files to make them a little later?

It's a kludge, but it's a kludge per build, not per compile. Andy Dingley (talk) 09:10, 6 March 2019 (UTC)

Timestamp granularity is a property of the filesystem (the underlying BSD kernel API timespec gives μs resolution). This discussion gives granularity for various Linux filesystems and NTFS, and this says HFS+ uses 1 second granularity. My vague recollection is that FAT uses 2 second granularity. I appreciate it's not ideal (and probably rather slow) but I wonder how you'd get on if you mount an EXT4 (loopback) via FUSE and build in that. -- Finlay McWalter··–·Talk 09:12, 6 March 2019 (UTC)

How much time do the unnecessary rebuilds add if you don't do anything to stop them? --76.69.46.228 (talk) 11:19, 6 March 2019 (UTC)

@Stephan Schulz: The effect you describe adds at most one unnecessary build of every dependant, multiplied by the depth of dependency.

• Suppose you modify a source. On the first make the source is compiled, then the library updated, and final target linked.
• On the second make the library appears to have the same timestamp as the object file, so it is (unnecessarily) rebuilt, then the target re-linked.
• On the third make the object file has a timestap from the first make, and the library – from the second one, so the library is not updated. Just the final target may be relinked, if the linking phase in the second attempt was done within a single system time tick.
• From now on, no additional actions will be performed in subsequent makes (until some is necessary due to some source change).

As a result, every lib file gets at most one additional update per each necessary one, and the executable will be linked at most two times unnecessarily after one really necessary link. Of course that holds if you run your making script without a reason. If you run it after changing some sources only, then at least one lib and the exe shall be rebuilt on reason, hence only some part of the make-time is actually a waste. --CiaPan (talk) 13:12, 6 March 2019 (UTC)

Yes, you are right. What irritates me is less the extra build time (though that is annoying), but the fact that everytime I run make, I wonder if I (or some other contributor) has messed up the dependencies. On a purely pragmatic level, I could probably leave the system as is...but I'd like to be MORE HAPPY ;-). --Stephan Schulz (talk) 16:26, 6 March 2019 (UTC)

• This effect can have a significantly bad impact on compile speed, if it approaches an $n^{2}$ number of compiles.

The problem is that the .a files are auto-generated. Thus the .o timestamp can't 'get away from them' (as we need for an efficient make).

In most cases, that gives $n$ compiles, which might not have been needed (but $n$ is usually manageable). However we rely on the second compile attempt of the .o seeing the .a as 'old' and unchanged. If the compile process is a bit on the dumb side (i.e. it's hand-written in Ant, Maven or shell, not merely running the compiler against existing files) then it's possible the .a gets recreated again too – and the whole then performance keeps repeating.

If you're doing this across an industrial scale build, as part of continuous integration or continuous testing (so the developers are hanging on the cycle time), then it's a problem. I've only seen it once (to a pathological level) and that was because something that should have used 'make' logic to auto-generate a header file, then compile, was doing it as a simple batch. The solution was (as always recommended) to shoot the guilty developer pour encourager les autres and to first fix that, then stop auto-generating anything claiming to be 'source'. Andy Dingley (talk) 17:25, 6 March 2019 (UTC)

It's hardly an explanation of any weird behaviors of the file-system, but the canonical advice from Apple is that you should use Xcode for dependency management... Tech Note 2339, (regrettably, even this documentation is "archived"). Xcode's graphical UI and its command-line tool 'xcodebuild' are capable of doing things more efficiently than Make, at the expense of portability. But, like this other archived official Apple documentation says, "you might consider making architectural changes to make your port (and future ports) more maintainable...."

GNU make provides official documentation (§4.3) for a type of prerequisite called an order-only, for the express purpose of working around filesystem modification timestamps. You can, in principle, rewrite your Makefile so that its dependencies are structured in that manner - at the risk of accidentally breaking the dependency tree.

In my opinion, one of the greatest deficiencies of GNU make - on macOS or any other linux/unix-like system - is that its dependency database depends on file timestamps, instead actually keeping track of the build-records. This is about the strongest argument that I can come up with for migrating to a newer-generation build management system, although in truth, I still use Make for a lot of my projects, because it is simple, portable, and largely free of other bugs.

Nimur (talk) 19:38, 6 March 2019 (UTC)

I'm not building an Apple "App", I'm building a scientific application suite that outside of the development environment mostly runs on Linux (often in clusters), and that some people even compile for Windows (usually using Cygwin). Many of my collaborators are developing on Linux, so portability is a very important trait. I also decided quite a while back not to rely on proprietary software for serious work - I'm only using MacOS as a convenient UNIX that Apple maintains, and that runs on some of the best (or at least reliably good) hardware (ok, and to load my iPod Nano ;-). --Stephan Schulz (talk) 20:47, 6 March 2019 (UTC)

@Stephan Schulz: I think I can see the solution. Not a pretty one, but hopefully it can work. Basically it is just what you did, i.e. adding a delay before building each .a file (or any intermediate file in the dependency tree). The difference is in making the delay as short as possible while keeping it as long as necessary.

IMHO this can be achieved with U*x shell tools only. Here is the outline (be warned, however, I had no opportunity to test it myself!): create a temporary file A; create a temporary file B; loop: test if B is newer than A; if so, break the loop; otherwise touch B and continue looping.

This makes a loop to run until the file-system timestamp tick elapses, thus ensuring a newly compiled/linked file will get a timestamp newer than its sources at least by one (in filesystem's timestamp atomic unit).

See https://unix.stackexchange.com/questions/181937/how-create-a-temporary-file-in-shell-script for a description of safe creating a unique temporary file. As a bonus, the answer https://unix.stackexchange.com/a/181938 describes how to assure deletion of the temp files on an unexpected script termination.

The find tool can be used for the central test: find tmp-folder-path-here -type f -name B -newer A. You'll need also a -maxdepth 0 option to make sure find will not recur into subdirectories. For testing the result you can e.g. filter the output with wc to determine the number of text lines (which should be a number of files found, hence either 0 or 1 ...or 2 if the terminating newline is counted as a start of one more line?). You can also use the --count option, which instructs the find tool to output just a number of files matched.

The textual output can be catched into a string by bash reverse quotes `...` and then compared to an expected value with if.

Hope that helps. --CiaPan (talk) 08:55, 7 March 2019 (UTC)

Well, it reminds me a bit of Rube Goldberg and this xkcd - and possibly also this one . My current fix is AR = sleep 1;ar rcs in the main include file for all makefiles. It has the charm of simplicity ;-). --Stephan Schulz (talk) 09:45, 7 March 2019 (UTC)

@Stephan Schulz:
You can invoke a nap-script precisely the same way as you invoke the sleep command. My aim, however, was saving your build-time, not hiding a call. --CiaPan (talk) 10:36, 7 March 2019 (UTC)

Yes. My instinctive reaction is against busy-waiting, but it is a viable approach.... --Stephan Schulz (talk) 08:20, 8 March 2019 (UTC)

Say, does "&" work inside (all relevant versions of) make? If so, then you could use command lines like constructfoo.o foo.o; (sleep 1; touch foo.o)& and it would not add appreciably to the overall time. Just a quick thought; I'm not even going to try it on a simple case. --76.69.46.228 (talk) 12:43, 8 March 2019 (UTC)

I don't see why background execution should not work - IIRC, standard behaviour for make is to execv() a new instance of /bin/sh for each line of build instructions. So whatever works in the shell should work in make, given enough quoting and escapes. But that might miss a real dependency if e.g. the programmer interrupts the compilation and retriggers it after fixing a syntax error). --Stephan Schulz (talk) 15:09, 8 March 2019 (UTC)

Let's back up here for a minute. We're all taking it for granted that "with a modern CPU and on a fast SSD, the build process is so fast that the .a file has the same time stamp as the .o files it is build from - so to be on the safe side, make decides to rebuild." But let's face it, computers have always been fast. In fact, I remember many years ago -- this would have been on a PDP-11 -- that source files and object files sometimes had the same mtime, and I wondered if that meant that make would needlessly remake things. But I discovered that it did not, and from this I concluded that somewhere deep inside, make must be using >, not >=, when comparing source and object mtimes and deciding whether to rebuild.

So has make changed at some point over the years? I don't believe that it has. Just now I made a few quick tests with a couple of my make-built programs, artificially setting the mtime of a .o file to be the same as its source .c file, or flipped the other way around, the same as an .a archive built from it. As I expected based on my experience of years ago (but in contradiction to Stephan Schulz's recent experience) I did not see any "unnecessary" recompilations.

(And before anyone asks, I'm testing this on a Mac, too.)

So, Stephan, let me ask: what kind of make rule are you using to construct .a's from .o's? Perhaps there's something else wrong, having nothing to do with mtimes and fast computers. —Steve Summit (talk) 16:16, 8 March 2019 (UTC)

Hi Steve!

Below is the makefile in question, where AR was defined asAR = ar rcs (which showed the behaviour) and AR = sleep 1;ar rcs now (which does not). The definitions are in ../Makefile.vars, which is too large (and ugly) to post here, but contains nothing active.

Extended content

#------------------------------------------------------------------------ # # File  : Makefile for BASICS.a library of generic data types and # algorithms. # # Author: Stephan Schulz # # Created: Sun Jul 6 22:55:11 MET DST 1997 # #------------------------------------------------------------------------ include ../Makefile.vars # Project specific variables PROJECT = BASICS LIB = $(PROJECT).a all: $(LIB) depend: *.c *.h $(MAKEDEPEND) # Remove all automatically generated files clean: @rm -f *.o *.a # Services (provided by the master Makefile) include ../Makefile.services # Build the library BASIC_LIB = clb_error.o clb_memory.o clb_os_wrapper.o \ clb_dstrings.o clb_verbose.o\ clb_stringtrees.o clb_numtrees.o clb_numxtrees.o \ clb_floattrees.o clb_pstacks.o\ clb_pqueue.o clb_dstacks.o clb_ptrees.o clb_quadtrees.o\ clb_regmem.o\ clb_objtrees.o clb_fixdarrays.o\ clb_plist.o clb_pdarrays.o clb_pdrangearrays.o \ clb_ddarrays.o clb_sysdate.o \ clb_intmap.o \ clb_simple_stuff.o clb_partial_orderings.o \ clb_plocalstacks.o $(LIB): $(BASIC_LIB) $(AR) $(LIB) $(BASIC_LIB) include Makefile.dependencies

I have to wonder: Is execv() the magic phrase to conjure you? I didn't even write it thrice is caps... ;-) --Stephan Schulz (talk) 17:24, 8 March 2019 (UTC)

The big difference in my Makefiles is that I always use variations on the theme of

libdb.a: $(DBOBJS) $(DBFILEOBJS)
    ar r $@ $?
    ranlib $@

(note use of $?). I wouldn't have expected that difference to cause your problem, though.

Can you demonstrate the problem when using a single Makefile in a single directory, or does it only involve your more elaborate, cascaded invocation?

(And yes, as a matter of fact there was some magic involved in drawing my attention to this thread, but I'm not at liberty to disclose it. :-) ) —Steve Summit (talk) 18:37, 8 March 2019 (UTC)

Yes, the problem occurs even if I just run make in the subdirectory:

Extended content

schulz@giordano.fritz.box 7:48pm [BASICS] make clean schulz@giordano.fritz.box 7:50pm [BASICS] make gcc -O3 -fomit-frame-pointer -fno-common -Wall -DNDEBUG -DFAST_EXIT -DPRINT_SOMEERRORS_STDOUT -DMEMORY_RESERVE_PARANOID -DPRINT_TSTP_STATUS -DSTACK_SIZE=32768 -DCLAUSE_PERM_IDENT -DTAGGED_POINTERS -DEXECPATH=/Users/schulz/tmp/E/PROVER -std=gnu99 -I../include -c -o clb_error.o clb_error.c [...] gcc -O3 -fomit-frame-pointer -fno-common -Wall -DNDEBUG -DFAST_EXIT -DPRINT_SOMEERRORS_STDOUT -DMEMORY_RESERVE_PARANOID -DPRINT_TSTP_STATUS -DSTACK_SIZE=32768 -DCLAUSE_PERM_IDENT -DTAGGED_POINTERS -DEXECPATH=/Users/schulz/tmp/E/PROVER -std=gnu99 -I../include -c -o clb_plocalstacks.o clb_plocalstacks.c ar rcs BASICS.a clb_error.o clb_memory.o clb_os_wrapper.o clb_dstrings.o clb_verbose.o clb_stringtrees.o clb_numtrees.o clb_numxtrees.o clb_floattrees.o clb_pstacks.o clb_pqueue.o clb_dstacks.o clb_ptrees.o clb_quadtrees.o clb_regmem.o clb_objtrees.o clb_fixdarrays.o clb_plist.o clb_pdarrays.o clb_pdrangearrays.o clb_ddarrays.o clb_sysdate.o clb_intmap.o clb_simple_stuff.o clb_partial_orderings.o clb_plocalstacks.o schulz@giordano.fritz.box 7:50pm [BASICS] make ar rcs BASICS.a clb_error.o clb_memory.o clb_os_wrapper.o clb_dstrings.o clb_verbose.o clb_stringtrees.o clb_numtrees.o clb_numxtrees.o clb_floattrees.o clb_pstacks.o clb_pqueue.o clb_dstacks.o clb_ptrees.o clb_quadtrees.o clb_regmem.o clb_objtrees.o clb_fixdarrays.o clb_plist.o clb_pdarrays.o clb_pdrangearrays.o clb_ddarrays.o clb_sysdate.o clb_intmap.o clb_simple_stuff.o clb_partial_orderings.o clb_plocalstacks.o

...and the Makefile is really not that complex. My make is GNU Make 4.2.1. Maybe its coincidence (and I have no good records), but I think the weird behaviour only started after upgrade to macOS High Sierra and APFS. I've just checked on CentOS (in VirtualBox on the same hardware), and that also does not show this behaviour. --Stephan Schulz (talk) 19:04, 8 March 2019 (UTC)

I don't know anything abut APFS (hadn't even heard of it until just now), but I have a hunch it must have something to do with the problem. Somehow or another, it sounds like one or more of the .o files is ending up with, not the same timestamp as the .a, but slightly in the future.

I'd consider using hdiutil to make a small virtual HFS disk, copying your project to it, and trying it there. —Steve Summit (talk) 19:18, 8 March 2019 (UTC)

APFS made a mockery of file-timestamps and read/write-ordering to files on "disk"? Fascinating! An oft-quoted engineer once wrote, "It's hardly an explanation of any weird behaviors of the file-system, but...the canonical advice from Apple is that you should use Xcode for dependency management..." Nimur (talk) 21:34, 8 March 2019 (UTC)

Well, looking at the time stamps on APFS with stat, at least one of the .o files always shows exactly the same age as the .a file after the first make. On Linux, they are all younger (and the time stamp has a lot higher resolution). So this really might be Apple APFS weirdness. Thanks! --Stephan Schulz (talk) 21:57, 8 March 2019 (UTC)

Well, the problem is not really that "at least one of the .o files always shows exactly the same age as the .a file". If that's all that's happening, make will not rebuild. You can test this yourself, perhaps by using touch -r file.a file.o to set file.o's mtime to exactly match file.a's.

My suspicion -- although this hypothesis definitely has some holes in it -- is that APFS records subsecond timestamps, that your stat program doesn't reveal them, that make somehow does see them, and that at least one of the .o files somehow ends up with an mtime slightly newer than the .a file. —Steve Summit (talk) 11:53, 9 March 2019 (UTC)

Yes, you are right - when I force the .o files to the age of the .a file, no rebuild happens. Nimur might be on to something with APFS. macOS stat shows me only centiseconds, Linux shows nanoseconds. --Stephan Schulz (talk) 13:13, 9 March 2019 (UTC)

And more weirdness...my VirtualBox CentOS mounts my real macOS user directory as an external file system. If I compile from virtual Linux on the APFS installation, the effect is not there. --Stephan Schulz (talk) 14:16, 10 March 2019 (UTC)

Stephan, I feel that you may have encountered a true bug or software defect, but we should be very careful about the investigation and I should not encourage anyone to jump to conclusions. The defect could be in many places, including inside the gcc distribution, or inside your source-code; but it could also be a defect in the file-system or in some other Apple-provided software. The correct course of action is to officially report it to Apple so that the right software engineers may investigate it completely.

Do you have a developer account with Apple? If you do not, you can create one at developer.apple.com. You can file a bug-report, and at the very least, there will be an official channel for formal investigation.

If you would not like to proceed in that official fashion, I can attempt to reproduce the bug and file a report on your behalf, but anything you can do to make it easier to reproduce would be appreciated. Specifically: do you know the exact versions of software you are using? (What are the exact versions of macOS, and of gcc and binutils that you have installed? Where have they come from and how did you install them?)

If you are able to create a case that reproduces this problem, without sharing any proprietary source-code, can you upload those as an archive so I can try to reproduce and forward a bug-report to our development tools team?

Nimur (talk) 17:49, 11 March 2019 (UTC)

Well, the software in question is my theorem prover E, available at https://www.eprover.org and at https://www.github.com/eprover, depending on wether you prefer the point release or the current development branch. Just follow the README.md for "Simplest installation" and see if after "./configure; make" a second make will rebuild something. But checking versions gave me another new bit of information. My default make is /sw/bin/make (GNU Make 4.2.1), installed via fink. If I run /usr/bin/make (which is really /Applications/Xcode.app/Contents/Developer/usr/bin/make (GNU Make 3.81), the effect again vanishes. I have a basic Apple developer account (from the time you needed that to get Xcode), but I'm not sure Apple is interested in a bug that happens only with a third-party tool. --Stephan Schulz (talk) 23:55, 11 March 2019 (UTC)

Thank you Stephan, I will try to reproduce and will let you know if I make progress.

In the meantime: please consider using the command-line tools that are distributed inside Xcode, instead of the ones distributed via fink. You may need to slightly modify your Makefile: among your options, you can replace gcc with cc (to use clang as your compiler); and you can use llvm ld to replace the two-stage ar and ranlib for the concatenation and packaging of .a files. Suffice to say that Apple's officially-released versions of important tools like make are not identical to the ones you find on other websites or package-managers. (Fun fact: make is made with Xcode; and Xcode is made with Xcode; as a theoretical computer scientist, you surely appreciate a conundrum. This is no ordinary bug: "Perhaps it is more important to trust the people who wrote the software.")

I recognize that this isn't a fix for your problem; but at least by using officially-supported tools, there will be fewer strange behaviors when you develop and build on Apple platforms. Meanwhile I will see if I can find anything obviously-wrong in the original set-up.

Nimur (talk) 16:42, 12 March 2019 (UTC)