Closed Bug 15241 Opened 25 years ago Closed 24 years ago

Method to reduce disk space requirements for build

Categories

(SeaMonkey :: Build Config, enhancement, P3)

x86
Linux
enhancement

Tracking

(Not tracked)

RESOLVED WONTFIX
Future

People

(Reporter: jim_nance, Assigned: cls)

References

Details

Attachments

(5 files)

The make utility is able to keep archive libraries up to date w/o requiring the .o files to be kept around. The attached patch demonstrates this and cuts ~150M from the disk space requirements for building mozilla. The patch is proof of concept quality. There may be issues that need to be solved/discovered before it goes into the main code.
Attached patch example patch (deleted) — Splinter Review
Ok, after some testing, here are the main problems: It bombs severely on parallel builds. Like the GNU make info page warns, it cal ls ar for the archive concurrently which causes the archive to be missing some s ymbols. When you remove a file from the build list, you must remove the ar chive and rebuild it from scratch. If you build a file using some method other than the implicit make rule, the .o will remain instead of being removed. I upgraded to GNU make 3.78.1 (which has some -jN fixes) but it didn't help. Which is a shame because I really like this patch. I'm going to do some more digging to see how hard it would be to turn off parallel builds on a per directory/target basis or (as the info page suggests) hack make to serialize all commands that operate on the same archive.
OK, I think I have an idea on how to improve the way this works. Make inserts the .o files into the .a file as soon as it generates the .o, but it does not delete any .o files until all the necessary .o files have been inserted into the archive. If we change the rule for adding a single .o file into an archive so that it does nothing, and then add all the .o files in at the same time it will be both faster and parallel safe. I dont have a patch for mozilla yet, but I did create an example Makefile that does this. It is attached.
I attached the file. Its a gzipped tar file, but had trouble telling bugzilla this.
Here is a new patch for config/rules.mk. It should be faster and hold up against parallel builds.
Found some problems with the improved method. It does not work well when source files change. The generated .o files do not get inserted into the archive. Investigating.
OK, I think this has all the features we wanted, and it seems to work too, which is always a plus :-) We should now be able to: 1) Run parallel makes 2) Run as fast as the old method 3) Remove .o files from the archive which have been removed from the Makefile Patch attached
Target Milestone: M11
Just one little tweak needed to fix the problem I found. If you removed a file from the filelist but did not touch any other file, it would not be removed from the library. Change $(LIBRARY): $(LIBRARY)($(OBJS) $(LOBJS)) $(LIBRARY:%.a=.%.timestamp) to $(LIBRARY): $(LIBRARY)($(OBJS) $(LOBJS)) $(LIBRARY:%.a=.%.timestamp) Makefile Makefile.in And I think we're set. A time comparison showed almost no performance hit. cls@amadeus:cls> tail -2 test-with-03 2566.90user 307.36system 55:31.03elapsed 86%CPU (0avgtext+0avgdata 0maxresident)k 0inputs+0outputs (4270596major+2851241minor)pagefaults 611swaps cls@amadeus:cls> tail -2 test-without-03 2582.87user 308.16system 54:56.31elapsed 87%CPU (0avgtext+0avgdata 0maxresident)k 0inputs+0outputs (4066718major+2814688minor)pagefaults 0swaps cls@amadeus:cls>
I tested the patch on Solaris. It works there too, and cuts 200M from the build :-)
Status: NEW → RESOLVED
Closed: 25 years ago
Resolution: --- → FIXED
Blocks: 16049
Status: RESOLVED → REOPENED
Dependencies do not work. After an afternoon of unsuccessful attempts of making dependencies work w/o breaking parallel builds, I'm making the feature a build option and reopening this bug. Ramiro, do you have any further details on the problem that you mentioned on irc? <ramiro> c++: installation problem, cannot exec `as': Connection refused <ramiro> make[4]: *** [nsHTMLSpanElement.o] Error 1 <ramiro> c++: installation problem, cannot exec `as': Connection refused <ramiro> make[4]: *** [nsHTMLSelectElement.o] Error 1 <ramiro> c++: installation problem, cannot exec `as': Connection refused <ramiro> make[4]: *** [nsHTMLStyleElement.o] Error 1
Resolution: FIXED → ---
Clearing Fixed resolution due to reopen.
Target Milestone: M11 → M13
Target Milestone: M13 → M14
moving out to m14. better to do this at the start of milestones.
Target Milestone: M14 → M20
mass re-assign of all bugs where i was listed as the qa contact
QA Contact: cyeh → chofmann
Target Milestone: M20 → Future
I don't think this is going to work for us. Researching another solution. Marking WONTFIX.
Status: REOPENED → RESOLVED
Closed: 25 years ago24 years ago
Resolution: --- → WONTFIX
Product: Browser → Seamonkey
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: