Not sure this is a bug but: ftp://ftp.mozilla.org/pub/mozilla/nightly/2000-09-20-13-M18/ mozilla-i686-pc-linux-gnu.tar.gz has grown with 3.5 MB since the previous build. At 8 o'clock today it was 8297 Kb. 5 hours later it had grown to 11764 Kb. I downloaded mozilla-i686-pc-linux-gnu-installer.tar.gz from the same dir but it and it unpacked fine but when running ./mozilla-installer it doesn't want to do much; terminates after only seconds with error "Fatal Error [-615] download failed"

Are we picking up both the jars and the flat files?

yes, we pick up both flat files and the jar files, because they're both in dist. The .zip files and the .sit files and the .tar.gz files are generally just what's in dist. If people want to reduce the amount of space in their download, the installer packages are the best way to go. Should we be making tar.gzs that have just what's in the packages-unix file, a la the installer? It seems like leaving the jars exploded will help xul/js developers, but maybe we can just say ``unzip these, if you like''

Since the jar files now contain the same structure as the flat files, I think we should be able to tell chrome developers to just unzip the jar files to do their work. Everyone downloading the build shouldn't have to pay that penalty. We still need a utility to convert the install-chrome.txt format back & forth though. Warren?

I want to make the browser do the translation based on an env var. That will be easiest. And you're right, we could delete everything in chrome except the jar files, then tell developers to unzip them if they want. Maybe the tarball script could do something like this: mkdir dist/bin/tmp-chrome mv dist/bin/chrome/*.jar dist/bin/tmp-chrome rm -rf dist/bin/chrome mv dist/bin/tmp-chrome dist/bin/chrome <tar>

we shouldn't be putting build funkiness in the automation that everyone is going to run into. this should be in the makefiles to do this if you're building with jar packaging enabled. if you don't want it in the tarball, make sure it isn't in dist.

Leaf should probably mark this WONTFIX then.

the build automation isn't going to do stuff like this for the tar.gz; installer builds have the jars-only, and can be used to save bandwidth. you should be able to use the sea if the linux installer is failing.

verified.

*** Bug 58185 has been marked as a duplicate of this bug. ***

> we shouldn't be putting build funkiness in the automation that everyone is > going to run into It is not everyone - it's only people creating a package. And they want to cerate a small package, not? Otherwise, maybe we can also pack in some nice images, so the user has fun right from the start? > the build automation isn't going to do stuff like this for the tar.gz Why? > installer builds have the jars-only, and can be used to save bandwidth. You're kidding, right? Why do you provide the tarball at all, then? REOPENing. Taking bug. I need this for my own distro.

Updating SUMMARY.

the tarball is provided so that people can see a full mozilla/dist/bin directory after a successful build. It's purpose in life is to reflect the state of the build. There have been many times when mozilla runs out of dist/bin and didn't run from the installer build. Having this tarball gives us something to compare to the installer build to find the problem. Hence my comment, if people don't want duplicate chrome in dist/bin then the jar packaging process should delete the unjar'ed chrome from dist/bin when it is packaged. We should not be hacking the xpinstall/packager makefile to do this as part of the packaging. Packaging just packages what is there, it shouldn't change what's there.

Jon's right. For example, will viewer work without the unpacked chrome files?

granrose, > the jar packaging process should delete the unjar'ed chrome from dist/bin when > it is packaged. warren said, this were too dangerous. > We should not be hacking the xpinstall/packager makefile to do this > as part of the packaging. Why not? > Having this tarball gives us something to compare > to the installer build to find the problem. The tarball is an install option of its own for Mozilla. It is not just a way to debug the installer. It is the preferred way for many people to install Mozilla. On Unix, the usual way to install a package is one of the following: 1. Get it with your distribution 2. rpm -i package.rpm, dpkg --install package.deb or apt-get install package 3. Get binary tarball, unzip, [install manually,] run 4. Get source tarball, |./oncfigure;make;su;make install| (or similar) None of tehse options includes a graphical installer. Such a thing is nice to have, but many people don't like it. So, the tarball is a tier-one install option, not just a way to debug the installer. I can see the need for a tarball that has everything: test apps etc.. But many testers don't need this. Why should they download it every day? Also, thr purpose of mozilla.org is to provide resources for Mozilla distributors. Mozilla distributors (e.g. Linux distributors, standalone distributors like Netscape and competetors etc.) want to keep the download size minimal, and their users don't need unjared chrome files or test programs. We should provide a package option for them, otherwise they have a hard time to figure out what exactly is optional and what not (which costs testing etc.). Also, as outlined above, they might not want a graphcial installer. So, if you really need to distribute a full dist/bin dir, then provide an additional option to make a minimal tarball. Or, rather, let me provide such an option. If mozilla.org distributed such tarballs is not subject of this bug, but there is a lot of demand for it.

I'm not a mozilla person so my opinion isn't final. However, I think the tarball of dist/bin should be the default package since the default user of mozilla is an engineer, not an end user. However, if you want to be able to ship builds to end users without all the extra cruft, I'm all for that, as long as you do not delete anything from dist/bin as part of the packaging. If you want to package builds for end users in a .tar.gz format, you should run pgkcp.pl with the packages-unix file against dist/bin to create all the various component directories (browser, mail, etc.), then tar all the subdirectories of those directories into one big tarball (probably using tar u or tar -A). Then you ship that tarball and that tarball will contain only the files from dist/bin that are shipped with the linux installer bits. This removes much more cruft from the tarball and is much more enduser friendly as it is like an installer build for any unix platform with Perl5 without having to deal with the installer. I've thought of doing this for a while now, but haven't had time to play with it pkgcp.pl and packages-unix are both in xpinstall/packager are are fairly self-explanatory. Let me know if any of this is unclear.

I have yet to see a valid reason why we have a duplicate 14 megs of chrome files in our final distribution. Viewer works fine with just the .jar files. If we have any test programs that don't work with just .jar files, then that's a bug and they need to be fixed. Building Mozilla isn't rocket science and we shouldn't have to jump thru these post-build hoops to get an end-user installation. We do not have installers for every platform and and even if we did, the default build should Do the Right Thing(tm). In this case, that is removing the chrome files after the jars have been created.

I agree, we should be removing the chrome directories from dist/bin after they are jarred as part of the jarring process. However, aside from that there is still a lot of cruft in dist/bin (tools, tests, etc.) that non-developer end users don't want or need. So having a packaging target that creates the installerless tarball equivalent of the installer build via the packages-unix file is a good thing (albeit possibly a different bug) for those platforms that don't have installers.

> I agree, we should be removing the chrome directories from dist/bin after they > are jarred as part of the jarring process. warren? > However, aside from that there is still a lot of cruft in dist/bin (tools, > tests, etc.) (Note that you can avoid most of that by enabling various build options, e.g. |--disable-tests|.) granrose, thanks for the tip for the perl packager. I will look at it. I didn't hear of it as yet -> it needs to be documented better, probably in <http://www.mozilla.org/build/distribution.html>*. Because that doc says "Create the package [...] cd mozilla/xpinstall/packager gmake", that's why I thought, this were the official/right way to create a distribution. *And we should link to <http://www.mozilla.org/build/distribution.html> from <http://www.mozilla.org/build/unix-details.html>, under "Other Unix Build pages". leaf, can you add that link, please?).

the tarballs are provided to give a snapshot of what's in dist. It's possible to use the manifests to get only what the installer gets, and package the result. This should probably be added as a new target.

cls: Patch looks good, except I think we need to make it controllable from the command line somehow -- the opposite of the $flatfilesonly flag. The original idea was to allow users to easily flip over to chrome development by changing their installed-chrome.txt to refer to the flat files instead of the jar files without a rebuild. Then someone came along and wanted us not to create the jar files (forget why) which lead to the $flatfilesonly flag. Now we have the opposite request, effectively $jarfilesonly. I think we should have both by default, and environment vars / config options to disable one or the other selectively.

leaf: see bug 56601 about providing a real install target for mozilla warren: That patch only removes the chrome files. To remove the extra directories would take some locking magic to avoid race conditions...unless we used a separate temp directory local to the build dir for each jar file. I thought we were going to provide a script that would allow the chrome developers to flip from using the jar files to using flatfiles? If so, couldn't that same script unzip the jar files when it updates installed-chrome.txt? And also, zip the jarfiles back up (and remove the flat files) when it switched back? Although a build/env config option may eventually be necessary for the build system to handle this, I don't see any inherent advantage to having both sets of files by default if we have the script to make the conversions for us.

Well, the real idea was to make the chrome registry decide at runtime whether to grab the files out of a jar, or out of a directory -- without modification to the installed-chrome.txt, and without any extra build steps. That means both jars and flat files have to be on disk. An interim idea was to add a build step to rewrite installed-chrome.txt instead of making the chrome registry do it on the fly. It's actually probably easier to do the latter though. We didn't intend to add an unzipping step either. I thought the developer would make a choice up front as to what type of developer they wanted to be: a) chrome developer -- keep flat files at all times b) c++ developer -- keep jar files at all times (or both since many c++ developers also develop chrome) c) build engineer -- eliminate flat files to minimize dist size

If you added the runtime logic for the chrome registry, it would have to be smart enough to do the right thing if either the jarfiles or the flat files were not available. Doing a runtime check should not mandate that both sets of files exist. I have no problem with the user deciding which category they want to fall into upfront (ie. at configure/build time) but the default should be the end-user case (where end-user is defined as not a mozilla developer). That's what I plan to move the entire build system towards as we near the mozilla 0.9 milestone (see bug 58372). There will probably still be a "maintainer" mode that resembles the current default build state but in the future, by default, everyone should not have to deal with the overhead that comes from our current setup (bug 58376). Are you still planning on going forth with the interim solution or are you going to skip straight to the runtime checks?

I looked at the Perl script granrose mentioned and hacked it and the makefile a bit to do what I want. I added - a new flag --flat/-l, which does not create dirs for each component, but stuffs everything directly into the package destination dir. - a new make target, "dist-tarball", which creates a tarball with only the files used for the installer. It uses pkgcp.pl with the new flag --flat. - new make options - bz2 compression. Works with both the "all" and "dist-tarball" target. (Actually for bug 36358.) - appname. It changes the top-level dir in the tarball, the tarball filename and the mozilla startup script filename. Works with both the "all" and "dist-tarball" target. The default is "mozilla". Note that this means that the top-level dir in the Mozilla nightly tarballs and Milestone tarballs will change from "package" to "mozilla". This was intended, but I don't know what this breaks (apart from scripts some regular Mozilla nightly tarball users might have written for themselves). - a bit of documentation for the new features I don't know Perl, autoconf or make - it's all written "by example" and "try and error". Please review carefully; see also <http://www.bucksch.org/1/projects/mozilla/review.html>. There is at least one known problem - how to get the absolute path to the dist dir. Look for "XXX" in the source. I think, my hack will break, if separate build and source dirs are used - suggestions welcome. The size of a bz2 tarball of a N6.0 branch build with MathML, SVG, XSLT and LDAP enabled is 7.3 MB. I just noticed that my oatch is against the branch, and doesn't apply to the trunk :-(. Will a attach new one after I figured out what's wrong.

*** Bug 59110 has been marked as a duplicate of this bug. ***

Should the summary be changed to reflect the fact that this could be used to make other forms of packages? [.sit, .zip, .pkg, .deb, .rpm] fwiw, I fall into warren's a and b categories, and am glad we aren't going to zap flat chrome from w32 .zip's. The other nice feature of the flat form is _easy_ incremental upgrades.

Since my bug 59110 is marked a duplicate of this, I'll post my win32 arguments here. FWIW, I sort of fall into warren's category 'a' (a user/hacker), and have no use for the flat chrome/skin files in the win32 zip files, nor do I see the real benefit in having them included. I download talkback builds about once a week. I don't use the installer.exe builds because I want the install control I get in the other versions. Once I discovered the uncompressed files weren't being used to run mozilla, I started unzipping the jar files to get what I need. It only takes a few seconds to unzip a jar, but, with my slow connections, can take an extra 30 minutes to download talkback.zip vs installer.exe. Maybe it is more convenient for some folks to have the flat files already included, and if you've got high-speed connections the extra download time isn't that big of a deal, but not everybody falls into both those groups. My $.02.

Since bug 59110 is marked a duplicate of this, I assume that an equivalent bug for Linux nightlies would be a duplicate of this one, too. Therefore I'm posting my petition here: Please get rid of the flat chrome files in the Linux nightly tarballs. There are many people downloading the tarballs regularly, and preferring them over the installer and "sea" builds. What's the point in unneccessary download time, unzip time and diskspace if you can re-create the flat chrome files with a simple "cd chrome && unjar *.jar" command? (An example "unjar" script is already attached to this bug.)

*** Bug 61861 has been marked as a duplicate of this bug. ***

My patches bitrotted, will attach new ones in a few minutes. granrose, can you r=, please?

Forgot to mention that I changed the dist-tarball target to just dist. Fix4 + the update looks good to me. r=cls At a glance, the only place I noticed "package" being used in the nightly automation was in a snippet used to generate sizes.txt but granrose may want to double check that.

the verification build automation just cd's to xpinstall/packager and does a make, so as long as that still works, or someone tells me what variables I have to set to get the tar.gz file, we're ok. I don't think anyone is still looking at the sizes.txt file any more. I'll have to check with chofmann.

Do I need any other OK, or are we're ready to checkin?

i'm almost positive sizes.txt is still used; chofmann would know for sure. cls's review for the change should be sufficient to commit it, thanks ben (just be around tomorrow for verifications, so i don't have to back you out =)

I asked chofmann about this yesterday, he said it's still being used. So after this is checked in, we need to change seamonkey-build if we are no longer using "package" as the top level directory in the tarball. We should try to make it generic (i.e. grep for '/' rather than 'package') so we don't have to do this again.

I checked in fix4+update . We need this change to distinguish the gcc295 builds from the default compiler builds. Working on the sed problem.

cls, thanks for checking in (sorry that I didn't earlier). Marking FIXED.

fyi, I backed out some of the changes made to pkgcp.pl because it was breaking the packaging process on Mac for some mysterious reasons. Here's a excerpt of the Mac Perl log: > # Use of uninitialized value, <MANIFEST> chunk 178. > File 'FastFreddy:Mozilla Tree:mozilla:xpinstall:packager:pkgcp.pl'; Line 223 After this, the script tries to delete *everything* in its own directory, and loops indefinitely with the error: > File 'FastFreddy:Mozilla Tree:mozilla:xpinstall:packager:pkgcp.pl'; Line 233 > # Can't unlink file :pkgcp.pl:: Device busy That's pretty bad. Removing the 'if($stat)' block inside of 'sub do_delete' fixed the problem. PLEASE TEST YOUR PATCHES ON ALL PLATFORMS BEFORE CHECKING THEM IN, or if you can't make sure you work with someone who can.

REOPENing per last comment. Who has a Mac and can tell why this breaks? It seems like $flat is not sat corrently (uninitialized or similar?) in that function. <http://bonsai.mozilla.org/cvsview2.cgi?diff_mode=context&whitespace_mode=show&file=pkgcp.pl&root=/cvsroot&subdir=mozilla/xpinstall/packager&command=DIFF_FRAMESET&rev1=1.13&rev2=1.14>

I'm also confused because looking at the change that jj checked in, it does nothing but disable the initial setting of $target based upon $flat. So is the problem that $flat was not set initially? If so, then the proper fix would have been to add this to the top of the file: $flat = 0; # flag: if set, do not create component dirs

Agreed, this might be a better fix. Please understand that I didn't try to find the optimum fix. I just backed out the lines that prevented the script from completing and the build to be delivered (for the second consecutive day). If what you suggest is the right fix, and it has been tested on all platforms, then go ahead and restore the if/else case I backed out.

It might be the right hotfix, but the right fix in the long term? With cls's suggested hotfix, in the best case, my changes won't work on Mac. I wonder why $flat isn't initialized in the first place.

jj, can you try out the workaround that cls suggested? I need the fix for this bug working, and mozilla.org does as well, if it wants to deliver small nightly tarballs. Who should I ask to investigate *why* my patch fails on Mac?

Well, guess what: initializing $flat at the top of the file doesn't fix the problem, to my great surprise. I tried it using rev.1.13 and I observed the exact same behavior. (auto-destruction & infinite loop) And like you, I wonder why $flat doesn't get initialized in the first place. As to "Who should I ask to investigate *why* my patch fails on Mac?", I cc'ed scc and sfraser, 2 in-house experts, hoping that they can find a few minutes to take a look at this and help us out... Thanks guys!

Which patch fails? And fails how?

Ben et al, Simon found the problem in the patch to pkgcp.pl: if ($flat) { my ($target) = "$targetpath$PD$targetfile"; } else { my ($target) = "$targetpath$PD$targetcomp$PD$targetfile"; } With this, the scope of $target is limited to the if case, which is why the subsequent test 'if ( -f $target )' fails. Solution: replace the 5 lines above with this one: my ($target) = ($flat) ? "$targetpath$PD$targetfile" : "$targetpath$PD$targetcomp$PD$targetfile"; I'll submit a new patch against rev. 1.14 and will test it on the release build mac. please review for other platform compatibility. (makes me wonder how the initial patch could work on windows/unix...)

This could definitely make it in for 0.8. A 1-line patch have been submitted (6 days ago) to fix a mac-specific problem in pkgcp.pl. (introduced in fix#4) All it needs is a review by the original author to make sure everything's fine with Win32 and Linux. I tested Mac.

Tested on win32 & linux. wfm. r=cls

jj checked in his patch a couple of weeks ago. Marking fixed.

verified.