User:JoeyArmstrong/makefiles/makemakefile r1

From MozillaWiki
Jump to: navigation, search

joey

post #2

Build system tools: make-makefile, file generation

This post will be article #2 in a series about the build-config tool make-makefile.
The series is being written to formally document current tool functionality and
provide a base for future topics related to the container makefile project.
Container structure, functionality and example usage.

When makefile generation is needed the tool will perform a few steps.

   1) gmake -f client.mk is invoked.  During traversal if gmake detects one
      of two conditions the tool will be invoked to generate a Makefile:
        o $(obj)/Makefile does not exist.
        o $(src)/Makefile.in is newer than $(obj)/Makefile.
   2) Target rules used to invoke mm live in config/rules.mk near line 1342 and 1347
        o 1342 Makefile: Makefile.in
        o 1343 	@$(PERL) $(AUTOCONF_TOOLS)/make-makefile -t $(topsrcdir) -d $(DEPTH)
   3) make-makefile will be passed -d and -t command line arguments:
        o -t  path to root of a development sandbox ~$(TOPSRCDIR)
        o -d  relative path --depth from file to hierarchy root directory.
              Makefile: $(MOZ_OBJDIR), Makefile.in: $(TOPSRCDIR)
   4) When invoked make-makefile is aware of the directory hierarchy
      and will expect cwd for the shell to be $(MOZ_OBJDIR) or a subdir
      of it [1].  config/rules.mk logic will place make-makefile in the
      target directory for each makefile to be generated [2].
   5) Step 1 - obtain values for depth, topsrcdir and objdir.
      Values will be:
        o used to construct directory and file paths.
        o perform text substitution within Makefile.in templates.
      Values can be:
        o explicitly passed on the command line.
        o determined from the filesystem based on arguments and/or cwd.
        o extracted from file arguments.
      o $depth can be set by:
        o Explicitly using the -d command line argument.
        o Assignment of DEPTH=../.. specified within an individual Makefile.
        o Assignment of DEPTH=../.. within any other files passed on the command line (is this a bug or feature?).
        o Assignment of DEPTH=../.. contained within a parent directory Makefile [3], [4].
   6) Step 2 - Using cwd and arguments, derive paths to the source template
      Makefile.in and generated target file Makefile.
   7) Step 3 - Slurp the source template Makefile.in.  Perform value substitutions
      on tokens embedded within the template of the form @token@.

browser/branding/official/Makefile.in
=====================================
DEPTH		= ../../..
topsrcdir	= @topsrcdir@
srcdir		= @srcdir@
VPATH		= @srcdir@

      Some directory/file paths are derived internally by the tool for quick
      substitution.  For any @${unknown}@ tokens make-makefile will delegate
      expansion with a call to system("./config.status") [5] to obtain values,
      create directories and who knows what else.  Shell overhead will be
      imposed by this step so avoid unnecessary tokens when possible.  When only
      a handful of tokens are in play check if mm could derive these values and
      avoid calling the config.status script [6].
   8) Step 4 - update or preserve generated Makefile timestamps.  If
      obj/Makefile does not exist create it.  If the file exists compare
      generated content against the original and only update when modified.

[1] - The requirement of cwd==$(MOZ_OBJDIR)/ can be removed with use of the --enhanced flag.
      This option and other new flags will be covered in a future post.
[2] - Enhancement: a response file could be used to support bulk makefile processing and
      directory creation (by config.status) saving some shell overhead.  Bulk processing
      might also reduce the number of times config.status must be invoked independently.
[3] - Potential bug.  DEPTH= searches within multiple files could set $depth incorrectly
      when it is not explicitly set within a makefile.  One ex would be passing browser/.../Makefile
      and js/src/.../Makefile as command line arguments.  Though this error condition
      exists it is not likely to be triggered building in the current hierarchy.  Makefiles
      are processed/generated individually by directory so there should be no chance for
      file interaction.
[4] - Potential bug: checking ../Makefile for 'DEPTH=' will fail for makefiles
      invoked from a parent directory several layers above cwd.
[5] - Enhancement - modify make-makefile to parse and extract fully expanded
      values from config* to avoid invoking ./config.status for a subset of
      substitution tokens.
[6] - make-makefile will issue a warning when the ./config.status script will be launched
      WARNING: token SET_MAKE not defined
      line 2, src: [.....]/js/src/ctypes/libffi/Makefile.in

resources

make-makefile

Unit tests for program and module

makefiles/target logic

Pending tool enhancements

  • Expand test coverage for make-makefile
  • Add Response file support.
  • Rewrite unit tests in python
  • Porting make-makefile from perl to python.
  • Container makefile support: transition to make-makefile --enhanced as a default

Future Topics

coop

coop 11:35, 16 February 2012 (PST) - the one thing that's missing from both this post and the last one is context: why are you talking about make-makefile and where is it going? Some subset of readers are very interested in your process and already have enough context to follow along, but I suspect most readers want an overview that tells them why this matters.

To facilitate that, and since you seem to have a series of blog posts lined up, I would also try to link your posts together. Include a link to the previous post at the start of this one: "Last time, I talked about make-makefile..." Similarly, at the end of the current post, let the reader know what's coming up next. Maybe even a table of contents that you could insert into all the related blog posts to allow people to jump between the different parts?

Don't be afraid to talk about the switch to python when you get to that part. That's one of those growing pains stories that usually goes over pretty well with readership.

Can you tell I want a lot of writing out of you? ;)

post #1

Build system tools: make-makefile

The build process consists of several steps, one of which is the dynamic generation of makefiles - the creation of recipes that control how and when elements of firefox and friends are gnerated. The tool currently used for generation is make-makefile, a utility written way back in the days of Netscape (circa 1999). Intent for this post and a few more that will follow will be to document the tool and usage.

Scripts and Config

  • build/autoconf/make-makefile
  • build/autoconf/makemakefile.pm
  • build/autoconf/make-makefile.excl

make-makefile will be automatically invoked while building whenever a Makefile does not exist beneath the object directory or a template (Makefile.in) is newer than a previously generated Makefile.


Post #x

The make-makefile tool is normally invoked during recursive make traversal, indirectly launched by make -f client.mk calls.
Whenever a non-existent Makefile is detected or when a Makefile.in source template is newer than a generated file, make-makefile will be invoked to create/re-generate the target Makefile.

When launched make-makefile will determine a few path values:
     source: ${src}Makefile.in - source template for makefile generation
  generated: ${obj}/Makefile   - generated makefile with expanded paths and variables
      depth:                   - path from file to top directory (~relpath)

The tool currently expects to be launched from beneath the object directory:
  o make-makefile will be invoked from a subdirectory of $OBJ
  o command line arguments -t ${topsrcdir} -d ${depth} are passed (~config/rules.mk)
    - $depth for relative path construction
    - $topsrcdir specifies root of the source tree
  o path to source template Makefile.in will be determined
  o $DEPTH from subdirectory to ${OBJ} will be determined and the value
    will later be used to derive paths for variable expansion in the makefile.
  o Read template Makefile.in from the source directory and perform variable subst.
  o For unknown variables source the config.status file to obtain values.
  o Generate/write Makefile in a corresponding subdirectory beneath ${OBJ}.

Path to the build directory root can be determined in a few different ways:
  o from command line argument --depth
  o parse the contents of Makefile/Makefile.in and extract a 'DEPTH=' string.
  o If Makefile/Makefile.in exists in a parent directory the file can be used
    as a base for determining DEPTH for the current makefile.

Variables:
  o Makefile templates can be parameterized by variables - @{var_name}@.
    Most Makefile.in templates should have an init section similar to this:

  DEPTH		= ..
  topsrcdir	= @top_srcdir@
  srcdir	= @srcdir@
  VPATH		= @srcdir@

  include $(DEPTH)/config/autoconf.mk

  top_srcdir, srcdir and a few others are defined internally by make-makefile.
  Additional values { anything defined by configure } can also be referenced
  as an @var@ expandable value in the makefile.

Extra Configuration:
  o build/autoconf/make-makefile.excl
    Exclusion list that can be used to inhibit generation of certain makefiles.
    If makefiles are being deprecated or have broken building presence on this
    list will inhibit their generation.

Enhancements {{bug|687511}} to support container makefile prototyping.
  o Remove restriction requiring that the tool can be used from the $OBJ directory.
  o Arguments added to allow passing in the path to Makefile and Makefile.in
  o When both file paths are available logic can be simplified.  Relative path,
    depth, $topsrcdir values can be set by derivation rather than searching.
Retrieved from "https://wiki.mozilla.org/index.php?title=User:JoeyArmstrong/makefiles/makemakefile_r1&oldid=398855"