Long commands on Windows with SCons

At my job, where one of my tasks is to handle builds for Windows software using SCons, I recently moved from using SCons 1.2 with some custom modifications to an unmodified SCons 2.3.4. The change comes along with a move to using a newer Visual Studio while still supporting builds with an older one. The newer SCons has trouble with issuing some commands, just like the older version. Neither version can run programs that have whitespace in their path if the entire command is longer than some threshold. The custom modifications I made were to correct the problem, but this time I wanted something less custom and easier to support. Here is a link to the solution I developed; read on for the details.

With shorter commands, SCons issues the commands about the same way it does on Linux, and it works fine. Longer commands run into trouble on Windows; I’m told it has to do with the C runtime libraries. SCons works around this by placing most of a long command into a temporary file and then passing the name of the file preceded by a ‘@’ character as the first command line argument. Microsoft’s compiler and linker will read the temporary file for their arguments.

The implementation of this has a flaw that makes it useless using default installations of Visual Studio. SCons first puts together the command line as though no special long command handling is required. If the result is too long, the command is modified to use a temporary file but is parsed incorrectly; the program to run is taken to be all the characters up to the first whitespace. This makes the command to run “C:\Program”, which isn’t a program, or even an existent path. Everything after the first whitespace is put into the temporary file. It may start with “Files (x86)\Microsoft Visual Studio 10.0\VC\bin\cl.exe”, for example.

I found an attempt at getting SCons to use long commands written by Phil Martin. He provided SCons with a different way to spawn processes. Unfortunately, it doesn’t handle the whitespace issue. By the time the spawn function is called, SCons has already modified the command to use the temporary file. Nevertheless, his implementation was a good place to start. Like his implementation, mine also requires PyWin32.

I modified it to detect the use of a temporary file. When used, the spawn function opens and reads the temporary file to rebuild the complete command line. Then it figures out the whole path to the program, including whitespace, and separates that from the arguments. I also made a modification to produce better error messages from CreateProcess().

Finding the path to the program works best when there is some delimiter in the program path. It is common on Windows to enclose paths that have whitespace with double quotes. The command constructed by SCons using the default program paths lack this delimiter. I solved this by supplying new complete paths for all programs used in Microsoft’s toolchain. It is quite a bother, but I did it the best and most complete way I could figure. This includes creating some build configuration options for several paths, and making two build environments: one for x86 targets and another for amd64 targets.

Hopefully this will help someone else. I really don’t understand why this bug has been around for so long considering that whitespace in paths on Windows is very common.

Advertisements

Tags: , ,

3 Responses to “Long commands on Windows with SCons”

  1. Nick White Says:

    Hi! Excuse me for asking a stupid question, but where do I put this “longwincmd.py” file? I’m trying to compile mesa3d on Windows and I’ve got this “too long command line” problem.
    I’m a complete newbie in Python.

    • jjackowski Says:

      Unfortunately, it isn’t as simple as putting a file somewhere. longwincmd.py provides the solution in a form that is intended to be an example of how to modify an existing Scons build script. It starts with the function that fixes the broken command, and then shows how to make an Environment object, defined by Scons, to use the function. After that, it shows modifying the object further to have double quotes around the paths to all the Visual Studio programs that Scons needs to use.

      I know figuring this out makes for a difficult introduction to Python and Scons, but I haven’t got something simpler. I don’t know the internals of Scons well enough to fix it there.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s


%d bloggers like this: