capistrano, svn and webserver timeouts

A customer reported problems with capistrany deploys that would just die like this:

 ** [XXX.XXX.XXX :: err] svn: REPORT request failed on '/!svn/vcc/default'
 ** svn: REPORT of '/!svn/vcc/default': Chunk delimiter was invalid (http://XXX.XXX.XXX)
    command finished

After disabling gzip compression on the server for text/xml documents, the error became

 ** [XXX.XXX.XXX :: err] svn: REPORT request failed on '/!svn/vcc/default'
 ** svn: REPORT of '/!svn/vcc/default': Could not read response body: connection was closed by server (http://XXX.XXX.XXX)

The server side logs said:

[Fri Jul 03 10:53:57 2009] [error] [client XX.XX.XX.XX] Provider encountered an error while streaming a REPORT response.  [500, #0]
[Fri Jul 03 10:53:57 2009] [error] [client XX.XX.XX.XX] A failure occurred while driving the update report editor  [500, #190004]

Googling was not very helpful – there are many reports of these errors going back years, and many different solutions, none of which applied to my setup. In general, these errors seem to mean that there was some sort of network problem.

I tried to reproduce the problem by running the offending svn command manually. Out of hundreds of tries, I only managed to make it fail like that just once. And yet running cap deploy, which in turn calls the svn command, it would happen much more often.

I finally tracked this down to an agressive send/receive timeout in Apache’s config. It was set to 3 seconds to prevent too many inactive connections from taking up server resources. Apparently the subversion client sometimes takes a while to get back to the http server its talking to – in this particular situation when run via capistrano, more than 3 seconds. So the server would disconnect the svn client, which would then just fall over with that obscure error message.

In other words, check your server timeouts if you see this kind of intermittent error…