I went searching for the answer to this question and the only thing I found was a comment on Matt Cutt’s blog in 2006 by a user stating he read somewhere inside Google sitemaps that the standard only allows 2000 characters. I don’t believe this to be the case now, as most of my files have been found and disallowed by Google and because of my recent site restructure, I currently have 4394 characters in my robots.txt file. Does anyone out there have the exact answer on this? I will try to post the question again to Matt Cutts.