If you turn on the "Generate a log file" checkbox then Blue Crab will copy the messages it displays in the Grabber window to an HTML formatted file. Use the "Choose location of log (site map)..." to specify the location of the log file.
The log file is named after the host of the starting URL with a ".log.html" suffix.
If you turn on the "Generate a site map" checkbox then Blue Crab will create an HTML formatted site map consisting of all the URL's it grabbed, organized according to the directory structure on disk. You can control which URL's are included by specifying their extensions (suffixes), or include all URL's by turning on the "Include all suffixes" checkbox.
Blue Crab can also save any email addresses it finds while it crawls. Email addresses are also saved in the same location as the log file.
All client software should identify itself using the User Agent string. You can specify the string using the "User Agent Spoofing" text field. Sometimes servers will return content conditioned on the value of the user agent string. Hence you could use this string to determine that alternative content.
The popup button, by default, provides some common user agent strings. You can import your own using the "Import User Agents..." button. The format of the file is straightforward:
<browser-name><tab><browser-user-agent><carriage-return>
A sample user agent import file:
Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7.5) Gecko/20041107 Firefox/1.0 Mozilla/5.0 (Macintosh; U; PPC Mac OS X; en) AppleWebKit/412 (KHTML, like Gecko) Safari/412 Mozilla/5.0 (Macintosh; U; PPC Mac OS X; en-US) AppleWebKit/125.4 (KHTML, like Gecko, Safari) OmniWeb/v563.34 Mozilla/5.0 (Macintosh; U; PPC Mac OS X Mach-O; en-US; rv:1.7.8) Gecko/20050427 Camino/0.8.4 Mozilla/4.0 (compatible; MSIE 6.0; Mac_PowerPC Mac OS X; en) Opera 8.0 Mozilla/5.0 (Macintosh; U; PPC Mac OS X; en) AppleWebKit/412.6 (KHTML, like Gecko) NetNewsWire/2.0b37 Googlebot/2.1 (+https://www.google.com/bot.html) BlueCrab/4.2.4 (Macintosh; U; PPC Mac OS X; en)