|
|
Information about our spiderWebWobot obeys the Robot Exclusion StandardOur spider's name is ScollSpider. When ScollSpider visits your site, it will first look for a robots.txt file in your root folder of your domain or subdomain. If it does not find one, it will assume that your site is available for all spiders to crawl every part of your site. If you don't yet know how to create a robots.txt file, you can consult http://www.robotstxt.org/ to see how it works. ScollSpider will parse your entire robots.txt file, and will look for any of the following: User-agent: * User-agent: ScollSpider Whatever directory is specified ( including the root / ), ScollSpider, as per the Robot Exclusion Standard, will deem all files and directories within the specified directory off-limits, and will not crawl them. All files and directories that are accessible via both internal and external (from other sites) links that are not excluded by robots.txt will be crawled and cached for possible indexing. Title and Meta-Tags & SEOThe most important tags we make use of is your title tag and your description tag. (In that order). While other search engines might make use of your keywords meta-tag, we do not. Our engine calculates what it deems to be your keywords, based upon the visible text content of your page. So if your text content is forever talking about apples, with no considerable mention of oranges, then we will not deem the page to be about oranges irrespective of what you might specify in your keywords meta-tag! Similarly, if every other word used was apples, then... well your page would be sounding kinda funny to your visitors reading the page, so we would also be having a little chuckle when it comes to indexing time! The moral of the story is: Don't try to design your pages for robots - design them for people who will be visiting your site. If your content is good to them, then we will be good to you! Rapid-FiringScollSpider would like to be welcome to visit your site time and time again. To try to help achieve this, it will try to blend in with your human visitors, and "click away" at your links as though it were some very interested speed-reading human visitor. So when ScollSpider visits your site, its presence will seem like a human who dropped by your site and read a page for 3 seconds or so, before moving on to the next page. In short, we do not rapid-fire websites!!! How long to get indexed?After submitting (and confirming) your website to WebWobot.com, your URL will be enqueued for crawling, just behind any other pending submissions. This may even occur instantly, but it may take up to 48 hours or so for pages to start appearing in search results. Spider Identification for forums, etcTo ensure that when ScollSpider visits your forum, it shows up as ScollSpider rather than guest, you may be able to configure your forum by adding a spider and using "ScollSpider" as identification string or our user-agentstring: Mozilla/3.0 (compatible; ScollSpider; http://www.webwobot.com). Bot IP: 82.43.129.240. Should you require further assistance please consult our forums.
|
Friends of WebWobot FaxReport Authority Sites
Directory |