TheDailySmarkBot

What it is

TheDailySmarkBot is the automated crawler I run for The Daily Smark. You can identify it by its user agent:

TheDailySmarkBot/1.0 (+https://thedailysmark.com/bot)

What it does

It gathers publicly available pro-wrestling news so I can surface it in The Daily Smark app and website. Specifically, it fetches:

• RSS and podcast feeds from wrestling news sites and shows
• A small number of additional wrestling news sites I read directly
• Images from the article pages it reads

How it behaves

I try to be a good citizen of the web:

• It honors robots.txt disallow rules on every request.
• It limits itself to roughly one request per second per site.
• It checks feeds every 10 to 30 minutes on a schedule, not continuously.
• It reads only publicly available pages. No logins, no paywalled content.

I crawl each source once, centrally, and store what I find. The people using The Daily Smark app don't each fetch your site, so my traffic stays modest and predictable no matter how many readers I have.

Identifying and controlling it

Every request carries the TheDailySmarkBot user agent above. To control what it may access, add a rule to your robots.txt:

User-agent: TheDailySmarkBot
Disallow: /

Contact

Questions, or seeing traffic you didn't expect? Email rich@thedailysmark.com.