Our crawler

Big Bear News runs a crawler that fetches public local sources — government agenda pages, weather and road feeds, event calendars, and local newsrooms — to assemble a daily local news digest for the Big Bear Valley. It reads pages a person could read in a browser. It does not log in, fill forms, or collect personal data.

It fetches on a polite cadence: most sources are checked a few times a day, and time-sensitive safety feeds a bit more often. It respects each site's rate limits and backs off when asked. The current seed URLs we monitor are listed on the about page.

Every request our crawler makes sends this user-agent:

bigbear.news bot v1 (https://bigbear.news/about/bot; admin@bigbear.news)

The link in the string points back to this page, and the address is a working contact for questions about the crawler.

If you run a site and would rather we not fetch it, you have two options:

• Add a rule for our user-agent to your robots.txt. We honor it.
• Or email us and we'll remove your site from our source list.

If you run a public Big Bear Valley site and want us to consider monitoring it, suggest a source on the about page. We review suggestions by hand and do not automatically start crawling new URLs.

Questions about the crawler, or a request to stop fetching your site? Email hello@bigbear.news. More about who we are is on the about page.