Our crawler

Big Bear News runs a crawler that fetches public local sources — government agenda pages, weather and road feeds, event calendars, local newsrooms, and posts in allowlisted public Facebook groups — to assemble a daily local digest for the Big Bear Valley. Public-group posts become an editorial “Around town” theme list, not a comment board: no names, no quotes. It reads pages a person could read in a browser. It does not log in, fill forms, or collect personal data.

It fetches on a polite cadence: most sources are checked a few times a day, and time-sensitive safety feeds a bit more often. It respects each site's rate limits and backs off when asked. The current seed URLs we monitor are listed on the about page.

Every request our crawler makes sends this user-agent:

bigbear.news bot v1 (https://bigbear.news/about/bot; admin@bigbear.news)

The link in the string points back to this page, and the address is a working contact for questions about the crawler.

If you run a site and would rather we not fetch it, you have two options:

• Add a rule for our user-agent to your robots.txt. We honor it.
• Or email us and we'll remove your site from our source list.

If you run a public Big Bear Valley site and want us to consider monitoring it, suggest a source on the about page. We review suggestions by hand and do not automatically start crawling new URLs.

Questions about the crawler, or a request to stop fetching your site? Email hello@bigbear.news. More about who we are is on the about page.