Building an indexer

An open-source solution for indexing your site.

Oct 26, 2024

Yesterday, I wrote about improving my blog’s indexing. After writing that post, I wanted to take the opportunity to build a new version that improves on the simpleindex project I mentioned in that post.

I built it in a few hours, and it’s a great way (with a caveat!) to index your site. It’s open-source, so you can see how it works, and you can use it to build your own indexer.

Check it out on GitHub.

Caveat

After sharing this tool on 𝕏, I learned that Google is starting to crack down on using tools like this for indexing. Unfortunately, this project was essentially dead on arrival:

Careful with that. Google says they’re cracking down on indexing API misuse and spam. You can index quickly but likelihood they’ll get delisted if page isn’t a job listing or livestream.
— Jon (@joninsley) October 25, 2024

It may be safe to use it for a while, but I don’t recommend it for production use. It probably isn’t worth the risk of having your domain or URLs penalized by Google.

But it uses Workflows!

We just announced Cloudflare Workflows:

Workflows, Cloudflare’s durable execution engine that allows you to build reliable, repeatable multi-step applications that scale for you, is now in open beta. Any developer with a free or paid Workers plan can build and deploy a Workflow right now: no waitlist, no sign-up form, no fake line around-the-block.

This was a great opportunity to build on Workflows, so even if the project isn’t recommended for production use, I learned a lot about how to use Workflows.