calling all nerds: is it possible to make a third party search engine that searches sites that have shitty on purpose search features?

[aptronym@autistics.life]

calling all nerds: is it possible to make a third party search engine that searches sites that have shitty on purpose search features? #asknerds #tech #enshittification

[bri7@social.treehouse.systems]

@aptronym possible: yes; legal: maybe not

[pelle@veganism.social]

@aptronym
a friend asked me to archive their shitty website, so i ran this command (on my android, in the termux app):

pkg install wget
wget --adjust-extension --convert-links --mirror --no-parent --page-requisites --trust-server-names https://shitty.web.site

if i want to find where "poop" is mentioned on the shitty website:

pkg install ripgrep
rg -o "poop" shitty.web.site

if this search engine has to be a web app, that's an extra step.

if the website blocks scraping, then that is a challenge. more have started doing that because of aggressive genAI scrapers.