<![CDATA[calling all nerds: is it possible to make a third party search engine that searches sites that have shitty on purpose search features?]]>calling all nerds: is it possible to make a third party search engine that searches sites that have shitty on purpose search features?

]]>https://forum.fedi.dk/topic/9f69c9fc-712f-4027-86de-bb0fb8419674/calling-all-nerds-is-it-possible-to-make-a-third-party-search-engine-that-searches-sites-that-have-shitty-on-purpose-search-featuresRSS for NodeThu, 20 Aug 2026 12:39:35 GMTSun, 09 Aug 2026 23:38:31 GMT60<![CDATA[Reply to calling all nerds: is it possible to make a third party search engine that searches sites that have shitty on purpose search features? on Sun, 16 Aug 2026 06:51:02 GMT]]>@aptronym
a friend asked me to archive their shitty website, so i ran this command (on my android, in the termux app):

pkg install wget
wget --adjust-extension --convert-links --mirror --no-parent --page-requisites --trust-server-names https://shitty.web.site

if i want to find where "poop" is mentioned on the shitty website:

pkg install ripgrep
rg -o "poop" shitty.web.site

if this search engine has to be a web app, that's an extra step.

if the website blocks scraping, then that is a challenge. more have started doing that because of aggressive genAI scrapers.

]]>https://forum.fedi.dk/post/https://veganism.social/users/pelle/statuses/117103855525024652https://forum.fedi.dk/post/https://veganism.social/users/pelle/statuses/117103855525024652Sun, 16 Aug 2026 06:51:02 GMT<![CDATA[Reply to calling all nerds: is it possible to make a third party search engine that searches sites that have shitty on purpose search features? on Mon, 10 Aug 2026 00:31:49 GMT]]>@aptronym possible: yes; legal: maybe not

]]>https://forum.fedi.dk/post/https://social.treehouse.systems/users/bri7/statuses/117068392655929779https://forum.fedi.dk/post/https://social.treehouse.systems/users/bri7/statuses/117068392655929779Mon, 10 Aug 2026 00:31:49 GMT