I know we've talked about this before but I'm just so disgusted with how useless most search engines are now.
-
@doomsdayrs @futurebird I read about companies that build their own search engines. They still need to buy data from Google and other older search engines. No one can afford to build an international bot network for collecting the data required for search.
-
@futurebird @doomsdayrs it must be so *difficult* to make a searchy thing that doesn't actually search, but instead obfuscates every search and then obfuscates the results.
But I suppose nobody is actually making them. They're mostly using APIs from the Usual Suspects search tools, which are now basically not-search-tools-at-all. 
️@FaithfullJohn @futurebird @doomsdayrs random brain fart i had a few days ago: i feel like we should somehow work on a decentralized index where each node is a crawler and an index and a search engine - and the nodes communicate with each other like the torrent network. I feel this might be the only way to get away from platform based restrictions surrounding monetization.
-
There are many scattered people working on similar projects and I think we need to start networking and try to pour our time, money and energy into a single project.
Maybe it could be as simple as "bring back google C. 2010" This wouldn't be cheap but I think enough people might want it to make it happen? Someone might already be doing this. But unless it's a non-profit user-owned type thing with membership dues I don't think it will work in the long run...
@futurebird To be free from Google, the EU invests into an own search machine which is a cooperation of #Ecosia and #Qwant.
No free community would have the financial power you need for such a project! -
@FaithfullJohn @futurebird @doomsdayrs random brain fart i had a few days ago: i feel like we should somehow work on a decentralized index where each node is a crawler and an index and a search engine - and the nodes communicate with each other like the torrent network. I feel this might be the only way to get away from platform based restrictions surrounding monetization.
@missqarnstein @FaithfullJohn @futurebird you're describing yacy
-
@doomsdayrs @futurebird I read about companies that build their own search engines. They still need to buy data from Google and other older search engines. No one can afford to build an international bot network for collecting the data required for search.
@paulc this. @doomsdayrs @futurebird
-
@FaithfullJohn @futurebird @doomsdayrs random brain fart i had a few days ago: i feel like we should somehow work on a decentralized index where each node is a crawler and an index and a search engine - and the nodes communicate with each other like the torrent network. I feel this might be the only way to get away from platform based restrictions surrounding monetization.
@FaithfullJohn @futurebird @doomsdayrs also it might be easy to bolt on user trained and shared bloom filters to customize search results: for flagging ai generated results, transphobic sources, sources of misinformation or flag sources to be biased.
-
@missqarnstein @FaithfullJohn @futurebird you're describing yacy
@doomsdayrs @FaithfullJohn @futurebird
do you have experience with it? Is it useful? -
@doomsdayrs Also if you've already looked into how to feasibly distribute/federate crawls (and their results for search index building), I'd be very interested to hear about it.
I don't think a single person or small organization will be able to map the entire web without exhausting itself to death, so I think it's important to decentralize at the very least the crawling from the beginning.
// @futurebird
@phryk @futurebird I have and I have a solution, wait to hear me announce its functional state eventually
-
@phryk @futurebird I have and I have a solution, wait to hear me announce its functional state eventually
@doomsdayrs You have my attention – but do I really have to wait (and possibly miss it)? What language is this and can I get a look at the code for contribution/inspiration/interop? 🥺
// @futurebird
-
@doomsdayrs @FaithfullJohn @futurebird
do you have experience with it? Is it useful?@missqarnstein @FaithfullJohn @futurebird its terrible.
It's somewhat a reason I am building my own without a "tor" network, but something more traditional and simple.
-
I know we've talked about this before but I'm just so disgusted with how useless most search engines are now. They all second guess me.
The most obscure or unusual word is ignored so it's very hard to search for the intersection of a popular thing and an obscure thing.
I have a theory that for many people search has never worked well for them since they just didn't use computers very much, or it was not explained well. Now those of us who found search effective get to see what it was like.
@futurebird oh i hate the word omission thing. even the "good" engines do that now
-
@doomsdayrs You have my attention – but do I really have to wait (and possibly miss it)? What language is this and can I get a look at the code for contribution/inspiration/interop? 🥺
// @futurebird
@phryk @futurebird Follow me and one day you'll see me announce it. Maybe in the next 3 months.
-
I really hope that enough people have noticed the degradation that it might matter.
Ha… these days it's become practically impossible to develop a new internet search from scratch: You've to crawl the web to build an index. Which means that you've got to scrape the web just like the AI crawlers do. Which means that these crawlers will run into the same self-defense mechanisms that were rolled out to have at least some protection.
-
The non-computer people I know don't really think search as gotten worse. The people I know who hate computers the most think it's gotten better... but they are in the minority.
I've always found obscure words powerful in searches. That power has been destroyed. Please don't tell me to put it in quotes or use a different engine... this is a design trend, made worse by "AI" I'm not asking for a solution or work around. I want to know WHY the software I depend on keeps getting worse.
@futurebird I'm convinced it's to drive up ad revenue or something.
If you have to search twice, they can show you twice the ads.
And yes, I've noticed it and the AI summaries made it way worse. I'm not a native English speaker and used to paste words or half sentences into Google to check spelling and such, it is now mostly useless for that. -
@futurebird To be free from Google, the EU invests into an own search machine which is a cooperation of #Ecosia and #Qwant.
No free community would have the financial power you need for such a project!@NatureMC @futurebird @doomsdayrs ha, didn't know about that project! Here's the website:
https://www.eu-searchperspective.com/
Looks like they like to use the AI marketing buzzwords though. This is their "product"
https://staan.ai/ -
@futurebird I'm convinced it's to drive up ad revenue or something.
If you have to search twice, they can show you twice the ads.
And yes, I've noticed it and the AI summaries made it way worse. I'm not a native English speaker and used to paste words or half sentences into Google to check spelling and such, it is now mostly useless for that.@futurebird which is funny in some twisted way, because one would expect a language model to be better at correcting my sentences. Now it sometimes just replies to my half pasted sentences, or if I want to check a quote. I don't want an answer or it's take on it, I want to know the origin.
-
The non-computer people I know don't really think search as gotten worse. The people I know who hate computers the most think it's gotten better... but they are in the minority.
I've always found obscure words powerful in searches. That power has been destroyed. Please don't tell me to put it in quotes or use a different engine... this is a design trend, made worse by "AI" I'm not asking for a solution or work around. I want to know WHY the software I depend on keeps getting worse.
@futurebird I know at least one person who agrees with you...
https://helvede.net/@jwcph/114714407615528803
