I know we've talked about this before but I'm just so disgusted with how useless most search engines are now.
-
I know we've talked about this before but I'm just so disgusted with how useless most search engines are now. They all second guess me.
The most obscure or unusual word is ignored so it's very hard to search for the intersection of a popular thing and an obscure thing.
I have a theory that for many people search has never worked well for them since they just didn't use computers very much, or it was not explained well. Now those of us who found search effective get to see what it was like.
@futurebird My search strategy has always been to think of three words that are likely to occur together on a relevant page but not elsewhere.
I do think some fuzziness in the matching is good, allowing for spelling variations, different inflections, and even synonyms usually increasing the chances of finding something useful.
Where it went wrong was when they started optimising for queries consisting of complete English questions rather than a list of keywords.
-
There are many scattered people working on similar projects and I think we need to start networking and try to pour our time, money and energy into a single project.
Maybe it could be as simple as "bring back google C. 2010" This wouldn't be cheap but I think enough people might want it to make it happen? Someone might already be doing this. But unless it's a non-profit user-owned type thing with membership dues I don't think it will work in the long run...
@futurebird @doomsdayrs it must be so *difficult* to make a searchy thing that doesn't actually search, but instead obfuscates every search and then obfuscates the results.
But I suppose nobody is actually making them. They're mostly using APIs from the Usual Suspects search tools, which are now basically not-search-tools-at-all. 
️ -
@futurebird My search strategy has always been to think of three words that are likely to occur together on a relevant page but not elsewhere.
I do think some fuzziness in the matching is good, allowing for spelling variations, different inflections, and even synonyms usually increasing the chances of finding something useful.
Where it went wrong was when they started optimising for queries consisting of complete English questions rather than a list of keywords.
@futurebird Of course, the amount of internet to search has grown tremendously, and most of it is rubbish, making the task of finding things more difficult regardless of strategy.
-
I know we've talked about this before but I'm just so disgusted with how useless most search engines are now. They all second guess me.
The most obscure or unusual word is ignored so it's very hard to search for the intersection of a popular thing and an obscure thing.
I have a theory that for many people search has never worked well for them since they just didn't use computers very much, or it was not explained well. Now those of us who found search effective get to see what it was like.
@futurebird myeah thing is, people don't wanna search, either it 's about factual finding or it 's about discovery, akin to taking something from storage or taking a walk inda woods to see what 's there, maybe specified like birdwatching or mushroompicking, but a wander anyway,
reckon instead of a linear crossreferencing digiwank ejac,
a circular/spiral/fractal/flowerpatterned "netscout" that allows for a topical zooming in/out & singular/communal referencing 'd be way more FUNktional ... -
I've been meaning to do the same, but sadly haven't gotten around it.
Are you far along enough yet that you've heard about vector space search? For me, that was the next big topic to look into and I suspect that this is where much of the (useful) complexity of a search engine lives.
// @futurebird
@doomsdayrs Also if you've already looked into how to feasibly distribute/federate crawls (and their results for search index building), I'd be very interested to hear about it.
I don't think a single person or small organization will be able to map the entire web without exhausting itself to death, so I think it's important to decentralize at the very least the crawling from the beginning.
// @futurebird
-
@futurebird I've been trying to make my own search engine as a solution because I experience exactly what you're experiencing, but the technical know-how and system complexity keeps delaying me.
@doomsdayrs @futurebird I read about companies that build their own search engines. They still need to buy data from Google and other older search engines. No one can afford to build an international bot network for collecting the data required for search.
-
I know we've talked about this before but I'm just so disgusted with how useless most search engines are now. They all second guess me.
The most obscure or unusual word is ignored so it's very hard to search for the intersection of a popular thing and an obscure thing.
I have a theory that for many people search has never worked well for them since they just didn't use computers very much, or it was not explained well. Now those of us who found search effective get to see what it was like.
@futurebird I think it’s two-fold.
Search started using word embeddings—they pick up on the interchangeability of a word in context. They do a statistical blurring of words—great if you can’t quite remember the right word, but not good if you can.
The second more direct factor is that Google hired a guy from Yahoo to run their ads division and squeeze growth numbers out at the expense of search quality:
https://www.wheresyoured.at/the-men-who-killed-google/ -
@futurebird @doomsdayrs it must be so *difficult* to make a searchy thing that doesn't actually search, but instead obfuscates every search and then obfuscates the results.
But I suppose nobody is actually making them. They're mostly using APIs from the Usual Suspects search tools, which are now basically not-search-tools-at-all. ️@FaithfullJohn @futurebird @doomsdayrs All search engines are either Google or a reskinned Bing, just like all browsers are either Firefox or a reskinned Chrome.
-
The non-computer people I know don't really think search as gotten worse. The people I know who hate computers the most think it's gotten better... but they are in the minority.
I've always found obscure words powerful in searches. That power has been destroyed. Please don't tell me to put it in quotes or use a different engine... this is a design trend, made worse by "AI" I'm not asking for a solution or work around. I want to know WHY the software I depend on keeps getting worse.
@futurebird search has become almost useless
with google, it's a policy, because kagi, a for-pay search engine is way better - and kagi basically re-sells the google enterprise search api back to its users
the public google search is garbage on purpose imo
-
I know we've talked about this before but I'm just so disgusted with how useless most search engines are now. They all second guess me.
The most obscure or unusual word is ignored so it's very hard to search for the intersection of a popular thing and an obscure thing.
I have a theory that for many people search has never worked well for them since they just didn't use computers very much, or it was not explained well. Now those of us who found search effective get to see what it was like.
@futurebird I use Kagi. I hope that paying for it reduces their motivation to shittify.
-
The non-computer people I know don't really think search as gotten worse. The people I know who hate computers the most think it's gotten better... but they are in the minority.
I've always found obscure words powerful in searches. That power has been destroyed. Please don't tell me to put it in quotes or use a different engine... this is a design trend, made worse by "AI" I'm not asking for a solution or work around. I want to know WHY the software I depend on keeps getting worse.
@futurebird The degradation of search at Google was a deliberate choice, because they needed to juice search numbers after achieving a monopoly.
They could not organically make more searches happen, because enough of everyone was online and searching.
So, they made search worse so that you had to search more.
This was revealed in the court proceedings in the Google Search monopoly trial, in which Google was convicted (but not punished).
-
@doomsdayrs @futurebird I read about companies that build their own search engines. They still need to buy data from Google and other older search engines. No one can afford to build an international bot network for collecting the data required for search.
-
@futurebird @doomsdayrs it must be so *difficult* to make a searchy thing that doesn't actually search, but instead obfuscates every search and then obfuscates the results.
But I suppose nobody is actually making them. They're mostly using APIs from the Usual Suspects search tools, which are now basically not-search-tools-at-all. ️@FaithfullJohn @futurebird @doomsdayrs random brain fart i had a few days ago: i feel like we should somehow work on a decentralized index where each node is a crawler and an index and a search engine - and the nodes communicate with each other like the torrent network. I feel this might be the only way to get away from platform based restrictions surrounding monetization.
-
There are many scattered people working on similar projects and I think we need to start networking and try to pour our time, money and energy into a single project.
Maybe it could be as simple as "bring back google C. 2010" This wouldn't be cheap but I think enough people might want it to make it happen? Someone might already be doing this. But unless it's a non-profit user-owned type thing with membership dues I don't think it will work in the long run...
@futurebird To be free from Google, the EU invests into an own search machine which is a cooperation of #Ecosia and #Qwant.
No free community would have the financial power you need for such a project! -
@FaithfullJohn @futurebird @doomsdayrs random brain fart i had a few days ago: i feel like we should somehow work on a decentralized index where each node is a crawler and an index and a search engine - and the nodes communicate with each other like the torrent network. I feel this might be the only way to get away from platform based restrictions surrounding monetization.
@missqarnstein @FaithfullJohn @futurebird you're describing yacy
-
@doomsdayrs @futurebird I read about companies that build their own search engines. They still need to buy data from Google and other older search engines. No one can afford to build an international bot network for collecting the data required for search.
@paulc this. @doomsdayrs @futurebird
-
@FaithfullJohn @futurebird @doomsdayrs random brain fart i had a few days ago: i feel like we should somehow work on a decentralized index where each node is a crawler and an index and a search engine - and the nodes communicate with each other like the torrent network. I feel this might be the only way to get away from platform based restrictions surrounding monetization.
@FaithfullJohn @futurebird @doomsdayrs also it might be easy to bolt on user trained and shared bloom filters to customize search results: for flagging ai generated results, transphobic sources, sources of misinformation or flag sources to be biased.
-
@missqarnstein @FaithfullJohn @futurebird you're describing yacy
@doomsdayrs @FaithfullJohn @futurebird
do you have experience with it? Is it useful? -
@doomsdayrs Also if you've already looked into how to feasibly distribute/federate crawls (and their results for search index building), I'd be very interested to hear about it.
I don't think a single person or small organization will be able to map the entire web without exhausting itself to death, so I think it's important to decentralize at the very least the crawling from the beginning.
// @futurebird
@phryk @futurebird I have and I have a solution, wait to hear me announce its functional state eventually
-
@phryk @futurebird I have and I have a solution, wait to hear me announce its functional state eventually
@doomsdayrs You have my attention – but do I really have to wait (and possibly miss it)? What language is this and can I get a look at the code for contribution/inspiration/interop? 🥺
// @futurebird
