Obsessions of the Searchers

This was written on July 25-26, 2023. Published on July 26, 2023.

---

The ability to search for files is one of the most important things required for a functioning OS. Whether the OS or program indexes the files, uses a tag system, or even indexes the content being able to find what you are looking for is essential. I'm sure many don't see it that way but one of my judges for a good OS or program.

As much as I try, I cannot keep organized. (though I will pat myself on the back and say I have gotten far better over the years).

We are long past the time where we would have only a few files to manage and even longer past the point where most of the files on your computer are directly made by you (I'm talking downloads).

With the way mainstream operating systems are becoming more simplified, file organization is going by the wayside.

I'm writing specifically about iOS and Windows gamers and how most peoples phones will have tons of empty folders and a MASSIVE downloads folder. People just don't organize files. Full stop. Not even gonna say it was better back then because I doubt people were giving a damn then either.

Same goes for Windows users (except some may have a dedicated C drive then put everything elsewhere) and even samer goes for ChromeOS users. Screw it, the same ALSO applies for macOS users. I am exempting Linux, Haiku, BSDs, etc because when it comes to those systems people will just sort their files. This is a fully mainstream problem.

Getting back on track...

Suffice to say while I try to organize my files I tend to forget where certain ones are stored. I've written hundreds of documents and downloaded hundreds of programs over the years and have tried on many separate occasions to try and get it all sorted. Despite my efforts over the years (the most recent attempt was a few weeks prior actually) I haven't found a folder structure that works yet. (I'm getting there though!)

As such I tend to rely on being able to search for either a line of text in the file or know the title off the top. Usually, I just hit the start menu and start typing the keyword.

Years ago when I was getting into Linux and other POSIX-compatible OS' file searching was something that illuded me. With Linux the tools presented vary heavily between what DE you use, Distro, coreutils, etc. As such I needed to use the command line. No problem there but it was a matter of which command or program to use. There's grep of course but like most things Linux there's always 3 or 4 alternatives that their supporters swear by.

Now combine that front with how badly Microsoft ruined the search function in Windows 10+. Clearly, I needed new methods. As such I began looking into what other programs are available to replace this. This article is the result of that labour and research.

The article here half-serves as reference for me. If in the future I ever want to look for a new searcher I have my perspective right here.

This is purely a Windows, Linux and related article. On Android I have always kept my files organized and never had a need for these tools there. I don't know anything about searching files in macOS. Last I checked there was the Finder but I nothing about it. But, excluding a few tools most of these are available on macOS.

So without further ado, let's analyze some 'file-finders', some 'string-locators' and 'index-searchers'.

Section I: The Ones I Don't Like!

Windows 10's Start Menu - As I wrote earlier Windows 10's start menu search is abysmal. It defaults to Bing and will, without fail always try to look for your file on the internet. It will never actually find it when searching local and jams all this other noise and BS into the results. You gotta click to 'Documents' which is completely nebulous and personally never ever found what I was looking for. Oh and it doesn't even work if offline. [1]

File Explorer - Never really worked for me. It has search-in-contents off by default, is hilariously slow and NEVER finds what I'm looking for. Even before I got the tools below, I never used it. I've always considered it a broken a feature and just completely ignore it.

It was always garbage for me regardless of Windows version. I would wonder if they fixed it in Windows 11, but I know better. They instead turned Explorer into a bloated electron app and added useless functionality like tabs (big shock but I really don't care about tabs on explorer's, Linux too).

Everything - This is gonna be the most controversial. Back when I was on Windows 7 there were a few times when the search outright broke. It couldn't find the files I was looking for even if I had them open and knew their place. One of those times I tried using Everything. At the time I was trying to find a file based on a line it had. Everything only searches filenames and I had no luck. I didn't like how it incorporated itself into my task bar either. I don't hate it much as those two above, but it really doesn't work for what I want. For most people, Everything is fine. fd does the same thing and completely destroys it though but that'll be for later.

Thunar - I just haven't had much luck using Thunar's built in search feature. No idea how the other Linux file managers handle search, nor do I care to find out. The CLI tools below are what I use exclusively for searching in Linux. Plus, Thunar does more impressive stuff than searching anyways. I like it in different ways.

PowerShell - I've programmed in PowerShell and have years of experience with it. I just don't like it. Shoot, I find Bash more intuitive. On all my Windows machines I replace Shift + Right Click to open CMD instead. Commands are too long, it's slow, it's annoying to use, it has advertisements in it, just ugh. Syntax is abysmal too, very inconsistent. I've used it for a few scripts here and there and I dread doing so. It doesn't help that work refuses to let my script in anything but PowerShell either, so I've become very familiar with it recently.

I could keep ranting but file searching on PowerShell isn't for me. To point its verbosity just look at this scenario. I am a folder and I want to find the image 'columns.png'. So with PowerShell I would have to do this:

Get-ChildItem -Filter "columns.png" -Recurse -ErrorAction SilentlyContinue -Force

Sure, some of that above stuff isn't necessary but I need it to ignore errors. If I was using fd or rg it's literally just

fd columns.png

That's it! So much simpler. PowerShell has its strengths, but it always feels like a fight whenever I need to do something with it. Just for bonus If I wasn't in a folder then the command could then be Get-ChildItem -Path "C:\users\whatever\idk\" -Filter "columns.png" -Recurse -ErrorAction SilentlyContinue -Force which is even longer. If you like PowerShell, good. But it's not for me.

Section II: How to Find Your Files in Style!

Open-Shell / Windows 7 - I don't know what it was but Windows 7's start menu search feature was excellent. Aside from it crapping out a few times I was always able to find what I want. Windows 10 ruined it but installing Open-Shell restores it just fine. It is what I used for years in pure bliss. I could type in a line or file name, and it would always find it without exception. The only syntax I needed for it was [filename] type:extension and that was just grand. And, with Windows 10 the 'crapping-out' has become extremely rare, only happening twice in the last 6 months. The only issue I have is that it ended up indexing some random localization files from Paradox Interactive and those now always come up regardless of what I search. But ehn, if it bothered me enough, I'd de-index or delete them.

For all the tools past this point I want to write that you should know how Regex works. It can be daunting but it's not so bad once you dive in. I will probably write an article combining all my Regex work to make things easier. If you're searching for a string then knowing Regex makes it so much easier.

Also, past this point they are all command-line tools. If you are scared of the command line don't. It's not so bad. And the tools aren't that hard to use. Hell, when I was researching the only one I found difficult was awk but that'll be explained later. There are 'frontends' for some of these tools but I don't know anything about them. I do not fear that black box we call a terminal.

Here is our target example when it comes to searching.

We have one files to hunt.

C:\raps\track8.txt

And within those files are these lines.

4 navy blue with the windbreaker
5 all black, like an undertaker
6 west wind, zephyr creator
7 secret shelter, safe haven
8 rap game hunter kraven
9 charinus, the obscurist maven


The number correspond to the line number they appear in the text file.

Now let's get searching.

grep - GNU's Grep is one of the most known tools for Linux. Developed in the 70s it is also one of the oldest tools on this list. Still used heavily today it comes standard with most Linux distros. It has easy syntax with just: grep [what to search] [path/to/file/] In our example if I wanted to find line 7 in the text file I could just go grep 'haven' C:\raps\track8.txt and it will find out the exact line. That simple! Course I can also do it to find filenames as well, and even have grep look through binary data and hunt stuff in that. There is a good article below at [2] about a user who used grep to find deleted files. For the longest time this was the goto for Linux and I wasn't even aware other searchers existed. Unfortunately I couldn't get grep working right on Windows so I ended up getting an alternative. Just not interested in installing Cygwin or MSYS2.

Grep also has Regex support, multiple folder searching, recursive searching, using it against output that will be run, combo'ing with other commands, and even binary searching. It is and was for the longest time the gold standard of searching in Linux.

ripgrep / rg - This is what I use now. Be warned, it is written in rust! (personally, don't care but I know some do). It is a recursive searching program inspired by the original grep. It's simple and fast and has all I'm looking for in a file like this. Syntax is relatively the same rg [string] [path\if\you\want].

One feature that that I loved more than grep was that it displayed which line on a text file the string appeared. So if I did rg haven C:\raps\track8.txt it would output 7: secret shelter, safe haven which makes it perfect for looking for stuff in long-winded files. It can target specific types of files too instead of bulk searching everything.

grep has the same thing but do I want to note that rg has the ability to print 'context' of your line. So, like grep it can print both lines above and below, just so you have a better idea of the context of the line.

In terms of speed I haven't noticed much of a difference when it comes to small directories, but I have noticed it a few seconds faster when I recurse against huge swaths of folders.

rg also offers windows binaries right off the bat which is another reason I use it. I have it in my $PATH for easy usage. In terms of speed I find it slower than the Windows 7 Search but that's not really comparable. Windows 7's search has an index while rg starts from scratch each time. I've gotten to the point where if I'm looking for a doc and I know it's general location I'll have rg go look for it instead of having Open-Shell search everything. Is it more efficient in this case? Idk. But I love this tool a lot.

rg was a lifesaver when it came to actually building my site. See I had the template but as I was rewriting it I needed to see where certain CSS classes or items were invoked from. rg made that super easy and was a real godsend as I programmed. Now I just mainly use it for searching through my unorganized notes and it hasn't failed me yet.

The only knock I can give it is that I don't like how it search case-sensitive by default. But they have a flag to stop that. I'm just too lazy to remember it.

find - find is a Unix utility that allows you to find files based on something you give it. In my notes I intended to use it as a way to find file names. In my case it was to go up against the tool 'Everything' but also be less annoying. Syntax isn't the best, but it works. find [location] -name [filename]. It's a little confusing but if you think of it as find in location the file with the -name of filename it makes more sense.

Personally, the main use I could see it for was find [location] -empty and have it find empty folders so I could delete them. It has more functionality but as you'll read with the next tool, it ended my research into find.

fd - fd is just a program to find files. Basically, the same thing as find. It claims to be a 'simple, fast and user-friendly alternative to find'. find appears to have more options but for my case fd is good enough. It offers windows binaries by default so you know I gravitated towards it. Syntax is way simpler than find. Just fd [name] [location / optional] or even just running it from a folder and having it target that. Some people may like find's syntax more since there's more 'specifying' in it (i.e. -name).

It's similarity to rg's syntax makes it perfect to combo since I don't have to remember as much.

fd makes the claim to be faster than find but as I've barely used find I cannot verify that. Ehn, it probably is. I've mostly used it to find files that I know the location of and part of the name. I've also used it to count instances of certain filenames, because sometimes I need a folder of txt files similar names (as in tracks on a specific mixtape, see the examples above).

That's about it for the ones I use heavily. After I got fd and rg working I stopped the search, content I found what I needed.

When it comes to searching my stack is Open-Shell / Windows 7's Search, rg and fd. Those three tools have been everything I need. The tools below I haven't tested but have looked in to.

If something catastrophic were to happen to any of the above tools, then I will return to this article and look for alternatives.

Section III: More Searching!

awk - awk is a full-on programming language that some use for searching. I've briefly used it, but it's got far more important uses than just file searching. I also don't like its syntax very much. Take a look: awk '[string +- regex] {print}' [filename]. A use case I could think of it is when I need very specific information from a whole bunch of documents. In that case I would use awk to scan each file, print out the juicy part and even have it split the parts by a slash or something.

But yeah, the possibilities for this are insane. It goes far deeper than this water droplet of an analysis but as I wrote, it's far too powerful for what I need. I know some people swear by it and there is not shortage of evangelists for it on YouTube. [3]

I also feel this is the hardest of the tools to learn, because, well it's a whole language!

ugrep - ugrep is another searcher but it comes with a user interface! That alone makes it the most user-friendly of the greps, even beating out rg. I don't have a problem with doing command line stuff I find it's interface so cool I will give it a shot at some point. If you're really scared of the terminal and want to wet your feet, then I recommend screwing with ugrep. It looks very pleasant to use.

walk + sor - Gavin Freeborn make a great (and unintentionally soothing) video about it that put it into better words than I. My opinion is that like most of Plan 9, it's real cool but not something I'd commonly use. (sorry, not much of a 9front/Plan 9 guy 'side from its aesthetics). It can be watched here: walk and sor what find should have been

The Silver Searcher / ag - I saw this one mentioned on the rg repo and that rg is faster than it. This program has the best name, beating out all other tools here. It's more geared towards searching code which is something I don't need at the moment. I haven't dealt with projects big enough or scattered enough to warrant it. Skimming its repo its syntax is simple and easy to use. Just evoke the command and what you are looking for.

ack - Another code-searching tool. And, as I said with the tool above, it's not geared towards me. Really it should with how much writers and programmers have in common, but I am content with rg and fd for now. This is also written in Perl which I think is neat. It is headquartered on its page, beyondgrep.com where it proves its efficiency beyond the original grep.

pdfgrep - This is a tool I know I’ll need to install at some point. It's basically grep but scans through and extracts text from PDFs. PDF is such a monstrous format that I can totally see myself needing this to beat a PDF with. I've had mixed results using rg against PDFs so this will hopefully mitigate that. Syntax is relatively identical to grep and rg.

sed - Like awk this is a tool that is very powerful, and beyond the scope of what I need. You feed it a file and it can search, replace, modify and more. From what I understand it's mostly used to replace text in a file but can also easily find lines within one.

locate, plocate and mlocate - The family of locate commands are yet another tool you can use to search. Found long after I had settled on fd+rg these three tools are meant for file and directory searching. Even when I was first starting off with Linux I never really used locate, so these remain foreign to me.

bfs - At the time of writing bfs is the most recent 'finder' released. And it was the one that inspired me to compile my notes and write this article. Unlike find, it makes use of the breadth-first algorithm (it's faster). It is compatible with find and has the same syntax. I don't have much to say on it but that it looks like a suitable replacement for find. I just don't see myself using it right now as I like fd and rg's compatibility. I will write though, watching the demo on its GitHub page it looks simply wonderful.

That's about all the tools I located. If there are any, I missed feel free to let me know. I'll look into them and update the article 'em. I've already covered these many tools. May as well add a few more.

When it comes to in-cloud searching I haven't done that at all. I make sure all my stuff in the cloud is properly sorted (this site's tree included). I haven't really needed any the tools to find stuff there. I assume that if I mounted a share to my computer, I could just use any of these tools to search through.

Section IV: Misc. Tools and a Conclusion

WizTree - Great program but I consider this off-topic. It has helped though when I needed to find a file on another drive. I just popped in the last made CSV and was easily able to know where to start looking before I even put in the drive.

qDirStat - Same deal as WizTree.

winDirStat - I'm gonna assume it's the same deal as WizTree. Never used it though.

DIY - Yeah, just write your own searcher. I probably could to if I put my mind to it. But I'm lazy. But hey, if you feel all of these tools aren't suited for you then you might just need to 'roll your own'. I've written very, and I mean very, rudimentary file searchers with Python and Regex. It's not that hard. I saw one script that I will link below at [4]. That guy made one of the fastest, with only 15 lines of code.

SwiftSearch and DocFetcher - Never used them and barely know anything about them.

But yes, who know there were so many searching tools? And who knew every single one of them was superior to Windows 10's Start Menu search. (well...that should've been obvious).

As for file curation (organization, anti-corruption, de-duplication and so on) that is worth another article entirely. That's about all I had to write.

Back to searching!
~ Charinus

Section V: Further Reading

[1] - I can't find it now but I recall several years ago there being an event when Microsoft was down and it broke every users Start Menu search ability. This occurred for a full day. Will update it with the link once I find it.

[2] - Recovering deleted files using only grep. by Alex Clemmer on his personal website.

[3] - Learning Awk Is Essential For Linux Users by DistroTube.

[4] - Every time I see someone publishing "the fastest find yet", I run some simple benchmarks against my untuned 15-line parallel finder and it's usually a bloodbath. by Per Vognsen on mastodon.social.