Thank you for the input! To be honest, I don’t use Dask often, and as a regular Pandas user, I don’t feel the most qualified to comment—but here we go.
Can this be merged into Pandas?
I’d be honored if something I built got incorporated into Pandas! That said, keeping aiopandas as a standalone package has the advantage of working with older Pandas versions, which is useful for workflows where upgrading isn’t feasible. I also can’t speak to the downstream implications of adding this directly into Pandas.
Pandas does not install tqdm by default.
That makes sense, and aiopandas doesn’t require tqdm either. You can pass any class with __init__, update, and close methods as the tqdm argument, and it will work the same. Keeping dependencies minimal helps avoid unnecessary breakage.
What about Dask?
I’m not a regular Dask user, so I can’t comment much on its internals. Dask already supports async coroutines (Dask Async API), but for simple async API calls or LLM requests, aiopandas is meant to be a lightweight extension of Pandas rather than a full-scale parallelization framework. If you’re already using Dask, it probably covers most of what you need, but if you’re just looking to add async support to Pandas without additional complexity, aiopandas might be a more lightweight option.
> Neither Dask cuDF nor Dask DataFrame provide support for multi-GPU or multi-node execution on their own. You must also deploy a dask.distributed cluster to leverage multiple GPUs. We strongly recommend using Dask-CUDA to simplify the setup of the cluster, taking advantage of all features of the GPU and networking hardware.
Thanks! I originally built this to scratch an itch I had, so I’m really glad you find it useful too. If you have any ideas for improvements or missing features, feel free to suggest them — or even open a PR!
It uses LLM embeddings to sort posts by semantic proximity, but you can also filter out posts with [case-insensitive] comma separated values (click on the filter button and add, for example, "US-Only, On-Site" to the "not contains" input).
Yeah, building something people want (and are willing to pay) is pretty hard, and trying to sell early/often is a good way to reduce your market risk. I'm more on the "build something I want", and as soon as I have something "I like", iterate on the communication/go-to-market, more than the product. I think this is more aligned with founders like Brian Chesky that launched (and failed) multiple times, but kept on going because they really believed in their idea (though I'm still on the failing part).
Yup! I’ve seen them both. It really is a grind to get to the point where something actually sticks. The biggest skill I think I need is just shipping really fast
There have been a few attempts at a crowdsourced-rank search engine (which is similar to what you're suggesting - people indexing the content), but it seems to be a hard cookie, most of the examples of similar ideas I could find on ProductHunt or ShowHN seem dead:
(btw, I just launched this llm-embedding based search service that lets you check if a startup idea has already been tried/failed).
I don't know if this idea has a higher death rate than the baseline, but my guess is Google/PageRank is good enough for most use-cases, and then if you want quality sources, you can just follow them on YouTube, Twitter, Instagram, etc. Wait, maybe I shouldn't try to compete with Google?
If you have any ideas for improvements, missing features, or run into any issues, don't hesitate to share!