Ask HN: Website with 6^16 subpages and 80k+ daily bots
I’m facing a unique challenge with my website: it has a staggering 6^16 (16,796,160) subpages and experiences over 80,000 daily bot requests. This is causing significant performance issues, overwhelming my server resources.
The website is a complex data repository with a deep hierarchical structure. Each subpage displays unique information, making traditional caching solutions ineffective.
My current challenges are:
Server Overloading: The sheer volume of requests, even from bots, is putting an immense strain on my server.
Slow Page Load Times: Even legitimate users experience significant delays accessing the information they need.
Bot Management: Identifying and blocking malicious bots is a constant struggle.
I’m seeking advice on:
Efficient Server Scaling: What are the best practices for scaling a server to handle such a large number of requests?
Caching Strategies: Are there any novel caching approaches suitable for a highly dynamic website like mine?
Bot Mitigation Techniques: How can I effectively identify and filter out malicious bot traffic?
Any insights or solutions from the community would be greatly appreciated. Thanks!