Resource Limit Reached...

[mechamind]

I see that last month, the J! Board enabled human verification, which for me is just a simple matter of waiting 3 seconds before the forum loads the first time around (as long as the initial test is passed). Alietr's comment implies that the protection is offered by their web host.

What options would we have here?

[gamesurf]

I’ve ran it up the chain of command to find out what our options are.

[gamesurf]

I shared this on discord, but worth sharing here as well.

There are two main constraints on our site:

1) Resource Limit (which is the maximum amount of data that can be used by people/bots accessing the site at any one time)
2) Bandwidth Limit (which is the amount of bandwidth the site can use in one month)

Both of these are sufficiently high they should be able to handle all regular human traffic and a large number of guests. But with 4,000 bots online reading through thousands of our old pages as "guests", they test both of these.

Imagine a machine dispensing nacho cheese from a hose.

1) If too much comes down the hose at any one time, the hose "clogs". These are the 508 errors that have plagued us for a while. Nobody else can be served until the blockage is cleared. This can be circumvented by refreshing the page until you get through.
2) If too much cheese is dispensed, we run out. That happened this month. On Feb 25, we (well, the bots, mostly) had used all of the site's monthly quota of data. For the next three days the site was inaccessible. On March 1 the quota reset and the site came back.

We're looking at what can be done both to stem the tide of bots, and to make sure our "hose" and "bucket" are adequate for serving delicious nachos for everybody.

A disproportionate amount of bot traffic accesses threads in Host Your Own and Out in Left Field. (I think they must be cached in Google somewhere.) As a test a month ago we tried temporarily restricting Out in Left Field and Host Your Own to be invisible to those not logged in. This didn't result in any reduction in traffic, as the bots were simply redirected to the login page. The experiment was deemed a failure after a day or two and the pages were returned to normal.

ClockGameJohn is aware of the issues and is seeking advice. I have confidence that this is something that can be figured out.

[TPIR98665]

Can you restrict the number of guests allowed 'in' at any point in time?  Say limit to 50 guests.  If guest 51 try to access they get a 'sorry guest limit reached' message..  As I type this I see there are over 400 guests..seems high at least to me...most of these are bots?

[gamesurf]

There are mods in the forum software that can require guests to log in after certain conditions are met. But they have the potential to break the site if poorly implemented and you have to be very careful about which ones you use.

In the last 48 hours I’ve turned on a mod that caps guests at viewing 15 pages a day. After a guest clicks on a 16th page they are shown a screen prompting them to log in.

The immediate effect did not seem to result in any reduction in traffic. More “guests” are sitting at the index page instead of scraping data. The fact that they’re not scraping is good, but they’re still here taking up space. Perhaps over the long term this will frustrate some bots, and they will decide to use their computing power on more “productive” sites. Perhaps the bots are dumb and will just err out forever. Only time will tell.

Can you restrict the number of guests allowed 'in' at any point in time?  Say limit to 50 guests.  If guest 51 try to access they get a 'sorry guest limit reached' message..

There is also a mod available that “caps” guests access to a certain number of guests at any one time. I do not think it would be a good idea to turn it on based on the results of the “15 page cap” experiment.

If we have 1,000 bots attempting to scrape the site, and we set a “cap” at 50 guests, what will actually happen is 50 bots will scrape the site, and 950 bots will attempt to read a page and get an error message. We’d still be dealing with 1,000 bots of traffic, and we probably wouldn’t see any reduction in traffic. But this would frustrate *actual* human guests trying to read the site in a much more obnoxious way than the “15 page cap” does. Downside with no upside.

The solution is probably “stop the bots from accessing the site altogether”, which would require CloudFlare protection or something similar. I’ll bring it up.