Resource Limit Reached...

[mechamind]

I see that last month, the J! Board enabled human verification, which for me is just a simple matter of waiting 3 seconds before the forum loads the first time around (as long as the initial test is passed). Alietr's comment implies that the protection is offered by their web host.

What options would we have here?

[gamesurf]

I’ve ran it up the chain of command to find out what our options are.

[gamesurf]

I shared this on discord, but worth sharing here as well.

There are two main constraints on our site:

1) Resource Limit (which is the maximum amount of data that can be used by people/bots accessing the site at any one time)
2) Bandwidth Limit (which is the amount of bandwidth the site can use in one month)

Both of these are sufficiently high they should be able to handle all regular human traffic and a large number of guests. But with 4,000 bots online reading through thousands of our old pages as "guests", they test both of these.

Imagine a machine dispensing nacho cheese from a hose.

1) If too much comes down the hose at any one time, the hose "clogs". These are the 508 errors that have plagued us for a while. Nobody else can be served until the blockage is cleared. This can be circumvented by refreshing the page until you get through.
2) If too much cheese is dispensed, we run out. That happened this month. On Feb 25, we (well, the bots, mostly) had used all of the site's monthly quota of data. For the next three days the site was inaccessible. On March 1 the quota reset and the site came back.

We're looking at what can be done both to stem the tide of bots, and to make sure our "hose" and "bucket" are adequate for serving delicious nachos for everybody.

A disproportionate amount of bot traffic accesses threads in Host Your Own and Out in Left Field. (I think they must be cached in Google somewhere.) As a test a month ago we tried temporarily restricting Out in Left Field and Host Your Own to be invisible to those not logged in. This didn't result in any reduction in traffic, as the bots were simply redirected to the login page. The experiment was deemed a failure after a day or two and the pages were returned to normal.

ClockGameJohn is aware of the issues and is seeking advice. I have confidence that this is something that can be figured out.

[TPIR98665]

Can you restrict the number of guests allowed 'in' at any point in time?  Say limit to 50 guests.  If guest 51 try to access they get a 'sorry guest limit reached' message..  As I type this I see there are over 400 guests..seems high at least to me...most of these are bots?

[gamesurf]

There are mods in the forum software that can require guests to log in after certain conditions are met. But they have the potential to break the site if poorly implemented and you have to be very careful about which ones you use.

In the last 48 hours I’ve turned on a mod that caps guests at viewing 15 pages a day. After a guest clicks on a 16th page they are shown a screen prompting them to log in.

The immediate effect did not seem to result in any reduction in traffic. More “guests” are sitting at the index page instead of scraping data. The fact that they’re not scraping is good, but they’re still here taking up space. Perhaps over the long term this will frustrate some bots, and they will decide to use their computing power on more “productive” sites. Perhaps the bots are dumb and will just err out forever. Only time will tell.

Can you restrict the number of guests allowed 'in' at any point in time?  Say limit to 50 guests.  If guest 51 try to access they get a 'sorry guest limit reached' message..

There is also a mod available that “caps” guests access to a certain number of guests at any one time. I do not think it would be a good idea to turn it on based on the results of the “15 page cap” experiment.

If we have 1,000 bots attempting to scrape the site, and we set a “cap” at 50 guests, what will actually happen is 50 bots will scrape the site, and 950 bots will attempt to read a page and get an error message. We’d still be dealing with 1,000 bots of traffic, and we probably wouldn’t see any reduction in traffic. But this would frustrate *actual* human guests trying to read the site in a much more obnoxious way than the “15 page cap” does. Downside with no upside.

The solution is probably “stop the bots from accessing the site altogether”, which would require CloudFlare protection or something similar. I’ll bring it up.

[alansh42]

I just had to put my own personal server behind Cloudflare. It's getting pretty much so that you can't have a public site without a front-end filter. Cloudflare's free plan offers bot blocking.

[gamesurf]

We very nearly made it through the month of March, but we ran out of bandwidth on the 30th.

Marc has been contacted regarding potential solutions. Stuff is happening, please be patient.

[gamesurf]

Back again! We saw a further surge of bots in April, resulting in the site hitting a 509 error on April 26th. As of May 1st we're back.

Marc and John are being kept in the loop.

[gamesurf]

Some good news!

A large amount of traffic was coming from a Chinese language search engine that starts with the letter B and ends with U, crawling pages at a much higher rate than Google. Since it doesn’t seem very useful to have our forum ranked on Chinese search engines, about 48 hours ago measures were taken to prevent them from crawling us.

Our “average” guest count immediately dipped by hundreds.

It is too early to say for sure if this could be a long-term solution, but it seems very promising. Other major clusters of bots have been tagged and identified (the biggest remaining culprit by far is the parent company of a popular short-form video app that starts with a T and ends with a K). We’ll observe their behavior for a week or so before determining what else needs to be done.

[TPIR98665]

here is to hoping you have it solved!

[gamesurf]

The good news: since restricting that search engine from scraping, we have stopped getting 508 errors during the site’s normal hours. The site seems much smoother and more responsive.

We’ve also put tags on a number of IP clusters that seem to be from suspected bot farms to log their activity and determine what needs to be done.

The bad news: nature abhors a vacuum, and after hours we have gotten bot swarms from 1AM EST to 8AM EST. Early this morning we set a new site record for traffic at over 7,000 guests online. The vast majority (but not all) of these swarms are from IP addresses beginning with 45.*.*.*. There is a small amount of legitimate traffic coming from that IP address so broadly limiting all traffic from that IP address does not seem to be a good solution.

Overall the site seems to be improved since last week but those swarms are something to keep an eye on and study further. Perhaps we can implement a smarter way to target these.

[richardjm92]

Just got 508'd several times in a row and am noticing that apparently there were just over 8,500 guests on a moment ago. These swarms are annoyingly persistent.

[gamesurf]

There is some legitimate traffic from active users that originates from 45.0.0.0. I made some posts from that IP range myself not too long ago when I was travelling and using my phone.

However, most of the bots in this current swarm seem to be coming from 45.196, 45.197, 45.206, 45.207, and there is no legitimate activity coming from any of those four subnets. I'm seeing what can be done about disallowing access to those four subnets.

Thanks again for everyone's patience as we figure this out.

[mechamind]

The visitor record is now at 8770 as of 42 minutes ago, and it could be growing. I do hope the subnet restrictions help.

[gamesurf]

We have been setting new records today with activity. Not good ones. We've been hitting new peaks with bot swarms as the night has gone on.

As an emergency measure guest access to the forum has been turned off temporarily and I am seeking advice from higher ups who can make changes that only they are authorized to make. I don't anticipate guest access to be disactivated for more than a day or two.