Forum very unreliable lately

B
Burgerbob
Posts: 6327
Joined: Apr 23, 2018

by Burgerbob »

Forum seems to be down for long stretches, multiple times a day lately. Usually the HTTPS error, but sometimes others.
A
AndrewMeronek
Posts: 1487
Joined: Mar 30, 2018

by AndrewMeronek »

Yes, I've noticed this too.
D
Doug_Elliott
Posts: 4155
Joined: Mar 22, 2018

by Doug_Elliott »

I thought it was just bad signal because I'm in Spain right now.
J
JohnL
Posts: 2529
Joined: Mar 23, 2018

by JohnL »

Same here, and I'm nowhere near Spain.
B
BGuttman
Posts: 7368
Joined: Mar 22, 2018

by BGuttman »

Hit me too. Never more than a couple of hours, though.
T
tbdana
Posts: 1928
Joined: Apr 08, 2023

by tbdana »

Same here. California.
W
WGWTR180
Posts: 2152
Joined: Sep 04, 2019

by WGWTR180 »

Ditto.
O
Oslide
Posts: 205
Joined: Apr 03, 2018

by Oslide »

Same here in the center of Europe. Thanks to those who will take care of it :good:
T
timothy42b
Posts: 1812
Joined: Mar 27, 2018

by timothy42b »

I haven't noticed here in central Virginia. But there has been a lot of solar activity lately; that might have something to do with it.
C
CalgaryTbone
Posts: 1460
Joined: May 10, 2018

by CalgaryTbone »

Ditto for Western Canada.

JS
B
boneberg
Posts: 216
Joined: Dec 19, 2020

by boneberg »

Same here in Germany
H
HawaiiTromboneGuy
Posts: 1025
Joined: Sep 03, 2018

by HawaiiTromboneGuy »

Spotty here in Hawaii.
I
ithinknot
Posts: 1339
Joined: Jul 24, 2020

by ithinknot »

Cloudflare Error 522 is the usual
D
Doug_Elliott
Posts: 4155
Joined: Mar 22, 2018

by Doug_Elliott »

I would assume that Matt is already aware and working on it.
G
ghmerrill
Posts: 2193
Joined: Apr 02, 2018

by ghmerrill »

Bizarre. I was having similar problems two days ago (so on the 25th). Just couldn't connect, and I was getting errors pointing to the server. But yesterday it completely cleared up and I've had no problems since then. Speculation is pretty senseless, I suppose, but I do note that in some areas (like mine) both Brightspeed and Spectrum are in a kind of frantic expansion mode, particularly in rural areas. Two days ago, contractors laying Spectrum fiber along my road cut through a water pipe -- leaving the area without effective water pressure for a day and a half. And yesterday, Brightspeed contractors were (again -- third time in about three months) upgrading the module on the utility pole at the end of my driveway to support more customers. So my direct internet connection was out for about two hours.

I wouldn't be at all surprised to learn that the forum problems aren't really related to the forum server at all. It's been working great for me since yesterday. :?
B
Burgerbob
Posts: 6327
Joined: Apr 23, 2018

by Burgerbob »

It's clearly the server.
G
ghmerrill
Posts: 2193
Joined: Apr 02, 2018

by ghmerrill »

[quote="Burgerbob"]It's clearly the server.[/quote]
How so? It's just difficult for me to imagine how I went from total inability to connect back to normal activity while others in other areas have been experiencing connection and performance issues. So I'm curious how you can tell its the TromboneChat server that's the issue.
B
Burgerbob
Posts: 6327
Joined: Apr 23, 2018

by Burgerbob »

I have not a single problem with any other thing I connect to, but the forum is down several hours a day (including 11pm before I went to bed last night).
G
ghmerrill
Posts: 2193
Joined: Apr 02, 2018

by ghmerrill »

[quote="Burgerbob"]I have not a single problem with any other thing I connect to, but the forum is down several hours a day (including 11pm before I went to bed last night).[/quote]
Okay, I"m not denying that it's an issue with SOME server (or some network path to the forum server). But I haven't had any noticeable problem for well over 24 hours at this point. I guess I could put it in ping loop to see if it's dropping out on a continuous basis, but even that wouldn't point the finger at it directly.

Anyhow, it doesn't really matter and I'm sure Matt will fix it. I was just curious about what diagnostics you might be using.
B
boneberg
Posts: 216
Joined: Dec 19, 2020

by boneberg »

[quote="Burgerbob"]I have not a single problem with any other thing I connect to, but the forum is down several hours a day (including 11pm before I went to bed last night).[/quote]

My experience is similar. Everything else worked fine. :idk:
B
BPBasso
Posts: 96
Joined: Mar 31, 2025

by BPBasso »

I've been experiencing the same issues as well. Usually 1-2 hrs of down time. Sometimes it'll load the "Active Topics" page, but fail to load any of the threads I open in another tab. Or, it'll load 1-2 threads, but none of the others.

It just went down for an hour or so. Just coming up around 2pm CST
B
Burgerbob
Posts: 6327
Joined: Apr 23, 2018

by Burgerbob »

Yup, just went down for me too. Connection lost error this time.
F
Finetales
Posts: 1482
Joined: Mar 23, 2018

by Finetales »

Me three. Error screen rather than just not loading like the past couple days.
P
Posaunus
Posts: 5018
Joined: Mar 23, 2018

by Posaunus »

Yup, it's been in & out here in California, no matter what the connection method. Clearly a TC server problem.
A
AtomicClock
Posts: 1094
Joined: Oct 19, 2023

by AtomicClock »

[quote="BPBasso"]It just went down for an hour or so. Just coming up around 2pm CST[/quote]
If you mean 2pm CDT, then the same thing happened for me.
B
BPBasso
Posts: 96
Joined: Mar 31, 2025

by BPBasso »

[quote="AtomicClock"]<QUOTE author="BPBasso" post_id="279756" time="1751051283" user_id="19343">
It just went down for an hour or so. Just coming up around 2pm CST[/quote]
If you mean 2pm CDT, then the same thing happened for me.
</QUOTE>

Sorry I've never called it anything but Central or CST.
G
ghmerrill
Posts: 2193
Joined: Apr 02, 2018

by ghmerrill »

Yeah, I've been in the hole for a while now: getting email messages about postings, but can't log on. :lol: From what I can see the forum is just bumping up against resource allocations and can't serve the demand.
T
tbdana
Posts: 1928
Joined: Apr 08, 2023

by tbdana »

It has been about as reliable as my chops lately. And that is not a good thing.
B
BrianJohnston
Posts: 1165
Joined: Jul 11, 2020

by BrianJohnston »

It's been about as reliable as the missing mail page on USPS.
A
AndrewMeronek
Posts: 1487
Joined: Mar 30, 2018

by AndrewMeronek »

[quote="ghmerrill"]Yeah, I've been in the hole for a while now: getting email messages about postings, but can't log on. :lol: From what I can see the forum is just bumping up against resource allocations and can't serve the demand.[/quote]

There are a number of things that can cause resource problems on servers. I've been a server admin in the past. There's a lot. It could be anything from a software glitch to a drive filling up, to a failing memory module. Whoever administrates the server running this Forum needs to be able to log in and do some diagnostics.
H
hyperbolica
Posts: 3990
Joined: Mar 23, 2018

by hyperbolica » (edited 2025-07-02 2:42 p.m.)

It's an intermittent problem for me (Virginia), could happen any time of day, so I don't think it's load related. Happens on mobile and laptop, local fiber connection and 5G. TC is the only site (including other phpbb sites) that has the problem. Local internet is good, in-home wifi is good. Error screen says it's the TC server. Server might be randomly rebooting, might be infected with a resource hogging virus, might be sunspots effecting server connection, etc...
<ATTACHMENT filename="Screenshot 2025-07-02 143730.jpg" index="0">[attachment=0]Screenshot 2025-07-02 143730.jpg</ATTACHMENT>
B
BrassSection
Posts: 424
Joined: May 11, 2022

by BrassSection »

Last two days in central PA it’s been good. Prior is was slow or non-usable. We were also having trouble with our internet for about two weeks. That’s been better today.
A
AtomicClock
Posts: 1094
Joined: Oct 19, 2023

by AtomicClock »

[quote="BPBasso"]Sorry I've never called it anything but Central or CST.[/quote]
I hope you're not scheduling any meetings.
B
BPBasso
Posts: 96
Joined: Mar 31, 2025

by BPBasso »

[quote="AtomicClock"]<QUOTE author="BPBasso" post_id="279767" time="1751055377" user_id="19343">
Sorry I've never called it anything but Central or CST.[/quote]
I hope you're not scheduling any meetings.
</QUOTE>

Not with anyone who couldn't handle figuring it out, luckily! :amazed:
U
UrbanaDave
Posts: 97
Joined: Mar 26, 2024

by UrbanaDave »

The forum went down again at 8PM CDT yesterday <EMOJI seq="1f643" tseq="1f643">🙃</EMOJI>
M
mgladdish
Posts: 155
Joined: Oct 10, 2021

by mgladdish »

I've been getting lots of Cloudflare timeouts over the past few days.
W
WGWTR180
Posts: 2152
Joined: Sep 04, 2019

by WGWTR180 »

This morning I was trying to logon and
P
Posaunus
Posts: 5018
Joined: Mar 23, 2018

by Posaunus »

Unreliable and IMPOSSIBLY slow. Multiple "time outs" trying to load. What's with the server?
R
RoscoTrombone
Posts: 251
Joined: Oct 17, 2018

by RoscoTrombone »

UK here & experiencing the same issues. It's not been accessible most of the day today. It's been on/off over the last couple of weeks.
B
BGuttman
Posts: 7368
Joined: Mar 22, 2018

by BGuttman »

I've been experiencing the same thing today. Fixing this issue is WAY above my pay grade (and capability!).
B
Burgerbob
Posts: 6327
Joined: Apr 23, 2018

by Burgerbob »

Yup, very bad this last week. Beuller?
J
jpwell
Posts: 154
Joined: May 11, 2018

by jpwell »

Is the ship on auto pilot or is someone in control???
E
EriKon
Posts: 636
Joined: Apr 03, 2022

by EriKon »

I was locked out for a whole day. Now back in again.
T
tbdana
Posts: 1928
Joined: Apr 08, 2023

by tbdana »

I think I found the 15 minutes today when the forum is loading normally. I wonder if it's the same time every day. And wonder if they're ever going to fix it.
C
CalgaryTbone
Posts: 1460
Joined: May 10, 2018

by CalgaryTbone »

Similar issue yesterday.

JS
J
JohnL
Posts: 2529
Joined: Mar 23, 2018

by JohnL »

[quote="jpwell"]Is the ship on auto pilot or is someone in control???[/quote]
Matt K, who seems to do the nuts and bolts of keeping things up and running, last signed in on June 21. He hasn't posted since March.
B
blast
Posts: 671
Joined: Mar 22, 2018

by blast »

It's out during the day in the UK. Works late evening. Odd.
P
Posaunus
Posts: 5018
Joined: Mar 23, 2018

by Posaunus »

[quote="JohnL"]<QUOTE author="jpwell" post_id="280725" time="1752087321" user_id="3273">
Is the ship on auto pilot or is someone in control???[/quote]
Matt K, who seems to do the nuts and bolts of keeping things up and running, last signed in on June 21. He hasn't posted since March.
</QUOTE>

Oops.

Please come back, Matt. We need you! :cool:
D
Doug_Elliott
Posts: 4155
Joined: Mar 22, 2018

by Doug_Elliott »

I texted Matt. I'll post when he replies.
S
slidesix
Posts: 107
Joined: Jan 03, 2025

by slidesix »

For me I’ve noticed it is unreliable during the day but okay in the evening (US Eastern Time)
C
CalgaryTbone
Posts: 1460
Joined: May 10, 2018

by CalgaryTbone »

Really bad today - tried to log on multiple times, and the first time that worked was after 10PM local time.

JS
J
jabice
Posts: 23
Joined: Nov 03, 2024

by jabice »

I'm a software engineer and would be happy to help. phpBB is written in PHP and likely has a mySQL or SQLite database; these are somewhat adjacent to my specialties.
P
paysonmcc
Posts: 71
Joined: May 21, 2019

by paysonmcc »

I’ve noticed that when it is really bad the number of users will say that there is 12 registered users online, and up to 1,400+ unregistered. Check out the bottom of the home page- it looks like we’re getting a lot of artificial traffic recently. Is there a way to stop that? The only way I can think is to require an account to access the forum.
D
dwcarder
Posts: 53
Joined: Jun 27, 2023

by dwcarder »

That's what I noticed too. In another group, this was attributed to bots scraping content and bringing the database to a crawl. I'd be happy to contribute to help get things stabilized.
P
Posaunus
Posts: 5018
Joined: Mar 23, 2018

by Posaunus »

MUCH better today.

Thanks Matt? :cool:
D
Doug_Elliott
Posts: 4155
Joined: Mar 22, 2018

by Doug_Elliott »

Matt is working on it and preparing to move it to a better service. He will probably check in today. I think he has a full time job plus a new kid, so maybe he just hasn't had time to deal with it.
M
mgladdish
Posts: 155
Joined: Oct 10, 2021

by mgladdish »

It would be nice to be able to spread the load and have more than just Matt who can look after the board. Everyone has stuff that comes up from time to time and having such a useful resource as this dependent on the availability of just one person will inevitably lead to issues.

I'm more than willing to help, and I'm also in the UK which will give cover over more of the day.
T
Trav1s
Posts: 473
Joined: Jul 26, 2018

by Trav1s »

[quote="paysonmcc"]I’ve noticed that when it is really bad the number of users will say that there is 12 registered users online, and up to 1,400+ unregistered. Check out the bottom of the home page- it looks like we’re getting a lot of artificial traffic recently. Is there a way to stop that? The only way I can think is to require an account to access the forum.[/quote]

Bots scraping date to feed AI?
D
Doug_Elliott
Posts: 4155
Joined: Mar 22, 2018

by Doug_Elliott »

Who is online

In total there are 1094 users online :: 13 registered, 2 hidden and 1079 guests

Here are the listed Bots:

Ahrefs [Bot],

Amazon [Bot],

Bing [Bot],

Google [Bot],

Google Adsense [Bot],

Semrush [Bot]
G
ghmerrill
Posts: 2193
Joined: Apr 02, 2018

by ghmerrill »

By comparison,

TubaForum.net: 249 users online, 6 registered, 243 guests

Euphonium-Tuba Forum: 341 users online, 2 members, 339 guests

Trumpet Herald: 496 users online, 4 registered, 492 guests.

I won't try to interpret those figures in any meaningful way. I suppose it's quite possible that there are just a LOT more trombonists in the world then euphers and tubists (I wouldn't be surprised), although Werden's forum lists the most ever online as 26,956 on April 17, 2024. Of course, many of those might be bots. :? Maybe there are just a lot more forum-watching trombonists. But the trumpet forum figures seems to suggest that something is a bit funny with this forums usage figures -- don't you think? It's "most users ever online" is 1797.
B
BPBasso
Posts: 96
Joined: Mar 31, 2025

by BPBasso »

[quote="Doug Elliott"]Who is online

In total there are 1094 users online :: 13 registered, 2 hidden and 1079 guests

Here are the listed Bots:

Ahrefs [Bot],

Amazon [Bot],

Bing [Bot],

Google [Bot],

Google Adsense [Bot],

Semrush [Bot][/quote]

Geez.

At the time of my post: "In total there are 1392 users online :: 14 registered, 3 hidden and 1375 guests (based on users active over the past 5 minutes)

Most users ever online was 2384 on Mon Apr 14, 2025 6:57 am"

Reminds me of the influencers and streamers who use bots to boost likes/views. I don't know how any of that works.. This forum doesn't have ads that I've seen. Is the forum being maliciously targeted? DDoS type of thing?
P
ParLawGod
Posts: 133
Joined: Mar 11, 2019

by ParLawGod »

[quote="ghmerrill"]By comparison,

TubaForum.net: 249 users online, 6 registered, 243 guests

Euphonium-Tuba Forum: 341 users online, 2 members, 339 guests

Trumpet Herald: 496 users online, 4 registered, 492 guests.

I won't try to interpret those figures in any meaningful way. I suppose it's quite possible that there are just a LOT more trombonists in the world then euphers and tubists (I wouldn't be surprised), although Werden's forum lists the most ever online as 26,956 on April 17, 2024. Of course, many of those might be bots. :? Maybe there are just a lot more forum-watching trombonists. But the trumpet forum figures seems to suggest that something is a bit funny with this forums usage figures -- don't you think? It's "most users ever online" is 1797.[/quote]
We were having problems at TubaForum.net with DDoS attacks a while back (I do the tech. side of that board). I'm not privy to things here, but maybe something similar is happening when it lists over 1000 "users" online...the server is being overloaded. We upgraded our firewall to help with that and haven't had any issues since.
G
ghmerrill
Posts: 2193
Joined: Apr 02, 2018

by ghmerrill »

[quote="ParLawGod"]We upgraded our firewall to help with that and haven't had any issues since.[/quote]
I know that Dave Werden was having problems of similar scope and managed to add more effective bot protection. That assault nowadays is simply relentless. Some years ago I set up and managed a phpBB board for the community band I was in (along with an email server), and simply didn't need to worry about this problem. Now, it's a major factor.

As an aside ... that board I set up failed -- in the sense of not achieving "user acceptance". At that time the band was heavily weighted in the direction of people in their 60s and 70s who simply couldn't/wouldn't adapt to online technology (basically, wouldn't even try to because it was "too complicated"). Fast forward almost 10 years, and I'm sure that people in that age group are now almost 100% comfortable with the use of online environments -- largely because they now have to be in order to manage their medical care and appointments. :lol: Of course, now invasive bots are a problem they didn't used to be. :roll:
M
Matt_K
Posts: 4809
Joined: Mar 21, 2018

by Matt_K »

A few people have identified the problem: there is an excessive amount of traffic that is nefarious in nature, almost certainly originating from LLM scraping. There is a reasonably good Hackernews thread on this:

https://news.ycombinator.com/item?id=43397361

It isn't unusual for us to see sustained 8MB/s outbound for an extensive period. A firewall would likely be helpful (if you wouldn't mind emailing me the details of what you've firewalled off, I would appreciate it). The concern I have with firewalls is that there is already a significant issue with reports of bans, and I don't want to exacerbate that problem for genuine users.

It appears that most of the bans are originating from individuals behind CGNAT, particularly when using AT&T, Verizon, and Comcast networks. (Humorously, I actually briefly banned myself when I logged in yesterday to post the announcement, as I am posting from behind Comcast CGNAT at the moment.)

I actually conducted a test yesterday with an LLM and found that asking a question that I had validated as not being found on other sites (Reddit, in particular) resulted in an immediate and significant spike in traffic that lasted for hours. We've added some Cloudflare mitigations, and I could really put the screws down on it too... You'll notice that with other phpBB sites, it's fairly common for EVERY SCREEN to have a recaptcha now (check out the odroid site, for example: https://forum.odroid.com/viewforum.php?f=175). That is a miserable experience that I would prefer to avoid.

And it's a real shame because I would personally prefer LLMs to be able to scrape our site, as long as it's done reasonably and respectfully. One of the main aims was to provide a corpus of knowledge about trombone, and LLMs are going to be the way people access that type of content... but non-stop scraping to merely validate nothing has changed is just excessive.

The concern about me disappearing is well-founded, although I should be back now. I had some personal stuff happen (as noted!), and I think everything is mostly winding down. I also had several people threaten me personally with lawsuits over moderation, and I really didn't want to deal with that with everything else I had going on. I probably would have had a more aggressive reaction had I not been dealing with all the personal issues. Perhaps it might be best for me to step back from moderating or take a back seat on that front.

At any rate, the reason I'm mentioning that is that there was some concern about the backup being lost and what would happen if I were to disappear. It's a valid concern, and setting up something where the backups would be publicly available (though with obvious redactions, such as excluding PMs and user data) is technically feasible; however, there might be some complications to it. And I am not a lawyer, so it's quite possible the concerns I have are not valid. In the event someone DOES want their posts deleted, what does that mean practically? If backups are posted, would we need to modify the backups retroactively? Does it depend on jurisdiction? Those are all questions that would need to be answered and actively monitored.

Though there may be workarounds or alternative solutions. I'm going to focus on hardening the site this week and will follow up with leadership later in the week/next week regarding those other items. Do note that we have a pretty rigorous backup process that both Neobri and I have access to, so if the server farm gets sucked into the earth, we can still recover and be at worst a few days out of sync. However, someone would need the technical know-how to restore the site, although it's pretty straightforward, so that if I were to go away, a suitable substitute should not be hard to find.
M
Matt_K
Posts: 4809
Joined: Mar 21, 2018

by Matt_K »

Speaking of, in case you've been wondering what has happened since the last post, we've been undergoing a major DDoS attack that was peaking at about 38k requests per second. I've had to resort to that hammer I was referring to, which will put an annoying cloudflare popup much more frequently than I would like (possibly as much as every time you load a page...) but it seems to have stopped the issue.

I have some errands to run but am monitoring the situation. The steps I'm taking tomorrow should help, and I'll be investigating how I can filter out these as obviously we do not naturally have 38k+ visitors per second, as cool as we are.
A
AndrewMeronek
Posts: 1487
Joined: Mar 30, 2018

by AndrewMeronek »

Great news!

I wish I had more relevant experience with online security systems (and could help, at least with advice), but alas although I have IT experience that's not in my background.
E
eln
Posts: 17
Joined: Jun 05, 2019

by eln »

I will admit to having asked trombone questions to LLMs recently, and watched them as they scoured the forum for answers.
G
ghmerrill
Posts: 2193
Joined: Apr 02, 2018

by ghmerrill »

[quote="eln"]I will admit to having asked trombone questions to LLMs recently, and watched them as they scoured the forum for answers.[/quote]
Ultimately, there is great benefit in AIs making the knowledge and instruction available on forums widely and quickly to all of us. We WANT the "right" AIs to do this. At the moment we just lack the model and mechanism to achieve these goals.
M
Matt_K
Posts: 4809
Joined: Mar 21, 2018

by Matt_K »

FWIW I do have a plan to provide LLMs with accessible access. Actually, several... going to focus on shoring up making the site usable for HUMANS first!!! Then circle back. The sledgehammer approach would be a read-replica that only bots can use which would be a mirror of this site. It isn't trivial to set something like that up but it also isn't an enormous burden. I may be able to finesse the cloudflare controls I setup yesterday but given that they seem to be working, I'm reluctant to "fix" something that isn't broken
G
ghmerrill
Posts: 2193
Joined: Apr 02, 2018

by ghmerrill »

[quote="Matt K"]FWIW I do have a plan to provide LLMs with accessible access.[/quote]
If you do that, it might be worth writing it up and publishing it ... say in Software Practice and Experience. I notice that in June they published a paper on "Safeguarding Networks From Malicious Intrusions ...". SP&E seems to have drifted in a somewhat more academic direction than when I dealt with them several eons ago, but I still think they might go for good a application and case study -- and people would benefit from seeing it. I mean, the title still includes practice and experience. :roll:
J
jabice
Posts: 23
Joined: Nov 03, 2024

by jabice »

Thanks for the hard work, Matt. I'll take the CF pop-ups over the hours-long outages!
T
tbdana
Posts: 1928
Joined: Apr 08, 2023

by tbdana »

[quote="jabice"]Thanks for the hard work, Matt. I'll take the CF pop-ups over the hours-long outages![/quote]

Ditto
M
Matt_K
Posts: 4809
Joined: Mar 21, 2018

by Matt_K »

At good to hear! I suspect this will stop the “bans” that were happening. This is a bit of a double edged sword here— a few people have been unable to get past cloudflare which I’m working on (emailing me is the best if somehow you see this and cannot get in). I was seeing the logs that indeed phpBB was treating people on - for example - Verizon as a single user which is fine for even a few thousand organic users but starts to fall over when its tens and hundreds of thousands all requesting the same thing non stop :amazed:
R
robcat2075
Posts: 1867
Joined: Sep 03, 2018

by robcat2075 »

[quote="Matt K"]... we've been undergoing a major DDoS attack that was peaking at about 38k requests per second.[/quote]

What does the attack get them?

i suppose it's an honor to be noticed but what does crowding out a trombone forum get them? It' s not a business that competes with anyone.

Is it really a general attack on every site that the host hosts?
M
Matt_K
Posts: 4809
Joined: Mar 21, 2018

by Matt_K »

Definitely not seeing this on other sites in this provider. I have several of my own sites on the same host (actually, there's another site hosted on the same server), and it gets virtually no traffic.

There are really two somewhat plausible explanations based on the patterns, and both would have a similar pattern. They aren't merely going to "trombonechat.com" - the front page - the traffic is going to specific, and different pages. Otherwise, the redis queue would serve up the page to an anonymous user from the cache, and there wouldn't be an issue. Each one goes to a unique thread and specific page. Because of the way phpBB works, each of these requires a brief connection to retrieve the page from the database.

To me that means either: 1) someone knows this and is/was intentionally, maliciously trying to bring the site down (there have been some heated exchanges, though nothing really IMO that would ever cause me to want to do something like that but :idk: ) or 2) it's LLMs that essentially constantly ping every page they've ever scraped to not miss out on whatever hot takes we have here. As silly as the latter may be, I have seen numerous reports from other similar sites with even less content than we have here, about identical traffic patterns.
T
Trav1s
Posts: 473
Joined: Jul 26, 2018

by Trav1s »

I work at major US bank as PM in IT and would agree that 2 is the likely option. As a younger engineer likes to say, the LLM is looking for the hot goss from the forum and is always on the quest to find it...

Thanks Matt for your work behind the scenes!

Like the Wizard of Oz, we know the important one is the man behind the curtain.
S
ScottZigler
Posts: 26
Joined: Jul 19, 2023

by ScottZigler »

Just want to put this here for Matt K:

I use an RSS reader (feedly.com) to read posts, and since 7/15 11:30A Eastern Time, there are no new posts coming over. I'm assuming it is seeing Feedly scraping and it looks just like an LLM scraping. There may be no fix, or maybe it's a long term fix (read-only copy for the bots) but just wanted to make you aware. Thanks for all you do to keep this running!
M
Matt_K
Posts: 4809
Joined: Mar 21, 2018

by Matt_K »

[quote="ScottZigler"]Just want to put this here for Matt K:

I use an RSS reader (feedly.com) to read posts, and since 7/15 11:30A Eastern Time, there are no new posts coming over. I'm assuming it is seeing Feedly scraping and it looks just like an LLM scraping. There may be no fix, or maybe it's a long term fix (read-only copy for the bots) but just wanted to make you aware. Thanks for all you do to keep this running![/quote]

Thanks for the report, that is unfortunately a necessary side-effect of the filtering I had to put in place. I do have a few ideas for how I might re-enable it, or enable it in a different form once I get this up and running. I'm almost there... I have all of the site migrated, database cloning in place, etc.

I'm hoping that I'm dealing with the last migration bug as I type... CSS and stylings are not showing up for some reason now, but I have the database synchronizing properly now. If I can get that running today, I'll probably have a solution for RSS feeds shortly thereafter.

The way that I'm architecting the site now will allow substantially more control over access and rerouting. Even at the peak amount of requests we were getting, that actually isn't THAT much traffic in the grand scheme of things, so enabling it is really just a matter of elbow grease in fitting a square peg in a round hole and getting phpbb to do something it wasn't wholly designed for.
S
ScottZigler
Posts: 26
Joined: Jul 19, 2023

by ScottZigler »

Any updates on RSS feeds? Is this just a lost cause? Thanks for all you do!

[quote="Matt K"]<QUOTE author="ScottZigler" post_id="281798" time="1753194822" user_id="16830">
Just want to put this here for Matt K:

I use an RSS reader (feedly.com) to read posts, and since 7/15 11:30A Eastern Time, there are no new posts coming over. I'm assuming it is seeing Feedly scraping and it looks just like an LLM scraping. There may be no fix, or maybe it's a long term fix (read-only copy for the bots) but just wanted to make you aware. Thanks for all you do to keep this running![/quote]

Thanks for the report, that is unfortunately a necessary side-effect of the filtering I had to put in place. I do have a few ideas for how I might re-enable it, or enable it in a different form once I get this up and running. I'm almost there... I have all of the site migrated, database cloning in place, etc.

I'm hoping that I'm dealing with the last migration bug as I type... CSS and stylings are not showing up for some reason now, but I have the database synchronizing properly now. If I can get that running today, I'll probably have a solution for RSS feeds shortly thereafter.

The way that I'm architecting the site now will allow substantially more control over access and rerouting. Even at the peak amount of requests we were getting, that actually isn't THAT much traffic in the grand scheme of things, so enabling it is really just a matter of elbow grease in fitting a square peg in a round hole and getting phpbb to do something it wasn't wholly designed for.
</QUOTE>
M
Matt_K
Posts: 4809
Joined: Mar 21, 2018

by Matt_K »

Definitely not a lost cause. I've been monitoring the site for a while, and it seems that the changes/optimizations/etc. that I did are sustainable for reliable operations. I've been a little sidetracked starting a band ( :good: ) and am back at work now that my paternity leave ended ( :weep: ). Have our first gig next week and then I'll circle back on trying to re-enable some of the lost features, like RSS feeds. The biggest problem is going to be keeping some of the protections we get from Cloudflare (virtually 99%+ of the traffic was bots) but allow, well, what you could technically call a bot through.

What that is probably going to look like is going to be publishing an rss.trombonechat.com and enabling people to paste the stuff after trombonechat.com/etc.etc.etc. into it to get such feeds. The problem is that if I use phpbb, it treats every access as a single, "anonymous" user for this type of traffic, which requires serious resources from the DB, which impedes reliability for like, you know, real users. So I'll likely spin up some kind of read replica and have the RSS feeds be slightly behind the real site.

Unfortunately, I do not intend to put the classifieds sections - as was the case before too - on RSS, because we got tons of scammers when that was enabled for non-logged-in users. Technically, there's nothing really stopping people from doing that, but it seems to put enough friction in place that many of the previous, very pervasive problems do not exist, but still people need to do due diligence, etc., as this is still a minor roadblock. As usual, the <.1% of bad people make life worse for the honest people.
B
BGuttman
Posts: 7368
Joined: Mar 22, 2018

by BGuttman »

I've noticed that CloudFlare is making me check a box every time where it didn't used to.

Wonder if my Internet connection is to blame.
M
Matt_K
Posts: 4809
Joined: Mar 21, 2018

by Matt_K »

[quote="BGuttman"]I've noticed that CloudFlare is making me check a box every time where it didn't used to.

Wonder if my Internet connection is to blame.[/quote]

Looking at your ISP, or at least what the forum thinks your ISP is, that is very possible. You appear to be behind a CGNAT, which basically means that several "people" share your IP address. Cloudflare looks at patterns not just from our site but from ALL sites they are involved with and ratchets up the filtering when it detects that IP is causing larger than expected traffic.

That said, it's unlikely that you have an internet neighbor (or perhaps literal neighbor, as it will still be likely people physically close to you) is causing large amounts of traffic aimed at us. I'll dial down some of the protections this weekend and see if that helps, and still allows the site to effectively function.
B
BGuttman
Posts: 7368
Joined: Mar 22, 2018

by BGuttman »

My ISP is in a nursing home and I probably share my connection with a large number of other users, although I'd bet none of them are looking for trombone stuff. I've been here for a long time. When you first initiated the CloudFlare check I'd sometimes get the box to click but way more often it would "pass" me directly on. Having to click the box every time isn't a deal breaker by any means, though.