-

System Apps

Directory Help Language Random Channel Report Bug Search

Fri, 11 Oct 2024 19:24:46 -0400

Hubzilla Support Forum

Hubzilla Support Forum

adminsforum@hubzilla.org

Marshall Sutherland wrote the following post Fri, 11 Oct 2024 19:24:45 -0400

Something seems seriously wrong with this channel. I looked at its clone on another server and the last new message from this channel it sees is 14 days ago. I swapped to another channel on my server which has a connection to this channel and it also sees nothing newer than 14 days ago. It looks like I have made 22 posts since the last one I see. There have been a few responses, so some of them are leaking out. I am seeing other people's new posts and comments.

I have turned my logging up to all, but I really don't know what I should be looking for.

show all 32 comments

Fri, 11 Oct 2024 19:37:43 -0400

Marshall Sutherland

dwatney@hub.farthinghalearms.com

I don't know if this relates at all, but I notice that the connection lists between my main and clone channels do not match. They are close, but they do not match.

Fri, 11 Oct 2024 20:27:35 -0400

Marshall Sutherland

dwatney@hub.farthinghalearms.com

So maybe an hour ago, I made a test channel on my server. Then I had each of my 2 existing channels connect to it. My test channel still has not received connection notices for me to accept.

Based on that, I predict that if I post a test message from my 2nd channel, I won't see if on my primary. Let's find out!

Fri, 11 Oct 2024 20:43:31 -0400

Marshall Sutherland

dwatney@hub.farthinghalearms.com

Apparently, the replies I have been posting from my server aren't arriving here.

To summarize, I made a test channel and tried to connect to it from my existing 2 channels. My test channel never received the connection notices. I posted a test message from my 2nd channel and my main channel did not see it. So, the problem isn't just my main channel. My whole server is a hot mess.

Fri, 11 Oct 2024 21:04:16 -0400 last edited: Fri, 11 Oct 2024 21:08:13 -0400

Markus

Markus

markus@hubzilla.markusgarlichs.de

@Marshall Sutherland I had a similar problem at the beginning, when I started my server, that I did not receive many posts. I only noticed it when I saw the corresponding messages in my web server log. There were HTTP 504 (gateway timeout) for several seconds/minutes.

I had a look at the Fediverse Observer page and saw that your server is occasionally unavailable. https://hubzilla.fediverse.observer/hub.farthinghalearms.com

In any case, the solution was that I made better settings to the database server.

I hope this could be a hint to the solution.

Fri, 11 Oct 2024 21:16:03 -0400

Marshall Sutherland

dwatney@hub.farthinghalearms.com

I've had 3 outage in the past 2 weeks or so due to hurricanes. I suspect now that my problems started after the second outage when I didn't get a chance to properly shut down my system before it lost power.

I haven't made any database changes in quite a while, but your suggestions gives me the idea that I should see if I can do some sort of integrity check on my database.

Fri, 11 Oct 2024 21:26:08 -0400

Marshall Sutherland

dwatney@hub.farthinghalearms.com

mysqlcheck reports all my tables are ok in all of my databases...

Fri, 11 Oct 2024 21:26:59 -0400

Markus

Markus

markus@hubzilla.markusgarlichs.de

@Marshall Sutherland I think that's a good approach, even if I see that there are occasional availability outages. I suspect that the outages cannot all be related to a hurricane. They shouldn't actually exist.

Fri, 11 Oct 2024 21:35:00 -0400

Markus

Markus

markus@hubzilla.markusgarlichs.de

@Marshall Sutherland

i added this lines to my /etc/mysql/my.cnf (MariaDB) for a better performance.

innodb_buffer_pool_size=2048M
tmp-table-size=512M
max-heap-table-size=512M
query_cache_size=512M
max_connections=200

Important is the max_connections value. The values depend on the equipment of the server 8GB Ram or 16GB etc.

Fri, 11 Oct 2024 21:44:32 -0400

Marshall Sutherland

dwatney@hub.farthinghalearms.com

I just realized those were months, not days on that up-time graph. I also found the error list section. I think I'd have to agree that they aren't all hurricane related. Still, something changed dramatically 2 weeks ago such that channels on my own server stopped communicate with each other.

I'll take a look at those settings.

Fri, 11 Oct 2024 21:52:15 -0400

Markus

Markus

markus@hubzilla.markusgarlichs.de

@Marshall Sutherland There may have been changes to the Hubzilla code that affected the behavior of the database.
In any case, it is helpful to monitor the web server log as the Fediverse Observer only checks accessibility once an hour.

Fri, 11 Oct 2024 22:14:11 -0400

Marshall Sutherland

dwatney@hub.farthinghalearms.com

None of those options were specified, so I tried setting them all, except for query_cache_size which was not recognized.

Fri, 11 Oct 2024 22:23:30 -0400

Markus

Markus

markus@hubzilla.markusgarlichs.de

@Marshall Sutherland Try out the values - you may have to adjust them again. The number of connections should now ensure that all requests are processed. I hope that this will also solve your problem.

Fri, 11 Oct 2024 22:50:52 -0400

Marshall Sutherland

dwatney@hub.farthinghalearms.com

My connection count is in single digits and I haven't actually noticed one from Hubzilla yet, but they obviously must be sneaking in now and then.

I just remembered something else that could be relevant -- delivery reports. Here on my clone's server, I get a delivery report that is 2 pages long. On my server, I get nothing but an error saying "no results". I checked every post I have made as far back as I have the option to get delivery reports (the last was 8 days ago) and that is all any of them do. The only thing that shows up in the hubzilla log is a message about my profile photo that loads on the header of the mostly empty delivery report page.

Fri, 11 Oct 2024 22:58:38 -0400

Marshall Sutherland

dwatney@hub.farthinghalearms.com

the last dreport_time in my dreport table is 9/27 which was.... 14 days ago!

Fri, 11 Oct 2024 23:25:16 -0400

Markus

Markus

markus@hubzilla.markusgarlichs.de

@Marshall Sutherland Yes, that is strange. Normally, the server should synchronize everything with the clone. However, I can't say whether a corresponding delivery report is created in the same way. I'm sure someone else can answer this question for you.

Sat, 12 Oct 2024 08:41:58 -0400

Mario Vavti

Mario Vavti

mario@hub.somaton.com

Looks to me like your queueworker got stuck. Maybe due to a PHP error? Do you see any fatal errors in your PHP log?

Sat, 12 Oct 2024 10:44:41 -0400

Marshall Sutherland

dwatney@hub.farthinghalearms.com

My PHP error log hasn't had a new message since May, which seems improbable, but I have confirmed with phpinfo() that it is the right file. I'm deleting it to see if it gets recreated.

Sat, 12 Oct 2024 10:53:22 -0400

Marshall Sutherland

dwatney@hub.farthinghalearms.com

I moved the log file to /var/log in case there was some funky permission thing going on. I restarted apache, confirmed the changed with phpinfo, posted a new test message, and no php error file has appeared yet.

Sat, 12 Oct 2024 11:10:06 -0400

Marshall Sutherland

dwatney@hub.farthinghalearms.com

I guess I should mention that my server is on v9.4.3. The last 2 updates required some manual intervention before they were successful in pulling down the new code cleanly due to some file permission issues.

Sat, 12 Oct 2024 11:30:41 -0400

Markus

Markus

markus@hubzilla.markusgarlichs.de

@Marshall Sutherland Since the last two updates were not so long ago, it is noticeable that they fall roughly in the period in which your problems began. I'm just wondering why file permissions had to be adjusted during the updates.

Sat, 12 Oct 2024 12:26:16 -0400

Marshall Sutherland

dwatney@hub.farthinghalearms.com

@Markus I'm not really sure what happened there. I have the updater running as www-data in a cron job nightly so I don't have to deal with it myself, but some of the files were owned by my account. Maybe, at some point, I updated things manually causing the ownership of some of the files to change. or maybe I was doing some debugging in the past resulting in some files being saved as me. Perhaps I would be better off having the files owned by me and running the updater as me. My solution to the problem was to just delete the problem files and let git (running as www-data) recreate them.

Sat, 12 Oct 2024 17:01:48 -0400

Markus

Markus

markus@hubzilla.markusgarlichs.de

@Marshall Sutherland Unfortunately, at the moment, I have no other idea that could help solve the problem. My original idea that the server's connection problems are making it difficult to synchronize the clone could be true, but it could of course also be due to something else. Are the posts you have recently written still not reaching the clone?

Sat, 12 Oct 2024 17:47:53 -0400

Marshall Sutherland

dwatney@hub.farthinghalearms.com

@Markus I have still not seen any of the messages I posted from my primary account on my clone. A few people have seen them, as I posted test messages asking people to respond. The guy who runs the server my clone is on has seen some, but some test messages have had no response at all. Checking a 2nd channel on my server, it still has seen nothing new from me that wasn't posted from my clone.

@Mario Vavti The lack of any errors in the php error log has been bothering me. I purposefully caused an error and no new log file has been created. So, something is wrong here even though the settings look fine. That will be my focus, as seeing those errors could be the key to solving the original problem.

Sat, 12 Oct 2024 17:53:24 -0400

Marshall Sutherland

dwatney@hub.farthinghalearms.com

Well, that turned out to be easy enough to fix. It was a permission issue. I moved the php log to the same folder as the hubzilla log and now it works. Now, hopefully, things will get interesting...

Sat, 12 Oct 2024 18:13:58 -0400

Marshall Sutherland

dwatney@hub.farthinghalearms.com

Unfortunately, things have not gotten interesting. I posted a new test message and there has not been a single new message in the error log.

Sat, 12 Oct 2024 18:17:45 -0400 last edited: Sat, 12 Oct 2024 18:21:41 -0400

Marshall Sutherland

dwatney@hub.farthinghalearms.com

My log was probably working where it originally was, before I moved it to /var/log. There just were no errors to report and I didn't think to force an error until after I moved it.

Sat, 12 Oct 2024 18:25:01 -0400

Marshall Sutherland

dwatney@hub.farthinghalearms.com

Tagging @Jason Cook, who runs the site where my clone is, so he can follow along.

Sat, 12 Oct 2024 23:29:30 -0400

Marshall Sutherland

dwatney@hub.farthinghalearms.com

I think I got it!!! Things are flowing now!

Inside the workerq table was a Cron task. Every time the cron job would run it, the Queueworker would say "Sorry, I've already got a Cron task and bail out." Maybe a Cron task got added just before the power went out and never got processed? Anyway, seeing that the Queue worker was bailing on my Cron task because it already had one, I deleted it. My queue seems to be going crazy now.

Sorry for dumping 2 weeks of messages on you all at once!

Sat, 12 Oct 2024 23:38:58 -0400

Scott M. Stolz

Scott M. Stolz

scott@biztechtonics.net

@Marshall Sutherland Where do I delete the old task?

Sat, 12 Oct 2024 23:49:11 -0400

Marshall Sutherland

dwatney@hub.farthinghalearms.com

In the workerq table, there was a record that got added for the Cron task. Mine is no longer there, but I suspect the workerq_cmd field said Cron.

My queue is exploding as all the formerly stuck tasks are turning into delivery tasks for people.

Sun, 13 Oct 2024 04:14:57 -0400

Markus

Markus

markus@hubzilla.markusgarlichs.de

@Marshall Sutherland I am pleased that you have found a solution and thank you for briefly explaining it. I can imagine that this information can help others.

Sun, 13 Oct 2024 10:47:35 -0400

Marshall Sutherland

dwatney@hub.farthinghalearms.com

That was pretty wild. I saw my queue go over 4500 as individual postings got turned into delivery tasks. It took hours to process it all, but it is down to 0 now.