If you go to Server Configuration, Index, you can check some details. Especially when the last Index optimization happened.

If you re-index the whole server, it can take hours, days, even weeks - depending on your Retain server size.

There is another part you can consider - update to a newer version  !

I wish I could upgrade to a newer version, but at the time that is forbidden.

I see the index optimization is set to run daily: Last Index Optimization 02-Dec-2021 02:02:01

I also see :

Re-indexes all messages and updates existing index with changes. Searches will work as normal during the re-index process.

Index optimization will require 206.86 GB of free disk space on the drive. (1 segments will be merged).

What do you see if you check index cluster?

similar to yours

Indexed: 18848487, unindexed: 0, failed: 16791

So I assume that your index is okay. 

You have mentioned that you cannot search after login. After any login? Or only after restarting Retain?

I can browse on the first tab, but on the second tab search just sits there and does nothing, after the 5 minutes my search for * comes back immediately(87 ms)

So does this happen any time, not only after your morning restart?

So does this happen any time, not only after your morning restart?

However do not forget that you have an opportunity to open a case. Especially for older versions of Retain (4.8 is one of these) I received some small "code pieces" to fix issues.

I have the server reboot every day now and after the reboot at 6am the first search takes about 5 minutes or the browser times out, the next search is fine.  The same thing happens if I restart retain-tomcat.  So all my users think the server is broken and the next time I try it works perfect.  The users hate this feature.

Joe, my experience is that Retain needs some time until all services are fully functional. Especially tomcat services need some time.

Move your reboot or restart of services to an earlier time of activity. So Retain will be ready at an early point.

Why do you restart retain-tomcat? Is there a reason? I see Retain server which are running for weeks and even months without restart ...

after the db is re-indexed maybe it will be working better,  This virtual machine has 25gb of ram and never looks busy.  We reboot it because apache or tomcat eventually fail and users can't search or login to retain for archived mail.

Well I did the "Re-index Failed messages" button yesterday and it seemed that some searches would not work.  I look in the logs today and I see a lot of searches after my reboot that look good but one at 8:36 AM that did not finish till around 8:42 AM.  The query results show a query that took around 5 minutes.

When I do a * query it normally shows me the number of messages I have in retain instantly.  But there are some times it does not work till after this large burst of disk traffic.

Here is proof that after the reboot I think retain users were still doing successful queries:

retain:~ # cat  /opt/beginfinite/retain/tomcat/logs/RetainServer.2021-12-07.log | grep "Search elapsed milliseconds in"
06:45:53, 781[ajp-nio-48009-exec-10] [TRACE] SearchAjaxCommon: Search elapsed milliseconds in RetainIndex=200
06:46:00, 446[ajp-nio-48009-exec-5] [TRACE] SearchAjaxCommon: Search elapsed milliseconds in RetainIndex=1550
08:14:08, 393[ajp-nio-48009-exec-1] [TRACE] SearchAjaxCommon: Search elapsed milliseconds in RetainIndex=2274
08:14:09, 066[ajp-nio-48009-exec-6] [TRACE] SearchAjaxCommon: Search elapsed milliseconds in RetainIndex=1028
08:14:11, 071[ajp-nio-48009-exec-8] [TRACE] SearchAjaxCommon: Search elapsed milliseconds in RetainIndex=1213
08:14:19, 156[ajp-nio-48009-exec-4] [TRACE] SearchAjaxCommon: Search elapsed milliseconds in RetainIndex=3315
08:14:19, 211[ajp-nio-48009-exec-9] [TRACE] SearchAjaxCommon: Search elapsed milliseconds in RetainIndex=595
08:14:23, 668[ajp-nio-48009-exec-5] [TRACE] SearchAjaxCommon: Search elapsed milliseconds in RetainIndex=1253
08:14:35, 500[ajp-nio-48009-exec-6] [TRACE] SearchAjaxCommon: Search elapsed milliseconds in RetainIndex=289
08:14:52, 082[ajp-nio-48009-exec-10] [TRACE] SearchAjaxCommon: Search elapsed milliseconds in RetainIndex=257
08:15:04, 562[ajp-nio-48009-exec-4] [TRACE] SearchAjaxCommon: Search elapsed milliseconds in RetainIndex=265
08:15:16, 971[ajp-nio-48009-exec-5] [TRACE] SearchAjaxCommon: Search elapsed milliseconds in RetainIndex=258
08:18:48, 548[ajp-nio-48009-exec-10] [TRACE] SearchAjaxCommon: Search elapsed milliseconds in RetainIndex=296
08:18:49, 956[ajp-nio-48009-exec-6] [TRACE] SearchAjaxCommon: Search elapsed milliseconds in RetainIndex=54
08:19:04, 018[ajp-nio-48009-exec-8] [TRACE] SearchAjaxCommon: Search elapsed milliseconds in RetainIndex=457
08:19:04, 835[ajp-nio-48009-exec-2] [TRACE] SearchAjaxCommon: Search elapsed milliseconds in RetainIndex=184
08:20:21, 608[ajp-nio-48009-exec-4] [TRACE] SearchAjaxCommon: Search elapsed milliseconds in RetainIndex=136
08:20:24, 278[ajp-nio-48009-exec-5] [TRACE] SearchAjaxCommon: Search elapsed milliseconds in RetainIndex=64
08:20:25, 103[ajp-nio-48009-exec-3] [TRACE] SearchAjaxCommon: Search elapsed milliseconds in RetainIndex=62
08:20:25, 526[ajp-nio-48009-exec-6] [TRACE] SearchAjaxCommon: Search elapsed milliseconds in RetainIndex=62
08:20:25, 695[ajp-nio-48009-exec-1] [TRACE] SearchAjaxCommon: Search elapsed milliseconds in RetainIndex=53
08:20:49, 349[ajp-nio-48009-exec-1] [TRACE] SearchAjaxCommon: Search elapsed milliseconds in RetainIndex=233
08:20:50, 487[ajp-nio-48009-exec-2] [TRACE] SearchAjaxCommon: Search elapsed milliseconds in RetainIndex=104
08:28:56, 712[ajp-nio-48009-exec-9] [TRACE] SearchAjaxCommon: Search elapsed milliseconds in RetainIndex=280
08:29:08, 237[ajp-nio-48009-exec-3] [TRACE] SearchAjaxCommon: Search elapsed milliseconds in RetainIndex=163
08:29:10, 166[ajp-nio-48009-exec-7] [TRACE] SearchAjaxCommon: Search elapsed milliseconds in RetainIndex=65
08:29:40, 141[ajp-nio-48009-exec-4] [TRACE] SearchAjaxCommon: Search elapsed milliseconds in RetainIndex=241
08:29:41, 449[ajp-nio-48009-exec-1] [TRACE] SearchAjaxCommon: Search elapsed milliseconds in RetainIndex=71
08:31:04, 804[ajp-nio-48009-exec-8] [TRACE] SearchAjaxCommon: Search elapsed milliseconds in RetainIndex=247
08:31:25, 658[ajp-nio-48009-exec-4] [TRACE] SearchAjaxCommon: Search elapsed milliseconds in RetainIndex=61
08:34:39, 707[ajp-nio-48009-exec-7] [TRACE] SearchAjaxCommon: Search elapsed milliseconds in RetainIndex=341
08:34:40, 767[ajp-nio-48009-exec-5] [TRACE] SearchAjaxCommon: Search elapsed milliseconds in RetainIndex=83
08:34:52, 789[ajp-nio-48009-exec-4] [TRACE] SearchAjaxCommon: Search elapsed milliseconds in RetainIndex=54
08:34:54, 659[ajp-nio-48009-exec-10] [TRACE] SearchAjaxCommon: Search elapsed milliseconds in RetainIndex=55
08:34:56, 717[ajp-nio-48009-exec-4] [TRACE] SearchAjaxCommon: Search elapsed milliseconds in RetainIndex=41
08:35:02, 713[ajp-nio-48009-exec-1] [TRACE] SearchAjaxCommon: Search elapsed milliseconds in RetainIndex=158
08:35:04, 933[ajp-nio-48009-exec-6] [TRACE] SearchAjaxCommon: Search elapsed milliseconds in RetainIndex=98
08:35:07, 956[ajp-nio-48009-exec-1] [TRACE] SearchAjaxCommon: Search elapsed milliseconds in RetainIndex=102
08:36:11, 225[ajp-nio-48009-exec-6] [TRACE] SearchAjaxCommon: Search elapsed milliseconds in RetainIndex=244
08:36:15, 513[ajp-nio-48009-exec-8] [TRACE] SearchAjaxCommon: Search elapsed milliseconds in RetainIndex=341
08:36:16, 748[ajp-nio-48009-exec-10] [TRACE] SearchAjaxCommon: Search elapsed milliseconds in RetainIndex=225
08:41:42, 351[ajp-nio-48009-exec-6] [TRACE] SearchAjaxCommon: Search elapsed milliseconds in RetainIndex=302449
08:41:43, 394[ajp-nio-48009-exec-7] [TRACE] SearchAjaxCommon: Search elapsed milliseconds in RetainIndex=179199
08:44:02, 977[ajp-nio-48009-exec-10] [TRACE] SearchAjaxCommon: Search elapsed milliseconds in RetainIndex=743
08:44:04, 208[ajp-nio-48009-exec-2] [TRACE] SearchAjaxCommon: Search elapsed milliseconds in RetainIndex=388
08:44:07, 678[ajp-nio-48009-exec-5] [TRACE] SearchAjaxCommon: Search elapsed milliseconds in RetainIndex=203

3

Joe, try to see it in this way ...

After a reboot or restart of services it will take some time until all processes are ready to work. Retain needs a strong coffee to be back on duty.

Some people need the same strong coffee in the morning.

However do not hesitate to open a case!

as you can see I had 41 successful searches until the one took about 5 minutes.  This server was rebooted at 6am and you can see the first person kicking the tires was 645 AM, so I would say 45 minutes is enough coffee.  It appears there is something wrong with our server doing a search with just the asterisk character.  Once it does that search once the rest of the day doing a * search is fast.

Yes, 45 minutes is too much coffee! Too dangerous for health. Let support check your system!

Hi Diethmar,

it might be our firewall as our subnets are routed via a firewall that may have a timeout on stateless https traffic.

I am going to do a server on the same subnet and see if that shows the results after 297000ms... :-)  I also want to see if the timeout is apache, tomcat or javascript and see if I can extend that.  This server is not on the internet so I am not concerned at the moment about DOS protection measures.

Thanks,

Joe