8.2 supposed IMAP improvements

Please post all issues related to X1 Search 8 Releases here!

Moderator: Mods

8.2 supposed IMAP improvements

Postby VictorWaner » Sun Mar 09, 2014 4:32 pm

Despite

1. X1 repeatedly telling that X1 is not supported on virtual machines; and
2. a user on this forum calling in effect some sort of technical nerd for running X1 on a virtual machine; and
3. X1 running on a virtual machine causing the indexing function to continually run at 100% CPU

X1 would complete indexing if the time schedule for IMAP was set at least 60 minutes.

Now with the beta IMAP indexing never finishes and continual keeps starting again for each IMAP account (i.e. not paying attention to the scheduled time of 60 minutes).

I believe that given how X1 responded to my requests for support, they will ignore this report on their beta for version 8.2 (just as they have ignored my request to restore keyboard shortcuts in the past and also ignoring my recent posting about what the shortcut key is for accessing the menu).

I post it just in case another is having the same or similar problem.

Copernic Desktop Search has no problem in running on a virtual machine and does not run at 100% of CPU.
VictorWaner
 
Posts: 39
Joined: Mon Apr 30, 2012 4:58 am

Re: 8.2 supposed IMAP improvements

Postby pdfct » Sun Mar 09, 2014 10:23 pm

Hi VictorWarner,

Could you clarify for me the exact behavior you are observing with IMAP indexing under the 8.2 beta? From your description, it sounds like the initial indexing cycle of your IMAP(s) account(s) may never complete successfully(?). Or, does the initial scan complete, but subsequent (incremental) scans commence in advance of the 60 minute schedule? X1 connectors are desinged to operate in the following manner. If you are observing behavior different from this, we'd be very interested to hear the details, and to review a set of x1 logs.

Expected behavior of IMAP connectors:
1. the user may configure one or more email connectors based on IMAP (Gmail, Outlook.com, Yahoo!, etc.)
2. to add an IMAP source, the user selects the appropriate email connector from the Manage Data Sources panel, and provides the required account configuration details
3. after selecting the email folders the user wishes X1 to index, the user click OK to finalize connector settings
4. the newly configured source connector is placed in the index queue. Once other active scanners have completed indexing, the IMAP connector will initiate indexing of its account(s)
5. the initial indexing cycle will continue until all of the content residing under the folders the user enabled for indexing have been consumed into index
6. the IMAP scanner will perform an incremental scan to update the index with additions/deletions at an interval defined by the collection schedule (e.g. Every X minutes; Every day at X hour; Never repeat). If the collection schedule is set to Every 30 minutes, this means that an incremental scan cycle should commence 30 minutes after the previous scan cycle has completed.
7. Although a scanner may have been configured with collection schedule, this setting may be overridden by clicking the Start Indexing (Play > control), or by clicking Resume All (from the status menu at the lower right of the GUI).

Please let us know if you are observing indexing behavior inconsistent with this description.

Also, if you are observing that the IMAP scanner(s) are consuming a lot of CPU &/or memory resource, could you please share your system specs (number of CPUs, RAM, OS & bitness)?

Thank you for taking the time to share your observations!

Best Regards,
Alan Anderson
Director Quality Assurance
pdfct
X1 Rep
X1 Rep
 
Posts: 42
Joined: Mon Oct 18, 2004 3:56 pm

Re: 8.2 supposed IMAP improvements

Postby Jogy » Mon Mar 10, 2014 5:26 am

Hi VictorWarner,

Just wanted to add a few things about the Imap plugin:
- when adding a new account, on the last page when showing the folders, also the number of messages in each folder is shown. Is there a folder with more than 1000 messages?
- during indexing, you can check the progress of the scanner either in the status popup in the lower right corner, or in the data sources dialog - there is a tooltip that shows detailed information in the format "Processing item x of y in folder FOLDERNAME" - you can check that information to monitor the progress and check whether it really starts from scratch in each folder.
- there is a logic that of a folder has more than 1000 items, indexing of that folder will stop, the scanner will move onto next folder(s) and accounts, and when they finish, it will immediately return to the unfinished folder and continue where it dropped of. Is it possible that you are experiencing that situation?

Jogy
Jogy
X1 Super User
X1 Super User
 
Posts: 387
Joined: Mon Oct 31, 2011 9:19 am

Re: 8.2 supposed IMAP improvements

Postby VictorWaner » Tue Mar 11, 2014 6:55 am

I am happy to provide information to X1 about this, but the last go-around with trying to get support, X1 clearly indicated that running X1 in a virtual machine is NOT supported. In essence the last message I received from X1 when requesting support was: if running X1 in a virtual machine works: fine; if not or slowly etc: tough, you are on your own.

For me, it makes little difference whether the scanning is occurring all the time or not for the IMAP accounts. What the real problem is that X1 is continually running at 100% of CPU. My only comparison is with a competitor (Copernic Desktop Search) which appears to be scanning all the time but does not consume 100% of CPU. If it had a couple of features that X1 has which Copernic does not then I would be using that program rather than X1.

Specific replies:

From your description, it sounds like the initial indexing cycle of your IMAP(s) account(s) may never complete successfully(?).


Under 8.1.1 the initial indexing cycle did complete.

Or, does the initial scan complete, but subsequent (incremental) scans commence in advance of the 60 minute schedule? X1 connectors are designed to operate in the following manner. If you are observing behavior different from this, we'd be very interested to hear the details, and to review a set of x1 logs.


Under 8.1 I set the 60 minutes schedule time, because subsequent (incremental) scans for each of the 4 IMAP accounts would not always complete within 60 minutes. Now under 8.2, the schedule for (incremental) scans is still scheduled at 60 minutes, as soon as one of the scans completes for 1 of the 4 IMAP accounts, another starts. IE they never stop, or rather after closely watching the window showing the progress of the various accounts/outlook/files, scanning does actually complete for the IMAP accounts, but then almost immediately (i.e. in less than 1 minute) starts again.

- when adding a new account, on the last page when showing the folders, also the number of messages in each folder is shown. Is there a folder with more than 1000 messages?


All of the IMAP accounts have more than 1000 messages (one over 8000, another almost 3000, the third around 1500, the 4th around 3500).

- during indexing, you can check the progress of the scanner either in the status popup in the lower right corner, or in the data sources dialog - there is a tooltip that shows detailed information in the format "Processing item x of y in folder FOLDERNAME" - you can check that information to monitor the progress and check whether it really starts from scratch in each folder.


After observation, it appears when the scheduled scanning takes place the strip under the name of each account does not start from the beginning.

- there is a logic that of a folder has more than 1000 items, indexing of that folder will stop, the scanner will move onto next folder(s) and accounts, and when they finish, it will immediately return to the unfinished folder and continue where it dropped of. Is it possible that you are experiencing that situation?


Could be (but then I do not know the internal workings of X1.


Also, if you are observing that the IMAP scanner(s) are consuming a lot of CPU &/or memory resource, could you please share your system specs (number of CPUs, RAM, OS & bitness)?


Spec:
Image

Running under vmWare Fusion 5.0 on

Image

From vmWare Fusion setup:

Image
VictorWaner
 
Posts: 39
Joined: Mon Apr 30, 2012 4:58 am

Re: 8.2 supposed IMAP improvements

Postby Jogy » Tue Mar 11, 2014 8:36 am

VictorWaner wrote:Under 8.1 I set the 60 minutes schedule time, because subsequent (incremental) scans for each of the 4 IMAP accounts would not always complete within 60 minutes. Now under 8.2, the schedule for (incremental) scans is still scheduled at 60 minutes, as soon as one of the scans completes for 1 of the 4 IMAP accounts, another starts. IE they never stop, or rather after closely watching the window showing the progress of the various accounts/outlook/files, scanning does actually complete for the IMAP accounts, but then almost immediately (i.e. in less than 1 minute) starts again.


The current behavior is when there are still unfinished folder (due to the 1000 items limit), to restart scanning immediately until all items are indexed.
When indexing finishes, then it should start respecting the schedule and scan for new emails every 60 minutes or whatever is set.

Also, for the new version we are considering to lift the 1000 items limit, which will mean that an Imap account won't stop until all items in all folders are scanned.

VictorWaner wrote:Spec:
Image

Running under vmWare Fusion 5.0 on

Image

From vmWare Fusion setup:

Image

[/quote]

Is it possible increase the number of processor cores in the virtual machine? While indexing and searching X1 runs several processes in parallel, and running all of them on a single core could be the reason for 100% CPU consumption. On a side note, there will be some internal Imap scanner optimizations in the next Beta version, which may reduce a bit the CPU and memory consumption.

Jogy
Jogy
X1 Super User
X1 Super User
 
Posts: 387
Joined: Mon Oct 31, 2011 9:19 am


Return to X1® Search 8 Releases

Who is online

Users browsing this forum: No registered users and 13 guests