Stop reindexing archived PST files on every computer restart

Enhancement Requests - We'd love to hear about them.

Moderator: Mods

Stop reindexing archived PST files on every computer restart

Postby szchren » Sat Feb 18, 2017 11:36 am

I've used XI with Outlook for years and have 30+ archived PST files which do not change, but I do need them in the index to find information from time to time. Yet at each computer reboot all the archived PST files are re-indexed with the currently active OST or PST file and that takes 15 minutes or so. It would be so much better if only the active OST or PST file were re-indexed and added to the index of all those static, archived PST files unless an option demanded otherwise.
Win 10 Pro, 16GB mem, 512GB SSD
X1 Search 8.5.2 6001si (64bit)
Outlook Client with POP accts and Gmail IMAP
szchren
 
Posts: 15
Joined: Wed May 11, 2016 9:33 am

Re: Stop reindexing archived PST files on every computer res

Postby Kenward » Sun Feb 19, 2017 3:49 am

How are these PST files connected to X1?

Like you, I have a raft of archived files and X1 does not seem to spend any time on them.
MK
X1 Search 8.5.2 - Build 6001si (64-bit)
Windows 10 Pro 64-bit | Windows 10 Home 32-bit
No, I have nothing to do with X1, just a user since 2004.
Kenward
X1 Guru
X1 Guru
 
Posts: 4099
Joined: Tue Apr 20, 2004 2:35 am
Location: UK

Re: Stop reindexing archived PST files on every computer res

Postby szchren » Thu Feb 23, 2017 12:12 pm

They were connected automatically (mostly) by XI when installed as Outlook profiles. So I have 40+ profiles each to an archived PST file. Is there a better way?

Thanks for responding.
szchren
 
Posts: 15
Joined: Wed May 11, 2016 9:33 am

Re: Stop reindexing archived PST files on every computer res

Postby Kenward » Fri Feb 24, 2017 12:41 pm

szchren wrote:So I have 40+ profiles each to an archived PST file.


That's a lot of profiles. I'm surprised that Outlook does not complain.

You can add PST individually to X1 without attaching them to a profile.

Or you can create one profile – email accounts not needed – and attach to it as many PST files as you like. That is the tactic that I have used for years.

I don't know if this would solve the problem. It would certainly make life simpler.
MK
X1 Search 8.5.2 - Build 6001si (64-bit)
Windows 10 Pro 64-bit | Windows 10 Home 32-bit
No, I have nothing to do with X1, just a user since 2004.
Kenward
X1 Guru
X1 Guru
 
Posts: 4099
Joined: Tue Apr 20, 2004 2:35 am
Location: UK

Re: Stop reindexing archived PST files on every computer res

Postby szchren » Fri Feb 24, 2017 3:13 pm

Kenward wrote:
szchren wrote:So I have 40+ profiles each to an archived PST file.


That's a lot of profiles. I'm surprised that Outlook does not complain.

You can add PST individually to X1 without attaching them to a profile.
That's what X1 does automatically resulting in the 40+ profiles I have.

Or you can create one profile – email accounts not needed – and attach to it as many PST files as you like. That is the tactic that I have used for years.
Yes, I have looked into that; create a new outlook profile without an email account and then add each PST individually. But that's 40+ to add manually (I cannot see how to bulk add). So my question is: I could do that but will it index that much faster?

I don't know if this would solve the problem. It would certainly make life simpler.
szchren
 
Posts: 15
Joined: Wed May 11, 2016 9:33 am

Re: Stop reindexing archived PST files on every computer res

Postby Kenward » Sat Feb 25, 2017 3:25 am

szchren wrote:That's what X1 does automatically resulting in the 40+ profiles I have.

X1 does not add all PSTs automatically here. Perhaps because I have not told it to "Enable all PST files found..."

The X1 profile gets created when I manually add a PST file.

In X1 8.5 is is available in the Outlook Email but of the Configured Data Sources settings.

szchren wrote:Yes, I have looked into that; create a new outlook profile without an email account and then add each PST individually. But that's 40+ to add manually (I cannot see how to bulk add). So my question is: I could do that but will it index that much faster?
[/color]

A good start might be to clean up your email system. 40 PSTs? I have about half a dozen.
MK
X1 Search 8.5.2 - Build 6001si (64-bit)
Windows 10 Pro 64-bit | Windows 10 Home 32-bit
No, I have nothing to do with X1, just a user since 2004.
Kenward
X1 Guru
X1 Guru
 
Posts: 4099
Joined: Tue Apr 20, 2004 2:35 am
Location: UK

Re: Stop reindexing archived PST files on every computer res

Postby szchren » Mon Feb 27, 2017 3:04 pm

Kenward wrote:
szchren wrote:That's what X1 does automatically resulting in the 40+ profiles I have.

X1 does not add all PSTs automatically here. Perhaps because I have not told it to "Enable all PST files found..."

The X1 profile gets created when I manually add a PST file.

In X1 8.5 is is available in the Outlook Email but of the Configured Data Sources settings.

szchren wrote:Yes, I have looked into that; create a new outlook profile without an email account and then add each PST individually. But that's 40+ to add manually (I cannot see how to bulk add). So my question is: I could do that but will it index that much faster?
[/color]

A good start might be to clean up your email system. 40 PSTs? I have about half a dozen.


I have now tried it both ways, letting X1 create all 40 profiles, or creating one non-email profile in outlook and then adding each of the 40 PSTs one at a time, and then letting X1 find that one profile with all 40 PSTs in it. Make NO difference to the time to index on start-up and when re-indexing.

I'll try at some point to merge PSTs in larger files, but it begs the question: If a PST is archived and is only needed to when searching for past information (just to read not edit), why can't XI have a category for this so those files do not have indexed automatically each day unless manually requested. In other words, just index the current, dynamic files each day, and then add the archived files to the index. I'm sure such a task is technically feasible.
szchren
 
Posts: 15
Joined: Wed May 11, 2016 9:33 am

Re: Stop reindexing archived PST files on every computer res

Postby Kenward » Tue Feb 28, 2017 5:17 am

Like I said, I just don't see this behaviour when X1 starts.

What indexing regime do you have for Outlook? Perhaps that is the difference.

I have enabled Real Time Indexing. Then it is set for "Every 30 minutes".

My files are on local disks, not a network.

Could something else be accessing those PST files and changing their "access" flags?
MK
X1 Search 8.5.2 - Build 6001si (64-bit)
Windows 10 Pro 64-bit | Windows 10 Home 32-bit
No, I have nothing to do with X1, just a user since 2004.
Kenward
X1 Guru
X1 Guru
 
Posts: 4099
Joined: Tue Apr 20, 2004 2:35 am
Location: UK

Re: Stop reindexing archived PST files on every computer res

Postby szchren » Tue Feb 28, 2017 10:22 am

Kenward wrote:Like I said, I just don't see this behaviour when X1 starts.

What indexing regime do you have for Outlook? Perhaps that is the difference.
Not sure what you mean by regime, I set X1 to index every day at a specific time, which it does (in addition to whenever I restart Windows). My archived PST files are NOT loaded into Outlook, only the current ones.

I have enabled Real Time Indexing. Then it is set for "Every 30 minutes".
I have real time indexing set too, and it works as intended

My files are on local disks, not a network.
My PST files are split. Active PSTs (those that change/are added to all the time) are on an SST, drive C, local. I have created a virtual drive, G:, on that SST, so also local, to store the archived PSTs. I do this so that that are not included in the system backups I do on a daily and weekly basis (they never change).

Could something else be accessing those PST files and changing their "access" flags?

As best I know only XI accesses those archived PSTs on a daily basis. The "date modified" of those files tracks when I last accessed them in X1.
szchren
 
Posts: 15
Joined: Wed May 11, 2016 9:33 am

Re: Stop reindexing archived PST files on every computer res

Postby Kenward » Thu Mar 02, 2017 7:12 am

Once again, you have an atypical arrangement of drives and data sources. I keep everything simple. So I have no experience of virtual drives and how X1 handles them.

You then say that you "set X1 to index every day at a specific time".

Do you know that X1 has ever completed a first index? What you see could be a symptom of that. If the index is not complete, then it will get back to work at startup.
MK
X1 Search 8.5.2 - Build 6001si (64-bit)
Windows 10 Pro 64-bit | Windows 10 Home 32-bit
No, I have nothing to do with X1, just a user since 2004.
Kenward
X1 Guru
X1 Guru
 
Posts: 4099
Joined: Tue Apr 20, 2004 2:35 am
Location: UK

Re: Stop reindexing archived PST files on every computer res

Postby szchren » Thu Mar 02, 2017 4:33 pm

Kenward wrote:Once again, you have an atypical arrangement of drives and data sources. I keep everything simple. So I have no experience of virtual drives and how X1 handles them.
Not really. My G: drive is just part of the SST for backup separation; it is sometimes a good idea to break up a hard drive or SST into partitions. I cannot see how it is important to X1 as long as disc access time are not elongated

You then say that you "set X1 to index every day at a specific time".
That is really no different that you setting it to every 30 min

Do you know that X1 has ever completed a first index? What you see could be a symptom of that. If the index is not complete, then it will get back to work at startup.

Yes, they are always completed. You can see % indexed as X1 is indexing, and I can verify that any of my thousands of emails are in the index by searching, and total count of items in the index that XI provides

Let me ask you some questions since you do no have this issue
    [1] What is the total size in MB or GB of all your archive PSTs?
    [2] Are those archived PSTs always loaded into Outlook, or not incorporated into Outlook but just added to XI via an Outlook profile?
    [3] Do you do the set up manually or did X1 find the profiles and add then automatically?
    [4] How long does the indexing take to complete when you run it every 30 min?
szchren
 
Posts: 15
Joined: Wed May 11, 2016 9:33 am

Re: Stop reindexing archived PST files on every computer res

Postby Kenward » Fri Mar 03, 2017 5:49 am

szchren wrote:What is the total size in MB or GB of all your archive PSTs?

something like 25GB.

X1 reports more than 1 million email items in Outlook > All Email.

szchren wrote:Are those archived PSTs always loaded into Outlook, or not incorporated into Outlook but just added to XI via an Outlook profile?

The answer to the first bit is yes, always loaded into Outlook.

I don't understand the rest of the question. What does "incorporated into Outlook" mean?

szchren wrote: Do you do the set up manually or did X1 find the profiles and add then automatically?

Again question not clear.

X1 finds all the profiles. I have also added a bunch of IMAP accounts manually.

szchren wrote:How long does the indexing take to complete when you run it every 30 min?

0 seconds? I don't notice it. After all, I have I have RTI turned on.
MK
X1 Search 8.5.2 - Build 6001si (64-bit)
Windows 10 Pro 64-bit | Windows 10 Home 32-bit
No, I have nothing to do with X1, just a user since 2004.
Kenward
X1 Guru
X1 Guru
 
Posts: 4099
Joined: Tue Apr 20, 2004 2:35 am
Location: UK

Re: Stop reindexing archived PST files on every computer res

Postby szchren » Sat Mar 04, 2017 12:02 pm

Kenward wrote:
szchren wrote:What is the total size in MB or GB of all your archive PSTs?

something like 25GB.

X1 reports more than 1 million email items in Outlook > All Email.
I have about half that

szchren wrote:Are those archived PSTs always loaded into Outlook, or not incorporated into Outlook but just added to XI via an Outlook profile?

The answer to the first bit is yes, always loaded into Outlook.
I don't understand the rest of the question. What does "incorporated into Outlook" mean?
The archive PSTs don't have to be loaded (i.e., "not incorporated") into Outlook for X1 to index them. I have now tried it both ways and there is no substantial difference in indexing time.

szchren wrote: Do you do the set up manually or did X1 find the profiles and add then automatically?

Again question not clear.
X1 finds all the profiles. I have also added a bunch of IMAP accounts manually.
X1 will create profiles automatically or you can add the PST file manually. Makes no difference to indexing time in my test.

szchren wrote:How long does the indexing take to complete when you run it every 30 min?

0 seconds? I don't notice it. After all, I have I have RTI turned on.

I have RTI turned on too but still it takes 10min+ to index my emails

So having explored all this with you, I really have no explanation for the differences. One last try: My archive PST files are structured with many folders and sub folders for different topics so I have hundreds of them and some emails have large attachments. Do your PSTs have a simpler structure with very few folders?
szchren
 
Posts: 15
Joined: Wed May 11, 2016 9:33 am

Re: Stop reindexing archived PST files on every computer res

Postby Kenward » Sun Mar 05, 2017 3:22 am

szchren wrote:One last try: My archive PST files are structured with many folders and sub folders for different topics so I have hundreds of them and some emails have large attachments. Do your PSTs have a simpler structure with very few folders?


Yes. I must clean it up some day. With X1 there, I don't really need a complicated filing system to find things.

By the way, my index, like X1, is on a m.2 SSD, which is way faster than a regular SSD. I also have a reasonably fast home-rolled 64-bit PC with 16GB of DDR4 memory. (The graphics are "on board" though which is bound to slow things down.)

My Outlook data is split between the SSD and an HDD.

For the PSTs, I do not Store Outlook Email body in the index.
MK
X1 Search 8.5.2 - Build 6001si (64-bit)
Windows 10 Pro 64-bit | Windows 10 Home 32-bit
No, I have nothing to do with X1, just a user since 2004.
Kenward
X1 Guru
X1 Guru
 
Posts: 4099
Joined: Tue Apr 20, 2004 2:35 am
Location: UK

Re: Stop reindexing archived PST files on every computer res

Postby szchren » Sun Mar 05, 2017 12:05 pm

Kenward wrote:
szchren wrote:One last try: My archive PST files are structured with many folders and sub folders for different topics so I have hundreds of them and some emails have large attachments. Do your PSTs have a simpler structure with very few folders?


Yes. I must clean it up some day. With X1 there, I don't really need a complicated filing system to find things.

By the way, my index, like X1, is on a m.2 SSD, which is way faster than a regular SSD. I also have a reasonably fast home-rolled 64-bit PC with 16GB of DDR4 memory. (The graphics are "on board" though which is bound to slow things down.)

My Outlook data is split between the SSD and an HDD.

For the PSTs, I do not Store Outlook Email body in the index.


Here is my set-up
Image
Image

So I don't think there is much difference in our set-ups so I have no idea how to shorten my index time. I do know that I schedule my index daily and you do it every 30 min. I've done all I know including deleting my index and building the index again--takes 2 hours or so. My CPU in a windows 10 home:
Image
Out of ideas!
szchren
 
Posts: 15
Joined: Wed May 11, 2016 9:33 am

Re: Stop reindexing archived PST files on every computer res

Postby cscott » Tue Mar 07, 2017 6:14 pm

Hello all,

We have optimized Outlook incremental "quick" scans to avoid having to re-index folders that have not changed.

This should dramatically improve the speed of Outlook scans, and will certainly go a long way towards alleviating this issue.

This fix will be included in the 8.5.1 release, so stay tuned in the coming weeks!
Chris Scott
Product Manager
X1
cscott
X1 Power User
X1 Power User
 
Posts: 66
Joined: Thu Dec 08, 2016 11:53 am

Re: Stop reindexing archived PST files on every computer res

Postby szchren » Wed Mar 08, 2017 11:07 am

cscott wrote:Hello all,

We have optimized Outlook incremental "quick" scans to avoid having to re-index folders that have not changed.

This should dramatically improve the speed of Outlook scans, and will certainly go a long way towards alleviating this issue.

This fix will be included in the 8.5.1 beta, so stay tuned in the coming weeks!

@cscott
Thanks for stepping into the thread with that. I look forward to the update.

I have a licensed product but not a support subscription so can I still get the beta release? Currently have latest 8.5 6001qm (64-bit)

Also, while this update should greatly speed up indexing, it would still be nice to have an option to prevent indexing at startup and just rely on the scheduled updates we can set up.

Thank again
szchren
 
Posts: 15
Joined: Wed May 11, 2016 9:33 am

Re: Stop reindexing archived PST files on every computer res

Postby dvlamis » Wed Mar 08, 2017 2:51 pm

I found this post when looking at using X1 with my 20 PST files. I too am concerned about my backup program continuously backing up my (static) PST files. Is there a way in the new version to cause X1 to stop updating the time modified on the PST files? I'm OK with X1 NOT allowing me to change the PST files--they are supposed to be static!

I'm looking forward to faster indexing in 8.5.1, but I also want to solve my backup issue.
dvlamis
 
Posts: 2
Joined: Wed Mar 08, 2017 2:38 pm

Re: Stop reindexing archived PST files on every computer res

Postby szchren » Thu Mar 09, 2017 10:54 am

dvlamis wrote:I found this post when looking at using X1 with my 20 PST files. I too am concerned about my backup program continuously backing up my (static) PST files. Is there a way in the new version to cause X1 to stop updating the time modified on the PST files? I'm OK with X1 NOT allowing me to change the PST files--they are supposed to be static!

I'm looking forward to faster indexing in 8.5.1, but I also want to solve my backup issue.

Best solution at the moment, and even if the dates are not changed when the new version is implemented, is to partition your hard drive to create a separate volume (say drive G:\) and then put all the archive PSTs into that volume. Then your backup of your main drive with never backup those files -- you can backup those of G:\ separately.
szchren
 
Posts: 15
Joined: Wed May 11, 2016 9:33 am

Re: Stop reindexing archived PST files on every computer res

Postby dvlamis » Thu Mar 09, 2017 11:57 am

Thanks for the information. I do want to request that X1 consider an enhancement request to allow for "read-only" PSTs. I don't understand why indexing software wants to open a PST read/write. Unless I have it wrong, there is no reason to *write* to an archival PST file. Perhaps I just don't understand the product well enough. But in any case, I appreciate the suggestion and the clear answer.
dvlamis
 
Posts: 2
Joined: Wed Mar 08, 2017 2:38 pm

Re: Stop reindexing archived PST files on every computer res

Postby Kenward » Mon Apr 10, 2017 3:30 am

cscott wrote:This fix will be included in the 8.5.1 release, so stay tuned in the coming weeks!


Any progress on that?

Weeks now run the risk of becoming months
MK
X1 Search 8.5.2 - Build 6001si (64-bit)
Windows 10 Pro 64-bit | Windows 10 Home 32-bit
No, I have nothing to do with X1, just a user since 2004.
Kenward
X1 Guru
X1 Guru
 
Posts: 4099
Joined: Tue Apr 20, 2004 2:35 am
Location: UK

Re: Stop reindexing archived PST files on every computer res

Postby SuperFlyBoy » Tue May 30, 2017 2:04 am

szchren wrote:I've used XI with Outlook for years and have 30+ archived PST files which do not change, but I do need them in the index to find information from time to time. Yet at each computer reboot all the archived PST files are re-indexed with the currently active OST or PST file and that takes 15 minutes or so. It would be so much better if only the active OST or PST file were re-indexed and added to the index of all those static, archived PST files unless an option demanded otherwise.
+1!
SuperFlyBoy
 
Posts: 8
Joined: Sat May 06, 2017 2:05 pm

Re: Stop reindexing archived PST files on every computer res

Postby Kenward » Tue May 30, 2017 5:23 am

SuperFlyBoy wrote:+1!

-1 here.

Some sort of configuration issue?
MK
X1 Search 8.5.2 - Build 6001si (64-bit)
Windows 10 Pro 64-bit | Windows 10 Home 32-bit
No, I have nothing to do with X1, just a user since 2004.
Kenward
X1 Guru
X1 Guru
 
Posts: 4099
Joined: Tue Apr 20, 2004 2:35 am
Location: UK


Return to Enhancements V8

Who is online

Users browsing this forum: No registered users and 16 guests

cron