How to remove files from index

You've got questions? We think we have some answers!

Moderator: Mods

How to remove files from index

Postby liver_state » Wed Feb 16, 2011 5:34 pm

Hi,

New user of X1 Pro, and I'm amazed at how capable this product is!
However, I have emails in my index which shouldn't be there.

My situation is:
* Indexed an Outlook account.
* Added several large archive PST files.

So far so great. Then...I realized that I had attached to the wrong outlook account...

* So I deleted the emails, and deleted them from the trash folder... which didn't remove them from X1 index... in fact, this just caused the items in the index to show "Unable to locate email message" in the X1 preview pane.
* Then I removed the outlook folder (that contained the deleted emails) from the sources list (unchecked)... but the emails are still in the index.

Is there any way for me to delete those entries from the index?

Running Win7/64 and x1 Pro 6.7.2
liver_state
 
Posts: 9
Joined: Wed Feb 16, 2011 5:23 pm

Re: How to remove files from index

Postby Kenward » Thu Feb 17, 2011 3:25 am

You have a couple of options. Reindex everything, or wait.

If you have only a small amount of data already indexed, then you can tell X1 to start all over again.

>>Tools
>>Options
>>Indexing
>>Clear Index Files

Probably the better approach is to wait. X1 eventually recognises that you have removed something and will expunge the references from the index.

If you don't want these items to show up in a search, then you can create a "Saved search" that excludes the PST file that holds these email messages. Come back if you need detailed instructions on how to do that.
MK
X1 Search 8.6.1 - Build 6003fa (64-bit)
Windows 10 Pro 64-bit | Windows 10 Home 32-bit
No, I have nothing to do with X1, just a user since 2004.
Kenward
X1 Guru
X1 Guru
 
Posts: 4149
Joined: Tue Apr 20, 2004 2:35 am
Location: UK

Re: How to remove files from index

Postby Chris Wheaton » Thu Feb 17, 2011 8:36 am

If you will be using this search repeatedly I do agree with Kenward; using a saved search is a great idea. I use several saved searches daily in my support work with our customers.

If you believe you may need this specific search only once or rarely there is another option.

With the X1 UI focused on your email search go to Tools > Columns. On the left find Path. Select it and click the arrow to add it to the selected columns list. Back on the list on the left find Profile/PST. Select and add this column as well.

Now, you can perform your search but also use a negation filter in one or the other column (-myspst) to remove results from the list that are hits from that PST file.

The PST name will show in the Path column if the PST is mounted in Outlook, thus you would add the negative filter to the Path column in that case. It will show up in the Profile/Pst column if the PST was manually added for indexing within the X1 interface.
Sincerely,

Chris Wheaton
Technical Support
Customer Care Rep.
______________________

www.X1.com
Chris Wheaton
X1 Rep
X1 Rep
 
Posts: 301
Joined: Tue Dec 23, 2008 1:00 pm
Location: Pasadena, CA

Re: How to remove files from index

Postby liver_state » Thu Feb 17, 2011 12:15 pm

Kenward wrote:You have a couple of options. Reindex everything, or wait.

If you have only a small amount of data already indexed, then you can tell X1 to start all over again.

>>Tools
>>Options
>>Indexing
>>Clear Index Files

Probably the better approach is to wait. X1 eventually recognises that you have removed something and will expunge the references from the index.

If you don't want these items to show up in a search, then you can create a "Saved search" that excludes the PST file that holds these email messages. Come back if you need detailed instructions on how to do that.


Thanks for the quick response, Kenward.

Seeing as I have nearly 100GB indexed with nearly 300K emails, I'm not sure if starting over is really an option for me now.
How long should I wait? I've been indexing these emails over the course of 2 weeks, and I still haven't seen the phantom emails disappear... It'd be really nice if I could manually run x1 Pro's "index cleanup" procedure.
liver_state
 
Posts: 9
Joined: Wed Feb 16, 2011 5:23 pm

Re: How to remove files from index

Postby liver_state » Thu Feb 17, 2011 12:26 pm

Chris Wheaton wrote:If you will be using this search repeatedly I do agree with Kenward; using a saved search is a great idea. I use several saved searches daily in my support work with our customers.

If you believe you may need this specific search only once or rarely there is another option.

With the X1 UI focused on your email search go to Tools > Columns. On the left find Path. Select it and click the arrow to add it to the selected columns list. Back on the list on the left find Profile/PST. Select and add this column as well.

Now, you can perform your search but also use a negation filter in one or the other column (-myspst) to remove results from the list that are hits from that PST file.

The PST name will show in the Path column if the PST is mounted in Outlook, thus you would add the negative filter to the Path column in that case. It will show up in the Profile/Pst column if the PST was manually added for indexing within the X1 interface.


Chris,
Thanks for the detailed instructions. This seems to work well, but some of my archives were exported from a separate system, so show up with a non-unique name (ie, "BMA Export"). I've long since (a week ago) changed the archive names to be more accurate, but the index still shows the old name in the "Path" column.

Is there some wat to change/update the x1 index metadata manually?

I guess my whole question is... how do I "sync" the index to match the changes in my Outlook?
Is this possible without re-indexing the entire set of archives (over 100GB)?
liver_state
 
Posts: 9
Joined: Wed Feb 16, 2011 5:23 pm

Re: How to remove files from index

Postby Kenward » Thu Feb 17, 2011 2:48 pm

liver_state wrote:Seeing as I have nearly 100GB indexed with nearly 300K emails, I'm not sure if starting over is really an option for me now.
How long should I wait?

Depends on your PC.

I have 212,532 emails in various files. My files are about the same as yours, not far off 100GB.

When I have to do it, not as often as I used to thanks to the wonderful new index repair tool, it takes a couple of days to reindex. And my machine is three or four years old. (Keep meaning to build a new one.)

The way to speed up indexing, or clearing out of old junk, is to leave X1 to while away the hours overnight while you are asleep.
MK
X1 Search 8.6.1 - Build 6003fa (64-bit)
Windows 10 Pro 64-bit | Windows 10 Home 32-bit
No, I have nothing to do with X1, just a user since 2004.
Kenward
X1 Guru
X1 Guru
 
Posts: 4149
Joined: Tue Apr 20, 2004 2:35 am
Location: UK

Re: How to remove files from index

Postby Chris Wheaton » Thu Feb 17, 2011 3:08 pm

I basically agree with Kenward once again. Even with our Real-Time Email indexing enabled, because it does not update every email folder or PST you have in real-time, there is no easy way to have all email index data "caught up and current" the instant you change something.

You don't however have to lose all of your index data to address this. I hate to lose file index data or other Outlook related index data when All I really need is a clean email index. So, try this:

Go to Tools > Options > Indexing > Email > Outlook. Click Select Folders to Scan. Disable the entire tree by un-checking the "Root" Profile check box. Click OK.

Back at the Outlook Email Options window manually update your email index. This pass should run quickly as it should essentially erase the existing email index data.

Once that is done, repeat the steps at Select Folders to Scan and re-enable the mailbox hierarchy and again update the index with the Manual index button.

You might also set the Indexing Priority back at Tools > Options > Indexing to Never Delay Indexing. This allows X1 to be a bit busier on the disk which can speed the re-indexing process.

As Kenward stated this is best done overnight so leave yourself logged in; lock you system for security, and leave X1 up as the active foreground process. In the morning, with any good fortune, the indexes will be rebuilt.
Sincerely,

Chris Wheaton
Technical Support
Customer Care Rep.
______________________

www.X1.com
Chris Wheaton
X1 Rep
X1 Rep
 
Posts: 301
Joined: Tue Dec 23, 2008 1:00 pm
Location: Pasadena, CA

Re: How to remove files from index

Postby Kenward » Fri Feb 18, 2011 5:04 am

Nice one.

I had thought of removing the whole PST rather than just the email bit, but this is much better.

Indeed, this would seem to be a nice approach to tweaking an index.

Good enough to warrant a separate "sticky" note?

But which files would it work best on as a quick fix? I assume that it would not work on a massive set of files that require a lot of indexing time.
MK
X1 Search 8.6.1 - Build 6003fa (64-bit)
Windows 10 Pro 64-bit | Windows 10 Home 32-bit
No, I have nothing to do with X1, just a user since 2004.
Kenward
X1 Guru
X1 Guru
 
Posts: 4149
Joined: Tue Apr 20, 2004 2:35 am
Location: UK

Re: How to remove files from index

Postby Chris Wheaton » Fri Feb 18, 2011 8:54 am

This is an approach I tend to use on limited data.

In the case for Liver_State we had the need to clear and rebuild his entire email index. A more typical application of this approach would involve simply removing one or two folders from the email index, updating the index, then re-enabling the folders removed earlier and updating the index once more.

This is essentially a "trick" to try to flush old data from the email index with a limited amount of index destruction and rebuilding. It is not something that fits a wide range of scenarios.

For a broader application the Index Repair tool is a better choice.
Sincerely,

Chris Wheaton
Technical Support
Customer Care Rep.
______________________

www.X1.com
Chris Wheaton
X1 Rep
X1 Rep
 
Posts: 301
Joined: Tue Dec 23, 2008 1:00 pm
Location: Pasadena, CA

Re: How to remove files from index

Postby Kenward » Fri Feb 18, 2011 12:24 pm

Chris Wheaton wrote:A more typical application of this approach would involve simply removing one or two folders from the email index, updating the index, then re-enabling the folders removed earlier and updating the index once more.

Just to clarify, I assume that this means removing from the list of folders to index. Not removing the folders from Outlook

Seems pretty clear to me, but I would not want anyone making the wrong move.
MK
X1 Search 8.6.1 - Build 6003fa (64-bit)
Windows 10 Pro 64-bit | Windows 10 Home 32-bit
No, I have nothing to do with X1, just a user since 2004.
Kenward
X1 Guru
X1 Guru
 
Posts: 4149
Joined: Tue Apr 20, 2004 2:35 am
Location: UK

Re: How to remove files from index

Postby liver_state » Fri Feb 18, 2011 12:58 pm

Chris Wheaton wrote:I basically agree with Kenward once again. Even with our Real-Time Email indexing enabled, because it does not update every email folder or PST you have in real-time, there is no easy way to have all email index data "caught up and current" the instant you change something.

You don't however have to lose all of your index data to address this. I hate to lose file index data or other Outlook related index data when All I really need is a clean email index. So, try this:

Go to Tools > Options > Indexing > Email > Outlook. Click Select Folders to Scan. Disable the entire tree by un-checking the "Root" Profile check box. Click OK.

Back at the Outlook Email Options window manually update your email index. This pass should run quickly as it should essentially erase the existing email index data.

Once that is done, repeat the steps at Select Folders to Scan and re-enable the mailbox hierarchy and again update the index with the Manual index button.

You might also set the Indexing Priority back at Tools > Options > Indexing to Never Delay Indexing. This allows X1 to be a bit busier on the disk which can speed the re-indexing process.

As Kenward stated this is best done overnight so leave yourself logged in; lock you system for security, and leave X1 up as the active foreground process. In the morning, with any good fortune, the indexes will be rebuilt.


Chris, please keep in mind, my concern isn't real-time. I am only indexing archives and searching against them (for one-time email search initiative).

I've tried removing some of the PST folders from indexing (using the "select folders to scan" option), but I still see the emails in the index after several days of separate indexing runs.

So far I'm seeing that my index only adds, doesn't update or delete to match my "selected folders" or even what's attached the Outlook account.

It seems the only way I can manage this is saved searches, but that's going to get complicated once my archive goes to 10 or 20 pst files (a few of which were renamed after an index run)... which will work, but is a bit unfortunate.
liver_state
 
Posts: 9
Joined: Wed Feb 16, 2011 5:23 pm

Re: How to remove files from index

Postby Chris Wheaton » Fri Feb 18, 2011 2:49 pm

Yes Kenward, you would be correct. The idea is to remove the folder from the index.
Sincerely,

Chris Wheaton
Technical Support
Customer Care Rep.
______________________

www.X1.com
Chris Wheaton
X1 Rep
X1 Rep
 
Posts: 301
Joined: Tue Dec 23, 2008 1:00 pm
Location: Pasadena, CA

Re: How to remove files from index

Postby liver_state » Fri Feb 18, 2011 3:30 pm

Guys,
I've had some limited success with updating my index :D :P

What does work (as you guys have recommended in the thread so far):
a) Adding PST file to Outlook as a folder
b) Indexing said folder
c) "Closing" Outlook folders (ie, detach them from outlook account)
d) Indexing again, which removes the content

What does not work:
1) in x1 Pro, Tools > Options > Email > Outlook > Select Folders to Scan > Add PST File(s) and choosing a PST
2) Indexing said content
3) Realizing this is bad content and removing it from the search folders (either de-selecting the checkbox or clicking "Remove PST File").
4) Indexing again (does not remove content)

I've gotten rid of some of the PST files that were incorrectly added to Outlook by re-indexing as you recommened above.
However the step (4) above does NOT seem to work.

You guys have been a great assistance so far in helping me suss out how x1 works so I can get my indexing done...
Any suggestions on the step (4) above?
liver_state
 
Posts: 9
Joined: Wed Feb 16, 2011 5:23 pm

Re: How to remove files from index

Postby Kenward » Sat Feb 19, 2011 3:15 am

There are a lot of unknown unknowns in there.

For example, we don't know how many Mail profiles you have, which PST files are attached to them, and whether you have "manually" added PST files to Outlook.

The arrangement of these will influence how X1 works.

For example, in my own system, and with Outlook 2007, I have two Mail profiles. One is for my daily work with my main PST file attached. This carries all of my email accounts.

I also have an "Archive" PST which has no email accounts associated with it, but does point to a raft of PST files that I use as archives of old email messages.

I arrived at this arrangement quite some time ago, following a suggestion here as to how to manage multiple PST files. It probably isn't needed any more, but the strategy works for me.

You can see from this that it might be possible to attach a PST file to more than one Mail profile. Add the manual addition of PST files to X1 and you can see how complicated things can get. That is why it is impossible to say what may be going on at your end and why X1 won't forget old data.

It may sound bothersome, but I try to organise my files and email in a way that makes it easier to get at them with X1 and other software, including backup software. So all of my data, including X1's index and data and anything else that is movable in the way of ProgramData and "AppData", is, wherever possible, on a different hard disk from my programs and operating system. Microsoft likes to plonk things down in its favoured locations, which don't always make life easy for me.
MK
X1 Search 8.6.1 - Build 6003fa (64-bit)
Windows 10 Pro 64-bit | Windows 10 Home 32-bit
No, I have nothing to do with X1, just a user since 2004.
Kenward
X1 Guru
X1 Guru
 
Posts: 4149
Joined: Tue Apr 20, 2004 2:35 am
Location: UK

Re: How to remove files from index

Postby Jeffrey Levine » Mon Nov 21, 2011 7:00 pm

I have to admit, I'm rather flummoxed by the previous dialog. I've been struggling with a similar set of problems for weeks now.

My setup is similar to that described by Mr. Liver_State and Kenward, except that I have two active .psts, each one associated with an active Gmail IMAP account, plus a passel of additional .psts that I've created over the course of time--some of them as archives, some of them as backups. I only have ever (intentionally!) had one Outlook profile; however, I've moved various .pst files at various times, and they may have been created and saved on previous computers, in previous versions of Outlook. (I've used 2002, 2003, & 2007). Presently, all the .psts I'm interested in consolidating reside in a single folder on the D: partition of my hard disk (D:\Outlook pst files\*.* or in subfolders thereof)

The end result is that I have not just duplicate, but multiple copies of emails that have been saved in various .pst files at various times in the past. The entire system has become a huge unwieldy mess. I've got to consolidate and I was hoping to be able to use X1 to help sort things out, but it's not going very well. I may want to separate emails out by "topic", or perhaps by "date" or by "Sender" (It will vary).... but in order to do this, I need to be able to identify the source of the email in the X1 index... and I need NOT to be struggling with emails that do not exist in the location(s) where X1 has them. :(

Problem #1 (which I've posted about previously) is the limited information provided by X1 regarding the source of data residing in the index. You can display the Profile/PST column.... but don't be deceived.... This does NOT display the name of the .pst file... only the profile, of which I have only one (that I know of) I can display the path & folder name, but unless these are in some way "unique", the same folder name might appear multiple times, and there's no way to distinguish which file it came from.

Problem #2 is the issue that brought me to this discussion thread. X1 has emails in the Index that no longer exist, and in many cases, haven't existed for weeks or months in the indicated locations. Thus, I receive the accursed, "Unable to locate this email message. It may have been either deleted, moved, or is located in a profile that is not currently connected." error. Using the "Move to Folder" function becomes problematic under these circumstances.

All this is complicated by the fact that Gmail places emails in folders that I have NOT created, such as "All Mail" "Important".... and I STILL have been unable to figure out if these refer to distinct copies of the email, or just multiple "labels" attached to a single email, or perhaps a combination of both. (At least one source I read claims that even Deleted mail ends up in All Mail), if deleted in Outlook, rather than on the Webmail interface.

As far as "waiting" and hoping for the Index to purge itself, I feel as if I'm in Casablanca in 1942, waiting for an exit visa to Lisbon, but I have no option but to wait,... and wait..., and wait......

I may be better off completely scrapping the index, then adding .pst files back one or two at a time, but it doesn't sound like this has worked very well for Mr. Liver. Any further suggestions would be appreciated, but nothing suggested previously has helped.
Jeffrey Levine, Richardson, TX, USA
X1 Enthusiast (I couldn't get by without it!)
and Vocal Advocate (But how to get people to listen?)
Jeffrey Levine
 
Posts: 12
Joined: Fri Jan 25, 2008 2:32 pm
Location: Richardson, TX

Re: How to remove files from index

Postby Kenward » Tue Nov 22, 2011 3:03 am

Clearing the index will not help much if X1 can get at those files. It will merely index them all over again.

You might want to sort out the underlying issue with all those PST files and Gmail before going much further.

You could start by working out where all those duplicated entries in the X1 index come from.

You can troubleshoot this by adding search columns to your email views. Add the columns Profile/PST and Folder Path. Then you will see where the messages reside. If they point to your extra PSTs, then you might want to consider filing them away in a safe place where X1 cannot get at them.

With those columns visible, you can filter out those that you do not want to see. You can also sort and delete, move messages as you feel moved.
MK
X1 Search 8.6.1 - Build 6003fa (64-bit)
Windows 10 Pro 64-bit | Windows 10 Home 32-bit
No, I have nothing to do with X1, just a user since 2004.
Kenward
X1 Guru
X1 Guru
 
Posts: 4149
Joined: Tue Apr 20, 2004 2:35 am
Location: UK

Re: How to remove files from index

Postby Kenward » Tue Nov 22, 2011 9:35 am

Jeffrey Levine wrote:I have to admit, I'm rather flummoxed by the previous dialog. I've been struggling with a similar set of problems for weeks now.

I have just noticed that you have duplicated a conversation that is going on elsewhere. So my answer is irrelevant.

I missed the earlier conversation because I didn't have time to read your long message.

It is usually a good idea to restrict yourself to discussing one issue at a time.

Dealing with duplicates could be one issue. Sorting out your mass of PST files might be another.

Sticking with that one, you should also add the "Path" column to your email display. Here at least that displays the PST that holds a message.

With Profile/PST, Folder Path and Path, it should be pretty obvious where a message resides.
MK
X1 Search 8.6.1 - Build 6003fa (64-bit)
Windows 10 Pro 64-bit | Windows 10 Home 32-bit
No, I have nothing to do with X1, just a user since 2004.
Kenward
X1 Guru
X1 Guru
 
Posts: 4149
Joined: Tue Apr 20, 2004 2:35 am
Location: UK


Return to Questions and Answers

Who is online

Users browsing this forum: No registered users and 67 guests

cron