Compliance Accelerator is not completing the Random Sampling processing.

Problem

Compliance Accelerator is not completing the Random Sampling processing, failing on step 6 with an 'Out Of Memory' error

Error Message

- Error from Dtrace of the AcceleratorService on the Compliance Accelerator server:

(AcceleratorService) <Sampling Thread Num 4:5496> EV-H {-} Exception: Error Sampling_Guaranteed_GuaranteSampling Info: {ACCELERATOREVENT.EN_US} {C1.EN_US} Guaranteed Sampling: Error during guaranteed sampling - aborting. System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.

Cause

This issue is caused by more items to be processed in the Random Capture holding table than the operating system imposed memory restrictions on the AcceleratorService process can handle.  The number of items can vary based on each customer environment.  For example, any number greater than 30,000 Random Captured items in an environment can cause this issue.  Once the issue occurs, it will continue to occur until manual intervention is accomplished.

Solution

Currently, three possible workarounds exist to this issue to retain the Guaranteed Sampling processing instead of switching to Statistical Sampling.  The first workaround is to lower the number of threads used for Guaranteed Sampling processing.  The second workaround is to break the Department into multiple Departments with fewer Monitored Employees in each new Department.  The third workaround is to remove the items from the Sampling tables, conduct a percentage search for the date range the sampling has been missing and allow the Sampling to continue the following day.

If Workaround 1 does not resolve the issue, Workaround 2 will need to be implemented.  If Workaround 2 does not resolve the issue, them a combination of Workarounds 1 and 2 may be needed.  Workaround 3 may need to be included with Workaround 1 or 2 to clear the appropriate tables to allow them to be of a more manageable size.  If implementing all of these workarounds does not resolve the issue, switching to Statistical Sampling will be the only option to allow for random sampling to succeed.  Care should be taken when switching to Statistical Sampling as more work is performed by the Journal Connector, which can cause a performance impact during journal archiving operations.

TROUBLESHOOTING:

Record a Dtrace of the AcceleratorService process during the Sampling time.  See Technical Article 000125000 in the Related Articles section below for instructions on obtaining a Dtrace.

Examine the Dtrace for the entry in the Error section above.  There will be no errors in the Veritas Enterprise Vault Event Log on the Compliance Accelerator (CA) server, only a lack of Random Captured items in the Review Set of each Department.

WORKAROUND 1:

Lower the number of threads used for Guaranteed Sampling processing by completing the steps below.

1. Adjust the number of threads used for Guaranteed Sampling through the CA Customer web site (for CA 6.x through 2007.x) or the CA Client (for 8.0 and greater).

For Compliance Accelerator 6.x through 2007.x:

1. Access the CA Customer through Internet Explorer using an account with sufficient Application level permissions, such as the Vault Service Account (VSA).
2. Click on the System Configuration link in the Application Administration column.
3. Hold down the Ctrl key on the keyboard and click once on the header Configuration Settings at the upper left side of the window. This will display the hidden settings.
4. Expand the Random Capture section.
5. Locate the row for the option Number of threads used for Guaranteed Sampling. This is a hidden setting and will be highlighted in gray.
6. Change the number in that row's Value column to 1.  The default value is 10, which may be too many threads to process all of the items for all Monitored Employees within any CA Department.

For Compliance Accelerator 8.0 and greater:

1. Access the CA Customer through the CA Client using an account with sufficient Application level permissions, such as the VSA, on the CA Customer.
2. Click on the Configuration tab.
3. Click on the Settings sub-tab.
4. Hold down the Ctrl key on the keyboard and click once on the header Configuration Settings at the upper left side of the window. This will display the hidden settings.
5. Scroll down to and expand the Random Capture section.
6. Locate the row for the option Number of threads used for Guaranteed Sampling. This is a hidden setting and will be highlighted in gray.
7. Change the number in that row's Value column to 1.  The default value is 10, which may be too many threads to process all of the items for all Monitored Employees within any CA Department.  
8. Click the Save button to save the change.

  2. Restart the Enterprise Vault Accelerator Manager Service (EVAMS) on the CA Server.  The next time Random Sampling processing occurs, all available items (i.e., those with the KVSSavesetID identified) should be processed into their appropriate review sets.

WORKAROUND 2:

Break the largest Department or Departments into multiple Departments with fewer Monitored Employees than the original Department or Departments. For example, a Department with 400 Monitored Employees could be broken into 4 smaller Departments of 100 Monitored Employees each.

To create smaller Departments, determine how many and which Monitored Employees will be in each new Department, then create each new Department following the normal steps to create a Department.  Refer to HOWTO58415 in the Related Documents section below for instructions to create a Department.

WORKAROUND 3: 

Remove the items from the Sampling tables, conduct a percentage search for the date range the sampling has been missing and allow the Sampling to continue the following day by completing the follow.

1. Determine what is acceptable by the Compliance team, including what percentage of messages is acceptable.

a) A randomly accepted percentage of the email for the entire date range.

Conducting a search over the entire date range would produce a randomly selected percentage overall for all mail in the department, but could not guarantee that each day would be that exact percentage.

b) A randomly accepted percentage of the email day by day.

This would require a separate search for each day in each department instead of one search over the entire date range.

2. Determine the Date Range.  Run the following SQL query against the CA Customer database:

SELECT * FROM tblConfig

Examine the entry for "Sampling : Last Sampled".  This will be the date and time of the last successful Random Sampling processing.  The time between this date and the current date is the period for which an Immediate search for all items within each Department will have to be run and the sampling percentage set on the search applied to the results.

3. Stop all Journal Tasks to prevent additional items from being processed.

This should be done late day to minimize the impact on daily processing.

4. Verify the 'Number of threads used for Guaranteed Sampling' is at its default value (10 threads).

For Compliance Accelerator 6.x through 2007.x:

1. Access the CA Customer through Internet Explorer using an account with sufficient Application level permissions, such as the Vault Service Account (VSA).
2. Click on the System Configuration link in the Application Administration column.
3. Hold down the Ctrl key on the keyboard and click once on the header Configuration Settings at the upper left side of the window. This will display the hidden settings.
4. Expand the Random Capture section.
5. Locate the row for the option Number of threads used for Guaranteed Sampling. This is a hidden setting and will be highlighted in gray.
6. Ensure the number in that row's Value column is 10.  Correct this value and save the change if needed.

For Compliance Accelerator 8.0 and greater:

1. Access the CA Customer through the CA Client using an account with sufficient Application level permissions, such as the VSA, on the CA Customer.
2. Click on the Configuration tab.
3. Click on the Settings sub-tab.
4. Hold down the Ctrl key on the keyboard and click once on the header Configuration Settings at the upper left side of the window. This will display the hidden settings.
5. Scroll down to and expand the Random Capture section.
6. Locate the row for the option Number of threads used for Guaranteed Sampling. This is a hidden setting and will be highlighted in gray.
7. Ensure the number in that row's Value column is 10.  Correct this value and click the Save button to save the change if needed.

5. Stop the Enterprise Vault Accelerator Manager (EVAMS) service on the CA server.

Note: The next steps are not for Statistical sampling.

6. Remove all records from the Random Capture / Random Sampling tables.

Contact Veritas Enterprise Vault Technical Support for assistance with this step.

7. Start the Journal Tasks on the EV Journal servers.

8. Start EVAMS on the CA server.

9. Conduct one or more Immediate searches within each appropriate Department covering the date range determined in Step 2 above with the appropriate sampling percentage as set by the Compliance Officer through Step 1 above.