Purging of Archives

Last modified on 2023/01/23 18:17

   Caution: If you inadvertently perform a purge, you cannot undo the action. The EasyVista team will not resolve any issues related to the deletion of data and its impact on your platform.

Note: The purge is only operational for a standard database structure.

  • If you added external fields (such as foreign keys in files), external tables or links, the purge will not be operational.
  • If this is the case, EasyVista will be required to perform administration operations manually on your database. To find out more, please contact your EasyVista consultant.
Definition

The purging of archives is used to clean databases permanently by physically deleting records from tables.

EndDefinition
  • A purge is built around a parent query that accesses a set of standardized data from the database. It is used to perform an individual purge of specific objects based on the selected parent query.
  • The data is extracted using filters.
  • When enabled, it will run based on the frequency defined.
  • You can configure a CSV file to be sent automatically in a notification email. This will list the deleted records.
  • Purges can be stored in folders that are organized by function.
     

Reasons for purging archives

  • Reduce the size of the database by deleting the oldest records. This will optimize processing speed and system performance.
  • Refresh the list of records by deleting those that are no longer viewed.

Examples

Creation of a purge called Equipment discarded more than 3 years ago

  • Execution frequency: Every December 31 at 7 pm
  • CSV file: Display the type, model and asset
Type Model Asset
IT/Printers LaserJet 1100 10221P
IT/Printers LaserJet 1100 10224C

Enable a purge called Tickets over 3 years old

The purging of tickets over three years old will be performed on a continuous basis as long as the purge remains enabled.

Notes

  • You can create or modify a purge as long as you are authorized to access the Administration > Purge > Purge Archives menu.
  • Once they are enabled, purges will appear as scheduled tasks.
  • Parent Query:
    • Each purge is associated with a parent query selected from a list defined by EasyVista. You cannot add new queries.
    • The queries are limited to the following main tables: Contracts, CIs, Incidents, Service Requests, History (Discovery), Equipment (Discovery), Employees, Events, Licenses, Equipment (Asset Management), Knowledge Base search results, Equipment Life Cycle.
    • You can modify a query as long as you have the appropriate rights, and only in each purge. Open url.png See User Profiles > Edit rights and display rights.
  • Purges do not take domains into account.
  • When you purge data, any file attachments associated with the records will automatically be purged.
     
  • CSV file:
    • You can define its structure freely based on the information you want to save.
    • Name of the CSV file = Purge name followed by the processing date/time.
    • Configuration of the Archives folder:
      • smoserver.ini file on the application server
      • Schema: A_COMPANY
      • Table: A_PARAMETERS
      • Field to modify: SHARED_DOCUMENT_PATH: Specify the folder for storing the files
    • When the purge is first run, the folder will automatically be created on the application server.
  • The sender of the email and the sender's email address must be specified in the A_PARAMETERS table.
    • Schema: A_COMPANY
    • Table: A_PARAMETERS
    • Fields to modify:
      • MAILER_SCHEDULER: Specify the sender name, e.g. EVO_SCHEDULER
      • MAILER_SCHEDULER_ADDR: Specify the sender's email address, e.g. scheduler@name_of_customer.com
  • If the execution of a purge is not triggered, check that the scheduler is started and correctly configured.

Caution

  • If you inadvertently perform a purge, you cannot undo the action. The EasyVista team will not resolve any issues related to the deletion of data and its impact on your platform.
    • If you inadvertently purged the wrong records, you can create an integration model to retrieve the data from the CSV file.
    • You should always check the execution of the purge by generating the CSV file in the Calendar screen to ensure that the records displayed in the file are the ones to be processed by the purge.
      • Define a filter to ensure that only the relevant records are deleted.
      • Remember to specify the dates in the filter. If you do not specify a date, all records in the database will be deleted.
  • Before enabling a purge, remember to perform a full backup of the database.
    • If you want purges to run regularly, you should schedule the backup before each purge.
  • The purging of data will run as a background task, based on the specified frequency, as long as the purge is enabled. If you want to interrupt the purging of data, you must stop the task using the Disable wizard.
  • Depending on the volume of data selected by the parent query, the processing time of the purge may be long. We recommend that you refrain from selecting too many fields to avoid complexifying the process.
  • Do not confuse archiving with the purging of archives.
    • Archiving is performed based on the end dates of different elements managed by Service Manager.

      example  Employee departure date, equipment discard date

    • Archived elements remain available and can be viewed using the Archived filters.

Best Practice

  • You should always check the execution of the purge by generating the CSV file in the Calendar screen to ensure that the records displayed in the file are the ones to be processed by the purge.
  • Store the purged data in a CSV file. This enables you to keep a history of your data.
  • If you want to save information whose fields are not available in the CSV file, you should add them manually to the parent query.
  • You should only purge archived elements, i.e. records displayed using the Archived filter.
  • To purge integration logs, use the dedicated parent query called Integration log. Open url.png See the procedure.
  • To purge temp files and log files, see the rules for purging that describe the resources to be purged based on operational constraints.
  • Avoid scheduling purges whose frequency is based on a specific time. We recommend you run the processing yearly.
  • If the running of purge B depends on the completion of purge A whose processing time is unknown (due to large data volumes, access time, etc.), we recommend that you schedule the tasks in a sequence instead of defining a specific time for running purge B.

Menu access

Administration > Purge > Purge Archives

Screens description

Main Information

          Purge - Step 1.png

Name: Name of the purge. 

Comment: Comment field that describes the purge. 

Folder Name: Name of the folder where the purge is saved. 

Parent Query: Name of the parent query used to configure the purge.

  • The fields available are parent query fields and other fields manually added to Service Manager internal reporting tools, e.g. trends, alerts, reports, and other purges.

Filter

Note: Except for the Integration log parent query

Filter used to extract the data to be included in the purge. Open url.png See Visual filter editor.

  • All fields in the purge's parent query as well as other manually added fields will be available for the filter.

          Purge - Step 2.png

Define the CSV Format

          Purge - Step 3.png

Order: Order of columns in the CSV file. Open url.png See the procedure.

Display: Columns to be displayed in the CSV file.

Field: List of fields from the parent query associated with the purge and other manually added fields. 

Calendar

          Purge - Step 4.png

Note: Universal Time (UTC): Current date and time in Service Manager, expressed in Universal Time (UT). 

Frequency: Used to define the frequency for running the purge.

  • The frequency will be applicable after the date specified in the Next Execution Date field.
  • The parameters depend on the selected frequency.
    • Every [N] Months / Days / At (Hour UT): Define the relevant frequency using Universal Time (UT).
    • Depends on: Used to sequence the running of two purges successively. Select the purge whose processing precedes the execution of your purge from the drop-down list.

Next Execution Date: (Note: Only if the frequency is not Depends on): Date and time at which the purge should next run.

  • The date initializes the start of the frequency.

    example  Frequency: Every seven days at 7 am; the next execution date is a Saturday ==> The purge will run every Saturday

  • You can also update the date using the Enable wizard.

Best Practice icon.png  Enter a date earlier than the current date in order to run the purge immediately.

Warn

          Purge - Step 5.png

Recipients (Employees) / Recipients (Groups): List of recipients of the notification email.

  • These recipients must be listed in the Employee Directory or Group Directory.
  • You must select one or more recipients, i.e. employee or group.
  • Each recipient must be separated using the semicolon (;).

Title: Subject of the notification email sent to recipients.

Message: Body of the notification email sent to recipients.

Procedures and Wizards

How to create a purge

Step 1: Create the new purge

1. Select Administration > Purge > Purge Archives in the menu.

2. Click + New.

Step 2: Specify the main information on the purge

1. Enter the information on the purge.

2. Select the parent query.

  • You can add fields to the parent query.
    • Click Edit icon.png next to the field.
      The list of fields from the parent query of the purge and other fields manually added to Service Manager internal reporting tools and other purges will appear.
    • Add the fields you want. Open url.png See How to modify a parent query

3. Click Next.

Step 3: Define the conditions for the filter used by the purge

Note: Except for purges using the Integration log parent query

1. Define the filter conditions. Open url.png See Visual filter editor.

  • All fields in the purge's parent query as well as other manually added fields will be available.

2. Click Next.

Step 4: Format the CSV file to be generated by the purge

1. Define the order of the columns in the file.

  • Select the Order box next to the field to be moved.
  • Click Insert Here next to the field above which you want to move the selected field.
             Purge - Insert here link.png

2. Specify the options depending on the type of field.

  • Only for tree fields that are not in the last level: Indicate the first and last level of the hierarchy to be displayed.
    • The value 0 is used to display all grouping levels in the hierarchy.

      example  Display the tree field, Location = North America/USA/New York/Manhattan/Broadway

      • Start = 0, End = 0 ==> North America/USA/New York/Manhattan/Broadway
      • Start = 1, End = 3 ==> USA/New York/Manhattan
      • Start = 4, End = 5 ==> Manhattan/Broadway
      • Location (last level) = Broadway

3. Select the fields to be displayed in the file.

Step 5: Define the frequency for running the purge

1. Specify the frequency of the purge.

2. Click Next.

Best Practice icon.png  Before moving on to the next step, test the purge by clicking Test to check the number of records that will be deleted.

Step 6: Complete the notification email

1. Specify the recipients of the notification email.

Step 7: Save the purge

1. Click Finish.

The list of purges will be refreshed.

Step 8: Enable the new purge

1. Select the purge.

2. Run the Enable wizard.

3. Enter the first execution date, click Next and click Finish.
The CSV file will display the records that will be purged when the purge is next run.

4. Check that the purge is enabled.

  • Select Administration > Scheduled Tasks > Purge Archives in the menu.
  • Check that the purge appears in the list of scheduled tasks.

How to create a purge using a template

Step 1: Select the purge

1. Display the list of purges.

2. Click Edit icon.png next to the template.
The properties window of the purge will appear.

Step 2: Create the new purge

1. Modify the relevant information in the template purge.

2. Click Save as to avoid overwriting the template purge. Give a name to the new purge.

How to check if a purge is enabled or disabled

Step 1: Check if the purge is enabled or disabled

1. Select Administration > Purge > Purge Archives in the menu.

2. Display the List view.

3. Look at the Enabled box for the relevant purge.

  • If the box is checked, the purge is enabled.
  • If the box is not checked, the purge is disabled.
     

Step 2: Check that the corresponding scheduled task exists

1. Select Administration > Scheduled Tasks > Purge Archives in the menu.

2. Check the presence of the purge.

  • If the purge appears in the list, it is scheduled to run based on the frequency defined.
  • If the purge does not appear in the list, it is not scheduled to run.

Wizards

Enable
Disable

Rules for purging temp files and log files

Server with Service Manager and Service Apps

Resources to be purged based on operational constraints

Server Folder Rule
Application server REP_APP\tools\SERVERS\mssql\log All XML files whose last modification date exceeds one month
Application server REP_APP\tools\servers\EASYVISTASERVER\log All XML files whose last modification date exceeds one month
Application server REP_APP\tools\smobackoffice\log All LOG files whose last modification date exceeds one month
Application server REP_APP\tools\smomail\log All LOG files whose last modification date exceeds one month
Application server REP_APP\tools\SERVERS\smoauth\log All LOG files whose last modification date exceeds one month
Web server REP_LIN_LOG/apache All files whose last modification date exceeds one month
Web server PHP.INI file in the folder defined by the session.save_path variable or the REP_SESS_PHP variable All files whose last modification date exceeds seven days
Web server REP_LIN_LOG/php All PHP logs whose last modification date exceeds one month
Web server REP_LIN_LOG/REST All logs with REST calls whose last modification date exceeds one month
Web server REP_LIN_LOG/SOAP All logs with outgoing SOAP calls whose last modification date exceeds one month
Web server
  • FilerSE/easyvista/resources/50004/Integration
  • FilerSE/easyvista /resources/50005/Integration
All files whose last modification date exceeds seven days
Web server /easyvista/argo All Service Apps logs whose last modification date exceeds seven days

Examples of purge scripts on Windows servers

In each example, replace the folder displayed below in yellow by the folders to be purged.

  • PowerShell v3 

# One script per Log folder would be required.
# MSSQL Log Folder files older than 30 days

Get-ChildItem –Path  "C:\EasyVista\tools\servers\MSSQL\Log" –Recurse | Where-Object CreationTime –lt (Get-Date).AddDays(-30) | Remove-Item

  • DOS Batch file

forfiles -p "C:\EasyVista\tools\servers\MSSQL\Log" -s -m *.xml -d 30 -c "cmd /C del @path"

Dedicated Service Apps server

Resources to be purged based on operational constraints

Server Folder Rule
Web server REP_LIN_LOG/apache All Service Apps logs whose last modification date exceeds seven days
Web server PHP.INI file in the folder defined by the session.save_path variable or the REP_SESS_PHP variable All files whose last modification date exceeds seven days
Web server REP_LIN_LOG/php All PHP logs whose last modification date exceeds one month

Configuration of the LOG.INI file

On the Service Manager GUI servers, log files are automatically purged by the application. You can configure the purge frequency in the LOG.INI file.

;   Frequency in number of days for deleting the files
expire = 7 days

;   Indicate what you want to delete
;     root   => /argo/<folder logs>: delete only this folder but not subfolders
;     tenant => actual tenant of user is logged
;     all    => root folder and all the tenants that exist
folder_delete = "all"

; log_cleanup_frequency
;    Keep this value to 100 unless necessary

Use case: Creation of a purge Delete attachments from closed tickets

The purge will delete documents associated to closed tickets both in the database and on the web server thanks to two dedicated parent requests. This allows to save disk space on the web server.

Implementation:

  • Select the dedicated parent query.
    • Use the dedicated parent request Documents Operation to delete documents associated to closed incidents.
    • Use the dedicated parent request Documents Transition to delete documents associated to closed requests.
  • Extract the data by specifying the filter. Indicate the period you want to delete the documents associated to closed incidents.
  • Purge execution: Indicate the frequency you want.
Tags:
Powered by XWiki © EasyVista 2022