Idea ID: 2849177

Automated backup tool for ESM

Status: Under Consideration

There is absolutely value in backing up configuration and settings specific to ESM.  Event retention is outside of the general goals of ESM - that being for real-time correlation, aggregation, and alerting.  So, to the point of the DBs being overly large and thus rsync taking a very long time to run or perhaps even failing, the ideal ESM state is one where the DB remains small, only holding the most near-term events, all the list / resource data, and (for now) trends and cases.  Longer term, as we make the transition to Detect, and some / much of this data transitions to better storage mechanisms that are better at longer-term retention, some functionality (reports, user management, etc) are replaced by common components, etc - it makes sense to build a backup solution for the entire platform, not just for the Detect portion.  I suspect in the immediacy we will be reviewing options for configuration and settings backup (which would include a lot of content as well), with longer conversations happening around a more complete platform-wide backup solution.

See status update history

Customer has been looking for a natively implemented backup solution for ESM that would allow to backup all required settings and content. The capability to backup data is considered optional as long as long as archived data can be reused after restore).

The desired backup tool shall get implemented with following preconditions:

  1. it is a single backup /restore script
  2. the script when executed requires the DB user/password, backup destination and all other necessary details to perform backup (same for restore). The script shall not make a copy of any of those.
  3. progress bar or at least percentage of accomplishment shall be displayed through out the process
  4. script might require to stop certain or all ESM services
  5. same script shall handle to backup both: the compact and distributed mode of ESM.

At present the most comprehensive and reliable solution being used is to:

  • Stop all ESM services
  • Rsync/opt/arcisght to remote location
  • Start ESM services.

This approach works as long as the datastore is below 3TB of size.

With dataset increased in size, backup process takes too much time and does not complete before the  maintenance window expires, which makes the procedure of no use any more.

Tags:

  • You might want run rsync while running ESM, than stop ESM services. Run rsync again, this time it will take much shorter. Start ESM again.

    This will shorter time of downtime.

    You also might want check https://www.microfocus.com/documentation/arcsight/arcsight-esm-7.4/pdfdoc/ESMBackupRecovery/ESMBackupRecovery.pdf

    Also there is a script from S3COPS https://github.com/S3COPS/ArcSight-ESM-Backup-Script/blob/master/arc_full_backup.sh  but it is not best, error handling really is not best in that script.

    But anyway, I agree that software like arcsight $$ should provide some simple backup solution.

     

  • Hi  

    If backing up the event data is not a priority as you have the archive files backed up elsewhere, you can just take a backup of the system tables.

    This will backup the database ( all content, system configuration etc...) using the process defined in the ArcSight Administration Guide. This will not back up event data.

    It recommends that you stop the manager service before performing the backup. However since the mysql database needs to be running while the backup occurs, you could test running the backup whilst all services are still running. (I would not recommended testing this out on a production system, test in dev first)

    I'm sure you could quite easily build this into a bash script that runs on a cronjob for a scheduled automated backup with no downtime.

    I'm sure there is a post somewhere in the community on this, i vaguely remember seeing it but cannot find it at the moment.

     

    Thanks

     

    Lewis