Content Manager DataPort Export Bulk Documents Performance

Hi , 

I am trying to export bulk data from Content Manager using Data Port. 

I have 50K, 120K document records to export and created few small, saved searches 1000 records and 10000 records to test. 

I am able to export 1000 records with documents in around 9 minutes. I am planning to run the 10K search and export. 

Is there a way to improve the performance of Data Export using data port. 

My saved searches are based on URI. 

Used export document format name: DatabaseID and RecordURI 

Thanks 

AJ

  • Suggested Answer

    0  

    Hi Arjun,

    Not really - DataPort is DataPort.

    Strategies I've used in the past:

    - Run Multiples (DataPort A runs from Search A, DataPort B runs from Search B) --> At the end then combine the multiple output files

    - Use Print Merge using a similar strategy to use multiple CM clients to run different parts of the search at the same time (only really works with no electronic export)

    - Export using VBA, C# or Powershell -> These can allow 'multi threading' to take place, e.g. First fetch the URIs, break into batches of 500, then loop through multiple database connections at the same time (4 running at once)

    Each of these can be more difficult than the last depending on what exactly you are trying to do, how quickly you need to 'dump out' the files, and existing skillset.

    One strategy I've sometimes used for exports that include electronic, is do a metadata export using one method, and fetch the electronic documents using an alternate method (robocopy from the document store direct) - That doesn't always work though depending on security setup / architecture which may not allow direct access to the document store.

    -Scotty