2.7 Performance impact

We have alluded to a few different kinds of performance impact when loading data into a database. To sumarize, the main impacts are as follows. First there is the actual CPU and IO load to do the insert. This includes reading the data, adding it to the table, and updating all the standard indexes. This is largely unavoidable, but can be countered with additional CPUs, and sufficient IO bandwidth.

Secondly there will be some impact due to locking the tables during the insert. This allows data integrity to be maintained in the database file. If there is no read contention for the table this will not be a problem. If the table is being heavily read it can be a problem.

Third is the impact of searching the new records to resolve queries before the metamorph indices are updated. This is the extra IO of reading the new records, and then performing the LIKE query against them to see if they match. With a relatively small set of new records the impact will not be large. With a larger set you may wish to make sure that the index is up to date, update a copy of the database, or else disable processing of the new list.

Finally there is the impact of updating the metamorph index. We are working to reduce the impact of the update, but the update does currently require a fair amount of IO and CPU bandwidth as it merges the new records into the index. Also with the switch of index the disk cache will have some catching up to do.

Back: Handling continuous streams

Next: Creating Queries