Slow data load speeds with Greenplum

Recently started loading data into a brand new Greenplum DCA. Data load speeds should be blazing fast right? Well, ours were very slow. It  was taking 50 seconds to load a 2.6GB csv file, pitifully slow. I finally figured out what error I had made. We were moving data from one GP system to another. That process involved dumping the schema out of the production system, dumping the data out of the production system, restoring the schema in the new contingency system, then restoring the data into the new contingency system. The problem was that when we restored the schema into the new system, we restored everything, including indexes. So as I was attempting to load the new data, the system was indexing it at the same time. After dropping indexes, that 2.6GB csv file loaded in about 8 seconds. That's more like it.

Comments

Post a Comment

Popular posts from this blog

Database, schema, and table sizes in Greenplum

Starting in the 4.x release, you get size info from the gp_toolkit schema. To get the size of all databases and their size in bytes: select sodddatname, sodddatasize from gp_toolkit.gp_size_database; To see the database size in GB, TB, and/or MB. TB: select sodddatname, (sodddatsize/1073741824)/1024 AS sizeinTB from gp_toolkit.gp_size_of_database; GB: select sodddatname, (sodddatsize/1073741824) AS sizeinGB from gp_toolkit.gp_size_of_database; MB: select sodddatname, (sodddatsize/1048576) AS sizeinMB from gp_toolkit.gp_size_of_database; For schema sizes , connect to your database and run: TB: select sosdnsp, (sosdschematablesize/1073741824)/1024 AS schemasizeinTB from gp_toolkit.gp_size_of_schema_disk; GB: select sosdnsp, (sosdschematablesize/1073741824) AS schemasizeinGB from gp_toolkit.gp_size_of_schema_disk; MB: select sosdnsp, (sosdschematablesize/1048576) AS schemasizeinMB from gp_toolkit.gp_size_of_schema_disk; If you want a specific schema only, add
Read more

Greenplum update with multiple tables

Recently ran into an issue using Greenplum to update a column. I had column1 in tableA that needed to be set to the value of column1 in tableB. Finally came across the correct syntax here . This is what I was looking for: UPDATE tableA  SET col1 = tableB.col1  FROM tableB  WHERE tableA.col2=tableB.col2;
Read more

Show running queries on Postgresql/Greenplum

I've got some queries that run for a long time. Sometimes longer than others. I wanted to see how long active queries have been running. Here's a view to consistently do that: CREATE VIEW public.view_activequeries AS ( SELECT age(query_start, backend_start) AS queryage,* FROM pg_stat_activity WHERE current_query NOT LIKE '%IDLE%' ORDER BY queryage DESC); Then you just run: select * from public.view_activequeries; You get a list of all active queries in descending order of how long they have been running, i.e. longest running queries listed first.
Read more