{"id":5051,"date":"2013-11-05T07:37:31","date_gmt":"2013-11-05T15:37:31","guid":{"rendered":"http:\/\/www.chesnok.com\/daily\/?p=5051"},"modified":"2013-11-12T11:33:24","modified_gmt":"2013-11-12T19:33:24","slug":"eliminating-duplicate-code-our-backfill-functions-for-socorro","status":"publish","type":"post","link":"https:\/\/www.chesnok.com\/daily\/2013\/11\/05\/eliminating-duplicate-code-our-backfill-functions-for-socorro\/","title":{"rendered":"Eliminating duplicate code: our backfill functions for Socorro"},"content":{"rendered":"<p>Last Friday, I spent some time refactoring a user defined function in <a href=\"http:\/\/github.com\/mozilla\/socorro\">Socorro<\/a> that was taking a little too long to run each day.<\/p>\n<p>This meant splitting up one function into about 8 separate functions. Our functions are designed to backfill themselves when a failure occurs. However, if we need to remove an incorrect daily report and re-run the functions from scratch, we&#8217;ve typically written a special function for every report called <code>backfill_REPORTNAME<\/code> that handles the cleanup work.<\/p>\n<p>This means we&#8217;ve got a lot of boilerplate code, that it would really be nice to replace. So, I took this opportunity to create a utility function and hopefully never have to write another <code>backfill_REPORTNAME<\/code> function again!<\/p>\n<p>Here it is:<\/p>\n<pre><code>CREATE OR REPLACE FUNCTION backfill_named_table(tablename text, updateday date) \n    RETURNS boolean\n    LANGUAGE plpgsql\nAS $function$\nDECLARE\n    update_proc_name TEXT := 'update_' || tablename;\nBEGIN\n\n-- Check if requested table for backfilling exists\nPERFORM 1 FROM information_schema.tables WHERE table_name=tablename;\nIF NOT FOUND THEN\n    RAISE INFO 'table: % not found', tablename;\n    RETURN FALSE;\nEND IF;\n\n-- Check that requested function for update exists\nPERFORM 1 FROM pg_proc WHERE proname = update_proc_name;\nIF NOT FOUND THEN\n    RAISE INFO 'proc: % not found', update_proc_name;\n    RETURN FALSE;\nEND IF;\n\nEXECUTE format('DELETE FROM %I WHERE report_date = %L', tablename, updateday);\n\nEXECUTE format('SELECT %I(%L, FALSE)', update_proc_name, updateday);\n\nRETURN TRUE;\n\nEND;\n$function$\n;\n<\/code><\/pre>\n<p>Here&#8217;s <a href=\"https:\/\/github.com\/selenamarie\/socorro\/blob\/5c7d59e531d308d4cebbae51d3458176e2a03e63\/socorro\/external\/postgresql\/raw_sql\/procs\/backfill_named_table.sql\">the file with the code<\/a>.<\/p>\n<p>I&#8217;ve been trying to switch over to using <a href=\"http:\/\/www.postgresql.org\/docs\/current\/static\/functions-string.html#FUNCTIONS-STRING-FORMAT\">format()<\/a> instead of <code>||<\/code> in my queries, because it tends to be much more readable.<\/p>\n<p>You&#8217;ll see that I&#8217;ve got a check for the existence of the table, and that the user defined function for the update exists. The type checking in the function handles ensuring that <code>updateday<\/code> is a valid date. If you think there&#8217;s any improvements I could make on this, definitely let me know in the comments.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Last Friday, I spent some time refactoring a user defined function in Socorro that was taking a little too long to run each day. This meant splitting up one function into about 8 separate functions. Our functions are designed to &hellip; <a href=\"https:\/\/www.chesnok.com\/daily\/2013\/11\/05\/eliminating-duplicate-code-our-backfill-functions-for-socorro\/\">Continue reading &rarr;<\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[9],"tags":[604],"class_list":["post-5051","post","type-post","status-publish","format-standard","hentry","category-postgresql","tag-everyday-postgres"],"_links":{"self":[{"href":"https:\/\/www.chesnok.com\/daily\/wp-json\/wp\/v2\/posts\/5051","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.chesnok.com\/daily\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.chesnok.com\/daily\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.chesnok.com\/daily\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.chesnok.com\/daily\/wp-json\/wp\/v2\/comments?post=5051"}],"version-history":[{"count":4,"href":"https:\/\/www.chesnok.com\/daily\/wp-json\/wp\/v2\/posts\/5051\/revisions"}],"predecessor-version":[{"id":5056,"href":"https:\/\/www.chesnok.com\/daily\/wp-json\/wp\/v2\/posts\/5051\/revisions\/5056"}],"wp:attachment":[{"href":"https:\/\/www.chesnok.com\/daily\/wp-json\/wp\/v2\/media?parent=5051"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.chesnok.com\/daily\/wp-json\/wp\/v2\/categories?post=5051"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.chesnok.com\/daily\/wp-json\/wp\/v2\/tags?post=5051"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}