{"id":895,"date":"2017-07-05T17:42:27","date_gmt":"2017-07-05T17:42:27","guid":{"rendered":"http:\/\/yingzhenli.net\/home\/en\/?page_id=895"},"modified":"2020-12-13T17:23:21","modified_gmt":"2020-12-13T17:23:21","slug":"topics-in-approximate-inference","status":"publish","type":"page","link":"https:\/\/yingzhenli.net\/home\/en\/?page_id=895","title":{"rendered":"Topics in approximate inference"},"content":{"rendered":"<p>Update:\u00a0another major update of this monograph document is in. Some small updates are still in schedule to finish before the end of 2020.<\/p>\n<p><a href=\"http:\/\/yingzhenli.net\/home\/pdf\/topics_approx_infer.pdf\" target=\"_blank\" rel=\"noopener noreferrer\">Current version (Dec 2020) (pdf)<\/a><\/p>\n<p>Next major update will be on applications of approximate inference in machine learning.<\/p>\n<p><span style=\"color: #800000;\">Read the following explainers before you decide to delve in this subject<\/span>\u00a0<span class=\"emoji\">\ud83e\udd14<\/span><\/p>\n<p>What is approximate inference? <span style=\"background-color: #fff;\"><a onclick=\"read_toggle(1584821598, 'more', 'hide'); return false;\" class=\"read-link\" id=\"readlink1584821598\" style=\"color: #FF8500;font-weight: 800;font-family: Arial, Helvetica, sans-serif;font-size: 12px;padding: 12px 12px 12px 12px;\" href=\"#\">more<\/a><\/span>\n<div class=\"read_div\" id=\"read1584821598\" style=\"display: none;\"><\/p>\n<p>To me, <strong>approximate inference finds the best approximation to an integration problem confronting computational constraints<\/strong>.<\/p>\n<p>Imagine you have a distribution <span class='MathJax_Preview'><img src='https:\/\/yingzhenli.net\/home\/en\/wp-content\/plugins\/latex\/cache\/tex_4130c89f2d12c3ac81aba3adbff28685.gif' style='vertical-align: middle; border: none; ' class='tex' alt=\"\" \/><\/span><script type='math\/tex'><\/script> and you want to compute the integral <span class='MathJax_Preview'><img src='https:\/\/yingzhenli.net\/home\/en\/wp-content\/plugins\/latex\/cache\/tex_b2c77f79a3a71d08c7b32ff9063e1363.gif' style='vertical-align: middle; border: none; ' class='tex' alt=\"\" \/><\/span><script type='math\/tex'><\/script> for some function <span class='MathJax_Preview'><img src='https:\/\/yingzhenli.net\/home\/en\/wp-content\/plugins\/latex\/cache\/tex_d76f2c4d6bdf142af5106c3f36e9e970.gif' style='vertical-align: middle; border: none; ' class='tex' alt=\"\" \/><\/span><script type='math\/tex'><\/script> of interest. We call the computation of this integral as <em>inference<\/em>. Examples include Bayesian inference where now <span class='MathJax_Preview'><img src='https:\/\/yingzhenli.net\/home\/en\/wp-content\/plugins\/latex\/cache\/tex_4130c89f2d12c3ac81aba3adbff28685.gif' style='vertical-align: middle; border: none; ' class='tex' alt=\"\" \/><\/span><script type='math\/tex'><\/script> is some posterior distribution and <span class='MathJax_Preview'><img src='https:\/\/yingzhenli.net\/home\/en\/wp-content\/plugins\/latex\/cache\/tex_d76f2c4d6bdf142af5106c3f36e9e970.gif' style='vertical-align: middle; border: none; ' class='tex' alt=\"\" \/><\/span><script type='math\/tex'><\/script> is the likelihood function of <span class='MathJax_Preview'><img src='https:\/\/yingzhenli.net\/home\/en\/wp-content\/plugins\/latex\/cache\/tex_9dd4e461268c8034f5c8564e155c67a6.gif' style='vertical-align: middle; border: none; padding-bottom:2px;' class='tex' alt=\"\" \/><\/span><script type='math\/tex'><\/script> on unseen data. Or if <span class='MathJax_Preview'><img src='https:\/\/yingzhenli.net\/home\/en\/wp-content\/plugins\/latex\/cache\/tex_4130c89f2d12c3ac81aba3adbff28685.gif' style='vertical-align: middle; border: none; ' class='tex' alt=\"\" \/><\/span><script type='math\/tex'><\/script> is unnormalised, taking <span class='MathJax_Preview'><img src='https:\/\/yingzhenli.net\/home\/en\/wp-content\/plugins\/latex\/cache\/tex_4beb13900affbaa3dd5bb579aa4f66be.gif' style='vertical-align: middle; border: none; ' class='tex' alt=\"\" \/><\/span><script type='math\/tex'><\/script> would return the integral as the normalising constant (or partition function) of <span class='MathJax_Preview'><img src='https:\/\/yingzhenli.net\/home\/en\/wp-content\/plugins\/latex\/cache\/tex_83878c91171338902e0fe0fb97a8c47a.gif' style='vertical-align: middle; border: none; padding-bottom:1px;' class='tex' alt=\"\" \/><\/span><script type='math\/tex'><\/script>. Unfortunately for many complicated models we are fancy on now (say neural networks) this integral is intractable, and here intractability means <em>you can't compute the exact value of the integral due to computational constraints<\/em> (say running time, memory usage, precision, etc). So instead we use approximate inference to approximate that integral.<\/p>\n<p><\/div><\/p>\n<p>Why I should care about it? <span style=\"background-color: #fff;\"><a onclick=\"read_toggle(1494370754, 'more', 'hide'); return false;\" class=\"read-link\" id=\"readlink1494370754\" style=\"color: #FF8500;font-weight: 800;font-family: Arial, Helvetica, sans-serif;font-size: 12px;padding: 12px 12px 12px 12px;\" href=\"#\">more<\/a><\/span>\n<div class=\"read_div\" id=\"read1494370754\" style=\"display: none;\"><\/p>\n<p>Now you might ask: why I should care about this approximate integral in the first place? Couldn't we just forget about integrals and do point estimations instead? Or couldn't we just use models that can do exact inference (e.g. stacking invertible transformations, auto-regressive models, sum-product networks)? Well for the first question it's more like a debate between Frequentist and Bayesian which I don't think I should say too much about it here. But I shall emphasize that <em>integration is almost everywhere in probability and statistics<\/em>, say calculating expectations and computing marginal distributions. For the second question, those models are usually more computationally demanding (both in time and memory), which is exactly the point for not using them and do approximate inference instead if you don't have that much computational resource. <strong>Approximate inference only makes sense when you do have computational constraints, and the way you do approximate inference also depends on how much precision you want and computational resource you have.<\/strong><\/p>\n<p><\/div><\/p>\n<p>What this list is about? <span style=\"background-color: #fff;\"><a onclick=\"read_toggle(208782260, 'more', 'hide'); return false;\" class=\"read-link\" id=\"readlink208782260\" style=\"color: #FF8500;font-weight: 800;font-family: Arial, Helvetica, sans-serif;font-size: 12px;padding: 12px 12px 12px 12px;\" href=\"#\">more<\/a><\/span>\n<div class=\"read_div\" id=\"read208782260\" style=\"display: none;\"><\/p>\n<p>There are mainly two ways to do approximate inference: directly approximating the integral you want, or, finding an accurate approximation <span class='MathJax_Preview'><img src='https:\/\/yingzhenli.net\/home\/en\/wp-content\/plugins\/latex\/cache\/tex_7694f4a66316e53c8cdd9d9954bd611d.gif' style='vertical-align: middle; border: none; padding-bottom:1px;' class='tex' alt=\"\" \/><\/span><script type='math\/tex'><\/script> to the target distribution <span class='MathJax_Preview'><img src='https:\/\/yingzhenli.net\/home\/en\/wp-content\/plugins\/latex\/cache\/tex_83878c91171338902e0fe0fb97a8c47a.gif' style='vertical-align: middle; border: none; padding-bottom:1px;' class='tex' alt=\"\" \/><\/span><script type='math\/tex'><\/script> and using it for integration later. The first approach is mainly dominated by sampling methods (and maybe adding student models to distill it) and quadrature. The second one, sometimes referred as the indirect approach, will be the focus of this list.<\/p>\n<p><\/div><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Update:\u00a0another major update of this monograph document is in. Some small updates are still in schedule to finish before the end of 2020. Current version (Dec 2020) (pdf) Next major update will be on applications of approximate inference in machine learning. Read the following explainers before you decide to delve in this subject\u00a0\ud83e\udd14 What is ... <span class=\"more\"><a class=\"more-link\" href=\"https:\/\/yingzhenli.net\/home\/en\/?page_id=895\">[Read more...]<\/a><\/span><\/p>\n","protected":false},"author":1,"featured_media":0,"parent":0,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"footnotes":""},"class_list":["entry","page","publish","author-admin","post-895"],"_links":{"self":[{"href":"https:\/\/yingzhenli.net\/home\/en\/index.php?rest_route=\/wp\/v2\/pages\/895","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/yingzhenli.net\/home\/en\/index.php?rest_route=\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/yingzhenli.net\/home\/en\/index.php?rest_route=\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/yingzhenli.net\/home\/en\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/yingzhenli.net\/home\/en\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=895"}],"version-history":[{"count":27,"href":"https:\/\/yingzhenli.net\/home\/en\/index.php?rest_route=\/wp\/v2\/pages\/895\/revisions"}],"predecessor-version":[{"id":1348,"href":"https:\/\/yingzhenli.net\/home\/en\/index.php?rest_route=\/wp\/v2\/pages\/895\/revisions\/1348"}],"wp:attachment":[{"href":"https:\/\/yingzhenli.net\/home\/en\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=895"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}