<?xml version="1.0" encoding="UTF-8" standalone="yes"?><oembed><version><![CDATA[1.0]]></version><provider_name><![CDATA[Ordinary Ideas]]></provider_name><provider_url><![CDATA[https://ordinaryideas.wordpress.com]]></provider_url><author_name><![CDATA[paulfchristiano]]></author_name><author_url><![CDATA[https://ordinaryideas.wordpress.com/author/paulfchristiano/]]></author_url><title><![CDATA[Motivation]]></title><type><![CDATA[link]]></type><html><![CDATA[<p>This blog contains a lot of material that is very weird. A natural question is: why write about such weird things? Many of the situations I&#8217;ve considered or problems I&#8217;ve written about involve improbable hypotheticals. We <em>probably</em> won&#8217;t ever have a powerful utility maximizer which we must provide a precisely defined utility function, for example. Why not focus on designing systems that are likely to actually get built?</p>
<p>The reason should not be so unfamiliar to a mainstream academic: though these problems may be too simple to be directly applicable, thinking about them sheds light on fundamental philosophical and mathematical issues. The heuristic is something like: where there are <em>simple</em>, <em>fundamental</em> problems on which it is possible to make progress, you should do it, even if you can&#8217;t see exactly why or how that progress will be useful.<!--more--></p>
<p>I don&#8217;t know what the future will look like. But I&#8217;d guess it will be influenced profoundly by the development of machine intelligences of some sort. What kind of understanding is necessary, to ensure that this influence is positive? I don&#8217;t know that either, but it seems quite likely that we will have to face much hairier, real-world analogs of questions like &#8220;if we had a powerful utility maximizer, what utility function might we want to give it?&#8221; or &#8220;if we had an oracle that could answer formally specified questions, what would we do with it?&#8221; Maybe working on these simpler questions is a waste of time&#8211;maybe we should wait until we understand better what the real questions will be&#8211;but there aren&#8217;t that many simple questions in the world. As long as we only waste time on fundamental problems, we at least won&#8217;t waste too much time. And it seems to me that there may be relatively deep theory underlying these questions&#8211;if that&#8217;s the case, we may not want to count on our ability to figure it all out when the time comes.</p>
<p>This isn&#8217;t so different from the state of affairs in any other theoretical field. Why do we care about having a polynomial time algorithm for secure multi-party computation? It <em>probably</em> isn&#8217;t going to ever get used (and I could use more extreme examples, which definitely won&#8217;t get used). But because the question is so simple, and seems fundamentally related to the <em>sort of thing</em> we are likely to want to do in the future, we expect (I think rightly) that a deeper understanding is likely to be independently useful, and to suggest fruitful directions for further study.</p>
]]></html></oembed>