Teaching robots the human way

Teaching robots the human way

Problem being addressed

Natural language is perhaps the most versatile and intuitive way for humans to communicate tasks to a robot. However​,​ each task must be specified with a goal image—something that is not practical in open-world environments.

Solution

Combining the setting of open-ended robotic manipulation with open-ended human language conditioning. The researchers train a single policy to solve image or language goals, then use only language conditioning at test time. They introduce a simple transfer learning augmentation, applicable to any language conditioned policy and find that this significantly improves downstream robotic manipulation.

Advantages of this solution

The method reduces the cost of language pairing to less than 1% of collected robot experience, with the majority of control still learned via self-supervised imitation. At test time, a single agent trained in this manner can perform many different robotic manipulation skills in a row in a 3D environment, directly from images, and specified only with natural language. Importantly, this same technique allows the agent to follow thousands of novel instructions in zero shot, across 16 different languages.

Solution originally applied in these industries

engineering

Engineering

Possible New Application of the Work

data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAGQAAABkCAYAAABw4pVUAAASfHpUWHRSYXcgcHJvZmlsZSB0eXBlIGV4aWYAAHjarZpnduMwskb/YxWzBKRCWA7iObODt/y5Bcq2JOf2s7stmRZBoMIXQJr1f//d5j98JVuLiZJLqilZvmKN1TfeFHt9Xa/OxvPzfKV4+5t7PG5e/+A5FHgNtxPW7fON4/J2Qr593vXH4yaP2zjlNtDtDy8DBr2y5828TfI2UPDXcXf73VR/vWnpbjm3/3v4qoekX396/j1mgjGF8YI3fgUX7PnpryuF63/jfzo/xb8cEb4bnwzv42deQ/dBAF/fPcXPjtvx8BaOa6CXZaWnON2OO/k4fidK9zNy/vXK/n5Gvbpk77/u47dn2Xtdq2sxGcKVbot6Wcp5xwcJabyikfjO/Bfe5/Nd+S622UHWJkvtxnZ+qc4T8e2im6657dZ5HW4wxeiXz7x6P3w4x0rIvvpB+F2I+u22zybUMEMhK4PMBQ7717m4c92q1+NihStPxye9YzBy/Phtng/86/fDQHtrmTtny2usmJfX+mIamjn9yadIiNu3mMqJrzPXi33+0sQGMignzIUFNtuvIbq4t9oKJ8/BiuGj0V794vK8DUCIuLYwGRfIgE0uiEvOZu+zc8SxkJ/GzH2IvpMBJ0b8ZJY+hpBITvF6bc7J7nzWi78OAy8kQmiaTGpqaCQrRomJfiuUUDO0TxSRJFmKVGkppJgkpZST4lTLIccsOeWcS665lVBikZJKLqXU0qqvARgTU1PNtdRaW+OiLTbGany+caD7Hnrs0lPPvfTa26B8Rhwy0sijjDra9DNMIMDMNPMss8623KKUVlyy0sqrrLraptZ22HHLTjvvsutur1m7ZfUxa8+Z+zpr7pY1fxKln8tvWeNwzi9DOIUT0ZyRMR8dGc+aAQraa85scTF6zZzmzFYfTAC3mKVocqbTjJHBuJyX7V5z95a5T/NmiO5v8+Y/ypzR1P1/ZM5o6u4y9z5vH2RttgO34SRIu5CYgpCB9uMDq6/Z9e0ehQhkRZO4OQB2TALmJc+elzattQBY5o+866nuDSGYVqwuxhf+QUo/fV12LxdymZlcuZWLkbS6sDxveywpl7ZlzSlpiqsSW4kxjTrHGDnUHOaevvehhapzrSPvFetKzRk9kJzUsDe5lGjdGk7KTln/QoVkaNGvudPaixE2Ccuj1ZB2dSvlWQNX6dmQr7HH7tl18ptXL2tKdxFaz4HpFz11BJ0Jv+kPfmFhPeRNxdS5du95V0OorBspzNVG4rCPnSakF3KbFIplSW3OxNVjns3NSR+kHcT2tqa3pNjaBTmYvAITENGJp7kFKmgrDb992MHPtJpH96Q4R0wL5oIURuFNdGESdj9ETx7e6NlxcPbUSlthE4806ZOSnJ69pq7SU3jn4m6mjpgIOcki7Hwyy6x5ROMadShzVaY7ErMlvYQ8Fe/n2Nv3Rcv5nGoZmQzTJJuBVz110PQS16t5PsCZnsrQ0LGG5Mlz21koT1sbNTvbLqUPTap3pZD+Pie/UNnRp6YBS6+lrUNcxU2taHm/L+5bafPpU9rMiOr+x+J+LEqjVflUlKIlSdi1kwJ1NKi5Game4Tv1uQq4wLVnuQ+LeYjPd69fxM98FcDfxM98FcDfxM98FcDfxM98FMBeTrtROCBjLtsBrdBIoZt8FwgxLag9alW6PYBnOAVZk1VCljRBeXAV7qC34aW1GZNYasVLX3VmCIcuAGjXaqHQ4ADGhtTySr0YlGRLS/UarZSi9p2CzAImnNtFZgecHD1x1CcU52INAtLsqhpvVPgNhOim2jwmF59zMImyOtCmcCS52756Zo1xQSEAkEIYdNJK9anABtUhewGAhjqtxlXmJq2rpxAG+PKVLifSJ9RIS2IMLi+YvLlgNPv9sGoZ+oEVoWmWkTuwukoEyNxY80sGoRY9TEvVkCFG9z6uUz87zXPVmIIiLjpKDva6L17NyxshUoURU29pl6YJyjn2DuevFSpF1AtxmqPHUZojCWDwEvAyVgdYGWAqNG0kqqJTh9DHtQCfa6NbZpF0GoFrweIQN8TpNINMGskz+Qydsoyg2iGbTo3urhC6JmCZJ19gYh4CPG5bOwXdai7q2TrJkQUjzRZngIqoNvBoprbGaI6G8suuJRLQBW6BvsR5Lg1D9lBRV2JiHj2m05ElNuUrB5dB+SaiVzYSBvLnR6Uwxq4VSgDAdx7Ll7P0Ait711vPNc0FmpQFrAtiy0dEc0mkX3CHp5W8pZuIetW6XD4smvIAB41KYxzNErRaxqCX1kAcqYYh6nC6GX3P7WaAIXuiUy6K3E5VFbJHRrc6GziKMl+kiMKCleEgSg59ngnFXMUadaw6aekIiElkmLSDiyKFNzJ4JIDHZRmDej6ds/+gnMxH9fVa6LcyB6FOoTMMpU6XUenPzWLeuuXIBroF6sUnMgjL4JMLfKFXRgKPKbXPINcczNU8+69aljhIC6ASue+rICorkMS1M2l0EDcWwiZh0WBBCgppuleRQdjmmrpjwKgNlUixkrjQ5ECvP5KGoqavtKJq2YaeIZuBAh0nCkgB+ARlsnenlnLiBY9La8xML4+ABKZUaqI/4BqhI7kCnva4711zAqmtthNqxE1aXvs3aTlWgkLYfI9zK9YFhdqaWO0E+hGONGcqhtlXaqO1gCJGFyU1eiF4txrYSJf7ryEEXD7LM6junVSr7eS1abRd6A36mHH3JsBIsAVXoM77zqh3JGJaOITXXMjIC6gF3nonjdDQyjFM6jnBxJrnnKlsukD7A8eHnwC8S1jQih+1yIgcRLzOnSfSL4W6Qrf/yPkvlG/+yvkvlG9+zfnr4DGtPWNHs6pnpn/E7JLb2jQzKINX0yhgvCGsCEVN5sSoA+7SgljejfkJG5hv6eHpNZ1GTUOW1nsS0Juax6+h6RW+Ac2pJZrBx3ywsgdF8Q4FU699KSmAaZMhPsQL83vA+BgvzO8B42O8MOcNC4ccxamOVw8Ra6cSuUZ3FHESX2BjGgD0oL8KPmKtqMYZgzBAElZMjFTSu+6xbKKEAxF1LA3QD+w6utBLBfgF+K04GWTYMeJQFzMtaDt0EpVpdvqRDvnuNdEipWa4mKYt28pYtBUzSzI388cgDWKHXiSZNAdiakG6Q7f9lHrU3CEEhkyDrKsHb9ChKEXEHoJvDWRJg8Dw7RMk85Nzp4sZ/+asrzLIsa3bz40Y1NGz8Wgyi7N/7xBepLc68rtG1DbEcKanRjYPnax4ON8EE0NckklrWQtEBRN1Vd9RDcD2VjzPLgzxMiRcxAsaQ7z0JOAG8bJW+DulHRHGjqKs0+i+DvoYU+eUeb2WkWwQhNVIpEPAMPh3esDZ8lPbHo7m+sgkP5wnNrvoRlSNUDPR0p3QqVXRs291Ucs1hjYopxRaQ8yO0q1okWWEc9PEJdtWQ9eG4qNJaFvdHxm/cUkfvF42iyhIdQ0Ub7rbsNpG6Hhp6P1KrjbgpYjgEwJoVxSmxTvDNcWrumzT6kD8vmME8ONS+0BVpqaajlU2P5RtAPveXUUW4r4bdQkT0mqU1hnimpP5myV+Q3fzN0v8hu7mpZDhSRcBkzWoHhqa1UzMibq2oAZEg7W0VJCXGZYu6m+mIEHb1CEMnaOpTiNA2ifVaPEey7RShVSn7fgtAh1IUqpPZRuZDvydTL8lz/wx60fqy47B0PlzpaoL2JjMXkOsNpD5g9xAJ+rDB0dhL4mAT8/23DmJm9Uh5fngNaOf9nvLeYV0UlIBTlTRUhmieeNCbZv6+cmPnP3NFp75xIF9wtit06zwIfqjXkuH+dSRFoPQUctSCQxoSlXOFdFMMTncExYTV4QEbVyBtlWPiJYCmbE1OxIgcS12pBoWAnDQuyaxVa/NgyKfSqZXbp/nM1i26kp1JAn1NAfFpDuiy5AV3QcjCioe+XtrfGro5hklBZvqQqhF4boMwxUz9Si9U7m2AybIWDSzmJ4AH+lgLJSgxip9UC6krF6ry0y36cAAEQoOjJsM2RGjWkMsNKVRXIk2Lin8s8NCvHh3WYoF0O3OSGTW1E98yJwKZ4/0ri1JTNXU7m1y1qGf+ad3OqbuHH4cJJ2bVebDzkHlYLGq/x5rjlR2GUs36ZKkUnxbACndufTeCS40jGRvo5J61aV1YRL0M1I01kjc7om/iRO087ojCwVgsPSeVzybGXIfKLXChdLUwo3QwtHWyF4K1Bf1iSZdeHZDM22UfPzGhWh1fn3629nm/elPLPnhtsJ7ljQPAuuBIy+CbJBY6jekQ1IEoYvQT612Mk3YAuaAwwbtOmiHSE1SpLrfcgAPoNgDY48SajYdmPOZFor48ziVvTjLgSFUMDU5sFlRnRH4gvqq6DAqn97i44V+A6FCatTNwDFRr/RFTJRgLJ1aQ4Y6PDZaT5tWPBlEATmmmIPGB8lz5OE7OeVGXTgRC3zHAYKsRh/juGjB2szQHh0hUv2gdTzqgsxjiqLe4CASFNegEBMmpii9M7/j5pPuQ/jEXBvwYzqKbExIO2DfsAUrlJ5ZYFcNihgtTlBvA9VFeEpCYsLFvjet84s4QAFcmmlByqwq3kbNNdCPUIcjcIF6zXEwGYtwIYJL93co67aF/D+2T6imA7WgOcv3uqEoHR2DmhQ4QBb6IlAcS+sixJWiaj6Q3enNG7RJ0t2RrXd3lt6EoqHocgxCONt700c0kVjUMi/eM19UBN4UxVVJpvMxZQ2cLQJm0FjQZECM4i40JVHLDCccwTeKZOjtH85lqrrREQqQWiupBXFAVZw8bJJVsOt9gtwMyatPmlkCNUTyEgUDq8lw4FmLAnTGFOHjaltjEYLBLYApqyVGOtuVIZ915iU4/oG2jqODmGMNP/Ueju5qhg5hwZUO80v5ZhTEMREVI9iGefRLF8nonYldtTuW8tjpfmCY/tnhs21GA8uhsP7l1McdSvP5FuXvdiiN/NaLfvL6zUA/j5l5t/J/jJn5+bbu1zEzP9/W/VOMfh4z8221/DBm5t+3wh9jZr4pNL0V2MDQ03zV69bqWrfmcxBkVS+hzadexI98us+e7hNopOfmzw3JMIHW6XFzXbetpehtz4HhyhA+kDfAJ9/UlzBQLoHFZG/rOtvDBSRSiXBtD+OmR3XdprM7rNux4ewOI9ew+yq3uAAXMfDkVMt2buPqrS+I1V7biap/mduasAYoXzFL/tNblubze5YaPMg9LuAaOIdikm65CZdPHdlYCjJx3u41qfSbHeSFf5GOJac6mPpUf7YQQfo4V5bnapLQw0S1wWIeQG4T8WdUcAHGAPVOMka2EJBuPSTQGOIllBpNVtvcVuX7WQ7NYxKfcsjy1Kvjl3CWyxeigSutDIXgYBo4bnQcEQ/eEPmJNnFRzXvZVWWwKtWIRa3DES/0ylL7cvZ+JTDx7pVGr+1en3mTtuqja7O3bJw5dI5+9A5RVNC/GKLNZfQJJ+t6xRhjpTBx+L609R4WDrrpfTfsnSkBIle3oFXlkOPo3XqEJ1TeNemoc33qYuQlukOKVrPD116QdSrIBNs86tBHGdp80J1Ns47cH4K9JrNB5X7V4qb59IkQIsmsoDTJSICFDoDTTF1dcHFhBNXjXjVORMTh7GMR5S8y1vTSSd/rswR0NFlACG7L1fzGoWYZhgiI6LMF1a/EarbQ6JAjqZkL8hQCijTZ+pgHvK8bIGtp2WKNin+7t29+dXMfRc7ftQI9KiarWcCkCSIyG69SvaLVyjjNqF2JDULU6OMAP38awLy0Vnve8ZS6m9dHWs6jg8MhX9VJ7dlqHlXv+1BKaAxqDX2Cg3Tg29ZbMQV9iD/MHe+BYHwSouv4Xr2hSgbsUrRN66Amrg6zbMb4dAMVuYaF//TUxzPNZ3uvv916NZ/tvf721fzDiR/Gy3y+6qd4CfGKPcJWWCPAYGFvoAWE56EFsyPD4vjpZwHysDxQ0uiOcDEe3iSP9t1+tnCeAWMUW70VRVgKkCthAgXoLwtAjb7pU09q/bC6mfjKznte7K0BRydfT2ngn0EWjKY+B6D3G/15xYoe6pSgt8o1KNf9W+AlN/saKu3iphv9BuULc50h7O3JAntum0KUe9K/ettYr+xGvj2NsMqdoNjXlNDZtwkxB53SNaFrOm+TsWc6T5NhkGsyZy5GbxycqXw0kWsaj5O4i8t9WMxf4nIfFvOXuNyHxfwlLvdhMX+Jy31YzF/ich8W85e43IfF/CUu92Exf4nLfVjMX+JyHxbzb3HpqOLlbw80TDgGOgI8BOIa+uRj0yc71Fpnv9GtpaIrAiTWi42wfMCubmSFiq2uUnM26GHF5tY1ow29l2vfNQ19uIb3s1Ib/wNHYNf2EgtEFAAAAYVpQ0NQSUNDIHByb2ZpbGUAAHicfZE9SMNAHMVfU2tFKg7tIOKQoYqDBVFRR61CESqEWqFVB5NLP4QmDUmLi6PgWnDwY7Hq4OKsq4OrIAh+gDg5Oim6SIn/SwotYjw47se7e4+7d4BQLzHN6hgFNL1iphJxMZNdEYOv6EQAYQxjSmaWMStJSXiOr3v4+HoX41ne5/4cPWrOYoBPJJ5hhlkhXiee3KwYnPeJI6woq8TnxCMmXZD4keuKy2+cCw4LPDNiplNzxBFisdDGShuzoqkRTxBHVU2nfCHjssp5i7NWqrLmPfkLQzl9eYnrNAeQwAIWIUGEgio2UEIFMVp1UiykaD/u4e93/BK5FHJtgJFjHmVokB0/+B/87tbKj4+5SaE4EHix7Y9BILgLNGq2/X1s240TwP8MXOktf7kOTH+SXmtp0SOgdxu4uG5pyh5wuQP0PRmyKTuSn6aQzwPvZ/RNWSB8C3Svur0193H6AKSpq+QNcHAIDBUoe83j3V3tvf17ptnfD6IScrrpa2XVAAAABmJLR0QA/wD/AP+gvaeTAAAACXBIWXMAAC4jAAAuIwF4pT92AAAAB3RJTUUH5AYFEzcNCUk3BgAABe9JREFUeNrtnW2IVFUYx39nZjfX1VpN0dJ8qygLrFSWMDXMIKUkC98220VSjBTZsjKpPlSEUlGgfbCyoFS0hMoXliSjMKIkLFDzkwapHyohjMj3Vf99uMeYpnH0Xu/M3Dvz/OB+mBeevXN/85x7ztnnnIEc2tvbkZS2o5ekjZIOSFojqTFtn6G1tfVfBxlSjHMOYCkwGRgItAJP+efrgDrn3Ln3pYI60s+IvMePSLoPGAJI0q/AdmAzsM05d1KSCSllouQ9HuKPc/QFhgPzgUOS1gLvO+f2FBFTD/QH9pf7w2SoLfoCTwK7JP0ALPIZ1tU3a72BFuBHYLxlSHm/iCP9AXBaQbpkgKw1WdZ012yTlYrUNUyIYUJMiGFCTIhhQmxAlEp+J5iHOkUwXXItwfyUCakAnwOTnHOnAfwEYhPB3NQM4AGgizVZ5eMUwdwUObO5fwEbnHMtQD9gIbCnSIx9wHeWISXGCzoMLHPOLZM0DJhIMON7uX/tC2C9F2tCyiznJ39Yk2WYEBNi1JCQYpUkaas0Sd1N3TmXO54YAQyTdD0wABic9/aRkj4GDvru605gl3PuWM7N3IRElJABbpM0BbjHy7hQVvcDpuQ9d0LSDmAL8Amw14SEE9FT0mxgDnBTDGEbgLH+WArsAFYC65xzx5KUNUm7h/SW9CpwAHg9JhmFaAbeBX6R9CzQzYT8l3qCeqmfgWf8qLkc9PEZsw+Yxf+L7mpLiO8J3eqbkDf8TbsSXA18AHwJDK5kD62SQpykduB7LyUJ3AXslDS9UlIqJaQBWAUsJ3lT4U3AekmvVOL6ZCr0gbcCbQkf/iwG1vkvT9UKaSL4B9LYlAycZ/hxS5dqFNIIfAbcnrLZjHuBtZSpCDtTRvGrgTtIJ1P8uCj9Qnxv5fkC0xlp4wlgVql7XyUXImkc8ALVwQpJQ9MspLsfcGWrREijb3qzqRPiU/tlYBDVRTOwoFRNV8mE+NReQHXykqTeqRHivz1LqN6qliZgcSmyJFOi7BhGUCFYzcyT1CfxQvy3ZiHVX0DRDXgs7izJlCA7egLTqQ3mSqpLtBA/AOxWI0KuAcYlVohP31rJjnPE+r+TWIVIuoL0zOTGxURJ2UQKIZjJbagxIQOA6xInxKftGGqT0UnNkOE1KmREXPeR2IRIcsCNNSpkaBIzpJ6gjLMWGRRX9WOhQU0DQQ1s2J5DlmC6vSaFAN8AYa0ccs5Ny90Es+48WTMG20srDJdF7NAcKMdI3UhQL8swISbEMCEmxKgQcXZtzwAfAhsJtqW4G3gU6BoxXiewBugAzgITgNlEq7P9G3gH2EZQyjMNmEr0BTp/AisI9kNpAmYCk2K3438doVFSp8LRKelBgNxNXyTdIumwwnNc0vgC8UZLOhIy1m+SbigQa5akMxHObb+kgQXizYsSCyjJryO855zbkDt9IAnn3G7guQjxljvnvioQ71uCJWhheNo5t7dArFXARxHObT5wMD8e8BZBMXki7iEdheZy/HMbI8T7tEi8zSGbvS1FYnWEPK9TBGtbznsdkiLk5AXa77Acifi3Cr232Pv/iCDkbJHXj1ovy7q9hgkxIYYJMUyICTFMiAkxTIgJMUyICTFMiGFCTIhhQmpSSLFtXaNsQdEr4t/KpyvFVwT3D3leDRT/LaseSRHyUKEVRP65KHsrthWJNzNEnCzQUiRWS8jzquP8q4wzEeKVTMg0SYvyP7Ck+wk2LwvLHElzcy+kj/cw8HjIWEskjc+T4iS9SFDrFZY3gVF58bIEO86NutQLGWeh3GsEBWibgJO+rmpCROkZYKWkVnxpjaQJBIv0wxa3dQe2SuogWFTTSLC5QdS9gnsAX0vaBGz3Teh04OY4LmLci3Ka/REXd/rjUskCk/0RB/UElY9TrZdlvSzDhJiQkt9bjIu8zoUu/HHgSrtWZeHsxQgRwfoHowxIoq2tzS5EUnF5tnoBb9vNvqycAOY4504UEjKA4IfhTUj5OApc5Zw7Yt1eG4cYYfvBncBuE1VWjud2f/8BhL7ejharrUQAAAAASUVORK5CYII=

Government Sector

Robots are currently used in the epicentres of natural disasters, and with better human control they will be able to execute more complex rescue tasks.

/static/media/healthcare.315d5c44.png

Healthcare Sector

Voice-controlled robots can be widely used by the patients with physical restrictions that need assistance with simple tasks; the robots should be able to understand the request, since the actions can't be fully programmed and automated like in a manufacturing cycle.

Author of original research described in this blitzcard:

question-mark-icon

Name of the author who conducted the original research that this blitzcard is based on.

Source URL: #############check-icon


search-iconBrowse all blitzcards