239 lines
9.7 KiB
ReStructuredText
239 lines
9.7 KiB
ReStructuredText
|
|
Examples
|
|
========
|
|
|
|
|
|
Fixing Bugs By Replacing Shell
|
|
------------------------------
|
|
|
|
Have you ever encountered shell like this? It arranges to conditionally execute
|
|
an ``if`` statement as root on a file server behind a bastion host:
|
|
|
|
.. code-block:: bash
|
|
|
|
ssh bastion "
|
|
if [ \"$PROD\" ];
|
|
then
|
|
ssh fileserver sudo su -c \"
|
|
if grep -qs /dev/sdb1 /proc/mounts;
|
|
then
|
|
echo \\\"sdb1 already mounted!\\\";
|
|
umount /dev/sdb1
|
|
fi;
|
|
rm -rf \\\"/media/Main Backup Volume\\\"/*;
|
|
mount /dev/sdb1 \\\"/media/Main Backup Volume\\\"
|
|
\";
|
|
fi;
|
|
sudo touch /var/run/start_backup;
|
|
"
|
|
|
|
Chances are high this is familiar territory, we've all seen it, and those
|
|
working in infrastructure have almost certainly written it. At first glance,
|
|
ignoring that annoying quoting, it looks perfectly fine: well structured,
|
|
neatly indented, and the purpose of the snippet seems clear.
|
|
|
|
1. At first glance, is ``"/media/Main Backup Volume"`` quoted correctly?
|
|
2. How will the ``if`` statement behave if there is a problem with the machine,
|
|
and, say, the ``/bin/grep`` binary is absent?
|
|
3. Ignoring quoting, are there any other syntax problems?
|
|
4. If this snippet is pasted from its original script into an interactive
|
|
shell, will it behave the same as before?
|
|
5. Can you think offhand of differences in how the arguments to ``sudo
|
|
...`` and ``ssh fileserver ...`` are parsed?
|
|
6. In which context will the ``*`` glob be expanded, if it is expanded at all?
|
|
7. What will the exit status of ``ssh bastion`` be if ``ssh fileserver`` fails?
|
|
|
|
|
|
Innocent But Deadly
|
|
~~~~~~~~~~~~~~~~~~~
|
|
|
|
1. The quoting used is nonsense! At best, ``mount`` will receive 3 arguments.
|
|
At worst, the snippet will not parse at all.
|
|
2. The ``if`` statement will treat a missing ``grep`` binary (exit status 127)
|
|
the same as if ``/dev/sdb1`` was not mounted at all (exit status 1). Unless
|
|
the program executing this script is parsing ``stderr`` output, the failure
|
|
won't be noticed. Consequently, since the volume was still mounted when
|
|
``rm`` was executed, it got wiped.
|
|
3. There is at least one more syntax error present: a semicolon missing after
|
|
the ``umount`` command.
|
|
4. If you paste the snippet into an interactive shell, the apparently quoted
|
|
"!" character in the ``echo`` command will be interpreted as a history
|
|
expansion.
|
|
5. ``sudo`` preserves the remainder of the argument vector as-is, while
|
|
``ssh`` **concatenates** each part into a single string that is passed to
|
|
the login shell. While quotes appearing within arguments are preserved by
|
|
``sudo``, without additional effort, pairs of quotes are effectively
|
|
stripped by ``ssh``.
|
|
6. As for where the glob is expanded, the answer is I have absolutely no idea
|
|
without running the code, which might wipe out the backups!
|
|
7. If the ``ssh fileserver`` command fails, the exit status of ``ssh bastion``
|
|
will continue to indicate success.
|
|
8. Depending in which environment the ``PROD`` variable is set, either it will
|
|
always evaluate to false, because it was set by the bastion host, or it
|
|
will do the right thing, because it was set by the script host.
|
|
|
|
Golly, we've managed to hit at least 8 potentially mission-critical gotchas in
|
|
only 14 lines of code, and they are just those I can count! Welcome to the
|
|
reality of "programming" in shell.
|
|
|
|
In the end, superficial legibility counted for nothing, it's 4AM, you've been
|
|
paged, the network is down and your boss is angry.
|
|
|
|
|
|
Shell Quoting Madness
|
|
~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
Let's assume on first approach that we really want to handle those quoting
|
|
issues. I wrote a little Python script based around the :py:func:`shlex.quote`
|
|
function to construct, to the best of my knowledge, the quoting required for
|
|
each stage:
|
|
|
|
.. code-block:: bash
|
|
|
|
ssh bastion '
|
|
if [ "$PROD" ];
|
|
then
|
|
ssh fileserver sudo su -c '"'"'
|
|
if grep -qs /dev/sdb1 /proc/mounts;
|
|
then
|
|
echo "sdb1 already mounted!";
|
|
umount /dev/sdb1
|
|
fi;
|
|
rm -rf "/media/Main Backup Volume"/*;
|
|
mount /dev/sdb1 "/media/Main Backup Volume"
|
|
'"'"';
|
|
fi;
|
|
sudo touch /var/run/start_backup
|
|
'
|
|
|
|
Even with Python handling the heavy lifting of quoting each shell layer, and
|
|
even if the aforementioned minor disk-wiping issue was fixed, it is still not
|
|
100% clear that argument handling rules for all of ``su``, ``sudo``, ``ssh``,
|
|
and ``bash`` are correctly respected.
|
|
|
|
Finally, if any login shell involved is not ``bash``, we must introduce
|
|
additional quoting in order to explicitly invoke ``bash`` at each stage,
|
|
causing an explosion in quoting:
|
|
|
|
.. code-block:: bash
|
|
|
|
ssh bastion 'bash -c '"'"'if [ "$PROD" ]; then ssh fileserver bash -c '"'"'
|
|
"'"'"'"'"'"'sudo su -c '"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"
|
|
'bash -c '"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"
|
|
'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'
|
|
"'"'"'"'"'"'"'"'"'"'if grep -qs /dev/sdb1 /proc/mounts; then echo "sdb1 alr
|
|
eady mounted!"; umount /dev/sdb1 fi; rm -rf "/media/Main Backup Volume"/*;
|
|
mount /dev/sdb1 "/media/Main Backup Volume"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"
|
|
'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'
|
|
"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"''"'"'"'"'"'"'"'"'"'"'
|
|
"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"''"'"'"'"'"'"'"'"'; fi; sudo touch /var/run/
|
|
start_backup'"'"''
|
|
|
|
|
|
There Is Hope
|
|
~~~~~~~~~~~~~
|
|
|
|
We could instead express the above using Mitogen:
|
|
|
|
::
|
|
|
|
import shutil, os, subprocess
|
|
import mitogen
|
|
|
|
def run(*args):
|
|
return subprocess.check_call(args)
|
|
|
|
def file_contains(s, path):
|
|
with open(path, 'rb') as fp:
|
|
return s in fp.read()
|
|
|
|
@mitogen.main()
|
|
def main(router):
|
|
device = '/dev/sdb1'
|
|
mount_point = '/media/Media Volume'
|
|
|
|
bastion = router.ssh(hostname='bastion')
|
|
bastion_sudo = router.sudo(via=bastion)
|
|
|
|
if PROD:
|
|
fileserver = router.ssh(hostname='fileserver', via=bastion)
|
|
if fileserver.call(file_contains, device, '/proc/mounts'):
|
|
print('{} already mounted!'.format(device))
|
|
fileserver.call(run, 'umount', device)
|
|
fileserver.call(shutil.rmtree, mount_point)
|
|
fileserver.call(os.mkdir, mount_point, 0777)
|
|
fileserver.call(run, 'mount', device, mount_point)
|
|
|
|
bastion_sudo.call(run, 'touch', '/var/run/start_backup')
|
|
|
|
* In which context must the ``PROD`` variable be defined?
|
|
* On which machine is each step executed?
|
|
* Are there any escaping issues?
|
|
* What will happen if the ``grep`` binary is missing?
|
|
* What will happen if any step fails?
|
|
* What will happen if any login shell is not ``bash``?
|
|
|
|
|
|
Recursively Nested Bootstrap
|
|
----------------------------
|
|
|
|
This demonstrates the library's ability to use slave contexts to recursively
|
|
proxy connections to additional slave contexts, with a uniform API to any
|
|
slave, and all features (function calls, import forwarding, stdio forwarding,
|
|
log forwarding) functioning transparently.
|
|
|
|
This example uses a chain of local contexts for clarity, however SSH and sudo
|
|
contexts work identically.
|
|
|
|
nested.py:
|
|
|
|
.. code-block:: python
|
|
|
|
import os
|
|
import mitogen
|
|
|
|
@mitogen.main()
|
|
def main(router):
|
|
mitogen.utils.log_to_file()
|
|
|
|
context = None
|
|
for x in range(1, 11):
|
|
print('Connect local%d via %s' % (x, context))
|
|
context = router.local(via=context, name='local%d' % x)
|
|
|
|
context.call(os.system, 'pstree -s python -s mitogen')
|
|
|
|
|
|
Output:
|
|
|
|
.. code-block:: shell
|
|
|
|
$ python nested.py
|
|
Connect local1 via None
|
|
Connect local2 via Context(1, 'local1')
|
|
Connect local3 via Context(2, 'local2')
|
|
Connect local4 via Context(3, 'local3')
|
|
Connect local5 via Context(4, 'local4')
|
|
Connect local6 via Context(5, 'local5')
|
|
Connect local7 via Context(6, 'local6')
|
|
Connect local8 via Context(7, 'local7')
|
|
Connect local9 via Context(8, 'local8')
|
|
Connect local10 via Context(9, 'local9')
|
|
18:14:07 I ctx.local10: stdout: -+= 00001 root /sbin/launchd
|
|
18:14:07 I ctx.local10: stdout: \-+= 08126 dmw /Applications/iTerm.app/Contents/MacOS/iTerm2
|
|
18:14:07 I ctx.local10: stdout: \-+= 10638 dmw /Applications/iTerm.app/Contents/MacOS/iTerm2 --server bash --login
|
|
18:14:07 I ctx.local10: stdout: \-+= 10639 dmw bash --login
|
|
18:14:07 I ctx.local10: stdout: \-+= 13632 dmw python nested.py
|
|
18:14:07 I ctx.local10: stdout: \-+- 13633 dmw mitogen:dmw@Eldil.local:13632
|
|
18:14:07 I ctx.local10: stdout: \-+- 13635 dmw mitogen:dmw@Eldil.local:13633
|
|
18:14:07 I ctx.local10: stdout: \-+- 13637 dmw mitogen:dmw@Eldil.local:13635
|
|
18:14:07 I ctx.local10: stdout: \-+- 13639 dmw mitogen:dmw@Eldil.local:13637
|
|
18:14:07 I ctx.local10: stdout: \-+- 13641 dmw mitogen:dmw@Eldil.local:13639
|
|
18:14:07 I ctx.local10: stdout: \-+- 13643 dmw mitogen:dmw@Eldil.local:13641
|
|
18:14:07 I ctx.local10: stdout: \-+- 13645 dmw mitogen:dmw@Eldil.local:13643
|
|
18:14:07 I ctx.local10: stdout: \-+- 13647 dmw mitogen:dmw@Eldil.local:13645
|
|
18:14:07 I ctx.local10: stdout: \-+- 13649 dmw mitogen:dmw@Eldil.local:13647
|
|
18:14:07 I ctx.local10: stdout: \-+- 13651 dmw mitogen:dmw@Eldil.local:13649
|
|
18:14:07 I ctx.local10: stdout: \-+- 13653 dmw pstree -s python -s mitogen
|
|
18:14:07 I ctx.local10: stdout: \--- 13654 root ps -axwwo user,pid,ppid,pgid,command
|