Introduction
Repetitive duties are ripe for automation. It is not uncommon for builders and system directors to automate routine duties like well being checks and file backups with shell scripts. Nevertheless, as these duties turn out to be extra complicated, shell scripts might turn out to be more durable to take care of.
Luckily, we will use Python as a substitute of shell scripts for automation. Python offers strategies to run shell instructions, giving us the identical performance of these shells scripts, with extra assist from Python’s ecosystem. Studying the way to run shell instructions in Python opens the door for us to automate pc duties in a structured and scalable approach.
On this article, we are going to have a look at the varied methods to execute shell instructions in Python, and the best state of affairs to make use of every technique.
Utilizing os.system to Run a Command
Python permits us to instantly execute a shell command that is saved in a string utilizing the os.system() perform. Let’s begin by creating a brand new Python file known as echo_adelle.py and enter the next:
import os
os.system("echo Howdy from the opposite facet!")
The very first thing we do in our Python file is import the os module, which comprises the system perform that may execute shell instructions. The following line does precisely that, runs the echo command in our shell via Python.
In your Terminal, run this file with utilizing the next command, and it’s best to see the corresponding output:
$ python3 echo_adelle.py
Howdy from the opposite facet!
Because the echo instructions prints to our stdout, os.system() additionally shows the output on our stdout stream. Whereas not seen within the console, the os.system() command returns the exit code of the shell command. An exit code of 0 means it ran with none issues and another quantity means an error. This manner, you ca management the move of your instructions, examine the outcomes and both retry or return an error if any name fails:
import os
home_dir = os.system("cd ~")
print("`cd ~` ran with exit code %d" % home_dir)
unknown_dir = os.system("cd doesnotexist")
print("`cd doesnotexis` ran with exit code %d" % unknown_dir)
On this script, we create two variables that retailer the results of executing instructions that change the listing to the house folder, and to a folder that doesn’t exist. Operating this file, we are going to see:
$ python3 cd_return_codes.py
`cd ~` ran with exit code 0
sh: line 0: cd: doesnotexist: No such file or listing
`cd doesnotexist` ran with exit code 256
The primary command, which adjustments the listing to the house listing, executes efficiently. Subsequently, os.system() returns its exit code, zero, which is saved in home_dir. However, unknown_dir shops the exit code of the failed bash command to vary the listing to a folder that doesn’t exist.
The os.system() perform executes a command, prints any output of the command to the console, and returns the exit code of the command. If we want extra high-quality grained management of a shell command’s enter and output in Python, we should always use the subprocess module.
Operating a Command with subprocess
The subprocess module is Python’s really useful method to executing shell instructions. It provides us the pliability to suppress the output of shell instructions or chain inputs and outputs of assorted instructions collectively, whereas nonetheless offering an identical expertise to os.system() for fundamental use instances.
In a brand new file known as list_subprocess.py, let’s run a command with the subprocess module:
import subprocess
list_files = subprocess.run(["ls", "-l"])
print("The exit code was: %d" % list_files.returncode)
Within the first line, we import the subprocess module, which is a part of the Python normal library. We then use the subprocess.run() perform to execute the command. Like os.system(), the subprocess.run() command returns the exit code of what was executed.
Not like os.system(), notice how subprocess.run() requires a listing of strings as enter as a substitute of a single string. The primary merchandise of the record is the title of the command. The remaining gadgets of the record are the flags and the arguments of the command.
Observe: As a rule of thumb, you’ll want to separate the arguments primarily based on area, for instance ls -alh could be ["ls", "-alh"], whereas ls -a -l -h, could be ["ls", "-a", -"l", "-h"]. As one other instance, echo good day world could be ["echo", "hello", "world"], whereas echo "good day world" or echo good day world could be ["echo", "hello world"].
Run this file and your console’s output could be much like:
$ python3 list_subprocess.py
complete 80
[email protected] 1 stackabuse employees 216 Dec 6 10:29 cd_return_codes.py
[email protected] 1 stackabuse employees 56 Dec 6 10:11 echo_adelle.py
[email protected] 1 stackabuse employees 116 Dec 6 11:20 list_subprocess.py
The exit code was: 0
Now let’s attempt to use one of many extra superior options of subprocess.run(), particularly ignore output to stdout. In the identical list_subprocess.py file, change:
list_files = subprocess.run(["ls", "-l"])
To this:
list_files = subprocess.run(["ls", "-l"], stdout=subprocess.DEVNULL)
The usual output of the command now pipes to the particular /dev/null gadget, which suggests the output wouldn’t seem on our consoles. Execute the file in your shell to see the next output:
$ python3 list_subprocess.py
The exit code was: 0
What if we needed to supply enter to a command? The subprocess.run() facilitates this by its enter argument:
Take a look at our hands-on, sensible information to studying Git, with best-practices, industry-accepted requirements, and included cheat sheet. Cease Googling Git instructions and really be taught it!
import subprocess
useless_cat_call = subprocess.run(["cat"],
stdout=subprocess.PIPE,
textual content=True,
enter="Howdy from the opposite facet")
print(useless_cat_call.stdout)
We use subprocess.run() with fairly just a few instructions, let’s undergo them:
stdout=subprocess.PIPEtells Python to redirect the output of the command to an object so it may be manually learn latertextual content=Truereturnsstdoutandstderras strings. The default return sort is bytes.enter="Howdy from the opposite facet"tells Python so as to add the string as enter to thecatcommand.
Operating this produces the next output:
Howdy from the opposite facet
We are able to additionally elevate an Exception with out manually checking the return worth. In a brand new file, false_subprocess.py, add the code beneath:
import subprocess
failed_command = subprocess.run(["false"], examine=True)
print("The exit code was: %d" % failed_command.returncode)
In your Terminal, run this file. You will notice the next error:
$ python3 false_subprocess.py
Traceback (most up-to-date name final):
File "false_subprocess.py", line 4, in <module>
failed_command = subprocess.run(["false"], examine=True)
File "/usr/native/python/3.7.5/Frameworks/Python.framework/Variations/3.7/lib/python3.7/subprocess.py", line 512, in run
output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['false']' returned non-zero exit standing 1.
By utilizing examine=True, we inform Python to lift any exceptions if an error is encountered. Since we did encounter an error, the print assertion on the ultimate line was not executed.
The subprocess.run() perform provides us immense flexibility that os.system() would not when executing shell instructions. This perform is a simplified abstraction of the subprocess.Popen class, which offers extra performance we will discover.
Operating a Command with Popen
The subprocess.Popen class exposes extra choices to the developer when interacting with the shell. Nevertheless, we have to be extra specific about retrieving outcomes and errors.
By default, subprocess.Popen doesn’t cease processing of a Python program if its command has not completed executing. In a brand new file known as list_popen.py, sort the next:
import subprocess
list_dir = subprocess.Popen(["ls", "-l"])
list_dir.wait()
This code is equal to that of list_subprocess.py. It runs a command utilizing subprocess.Popen, and waits for it to finish earlier than executing the remainder of the Python script.
For instance we don’t wish to watch for our shell command to finish execution so this system can work on different issues. How would it not know when the shell command has completed execution?
The ballot() technique returns the exit code if a command has completed working, or None if it is nonetheless executing. For instance, if we needed to examine if list_dir was full as a substitute of watch for it, we’d have the next line of code:
list_dir.ballot()
To handle enter and output with subprocess.Popen, we have to use the talk() technique.
In a brand new file known as cat_popen.py, add the next code snippet:
import subprocess
useless_cat_call = subprocess.Popen(["cat"], stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE, textual content=True)
output, errors = useless_cat_call.talk(enter="Howdy from the opposite facet!")
useless_cat_call.wait()
print(output)
print(errors)
The talk() technique takes an enter argument that is used to go enter to the shell command. The talk technique additionally returns each the stdout and stderr when they’re set.
Having seen the core concepts behind subprocess.Popen, we’ve now lined 3 ways to run shell instructions in Python. Let’s re-examine their traits so we’ll know which technique is finest fitted to a challenge’s necessities.
Finest Strategy to Run Shell Instructions with Python?
If you’ll want to run one or just a few easy instructions and don’t thoughts if their output goes to the console, you should use the os.system() command. If you wish to handle the enter and output of a shell command, use subprocess.run(). If you wish to run a command and proceed doing different work whereas it is being executed, use subprocess.Popen.
Here’s a desk with some usability variations you could additionally use to tell your determination:
| os.system | subprocess.run | subprocess.Popen | |
|---|---|---|---|
| Requires parsed arguments | no | sure | sure |
| Waits for the command | sure | sure | no |
| Communicates with stdin and stdout | no | sure | sure |
| Returns | return worth | object | object |
Conclusion
Python permits you to execute shell instructions, which you should use to start out different packages or higher handle shell scripts that you simply use for automation. Relying on our use case, we will use os.system(), subprocess.run() or subprocess.Popen to run bash instructions.
Utilizing these strategies, what exterior process would you run through Python?
