Instruction:
Dropping infinite values from dataframes in pandas?

How do I drop nan, inf, and -inf values from a DataFrame without resetting mode.use_inf_as_null? Can I tell dropna to include inf in its definition of missing values so that the following works? df.dropna(subset=["col1", "col2"], how="all")

Output: First replace() infs with NaN: df.replace([np.inf, -np.inf], np.nan, inplace=True) and then drop NaNs via dropna(): df.dropna(subset=["col1", "col2"], how="all", inplace=True) For example: >>> df = pd.DataFrame({"col1": [1, np.inf, -np.inf], "col2": [2, 3, np.nan]}) >>> df col1 col2 0 1.0 2.0 1 inf 3.0 2 -inf NaN >>> df.replace([np.inf, -np.inf], np.nan, inplace=True) >>> df col1 col2 0 1.0 2.0 1 NaN 3.0 2 NaN NaN >>> df.dropna(subset=["col1", "col2"], how="all", inplace=True) >>> df col1 col2 0 1.0 2.0 1 NaN 3.0 The same method also works for Series.


Instruction:
Python Requests - No connection adapters

I'm using the Requests: HTTP for Humans library and I got this error: No connection adapters were found for '192.168.1.61:8080/api/call' What does this mean, and how can I fix it?

Output: You need to include the protocol scheme: 'http://192.168.1.61:8080/api/call' Without the http:// part, Requests doesn’t have any idea how to connect to the remote server. Note that the protocol scheme must be all lowercase; if your URL starts with HTTP:// for example, it won’t find the http:// connection adapter either.


Instruction:
Cross-platform way of getting temp directory in Python

Is there a cross-platform way of getting the path to the temp directory in Python 2.6? For example, under Linux that would be /tmp, while under XP C:\Documents and settings\[user]\Application settings\Temp.

Output: That would be the tempfile module. It has functions to get the temporary directory, and also has some shortcuts to create temporary files and directories in it, either named or unnamed. Example: import tempfile print tempfile.gettempdir() # prints the current temporary directory f = tempfile.TemporaryFile() f.write('something on temporaryfile') f.seek(0) # return to beginning of file print f.read() # reads data back from the file f.close() # temporary file is automatically deleted here For completeness, here's how it searches for the temporary directory, according to the documentation: The directory named by the TMPDIR environment variable. The directory named by the TEMP environment variable. The directory named by the TMP environment variable. A platform-specific location: On RiscOS, the directory named by the Wimp$ScrapDir environment variable. On Windows, the directories C:\TEMP, C:\TMP, \TEMP, and \TMP, in that order. On all other platforms, the directories /tmp, /var/tmp, and /usr/tmp, in that order. As a last resort, the current working directory.


Instruction:
How do I use Pandas group-by to get the sum?

I am using this dataframe: Fruit Date Name Number Apples 10/6/2016 Bob 7 Apples 10/6/2016 Bob 8 Apples 10/6/2016 Mike 9 Apples 10/7/2016 Steve 10 Apples 10/7/2016 Bob 1 Oranges 10/7/2016 Bob 2 Oranges 10/6/2016 Tom 15 Oranges 10/6/2016 Mike 57 Oranges 10/6/2016 Bob 65 Oranges 10/7/2016 Tony 1 Grapes 10/7/2016 Bob 1 Grapes 10/7/2016 Tom 87 Grapes 10/7/2016 Bob 22 Grapes 10/7/2016 Bob 12 Grapes 10/7/2016 Tony 15 I would like to aggregate this by Name and then by Fruit to get a total number of Fruit per Name. For example: Bob,Apples,16 I tried grouping by Name and Fruit but how do I get the total number of Fruit?

Output: Use GroupBy.sum: df.groupby(['Fruit','Name']).sum() Out[31]: Number Fruit Name Apples Bob 16 Mike 9 Steve 10 Grapes Bob 35 Tom 87 Tony 15 Oranges Bob 67 Mike 57 Tom 15 Tony 1 To specify the column to sum, use this: df.groupby(['Name', 'Fruit'])['Number'].sum()


Instruction:
How do I save a trained model in PyTorch?

How do I save a trained model in PyTorch? I have read that: torch.save()/torch.load() is for saving/loading a serializable object. model.state_dict()/model.load_state_dict() is for saving/loading model state.

Output: Found this page on their github repo: Recommended approach for saving a model There are two main approaches for serializing and restoring a model. The first (recommended) saves and loads only the model parameters: torch.save(the_model.state_dict(), PATH) Then later: the_model = TheModelClass(*args, **kwargs) the_model.load_state_dict(torch.load(PATH)) The second saves and loads the entire model: torch.save(the_model, PATH) Then later: the_model = torch.load(PATH) However in this case, the serialized data is bound to the specific classes and the exact directory structure used, so it can break in various ways when used in other projects, or after some serious refactors. See also: Save and Load the Model section from the official PyTorch tutorials.


Instruction:
Transpose list of lists

Suppose I have l = [[1, 2, 3], [4, 5, 6], [7, 8, 9]] How can I get a result like so? r = [[1, 4, 7], [2, 5, 8], [3, 6, 9]] I know how to get r = [(1, 4, 7), (2, 5, 8), (3, 6, 9)] but I must have lists as elements of the result.

Output: For rectangular data (or to limit each "column" to the length of the shortest input "row") In Python 3.x, use: # short circuits at shortest nested list if table is jagged: list(map(list, zip(*l))) In Python 2.x, use: # short circuits at shortest nested list if table is jagged: map(list, zip(*l)) There are two important things to understand here: The signature of zip: zip(*iterables) This means zip expects an arbitrary number of arguments each of which must be iterable. E.g. zip([1, 2], [3, 4], [5, 6]). Unpacked argument lists: Given a sequence of arguments args, f(*args) will call f such that each element in args is a separate positional argument of f. Given l = [[1, 2, 3], [4, 5, 6], [7, 8, 9]], zip(*l) would be equivalent to zip([1, 2, 3], [4, 5, 6], [7, 8, 9]). See also: Expanding tuples into arguments The rest is just making sure the result is a list of lists instead of a list of tuples, by using map to create a list from each tuple. For jagged data To pad shorter rows with None values in the output, import itertools (this is in the standard library), and then: In Python 3.x, use: list(map(list, itertools.zip_longest(*l, fillvalue=None))) In Python 2.x, use: list(map(list, itertools.izip_longest(*l, fillvalue=None)))


Instruction:
How to use `subprocess` command with pipes

I want to use subprocess.check_output() with ps -A | grep 'process_name'. I tried various solutions but so far nothing worked. Can someone guide me how to do it?

Output: To use a pipe with the subprocess module, you can pass shell=True but be aware of the Security Considerations. It is discouraged using shell=True. In most cases there are better solutions for the same problem. However, this isn't really advisable for various reasons, not least of which is security. Instead, create the ps and grep processes separately, and pipe the output from one into the other, like so: ps = subprocess.Popen(('ps', '-A'), stdout=subprocess.PIPE) output = subprocess.check_output(('grep', 'process_name'), stdin=ps.stdout) ps.wait() In your particular case, however, the simple solution is to call subprocess.check_output(('ps', '-A')) and then str.find on the output.


Instruction:
How to update an existing Conda environment with a .yml file

How can a pre-existing conda environment be updated with another .yml file. This is extremely helpful when working on projects that have multiple requirement files, i.e. base.yml, local.yml, production.yml, etc. For example, below is a base.yml file has conda-forge, conda, and pip packages: base.yml name: myenv channels: - conda-forge dependencies: - django=1.10.5 - pip: - django-crispy-forms==1.6.1 The actual environment is created with: conda env create -f base.yml. Later on, additional packages need to be added to base.yml. Another file, say local.yml, needs to import those updates. Previous attempts to accomplish this include: creating a local.yml file with an import definition: channels: dependencies: - pip: - boto3==1.4.4 imports: - requirements/base. And then run the command: conda install -f local.yml. This does not work. Any thoughts?

Output: Try using conda env update: conda activate myenv conda env update --file local.yml --prune --prune uninstalls dependencies which were removed from local.yml, as pointed out in this answer by @Blink. Attention: if there is a name tag with a name other than that of your environment in local.yml, the command above will create a new environment with that name. To avoid this, use (thanks @NumesSanguis): conda env update --name myenv --file local.yml --prune See Updating an environment in Conda User Guide.


Instruction:
Python strings and integer concatenation

I want to create a string using an integer appended to it, in a for loop. Like this: for i in range(1, 11): string = "string" + i But it returns an error: TypeError: unsupported operand type(s) for +: 'int' and 'str' What's the best way to concatenate the string and integer?

Output: NOTE: The method used in this answer (backticks) is deprecated in later versions of Python 2, and removed in Python 3. Use the str() function instead. You can use: string = 'string' for i in range(11): string +=`i` print string It will print string012345678910. To get string0, string1 ..... string10 you can use this as YOU suggested: >>> string = "string" >>> [string+`i` for i in range(11)] For Python 3 You can use: string = 'string' for i in range(11): string += str(i) print string It will print string012345678910. To get string0, string1 ..... string10, you can use this as YOU suggested: >>> string = "string" >>> [string+str(i) for i in range(11)]


Instruction:
Upgrade Python in a virtual environment

Is there a way to upgrade the version of Python used in a virtual environment (e.g., if a bugfix release comes out)? I could pip freeze --local > requirements.txt, remove the directory, and pip install -r requirements.txt, but this requires a lot of reinstallation of large libraries, for instance, NumPy, which I use a lot. I can see this is an advantage when upgrading from, e.g., 2.6 -> 2.7, but what about 2.7.x -> 2.7.y?

Output: Did you see this? If I haven't misunderstand that answer, you may try to create a new virtualenv on top of the old one. You just need to know which Python interpreter is going to use your virtualenv (you will need to see your virtualenv version). If your virtualenv is installed with the same Python version of the old one and upgrading your virtualenv package is not an option, you may want to read this in order to install a virtualenv with the Python version you want. I've tested this approach (the one that create a new virtualenv on top of the old one), and it worked fine for me. I think you may have some problems if you change from python 2.6 to 2.7 or 2.7 to 3.x, but if you just upgrade inside the same version (staying at 2.7 as you want) you shouldn't have any problem, as all the packages are held in the same folders for both Python versions (2.7.x and 2.7.y packages are inside your_env/lib/python2.7/). If you change your virtualenv Python version, you will need to install all your packages again for that version (or just link the packages you need into the new version packages folder, i.e., your_env/lib/python_newversion/site-packages)


Instruction:
What is pip's `--no-cache-dir` good for?

I've recently seen the --no-cache-dir being used in a Docker file. I've never seen that flag before and the help is not explaining it: --no-cache-dir Disable the cache. Question: What is cached? Question: What is the cache used for? Question: Why would I want to disable it?

Output: Cached is: store away in hiding or for future use Used for store the installation files(.whl, etc) of the modules that you install through pip store the source files (.tar.gz, etc) to avoid re-download when not expired Possible Reason you might want to disable cache: you don't have space on your hard drive previously run pip install with unexpected settings eg: previously run export PYCURL_SSL_LIBRARY=nss and pip install pycurl want new run export PYCURL_SSL_LIBRARY=openssl and pip install pycurl --compile --no-cache-dir you want to keep a Docker image as small as possible Links to documentation https://pip.pypa.io/en/stable/topics/caching


Instruction:
Get a random boolean in python?

I am looking for the best way (fast and elegant) to get a random boolean in python (flip a coin). For the moment I am using random.randint(0, 1) or random.getrandbits(1). Are there better choices that I am not aware of?

Output: Adam's answer is quite fast, but I found that random.getrandbits(1) to be quite a lot faster. If you really want a boolean instead of a long then import random bool(random.getrandbits(1)) is still about twice as fast as random.choice([True, False]) Both solutions need to import random If utmost speed isn't to priority then random.choice definitely reads better. Note that random.choice() is slower than just choice() (after from random import choice) due to the attribute lookup. $ python3 --version Python 3.9.7 $ python3 -m timeit -s "from random import choice" "choice([True, False])" 1000000 loops, best of 5: 376 nsec per loop $ python3 -m timeit -s "from random import choice" "choice((True, False))" 1000000 loops, best of 5: 352 nsec per loop $ python3 -m timeit -s "from random import getrandbits" "getrandbits(1)" 10000000 loops, best of 5: 33.7 nsec per loop $ python3 -m timeit -s "from random import getrandbits" "bool(getrandbits(1))" 5000000 loops, best of 5: 89.5 nsec per loop $ python3 -m timeit -s "from random import getrandbits" "not getrandbits(1)" 5000000 loops, best of 5: 46.3 nsec per loop $ python3 -m timeit -s "from random import random" "random() < 0.5" 5000000 loops, best of 5: 46.4 nsec per loop


Instruction:
Converting numpy dtypes to native python types

If I have a numpy dtype, how do I automatically convert it to its closest python data type? For example, numpy.float32 -> "python float" numpy.float64 -> "python float" numpy.uint32 -> "python int" numpy.int16 -> "python int" I could try to come up with a mapping of all of these cases, but does numpy provide some automatic way of converting its dtypes into the closest possible native python types? This mapping need not be exhaustive, but it should convert the common dtypes that have a close python analog. I think this already happens somewhere in numpy.

Output: Use val.item() to convert most NumPy values to a native Python type: import numpy as np # for example, numpy.float32 -> python float val = np.float32(0) pyval = val.item() print(type(pyval)) # <class 'float'> # and similar... type(np.float64(0).item()) # <class 'float'> type(np.uint32(0).item()) # <class 'int'> type(np.int16(0).item()) # <class 'int'> type(np.cfloat(0).item()) # <class 'complex'> type(np.datetime64(0, 'D').item()) # <class 'datetime.date'> type(np.datetime64('2001-01-01 00:00:00').item()) # <class 'datetime.datetime'> type(np.timedelta64(0, 'D').item()) # <class 'datetime.timedelta'> ... (A related method np.asscalar(val) was deprecated with 1.16, and removed with 1.23). For the curious, to build a table of conversions of NumPy array scalars for your system: for name in dir(np): obj = getattr(np, name) if hasattr(obj, 'dtype'): try: if 'time' in name: npn = obj(0, 'D') else: npn = obj(0) nat = npn.item() print('{0} ({1!r}) -> {2}'.format(name, npn.dtype.char, type(nat))) except: pass There are a few NumPy types that have no native Python equivalent on some systems, including: clongdouble, clongfloat, complex192, complex256, float128, longcomplex, longdouble and longfloat. These need to be converted to their nearest NumPy equivalent before using .item().


Instruction:
How do I subtract one list from another?

I want to take the difference between lists x and y: >>> x = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9] >>> y = [1, 3, 5, 7, 9] >>> x - y # should return [0, 2, 4, 6, 8]

Output: Use a list comprehension to compute the difference while maintaining the original order from x: [item for item in x if item not in y] If you don't need list properties (e.g. ordering), use a set difference, as the other answers suggest: list(set(x) - set(y)) To allow x - y infix syntax, override __sub__ on a class inheriting from list: class MyList(list): def __init__(self, *args): super(MyList, self).__init__(args) def __sub__(self, other): return self.__class__(*[item for item in self if item not in other]) Usage: x = MyList(1, 2, 3, 4) y = MyList(2, 5, 2) z = x - y


Instruction:
Convert string to Enum in Python

What's the correct way to convert a string to a corresponding instance of an Enum subclass? Seems like getattr(YourEnumType, str) does the job, but I'm not sure if it's safe enough. As an example, suppose I have an enum like class BuildType(Enum): debug = 200 release = 400 Given the string 'debug', how can I get BuildType.debug as a result?

Output: This functionality is already built in to Enum: >>> from enum import Enum >>> class Build(Enum): ... debug = 200 ... build = 400 ... >>> Build['debug'] <Build.debug: 200> The member names are case sensitive, so if user-input is being converted you need to make sure case matches: an_enum = input('Which type of build?') build_type = Build[an_enum.lower()]


Instruction:
How to get a string after a specific substring?

How can I get a string after a specific substring? For example, I want to get the string after "world" in my_string="hello python world, I'm a beginner" ...which in this case is: ", I'm a beginner")

Output: The easiest way is probably just to split on your target word my_string="hello python world , i'm a beginner" print(my_string.split("world",1)[1]) split takes the word (or character) to split on and optionally a limit to the number of splits. In this example, split on "world" and limit it to only one split.


Instruction:
How do I remove NaN values from a NumPy array?

How do I remove NaN values from a NumPy array? [1, 2, NaN, 4, NaN, 8] ⟶ [1, 2, 4, 8]

Output: To remove NaN values from a NumPy array x: x = x[~numpy.isnan(x)] Explanation The inner function numpy.isnan returns a boolean/logical array which has the value True everywhere that x is not-a-number. Since we want the opposite, we use the logical-not operator ~ to get an array with Trues everywhere that x is a valid number. Lastly, we use this logical array to index into the original array x, in order to retrieve just the non-NaN values.


Instruction:
How can I fix "UnboundLocalError: local variable referenced before assignment"?

The following code gives the error UnboundLocalError: local variable 'Var1' referenced before assignment: Var1 = 1 Var2 = 0 def function(): if Var2 == 0 and Var1 > 0: print("Result 1") elif Var2 == 1 and Var1 > 0: print("Result 2") elif Var1 < 1: print("Result 3") Var1 -= 1 function() How can I fix this?

Output: You can fix this by passing parameters rather than relying on Globals def function(Var1, Var2): if Var2 == 0 and Var1 > 0: print("Result One") elif Var2 == 1 and Var1 > 0: print("Result Two") elif Var1 < 1: print("Result Three") return Var1 -= 1 function(1, 1)


Instruction:
How to use a variable inside a regular expression

I'd like to use a variable inside a regex, how can I do this in Python? TEXTO = sys.argv[1] if re.search(r"\b(?=\w)TEXTO\b(?!\w)", subject, re.IGNORECASE): # Successful match else: # Match attempt failed

Output: From python 3.6 on you can also use Literal String Interpolation, "f-strings". In your particular case the solution would be: if re.search(rf"\b(?=\w){TEXTO}\b(?!\w)", subject, re.IGNORECASE): ...do something EDIT: Since there have been some questions in the comment on how to deal with special characters I'd like to extend my answer: raw strings ('r'): One of the main concepts you have to understand when dealing with special characters in regular expressions is to distinguish between string literals and the regular expression itself. It is very well explained here: In short: Let's say instead of finding a word boundary \b after TEXTO you want to match the string \boundary. The you have to write: TEXTO = "Var" subject = r"Var\boundary" if re.search(rf"\b(?=\w){TEXTO}\\boundary(?!\w)", subject, re.IGNORECASE): print("match") This only works because we are using a raw-string (the regex is preceded by 'r'), otherwise we must write "\\\\boundary" in the regex (four backslashes). Additionally, without '\r', \b' would not converted to a word boundary anymore but to a backspace! re.escape: Basically puts a backslash in front of any special character. Hence, if you expect a special character in TEXTO, you need to write: if re.search(rf"\b(?=\w){re.escape(TEXTO)}\b(?!\w)", subject, re.IGNORECASE): print("match") NOTE: For any version >= python 3.7: !, ", %, ', ,, /, :, ;, <, =, >, @, and ` are not escaped. Only special characters with meaning in a regex are still escaped. _ is not escaped since Python 3.3.(s. here) Curly braces: If you want to use quantifiers within the regular expression using f-strings, you have to use double curly braces. Let's say you want to match TEXTO followed by exactly 2 digits: if re.search(rf"\b(?=\w){re.escape(TEXTO)}\d{{2}}\b(?!\w)", subject, re.IGNORECASE): print("match")


Instruction:
How to import module when module name has a '-' dash or hyphen in it?

I want to import foo-bar.py, this works: foobar = __import__("foo-bar") This does not: from "foo-bar" import * My question: Is there any way that I can use the above format i.e., from "foo-bar" import * to import a module that has a - in it?

Output: In Python 2, you can't. foo-bar is not an identifier. rename the file to foo_bar.py It's possible since Python 3.1+, see Julien's answer. If import is not your goal (as in: you don't care what happens with sys.modules, you don't need it to import itself), just getting all of the file's globals into your own scope, you can use execfile # contents of foo-bar.py baz = 'quux' >>> execfile('foo-bar.py') >>> baz 'quux' >>>


Instruction:
Can't compare naive and aware datetime.now() <= challenge.datetime_end

I am trying to compare the current date and time with dates and times specified in models using comparison operators: if challenge.datetime_start <= datetime.now() <= challenge.datetime_end: The script errors out with: TypeError: can't compare offset-naive and offset-aware datetimes The models look like this: class Fundraising_Challenge(models.Model): name = models.CharField(max_length=100) datetime_start = models.DateTimeField() datetime_end = models.DateTimeField() I also have django using locale date and times. What I haven't been able to find is the format django uses for DateTimeField(). Is it naive or aware? And how do I get datetime.now() to recognize locale datetime?

Output: By default, the datetime object is naive in Python, so you need to make both of them either naive or aware datetime objects. This can be done using: import datetime import pytz utc=pytz.UTC challenge.datetime_start = utc.localize(challenge.datetime_start) challenge.datetime_end = utc.localize(challenge.datetime_end) # now both the datetime objects are aware, and you can compare them Note: This would raise a ValueError if tzinfo is already set. If you are not sure about that, just use start_time = challenge.datetime_start.replace(tzinfo=utc) end_time = challenge.datetime_end.replace(tzinfo=utc) BTW, you could format a UNIX timestamp in datetime.datetime object with timezone info as following d = datetime.datetime.utcfromtimestamp(int(unix_timestamp)) d_with_tz = datetime.datetime( year=d.year, month=d.month, day=d.day, hour=d.hour, minute=d.minute, second=d.second, tzinfo=pytz.UTC)


Instruction:
Is there a difference between "pass" and "continue" in a for loop in Python?

Is there any significant difference between the two Python keywords continue and pass like in the examples for element in some_list: if not element: pass and for element in some_list: if not element: continue I should be aware of?

Output: Yes, they do completely different things. pass simply does nothing, while continue goes on with the next loop iteration. In your example, the difference would become apparent if you added another statement after the if: After executing pass, this further statement would be executed. After continue, it wouldn't. >>> a = [0, 1, 2] >>> for element in a: ... if not element: ... pass ... print(element) ... 0 1 2 >>> for element in a: ... if not element: ... continue ... print(element) ... 1 2


Instruction:
How to initialize a dict with keys from a list and empty value in Python?

I'd like to get from this: keys = [1,2,3] to this: {1: None, 2: None, 3: None} Is there a pythonic way of doing it? This is an ugly way to do it: >>> keys = [1,2,3] >>> dict([(1,2)]) {1: 2} >>> dict(zip(keys, [None]*len(keys))) {1: None, 2: None, 3: None}

Output: dict.fromkeys directly solves the problem: >>> dict.fromkeys([1, 2, 3, 4]) {1: None, 2: None, 3: None, 4: None} This is actually a classmethod, so it works for dict-subclasses (like collections.defaultdict) as well. The optional second argument, which defaults to None, specifies the value to use for the keys. Note that the same object will be used for each key, which can cause problems with mutable values: >>> x = dict.fromkeys([1, 2, 3, 4], []) >>> x[1].append('test') >>> x {1: ['test'], 2: ['test'], 3: ['test'], 4: ['test']} If this is unacceptable, see How can I initialize a dictionary whose values are distinct empty lists? for a workaround.


Instruction:
What do lambda function closures capture?

Recently I started playing around with Python and I came around something peculiar in the way closures work. Consider the following code: adders = [None, None, None, None] for i in [0, 1, 2, 3]: adders[i] = lambda a: i+a print adders[1](3) It builds a simple array of functions that take a single input and return that input added by a number. The functions are constructed in for loop where the iterator i runs from 0 to 3. For each of these numbers a lambda function is created which captures i and adds it to the function's input. The last line calls the second lambda function with 3 as a parameter. To my surprise the output was 6. I expected a 4. My reasoning was: in Python everything is an object and thus every variable is essential a pointer to it. When creating the lambda closures for i, I expected it to store a pointer to the integer object currently pointed to by i. That means that when i assigned a new integer object it shouldn't effect the previously created closures. Sadly, inspecting the adders array within a debugger shows that it does. All lambda functions refer to the last value of i, 3, which results in adders[1](3) returning 6. Which make me wonder about the following: What do the closures capture exactly? What is the most elegant way to convince the lambda functions to capture the current value of i in a way that will not be affected when i changes its value? For a more accessible, practical version of the question, specific to the case where a loop (or list comprehension, generator expression etc.) is used, see Creating functions (or lambdas) in a loop (or comprehension). This question is focused on understanding the underlying behaviour of the code in Python. If you got here trying to fix a problem with making buttons in Tkinter, try tkinter creating buttons in for loop passing command arguments for more specific advice. See What exactly is contained within a obj.__closure__? for technical details of how Python implements closures. See What is the difference between Early and Late Binding? for related terminology discussion.

Output: What do the closures capture exactly? Closures in Python use lexical scoping: they remember the name and scope of the closed-over variable where it is created. However, they are still late binding: the name is looked up when the code in the closure is used, not when the closure is created. Since all the functions in your example are created in the same scope and use the same variable name, they always refer to the same variable. There are at least two ways to get early binding instead: The most concise, but not strictly equivalent way is the one recommended by Adrien Plisson. Create a lambda with an extra argument, and set the extra argument's default value to the object you want preserved. More verbosely but also more robustly, we can create a new scope for each created lambda: >>> adders = [0,1,2,3] >>> for i in [0,1,2,3]: ... adders[i] = (lambda b: lambda a: b + a)(i) ... >>> adders[1](3) 4 >>> adders[2](3) 5 The scope here is created using a new function (another lambda, for brevity), which binds its argument, and passing the value you want to bind as the argument. In real code, though, you most likely will have an ordinary function instead of the lambda to create the new scope: def createAdder(x): return lambda y: y + x adders = [createAdder(i) for i in range(4)]


Instruction:
Test if executable exists in Python?

In Python, is there a portable and simple way to test if an executable program exists? By simple I mean something like the which command which would be just perfect. I don't want to search PATH manually or something involving trying to execute it with Popen & al and see if it fails (that's what I'm doing now, but imagine it's launchmissiles)

Output: Easiest way I can think of: def which(program): import os def is_exe(fpath): return os.path.isfile(fpath) and os.access(fpath, os.X_OK) fpath, fname = os.path.split(program) if fpath: if is_exe(program): return program else: for path in os.environ.get("PATH", "").split(os.pathsep): exe_file = os.path.join(path, program) if is_exe(exe_file): return exe_file return None Edit: Updated code sample to include logic for handling case where provided argument is already a full path to the executable, i.e. "which /bin/ls". This mimics the behavior of the UNIX 'which' command. Edit: Updated to use os.path.isfile() instead of os.path.exists() per comments. Edit: path.strip('"') seems like the wrong thing to do here. Neither Windows nor POSIX appear to encourage quoted PATH items.


Instruction:
Is there shorthand for returning a default value if None in Python?

In C#, I can say x ?? "", which will give me x if x is not null, and the empty string if x is null. I've found it useful for working with databases. Is there a way to return a default value if Python finds None in a variable?

Output: You could use the or operator: return x or "default" Note that this also returns "default" if x is any falsy value, including an empty list, 0, empty string, or even datetime.time(0) (midnight). Python 3.8+ update: bool(datetime.time(0)) now resolves to True. This was resolved with issue13936. Other "empty" values are still considered "falsy" as expected.


Instruction:
Why do we need to call zero_grad() in PyTorch?

Why does zero_grad() need to be called during training? | zero_grad(self) | Sets gradients of all model parameters to zero.

Output: In PyTorch, for every mini-batch during the training phase, we typically want to explicitly set the gradients to zero before starting to do backpropagation (i.e., updating the Weights and biases) because PyTorch accumulates the gradients on subsequent backward passes. This accumulating behavior is convenient while training RNNs or when we want to compute the gradient of the loss summed over multiple mini-batches. So, the default action has been set to accumulate (i.e. sum) the gradients on every loss.backward() call. Because of this, when you start your training loop, ideally you should zero out the gradients so that you do the parameter update correctly. Otherwise, the gradient would be a combination of the old gradient, which you have already used to update your model parameters and the newly-computed gradient. It would therefore point in some other direction than the intended direction towards the minimum (or maximum, in case of maximization objectives). Here is a simple example: import torch from torch.autograd import Variable import torch.optim as optim def linear_model(x, W, b): return torch.matmul(x, W) + b data, targets = ... W = Variable(torch.randn(4, 3), requires_grad=True) b = Variable(torch.randn(3), requires_grad=True) optimizer = optim.Adam([W, b]) for sample, target in zip(data, targets): # clear out the gradients of all Variables # in this optimizer (i.e. W, b) optimizer.zero_grad() output = linear_model(sample, W, b) loss = (output - target) ** 2 loss.backward() optimizer.step() Alternatively, if you're doing a vanilla gradient descent, then: W = Variable(torch.randn(4, 3), requires_grad=True) b = Variable(torch.randn(3), requires_grad=True) for sample, target in zip(data, targets): # clear out the gradients of Variables # (i.e. W, b) W.grad.data.zero_() b.grad.data.zero_() output = linear_model(sample, W, b) loss = (output - target) ** 2 loss.backward() W -= learning_rate * W.grad.data b -= learning_rate * b.grad.data Note: The accumulation (i.e., sum) of gradients happens when .backward() is called on the loss tensor. As of v1.7.0, Pytorch offers the option to reset the gradients to None optimizer.zero_grad(set_to_none=True) instead of filling them with a tensor of zeroes. The docs claim that this setting reduces memory requirements and slightly improves performance, but might be error-prone if not handled carefully.


Instruction:
How do I determine what type of exception occurred?

some_function() raises an exception while executing, so the program jumps to the except: try: some_function() except: print("exception happened!") How do I see what caused the exception to occur?

Output: The other answers all point out that you should not catch generic exceptions, but no one seems to want to tell you why, which is essential to understanding when you can break the "rule". Here is an explanation. Basically, it's so that you don't hide: the fact that an error occurred the specifics of the error that occurred (error hiding antipattern) So as long as you take care to do none of those things, it's OK to catch the generic exception. For instance, you could provide information about the exception to the user another way, like: Present exceptions as dialogs in a GUI Transfer exceptions from a worker thread or process to the controlling thread or process in a multithreading or multiprocessing application So how to catch the generic exception? There are several ways. If you just want the exception object, do it like this: try: someFunction() except Exception as ex: template = "An exception of type {0} occurred. Arguments:\n{1!r}" message = template.format(type(ex).__name__, ex.args) print(message) Make sure message is brought to the attention of the user in a hard-to-miss way! Printing it, as shown above, may not be enough if the message is buried in lots of other messages. Failing to get the users attention is tantamount to swallowing all exceptions, and if there's one impression you should have come away with after reading the answers on this page, it's that this is not a good thing. Ending the except block with a raise statement will remedy the problem by transparently reraising the exception that was caught. The difference between the above and using just except: without any argument is twofold: A bare except: doesn't give you the exception object to inspect The exceptions SystemExit, KeyboardInterrupt and GeneratorExit aren't caught by the above code, which is generally what you want. See the exception hierarchy. If you also want the same stacktrace you get if you do not catch the exception, you can get that like this (still inside the except clause): import traceback print traceback.format_exc() If you use the logging module, you can print the exception to the log (along with a message) like this: import logging log = logging.getLogger() log.exception("Message for you, sir!") If you want to dig deeper and examine the stack, look at variables etc., use the post_mortem function of the pdb module inside the except block: import pdb pdb.post_mortem() I've found this last method to be invaluable when hunting down bugs.


Instruction:
Python requests - print entire http request (raw)?

While using the requests module, is there any way to print the raw HTTP request? I don't want just the headers, I want the request line, headers, and content printout. Is it possible to see what ultimately is constructed from HTTP request?

Output: Since v1.2.3 Requests added the PreparedRequest object. As per the documentation "it contains the exact bytes that will be sent to the server". One can use this to pretty print a request, like so: import requests req = requests.Request('POST','http://stackoverflow.com',headers={'X-Custom':'Test'},data='a=1&b=2') prepared = req.prepare() def pretty_print_POST(req): """ At this point it is completely built and ready to be fired; it is "prepared". However pay attention at the formatting used in this function because it is programmed to be pretty printed and may differ from the actual request. """ print('{}\n{}\r\n{}\r\n\r\n{}'.format( '-----------START-----------', req.method + ' ' + req.url, '\r\n'.join('{}: {}'.format(k, v) for k, v in req.headers.items()), req.body, )) pretty_print_POST(prepared) which produces: -----------START----------- POST http://stackoverflow.com/ Content-Length: 7 X-Custom: Test a=1&b=2 Then you can send the actual request with this: s = requests.Session() s.send(prepared) These links are to the latest documentation available, so they might change in content: Advanced - Prepared requests and API - Lower level classes


Instruction:
Timeout for Python requests.get() entire response

I'm gathering statistics on a list of websites, and I'm using Requests for it for simplicity. Here is my code: data = [] websites = ['http://google.com', 'http://bbc.co.uk'] for w in websites: r = requests.get(w, verify=False) data.append((r.url, len(r.content), r.elapsed.total_seconds(), str([(l.status_code, l.url) for l in r.history]), str(r.headers.items()), str(r.cookies.items()))) Now, I want requests.get to timeout after 10 seconds, so the loop doesn't get stuck. This question has been of interest before too, but none of the answers were clean. I hear that maybe not using Requests is a good idea, but then how should I get the nice things Requests offer (the ones in the tuple)?

Output: What about using 'eventlet'? If you want to timeout the request after 10 seconds, even if data is being received, this snippet will work for you: import requests import eventlet eventlet.monkey_patch() with eventlet.Timeout(10): requests.get("http://ipv4.download.thinkbroadband.com/1GB.zip", verify=False)


Instruction:
How to hide output of subprocess

I'm using eSpeak on Ubuntu and have a Python 2.7 script that prints and speaks a message: import subprocess text = 'Hello World.' print text subprocess.call(['espeak', text]) eSpeak produces the desired sounds, but clutters the shell with some errors (ALSA lib..., no socket connect) so i cannot easily read what was printed earlier. Exit code is 0. Unfortunately there is no documented option to turn off its verbosity, so I'm looking for a way to only visually silence it and keep the open shell clean for further interaction. How can I do this? See Python os.system without the output for approaches specific to os.system - although modern code should normally use the subprocess library instead.

Output: For python >= 3.3, Redirect the output to DEVNULL: import os import subprocess retcode = subprocess.call(['echo', 'foo'], stdout=subprocess.DEVNULL, stderr=subprocess.STDOUT) For python <3.3, including 2.7 use: FNULL = open(os.devnull, 'w') retcode = subprocess.call(['echo', 'foo'], stdout=FNULL, stderr=subprocess.STDOUT) It is effectively the same as running this shell command: retcode = os.system("echo 'foo' &> /dev/null")


Instruction:
What's the bad magic number error?

What's the "Bad magic number" ImportError in python, and how do I fix it? The only thing I can find online suggests this is caused by compiling a .py -> .pyc file and then trying to use it with the wrong version of python. In my case, however, the file seems to import fine some times but not others, and I'm not sure why. The information python's providing in the traceback isn't particularly helpful (which is why I was asking here...), but here it is in case it helps: Traceback (most recent call last): File "run.py", line 7, in <module> from Normalization import Normalizer

Output: The magic number comes from UNIX-type systems where the first few bytes of a file held a marker indicating the file type. Python puts a similar marker into its pyc files when it creates them. Then the python interpreter makes sure this number is correct when loading it. Anything that damages this magic number will cause your problem. This includes editing the pyc file or trying to run a pyc from a different version of python (usually later) than your interpreter. If they are your pyc files (or you have the py files for them), just delete them and let the interpreter re-compile the py files. On UNIX type systems, that could be something as simple as: rm *.pyc or: find . -name '*.pyc' -delete If they are not yours, and the original py files are not provided, you'll have to either get the py files for re-compilation, or use an interpreter that can run the pyc files with that particular magic value. One thing that might be causing the intermittent nature. The pyc that's causing the problem may only be imported under certain conditions. It's highly unlikely it would import sometimes. You should check the actual full stack trace when the import fails. As an aside, the first word of all my 2.5.1(r251:54863) pyc files is 62131, 2.6.1(r261:67517) is 62161. The list of all magic numbers can be found in Python/import.c, reproduced here for completeness (current as at the time the answer was posted, has changed since then): 1.5: 20121 1.5.1: 20121 1.5.2: 20121 1.6: 50428 2.0: 50823 2.0.1: 50823 2.1: 60202 2.1.1: 60202 2.1.2: 60202 2.2: 60717 2.3a0: 62011 2.3a0: 62021 2.3a0: 62011 2.4a0: 62041 2.4a3: 62051 2.4b1: 62061 2.5a0: 62071 2.5a0: 62081 2.5a0: 62091 2.5a0: 62092 2.5b3: 62101 2.5b3: 62111 2.5c1: 62121 2.5c2: 62131 2.6a0: 62151 2.6a1: 62161 2.7a0: 62171


Instruction:
How to escape curly-brackets in f-strings?

I have a string in which I would like curly-brackets, but also take advantage of the f-strings feature. Is there some syntax that works for this? Here are two ways it does not work. I would like to include the literal text {bar} as part of the string. foo = "test" fstring = f"{foo} {bar}" NameError: name 'bar' is not defined fstring = f"{foo} \{bar\}" SyntaxError: f-string expression part cannot include a backslash Desired result: 'test {bar}' Edit: Looks like this question has the same answer as How can I print literal curly-brace characters in a string and also use .format on it?, but you can only know that if you know that str.format uses the same rules as the f-string. So hopefully this question has value in tying f-string searchers to this answer.

Output: Although there is a custom syntax error from the parser, the same trick works as for calling .format on regular strings. Use double curlies: >>> foo = 'test' >>> f'{foo} {{bar}}' 'test {bar}' To embed a value within braces, you can use triple-braces. >>> foo = 'test' >>> f'{{{foo}}}' '{test}' It's mentioned in the spec here and the docs here.


Instruction:
Comparing two dictionaries and checking how many (key, value) pairs are equal

I have two dictionaries, but for simplification, I will take these two: >>> x = dict(a=1, b=2) >>> y = dict(a=2, b=2) Now, I want to compare whether each key, value pair in x has the same corresponding value in y. So I wrote this: >>> for x_values, y_values in zip(x.iteritems(), y.iteritems()): if x_values == y_values: print 'Ok', x_values, y_values else: print 'Not', x_values, y_values And it works since a tuple is returned and then compared for equality. My questions: Is this correct? Is there a better way to do this? Better not in speed, I am talking about code elegance. UPDATE: I forgot to mention that I have to check how many key, value pairs are equal.

Output: If you want to know how many values match in both the dictionaries, you should have said that :) Maybe something like this: shared_items = {k: x[k] for k in x if k in y and x[k] == y[k]} print(len(shared_items))


Instruction:
How can I use list comprehensions to process a nested list?

I have this nested list: l = [['40', '20', '10', '30'], ['20', '20', '20', '20', '20', '30', '20'], ['30', '20', '30', '50', '10', '30', '20', '20', '20'], ['100', '100'], ['100', '100', '100', '100', '100'], ['100', '100', '100', '100']] I want to convert each element in l to float. I have this code: newList = [] for x in l: for y in x: newList.append(float(y)) How can I solve the problem with a nested list comprehension instead? See also: How can I get a flat result from a list comprehension instead of a nested list?

Output: Here is how you would do this with a nested list comprehension: [[float(y) for y in x] for x in l] This would give you a list of lists, similar to what you started with except with floats instead of strings. If you want one flat list, then you would use [float(y) for x in l for y in x] Note the loop order - for x in l comes first in this one.


Instruction:
Saving an Object (Data persistence)

I've created an object like this: company1.name = 'banana' company1.value = 40 I would like to save this object. How can I do that?

Output: You could use the pickle module in the standard library. Here's an elementary application of it to your example: import pickle class Company(object): def __init__(self, name, value): self.name = name self.value = value with open('company_data.pkl', 'wb') as outp: company1 = Company('banana', 40) pickle.dump(company1, outp, pickle.HIGHEST_PROTOCOL) company2 = Company('spam', 42) pickle.dump(company2, outp, pickle.HIGHEST_PROTOCOL) del company1 del company2 with open('company_data.pkl', 'rb') as inp: company1 = pickle.load(inp) print(company1.name) # -> banana print(company1.value) # -> 40 company2 = pickle.load(inp) print(company2.name) # -> spam print(company2.value) # -> 42 You could also define your own simple utility like the following which opens a file and writes a single object to it: def save_object(obj, filename): with open(filename, 'wb') as outp: # Overwrites any existing file. pickle.dump(obj, outp, pickle.HIGHEST_PROTOCOL) # sample usage save_object(company1, 'company1.pkl') Update Since this is such a popular answer, I'd like touch on a few slightly advanced usage topics. cPickle (or _pickle) vs pickle It's almost always preferable to actually use the cPickle module rather than pickle because the former is written in C and is much faster. There are some subtle differences between them, but in most situations they're equivalent and the C version will provide greatly superior performance. Switching to it couldn't be easier, just change the import statement to this: import cPickle as pickle In Python 3, cPickle was renamed _pickle, but doing this is no longer necessary since the pickle module now does it automatically—see What difference between pickle and _pickle in python 3?. The rundown is you could use something like the following to ensure that your code will always use the C version when it's available in both Python 2 and 3: try: import cPickle as pickle except ModuleNotFoundError: import pickle Data stream formats (protocols) pickle can read and write files in several different, Python-specific, formats, called protocols as described in the documentation, "Protocol version 0" is ASCII and therefore "human-readable". Versions > 0 are binary and the highest one available depends on what version of Python is being used. The default also depends on Python version. In Python 2 the default was Protocol version 0, but in Python 3.8.1, it's Protocol version 4. In Python 3.x the module had a pickle.DEFAULT_PROTOCOL added to it, but that doesn't exist in Python 2. Fortunately there's shorthand for writing pickle.HIGHEST_PROTOCOL in every call (assuming that's what you want, and you usually do), just use the literal number -1 — similar to referencing the last element of a sequence via a negative index. So, instead of writing: pickle.dump(obj, outp, pickle.HIGHEST_PROTOCOL) You can just write: pickle.dump(obj, outp, -1) Either way, you'd only have specify the protocol once if you created a Pickler object for use in multiple pickle operations: pickler = pickle.Pickler(outp, -1) pickler.dump(obj1) pickler.dump(obj2) etc... Note: If you're in an environment running different versions of Python, then you'll probably want to explicitly use (i.e. hardcode) a specific protocol number that all of them can read (later versions can generally read files produced by earlier ones). Multiple Objects While a pickle file can contain any number of pickled objects, as shown in the above samples, when there's an unknown number of them, it's often easier to store them all in some sort of variably-sized container, like a list, tuple, or dict and write them all to the file in a single call: tech_companies = [ Company('Apple', 114.18), Company('Google', 908.60), Company('Microsoft', 69.18) ] save_object(tech_companies, 'tech_companies.pkl') and restore the list and everything in it later with: with open('tech_companies.pkl', 'rb') as inp: tech_companies = pickle.load(inp) The major advantage is you don't need to know how many object instances are saved in order to load them back later (although doing so without that information is possible, it requires some slightly specialized code). See the answers to the related question Saving and loading multiple objects in pickle file? for details on different ways to do this. Personally I liked @Lutz Prechelt's answer the best, so that's the approach used in the sample code below: class Company: def __init__(self, name, value): self.name = name self.value = value def pickle_loader(filename): """ Deserialize a file of pickled objects. """ with open(filename, "rb") as f: while True: try: yield pickle.load(f) except EOFError: break print('Companies in pickle file:') for company in pickle_loader('company_data.pkl'): print(' name: {}, value: {}'.format(company.name, company.value))


Instruction:
How to install Python package from GitHub?

I want to use a new feature of httpie. This feature is in the github repo https://github.com/jkbr/httpie but not in the release on the python package index https://pypi.python.org/pypi/httpie How can I install the httpie package from the github repo? I tried pip install https://github.com/jkbr/httpie But I got an error 'could not unpack' In Nodejs, I can install packages from github like this npm install git+https://github.com/substack/node-optimist.git

Output: You need to use the proper git URL: pip install git+https://github.com/jkbr/httpie.git#egg=httpie Also see the VCS Support section of the pip documentation. Don’t forget to include the egg=<projectname> part to explicitly name the project; this way pip can track metadata for it without having to have run the setup.py script.


Instruction:
pandas: merge (join) two data frames on multiple columns

I am trying to join two pandas dataframes using two columns: new_df = pd.merge(A_df, B_df, how='left', left_on='[A_c1,c2]', right_on = '[B_c1,c2]') but got the following error: pandas/index.pyx in pandas.index.IndexEngine.get_loc (pandas/index.c:4164)() pandas/index.pyx in pandas.index.IndexEngine.get_loc (pandas/index.c:4028)() pandas/src/hashtable_class_helper.pxi in pandas.hashtable.PyObjectHashTable.get_item (pandas/hashtable.c:13166)() pandas/src/hashtable_class_helper.pxi in pandas.hashtable.PyObjectHashTable.get_item (pandas/hashtable.c:13120)() KeyError: '[B_1, c2]' Any idea what should be the right way to do this?

Output: Try this new_df = pd.merge( left=A_df, right=B_df, how='left', left_on=['A_c1', 'c2'], right_on=['B_c1', 'c2'], ) https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.merge.html left_on : label or list, or array-like Field names to join on in left DataFrame. Can be a vector or list of vectors of the length of the DataFrame to use a particular vector as the join key instead of columns right_on : label or list, or array-like Field names to join on in right DataFrame or vector/list of vectors per left_on docs


Instruction:
Understanding NumPy's einsum

How does np.einsum work? Given arrays A and B, their matrix multiplication followed by transpose is computed using (A @ B).T, or equivalently, using: np.einsum("ij, jk -> ki", A, B)

Output: (Note: this answer is based on a short blog post about einsum I wrote a while ago.) What does einsum do? Imagine that we have two multi-dimensional arrays, A and B. Now let's suppose we want to... multiply A with B in a particular way to create new array of products; and then maybe sum this new array along particular axes; and then maybe transpose the axes of the new array in a particular order. There's a good chance that einsum will help us do this faster and more memory-efficiently than combinations of the NumPy functions like multiply, sum and transpose will allow. How does einsum work? Here's a simple (but not completely trivial) example. Take the following two arrays: A = np.array([0, 1, 2]) B = np.array([[ 0, 1, 2, 3], [ 4, 5, 6, 7], [ 8, 9, 10, 11]]) We will multiply A and B element-wise and then sum along the rows of the new array. In "normal" NumPy we'd write: >>> (A[:, np.newaxis] * B).sum(axis=1) array([ 0, 22, 76]) So here, the indexing operation on A lines up the first axes of the two arrays so that the multiplication can be broadcast. The rows of the array of products are then summed to return the answer. Now if we wanted to use einsum instead, we could write: >>> np.einsum('i,ij->i', A, B) array([ 0, 22, 76]) The signature string 'i,ij->i' is the key here and needs a little bit of explaining. You can think of it in two halves. On the left-hand side (left of the ->) we've labelled the two input arrays. To the right of ->, we've labelled the array we want to end up with. Here is what happens next: A has one axis; we've labelled it i. And B has two axes; we've labelled axis 0 as i and axis 1 as j. By repeating the label i in both input arrays, we are telling einsum that these two axes should be multiplied together. In other words, we're multiplying array A with each column of array B, just like A[:, np.newaxis] * B does. Notice that j does not appear as a label in our desired output; we've just used i (we want to end up with a 1D array). By omitting the label, we're telling einsum to sum along this axis. In other words, we're summing the rows of the products, just like .sum(axis=1) does. That's basically all you need to know to use einsum. It helps to play about a little; if we leave both labels in the output, 'i,ij->ij', we get back a 2D array of products (same as A[:, np.newaxis] * B). If we say no output labels, 'i,ij->, we get back a single number (same as doing (A[:, np.newaxis] * B).sum()). The great thing about einsum however, is that it does not build a temporary array of products first; it just sums the products as it goes. This can lead to big savings in memory use. A slightly bigger example To explain the dot product, here are two new arrays: A = array([[1, 1, 1], [2, 2, 2], [5, 5, 5]]) B = array([[0, 1, 0], [1, 1, 0], [1, 1, 1]]) We will compute the dot product using np.einsum('ij,jk->ik', A, B). Here's a picture showing the labelling of the A and B and the output array that we get from the function: You can see that label j is repeated - this means we're multiplying the rows of A with the columns of B. Furthermore, the label j is not included in the output - we're summing these products. Labels i and k are kept for the output, so we get back a 2D array. It might be even clearer to compare this result with the array where the label j is not summed. Below, on the left you can see the 3D array that results from writing np.einsum('ij,jk->ijk', A, B) (i.e. we've kept label j): Summing axis j gives the expected dot product, shown on the right. Some exercises To get more of a feel for einsum, it can be useful to implement familiar NumPy array operations using the subscript notation. Anything that involves combinations of multiplying and summing axes can be written using einsum. Let A and B be two 1D arrays with the same length. For example, A = np.arange(10) and B = np.arange(5, 15). The sum of A can be written: np.einsum('i->', A) Element-wise multiplication, A * B, can be written: np.einsum('i,i->i', A, B) The inner product or dot product, np.inner(A, B) or np.dot(A, B), can be written: np.einsum('i,i->', A, B) # or just use 'i,i' The outer product, np.outer(A, B), can be written: np.einsum('i,j->ij', A, B) For 2D arrays, C and D, provided that the axes are compatible lengths (both the same length or one of them of has length 1), here are a few examples: The trace of C (sum of main diagonal), np.trace(C), can be written: np.einsum('ii', C) Element-wise multiplication of C and the transpose of D, C * D.T, can be written: np.einsum('ij,ji->ij', C, D) Multiplying each element of C by the array D (to make a 4D array), C[:, :, None, None] * D, can be written: np.einsum('ij,kl->ijkl', C, D)


Instruction:
Most efficient way to reverse a numpy array

Believe it or not, after profiling my current code, the repetitive operation of numpy array reversion ate a giant chunk of the running time. What I have right now is the common view-based method: reversed_arr = arr[::-1] Is there any other way to do it more efficiently, or is it just an illusion from my obsession with unrealistic numpy performance?

Output: reversed_arr = arr[::-1] gives a reversed view into the original array arr. Any changes made to the original array arr will also be immediately visible in reversed_arr. The underlying data buffers for arr and reversed_arr are shared, so creating this view is always instantaneous, and does not require any additional memory allocation or copying for the array contents. See also, this discussion on NumPy views: How do I create a view onto a NumPy array? Possible solutions to performance problems regarding views Are you re-creating the view more often than you need to? You should be able to do something like this: arr = np.array(some_sequence) reversed_arr = arr[::-1] do_something(arr) look_at(reversed_arr) do_something_else(arr) look_at(reversed_arr) I'm not a numpy expert, but this seems like it would be the fastest way to do things in numpy. If this is what you are already doing, I don't think you can improve on it.


Instruction:
How can I get the return value of a function passed to multiprocessing.Process?

In the example code below, I'd like to get the return value of the function worker. How can I go about doing this? Where is this value stored? Example Code: import multiprocessing def worker(procnum): '''worker function''' print str(procnum) + ' represent!' return procnum if __name__ == '__main__': jobs = [] for i in range(5): p = multiprocessing.Process(target=worker, args=(i,)) jobs.append(p) p.start() for proc in jobs: proc.join() print jobs Output: 0 represent! 1 represent! 2 represent! 3 represent! 4 represent! [<Process(Process-1, stopped)>, <Process(Process-2, stopped)>, <Process(Process-3, stopped)>, <Process(Process-4, stopped)>, <Process(Process-5, stopped)>] I can't seem to find the relevant attribute in the objects stored in jobs.

Output: Use a shared variable to communicate. For example, like this, Example Code: import multiprocessing def worker(procnum, return_dict): """worker function""" print(str(procnum) + " represent!") return_dict[procnum] = procnum if __name__ == "__main__": manager = multiprocessing.Manager() return_dict = manager.dict() jobs = [] for i in range(5): p = multiprocessing.Process(target=worker, args=(i, return_dict)) jobs.append(p) p.start() for proc in jobs: proc.join() print(return_dict.values()) Output: 0 represent! 1 represent! 3 represent! 2 represent! 4 represent! [0, 1, 3, 2, 4]


Instruction:
In Python, how do I split a string and keep the separators?

Here's the simplest way to explain this. Here's what I'm using: re.split('\W', 'foo/bar spam\neggs') >>> ['foo', 'bar', 'spam', 'eggs'] Here's what I want: someMethod('\W', 'foo/bar spam\neggs') >>> ['foo', '/', 'bar', ' ', 'spam', '\n', 'eggs'] The reason is that I want to split a string into tokens, manipulate it, then put it back together again.

Output: The docs of re.split mention: Split string by the occurrences of pattern. If capturing parentheses are used in pattern, then the text of all groups in the pattern are also returned as part of the resulting list. So you just need to wrap your separator with a capturing group: >>> re.split('(\W)', 'foo/bar spam\neggs') ['foo', '/', 'bar', ' ', 'spam', '\n', 'eggs']


Instruction:
Python truncate a long string

How does one truncate a string to 75 characters in Python? This is how it is done in JavaScript: const data = "saddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddsaddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddsadddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddd"; const info = (data.length > 75) ? data.substring(0, 75) + '..' : data; console.log(info);

Output: info = (data[:75] + '..') if len(data) > 75 else data This code matches the JavaScript, but you should consider using data[:73] so that the total result including the .. fits in 75 characters.


Instruction:
Can I set max_retries for requests.request?

The Python requests module is simple and elegant but one thing bugs me. It is possible to get a requests.exception.ConnectionError with a message like: Max retries exceeded with url: ... This implies that requests can attempt to access the data several times. But there is not a single mention of this possibility anywhere in the docs. Looking at the source code I didn't find any place where I could alter the default (presumably 0) value. So is it possible to somehow set the maximum number of retries for requests?

Output: It is the underlying urllib3 library that does the retrying. To set a different maximum retry count, use alternative transport adapters: from requests.adapters import HTTPAdapter s = requests.Session() s.mount('http://stackoverflow.com', HTTPAdapter(max_retries=5)) The max_retries argument takes an integer or a Retry() object; the latter gives you fine-grained control over what kinds of failures are retried (an integer value is turned into a Retry() instance which only handles connection failures; errors after a connection is made are by default not handled as these could lead to side-effects). Old answer, predating the release of requests 1.2.1: The requests library doesn't really make this configurable, nor does it intend to (see this pull request). Currently (requests 1.1), the retries count is set to 0. If you really want to set it to a higher value, you'll have to set this globally: import requests requests.adapters.DEFAULT_RETRIES = 5 This constant is not documented; use it at your own peril as future releases could change how this is handled. Update: and this did change; in version 1.2.1 the option to set the max_retries parameter on the HTTPAdapter() class was added, so that now you have to use alternative transport adapters, see above. The monkey-patch approach no longer works, unless you also patch the HTTPAdapter.__init__() defaults (very much not recommended).


Instruction:
How does functools partial do what it does?

I am not able to get my head on how the partial works in functools. I have the following code from here: >>> sum = lambda x, y : x + y >>> sum(1, 2) 3 >>> incr = lambda y : sum(1, y) >>> incr(2) 3 >>> def sum2(x, y): return x + y >>> incr2 = functools.partial(sum2, 1) >>> incr2(4) 5 Now in the line incr = lambda y : sum(1, y) I get that whatever argument I pass to incr it will be passed as y to lambda which will return sum(1, y) i.e 1 + y. I understand that. But I didn't understand this incr2(4). How does the 4 gets passed as x in partial function? To me, 4 should replace the sum2. What is the relation between x and 4?

Output: Roughly, partial does something like this (apart from keyword args support, etc): def partial(func, *part_args): def wrapper(*extra_args): return func(*part_args, *extra_args) return wrapper So, by calling partial(sum2, 4) you create a new function (a callable, to be precise) that behaves like sum2, but has one positional argument less. That missing argument is always substituted by 4, so that partial(sum2, 4)(2) == sum2(4, 2) As for why it's needed, there's a variety of cases. Just for one, suppose you have to pass a function somewhere where it's expected to have 2 arguments: class EventNotifier(object): def __init__(self): self._listeners = [] def add_listener(self, callback): ''' callback should accept two positional arguments, event and params ''' self._listeners.append(callback) # ... def notify(self, event, *params): for f in self._listeners: f(event, params) But a function you already have needs access to some third context object to do its job: def log_event(context, event, params): context.log_event("Something happened %s, %s", event, params) So, there are several solutions: A custom object: class Listener(object): def __init__(self, context): self._context = context def __call__(self, event, params): self._context.log_event("Something happened %s, %s", event, params) notifier.add_listener(Listener(context)) Lambda: log_listener = lambda event, params: log_event(context, event, params) notifier.add_listener(log_listener) With partials: context = get_context() # whatever notifier.add_listener(partial(log_event, context)) Of those three, partial is the shortest and the fastest. (For a more complex case you might want a custom object though).


Instruction:
Selecting with complex criteria from pandas.DataFrame

For example I have simple DF: import pandas as pd from random import randint df = pd.DataFrame({'A': [randint(1, 9) for x in range(10)], 'B': [randint(1, 9)*10 for x in range(10)], 'C': [randint(1, 9)*100 for x in range(10)]}) Can I select values from 'A' for which corresponding values for 'B' will be greater than 50, and for 'C' - not equal to 900, using methods and idioms of Pandas?

Output: Sure! Setup: >>> import pandas as pd >>> from random import randint >>> df = pd.DataFrame({'A': [randint(1, 9) for x in range(10)], 'B': [randint(1, 9)*10 for x in range(10)], 'C': [randint(1, 9)*100 for x in range(10)]}) >>> df A B C 0 9 40 300 1 9 70 700 2 5 70 900 3 8 80 900 4 7 50 200 5 9 30 900 6 2 80 700 7 2 80 400 8 5 80 300 9 7 70 800 We can apply column operations and get boolean Series objects: >>> df["B"] > 50 0 False 1 True 2 True 3 True 4 False 5 False 6 True 7 True 8 True 9 True Name: B >>> (df["B"] > 50) & (df["C"] != 900) or >>> (df["B"] > 50) & ~(df["C"] == 900) 0 False 1 False 2 True 3 True 4 False 5 False 6 False 7 False 8 False 9 False [Update, to switch to new-style .loc]: And then we can use these to index into the object. For read access, you can chain indices: >>> df["A"][(df["B"] > 50) & (df["C"] != 900)] 2 5 3 8 Name: A, dtype: int64 but you can get yourself into trouble because of the difference between a view and a copy doing this for write access. You can use .loc instead: >>> df.loc[(df["B"] > 50) & (df["C"] != 900), "A"] 2 5 3 8 Name: A, dtype: int64 >>> df.loc[(df["B"] > 50) & (df["C"] != 900), "A"].values array([5, 8], dtype=int64) >>> df.loc[(df["B"] > 50) & (df["C"] != 900), "A"] *= 1000 >>> df A B C 0 9 40 300 1 9 70 700 2 5000 70 900 3 8000 80 900 4 7 50 200 5 9 30 900 6 2 80 700 7 2 80 400 8 5 80 300 9 7 70 800


Instruction:
Redirecting to URL in Flask

I'm trying to do the equivalent of Response.redirect as in C# - i.e.: redirect to a specific URL - how do I go about this? Here is my code: import os from flask import Flask app = Flask(__name__) @app.route('/') def hello(): return 'Hello World!' if __name__ == '__main__': # Bind to PORT if defined, otherwise default to 5000. port = int(os.environ.get('PORT', 5000)) app.run(host='0.0.0.0', port=port)

Output: You have to return a redirect: import os from flask import Flask,redirect app = Flask(__name__) @app.route('/') def hello(): return redirect("http://www.example.com", code=302) if __name__ == '__main__': # Bind to PORT if defined, otherwise default to 5000. port = int(os.environ.get('PORT', 5000)) app.run(host='0.0.0.0', port=port) The default value for code is 302 so code=302 can be omitted or replaced by other redirect code (one in 301, 302, 303, 305, and 307). See the documentation on flask docs.


Instruction:
why should I make a copy of a data frame in pandas

When selecting a sub dataframe from a parent dataframe, I noticed that some programmers make a copy of the data frame using the .copy() method. For example, X = my_dataframe[features_list].copy() ...instead of just X = my_dataframe[features_list] Why are they making a copy of the data frame? What will happen if I don't make a copy?

Output: This answer has been deprecated in newer versions of pandas. See docs This expands on Paul's answer. In Pandas, indexing a DataFrame returns a reference to the initial DataFrame. Thus, changing the subset will change the initial DataFrame. Thus, you'd want to use the copy if you want to make sure the initial DataFrame shouldn't change. Consider the following code: df = DataFrame({'x': [1,2]}) df_sub = df[0:1] df_sub.x = -1 print(df) You'll get: x 0 -1 1 2 In contrast, the following leaves df unchanged: df_sub_copy = df[0:1].copy() df_sub_copy.x = -1


Instruction:
Get a human-readable version of a file size

A function to return a human-readable size from the bytes size: >>> human_readable(2048) '2 kilobytes' How can I do this?

Output: Addressing the above "too small a task to require a library" issue by a straightforward implementation (using f-strings, so Python 3.6+): def sizeof_fmt(num, suffix="B"): for unit in ("", "Ki", "Mi", "Gi", "Ti", "Pi", "Ei", "Zi"): if abs(num) < 1024.0: return f"{num:3.1f}{unit}{suffix}" num /= 1024.0 return f"{num:.1f}Yi{suffix}" Supports: all currently known binary prefixes negative and positive numbers numbers larger than 1000 Yobibytes arbitrary units (maybe you like to count in Gibibits!) Example: >>> sizeof_fmt(168963795964) '157.4GiB' by Fred Cirera


Instruction:
Django auto_now and auto_now_add

For Django 1.1. I have this in my models.py: class User(models.Model): created = models.DateTimeField(auto_now_add=True) modified = models.DateTimeField(auto_now=True) When updating a row I get: [Sun Nov 15 02:18:12 2009] [error] /home/ptarjan/projects/twitter-meme/django/db/backends/mysql/base.py:84: Warning: Column 'created' cannot be null [Sun Nov 15 02:18:12 2009] [error] return self.cursor.execute(query, args) The relevant part of my database is: `created` datetime NOT NULL, `modified` datetime NOT NULL, Is this cause for concern? Side question: in my admin tool, those two fields aren't showing up. Is that expected?

Output: Any field with the auto_now attribute set will also inherit editable=False and therefore will not show up in the admin panel. There has been talk in the past about making the auto_now and auto_now_add arguments go away, and although they still exist, I feel you're better off just using a custom save() method. So, to make this work properly, I would recommend not using auto_now or auto_now_add and instead define your own save() method to make sure that created is only updated if id is not set (such as when the item is first created), and have it update modified every time the item is saved. I have done the exact same thing with other projects I have written using Django, and so your save() would look like this: from django.utils import timezone class User(models.Model): created = models.DateTimeField(editable=False) modified = models.DateTimeField() def save(self, *args, **kwargs): ''' On save, update timestamps ''' if not self.id: self.created = timezone.now() self.modified = timezone.now() return super(User, self).save(*args, **kwargs) Edit in response to comments: The reason why I just stick with overloading save() vs. relying on these field arguments is two-fold: The aforementioned ups and downs with their reliability. These arguments are heavily reliant on the way each type of database that Django knows how to interact with treats a date/time stamp field, and seems to break and/or change between every release. (Which I believe is the impetus behind the call to have them removed altogether). The fact that they only work on DateField, DateTimeField, and TimeField, and by using this technique you are able to automatically populate any field type every time an item is saved. Use django.utils.timezone.now() vs. datetime.datetime.now(), because it will return a TZ-aware or naive datetime.datetime object depending on settings.USE_TZ. To address why the OP saw the error, I don't know exactly, but it looks like created isn't even being populated at all, despite having auto_now_add=True. To me it stands out as a bug, and underscores item #1 in my little list above: auto_now and auto_now_add are flaky at best.


Instruction:
When should I be using classes in Python?

I have been programming in python for about two years; mostly data stuff (pandas, mpl, numpy), but also automation scripts and small web apps. I'm trying to become a better programmer and increase my python knowledge and one of the things that bothers me is that I have never used a class (outside of copying random flask code for small web apps). I generally understand what they are, but I can't seem to wrap my head around why I would need them over a simple function. To add specificity to my question: I write tons of automated reports which always involve pulling data from multiple data sources (mongo, sql, postgres, apis), performing a lot or a little data munging and formatting, writing the data to csv/excel/html, send it out in an email. The scripts range from ~250 lines to ~600 lines. Would there be any reason for me to use classes to do this and why?

Output: Classes are the pillar of Object Oriented Programming. OOP is highly concerned with code organization, reusability, and encapsulation. First, a disclaimer: OOP is partially in contrast to Functional Programming, which is a different paradigm used a lot in Python. Not everyone who programs in Python (or surely most languages) uses OOP. You can do a lot in Java 8 that isn't very Object Oriented. If you don't want to use OOP, then don't. If you're just writing one-off scripts to process data that you'll never use again, then keep writing the way you are. However, there are a lot of reasons to use OOP. Some reasons: Organization: OOP defines well known and standard ways of describing and defining both data and procedure in code. Both data and procedure can be stored at varying levels of definition (in different classes), and there are standard ways about talking about these definitions. That is, if you use OOP in a standard way, it will help your later self and others understand, edit, and use your code. Also, instead of using a complex, arbitrary data storage mechanism (dicts of dicts or lists or dicts or lists of dicts of sets, or whatever), you can name pieces of data structures and conveniently refer to them. State: OOP helps you define and keep track of state. For instance, in a classic example, if you're creating a program that processes students (for instance, a grade program), you can keep all the info you need about them in one spot (name, age, gender, grade level, courses, grades, teachers, peers, diet, special needs, etc.), and this data is persisted as long as the object is alive, and is easily accessible. In contrast, in pure functional programming, state is never mutated in place. Encapsulation: With encapsulation, procedure and data are stored together. Methods (an OOP term for functions) are defined right alongside the data that they operate on and produce. In a language like Java that allows for access control, or in Python, depending upon how you describe your public API, this means that methods and data can be hidden from the user. What this means is that if you need or want to change code, you can do whatever you want to the implementation of the code, but keep the public APIs the same. Inheritance: Inheritance allows you to define data and procedure in one place (in one class), and then override or extend that functionality later. For instance, in Python, I often see people creating subclasses of the dict class in order to add additional functionality. A common change is overriding the method that throws an exception when a key is requested from a dictionary that doesn't exist to give a default value based on an unknown key. This allows you to extend your own code now or later, allow others to extend your code, and allows you to extend other people's code. Reusability: All of these reasons and others allow for greater reusability of code. Object oriented code allows you to write solid (tested) code once, and then reuse over and over. If you need to tweak something for your specific use case, you can inherit from an existing class and overwrite the existing behavior. If you need to change something, you can change it all while maintaining the existing public method signatures, and no one is the wiser (hopefully). Again, there are several reasons not to use OOP, and you don't need to. But luckily with a language like Python, you can use just a little bit or a lot, it's up to you. An example of the student use case (no guarantee on code quality, just an example): Object Oriented class Student(object): def __init__(self, name, age, gender, level, grades=None): self.name = name self.age = age self.gender = gender self.level = level self.grades = grades or {} def setGrade(self, course, grade): self.grades[course] = grade def getGrade(self, course): return self.grades[course] def getGPA(self): return sum(self.grades.values())/len(self.grades) # Define some students john = Student("John", 12, "male", 6, {"math":3.3}) jane = Student("Jane", 12, "female", 6, {"math":3.5}) # Now we can get to the grades easily print(john.getGPA()) print(jane.getGPA()) Standard Dict def calculateGPA(gradeDict): return sum(gradeDict.values())/len(gradeDict) students = {} # We can set the keys to variables so we might minimize typos name, age, gender, level, grades = "name", "age", "gender", "level", "grades" john, jane = "john", "jane" math = "math" students[john] = {} students[john][age] = 12 students[john][gender] = "male" students[john][level] = 6 students[john][grades] = {math:3.3} students[jane] = {} students[jane][age] = 12 students[jane][gender] = "female" students[jane][level] = 6 students[jane][grades] = {math:3.5} # At this point, we need to remember who the students are and where the grades are stored. Not a huge deal, but avoided by OOP. print(calculateGPA(students[john][grades])) print(calculateGPA(students[jane][grades]))


Instruction:
Convert 2D float array to 2D int array in NumPy

How do I convert a float NumPy array into an int NumPy array?

Output: Use the astype method. >>> x = np.array([[1.0, 2.3], [1.3, 2.9]]) >>> x array([[ 1. , 2.3], [ 1.3, 2.9]]) >>> x.astype(int) array([[1, 2], [1, 2]])


Instruction:
Why doesn't Python app print anything when run in a detached docker container?

I have a Python (2.7) app which is started in my dockerfile: CMD ["python","main.py"] main.py prints some strings when it is started and goes into a loop afterwards: print "App started" while True: time.sleep(1) As long as I start the container with the -it flag, everything works as expected: $ docker run --name=myapp -it myappimage > App started And I can see the same output via logs later: $ docker logs myapp > App started If I try to run the same container with the -d flag, the container seems to start normally, but I can't see any output: $ docker run --name=myapp -d myappimage > b82db1120fee5f92c80000f30f6bdc84e068bafa32738ab7adb47e641b19b4d1 $ docker logs myapp $ (empty) But the container still seems to run; $ docker ps Container Status ... myapp up 4 minutes ... Attach does not display anything either: $ docker attach --sig-proxy=false myapp (working, no output) Any ideas whats going wrong? Does "print" behave differently when ran in background? Docker version: Client version: 1.5.0 Client API version: 1.17 Go version (client): go1.4.2 Git commit (client): a8a31ef OS/Arch (client): linux/arm Server version: 1.5.0 Server API version: 1.17 Go version (server): go1.4.2 Git commit (server): a8a31ef

Output: Finally I found a solution to see Python output when running daemonized in Docker, thanks to @ahmetalpbalkan over at GitHub. Answering it here myself for further reference : Using unbuffered output with CMD ["python","-u","main.py"] instead of CMD ["python","main.py"] solves the problem; you can see the output now (both, stderr and stdout) via docker logs myapp why -u ref - print is indeed buffered and docker logs will eventually give you that output, just after enough of it will have piled up - executing the same script with python -u gives instant output as said above - import logging + logging.warning("text") gives the expected result even without -u what it means by python -u ref. > python --help | grep -- -u -u : force the stdout and stderr streams to be unbuffered;


Instruction:
How to drop a list of rows from Pandas dataframe?

I have a dataframe df : >>> df sales discount net_sales cogs STK_ID RPT_Date 600141 20060331 2.709 NaN 2.709 2.245 20060630 6.590 NaN 6.590 5.291 20060930 10.103 NaN 10.103 7.981 20061231 15.915 NaN 15.915 12.686 20070331 3.196 NaN 3.196 2.710 20070630 7.907 NaN 7.907 6.459 Then I want to drop rows with certain sequence numbers which indicated in a list, suppose here is [1,2,4], then left: sales discount net_sales cogs STK_ID RPT_Date 600141 20060331 2.709 NaN 2.709 2.245 20061231 15.915 NaN 15.915 12.686 20070630 7.907 NaN 7.907 6.459 How or what function can do that ?

Output: Use DataFrame.drop and pass it a Series of index labels: In [65]: df Out[65]: one two one 1 4 two 2 3 three 3 2 four 4 1 In [66]: df.drop(df.index[[1,3]]) Out[66]: one two one 1 4 three 3 2


Instruction:
How to set the working directory for debugging a Python program in VS Code?

How do I execute a Python file with the debugger and specify the working directory for the run?

Output: Setting "cwd" to ${FileDirname} in launch.json did not work for me; rather, I tried modifying settings.json, and actually got a solution. In the settings.json file, add the field: "python.terminal.executeInFileDir": true After adding this field, my settings.json looks something like: { "python.terminal.executeInFileDir": true } This will definitely set the cwd to the directory that contains your file, as opposed to the root folder of the project.


Instruction:
Get loop count inside a for-loop

This for loop iterates over all elements in a list: for item in my_list: print item Is there a way to know within the loop how many times I've been looping so far? For instance, I want to take a list and after I've processed ten elements I want to do something with them. The alternatives I thought about would be something like: count=0 for item in my_list: print item count +=1 if count % 10 == 0: print 'did ten' Or: for count in range(0,len(my_list)): print my_list[count] if count % 10 == 0: print 'did ten' Is there a better way (just like the for item in my_list) to get the number of iterations so far?

Output: The pythonic way is to use enumerate: for idx, item in enumerate(my_list):


Instruction:
How can I find the current OS in Python?

As the title says, how can I find the current operating system in python?

Output: I usually use sys.platform to get the platform. sys.platform will distinguish between linux, other unixes, and OS X, while os.name is "posix" for all of them. For much more detailed information, use the platform module. This has cross-platform functions that will give you information on the machine architecture, OS and OS version, version of Python, etc. Also it has os-specific functions to get things like the particular linux distribution.


Instruction:
Deleting folders in python recursively

I'm having a problem with deleting empty directories. Here is my code: for dirpath, dirnames, filenames in os.walk(dir_to_search): # other codes try: os.rmdir(dirpath) except OSError as ex: print(ex) The argument dir_to_search is where I'm passing the directory where the work needs to be done. That directory looks like this: test/20/... test/22/... test/25/... test/26/... Note that all the above folders are empty. When I run this script the folders 20,25 alone gets deleted! But the folders 25 and 26 aren't deleted, even though they are empty folders. Edit: The exception that I'm getting are: [Errno 39] Directory not empty: '/home/python-user/shell-scripts/s3logs/test' [Errno 39] Directory not empty: '/home/python-user/shell-scripts/s3logs/test/2012' [Errno 39] Directory not empty: '/home/python-user/shell-scripts/s3logs/test/2012/10' [Errno 39] Directory not empty: '/home/python-user/shell-scripts/s3logs/test/2012/10/29' [Errno 39] Directory not empty: '/home/python-user/shell-scripts/s3logs/test/2012/10/29/tmp' [Errno 39] Directory not empty: '/home/python-user/shell-scripts/s3logs/test/2012/10/28' [Errno 39] Directory not empty: '/home/python-user/shell-scripts/s3logs/test/2012/10/28/tmp' [Errno 39] Directory not empty: '/home/python-user/shell-scripts/s3logs/test/2012/10/26' [Errno 39] Directory not empty: '/home/python-user/shell-scripts/s3logs/test/2012/10/25' [Errno 39] Directory not empty: '/home/python-user/shell-scripts/s3logs/test/2012/10/27' [Errno 39] Directory not empty: '/home/python-user/shell-scripts/s3logs/test/2012/10/27/tmp' Where am I making a mistake?

Output: Try shutil.rmtree to delete files and directories: import shutil shutil.rmtree('/path/to/your/dir/')


Instruction:
Using headers with the Python 'Requests' library's get() method

I recently stumbled upon this great library for handling HTTP requests in Python; found on Requests: HTTP for Humans. I love working with it, but how can I add headers to my get requests?

Output: According to the API, the headers can all be passed in with requests.get(): import requests r = requests.get("http://www.example.com/", headers={"Content-Type":"text"})


Instruction:
Simple way to measure cell execution time in ipython notebook

I would like to get the time spent on the cell execution in addition to the original output from cell. To this end, I tried %%timeit -r1 -n1 but it doesn't expose the variable defined within cell. %%time works for cell which only contains 1 statement. In[1]: %%time 1 CPU times: user 4 µs, sys: 0 ns, total: 4 µs Wall time: 5.96 µs Out[1]: 1 In[2]: %%time # Notice there is no out result in this case. x = 1 x CPU times: user 3 µs, sys: 0 ns, total: 3 µs Wall time: 5.96 µs What's the best way to do it? Update I have been using Execute Time in Nbextension for quite some time now. It is great. Update 2021-03 As of now, this is the correct answer. Essentially, %%time and %%timeit both now work as one would expect.

Output: That was only a problem in old versions. All you need to do now is put %%time at the top of the cell. %%time measures how long it took something to run. It’s better for reporting on long-running operations than for doing low-level optimization. %%timeit is a benchmarking tool that runs statements over and over to give the average runtime for some statements, as well as the standard deviation. Because of the way in which the statements are repeatedly executed, the variables created in %%timeit cells are not available in other cells. %%timeit uses the python timeit module. The docs for that say, It avoids a number of common traps for measuring execution times. See also Tim Peters’ introduction to the “Algorithms” chapter in the Python Cookbook, published by O’Reilly. I hope that that module is still relevant, as the reference it refers to describes issues such as (1) workarounds for Windows 98 only updating time.time() 18.2 times per second, and (2) jamming all the statements onto one line to avoid the bytecode overhead of incrementing the line number counter. The currently top-rated answer, as well as some of the other outdated ones—which should be deleted because they are now highly misleading—do have useful comments indicating that those answers are not correct: %%time works even when the last statement is not print variables in the tested cell are now taken into consideration into the next cells


Instruction:
How to remove specific elements in a numpy array

How can I remove some specific elements from a numpy array? Say I have import numpy as np a = np.array([1,2,3,4,5,6,7,8,9]) I then want to remove 3,4,7 from a. All I know is the index of the values (index=[2,3,6]).

Output: Use numpy.delete(), which returns a new array with sub-arrays along an axis deleted. numpy.delete(a, index) For your specific question: import numpy as np a = np.array([1, 2, 3, 4, 5, 6, 7, 8, 9]) index = [2, 3, 6] new_a = np.delete(a, index) print(new_a) # Output: [1, 2, 5, 6, 8, 9] Note that numpy.delete() returns a new array since array scalars are immutable, similar to strings in Python, so each time a change is made to it, a new object is created. I.e., to quote the delete() docs: "A copy of arr with the elements specified by obj removed. Note that delete does not occur in-place..." If the code I post has output, it is the result of running the code.


Instruction:
Type hinting a collection of a specified type

Using Python 3's function annotations, it is possible to specify the type of items contained within a homogeneous list (or other collection) for the purpose of type hinting in PyCharm and other IDEs? A pseudo-python code example for a list of int: def my_func(l:list<int>): pass I know it's possible using Docstring... def my_func(l): """ :type l: list[int] """ pass ... but I prefer the annotation style if it's possible.

Output: As of May 2015, PEP 484 (Type Hints) has been formally accepted. The draft implementation is also available at github under ambv/typehinting. In September 2015, Python 3.5 was released with support for Type Hints and includes a new typing module. This allows for the specification of types contained within collections. As of November 2015, JetBrains PyCharm 5.0 fully supports Python 3.5 to include Type Hints as illustrated below. from typing import List def do_something(l: List[str]): for s in l: s # str Original Answer As of Aug 2014, I have confirmed that it is not possible to use Python 3 type annotations to specify types within collections (ex: a list of strings). The use of formatted docstrings such as reStructuredText or Sphinx are viable alternatives and supported by various IDEs. It also appears that Guido is mulling over the idea of extending type annotations in the spirit of mypy: http://mail.python.org/pipermail/python-ideas/2014-August/028618.html


Instruction:
Create a "with" block on several context managers?

Suppose you have three objects you acquire via context manager, for instance A lock, a db connection and an ip socket. You can acquire them by: with lock: with db_con: with socket: #do stuff But is there a way to do it in one block? something like with lock,db_con,socket: #do stuff Furthermore, is it possible, given an array of unknown length of objects that have context managers, is it possible to somehow do: a=[lock1, lock2, lock3, db_con1, socket, db_con2] with a as res: #now all objects in array are acquired If the answer is "no", is it because the need for such a feature implies bad design, or maybe I should suggest it in a pep? :-P

Output: In Python 2.7 and 3.1 and above, you can write: with A() as X, B() as Y, C() as Z: do_something() This is normally the best method to use, but if you have an unknown-length list of context managers you'll need one of the below methods. In Python 3.3, you can enter an unknown-length list of context managers by using contextlib.ExitStack: with ExitStack() as stack: for mgr in ctx_managers: stack.enter_context(mgr) # ... This allows you to create the context managers as you are adding them to the ExitStack, which prevents the possible problem with contextlib.nested (mentioned below). contextlib2 provides a backport of ExitStack for Python 2.6 and 2.7. In Python 2.6 and below, you can use contextlib.nested: from contextlib import nested with nested(A(), B(), C()) as (X, Y, Z): do_something() is equivalent to: m1, m2, m3 = A(), B(), C() with m1 as X: with m2 as Y: with m3 as Z: do_something() Note that this isn't exactly the same as normally using nested with, because A(), B(), and C() will all be called initially, before entering the context managers. This will not work correctly if one of these functions raises an exception. contextlib.nested is deprecated in newer Python versions in favor of the above methods.


Instruction:
I can't install python-ldap

When I run the following command: sudo pip install python-ldap I get this error: In file included from Modules/LDAPObject.c:9: Modules/errors.h:8: fatal error: lber.h: No such file or directory How can I fix this?

Output: The python-ldap is based on OpenLDAP, so you need to have the development files (headers) in order to compile the Python module. If you're on Ubuntu, the package is called libldap2-dev. Debian/Ubuntu: sudo apt-get install libsasl2-dev python-dev-is-python3 libldap2-dev libssl-dev Red Hat/CentOS: sudo yum install python-devel openldap-devel


Instruction:
Remove __pycache__ folders and .pyc files from Python project

What is the best way to clear out all the __pycache__ folders and .pyc/.pyo files from a Python project? I have seen multiple users suggest the pyclean script bundled with Debian, but this does not remove the folders. I want a simple way to clean up the project before pushing the files to my DVS.

Output: Running py3clean . cleaned it up very nicely. Make sure you do not accidentally mistype pyclean as pycclean, otherwise you will get this error: No command 'pycclean' found, did you mean: Command 'py3clean' from package 'python3-minimal' (main) Command 'pyclean' from package 'python-minimal' (main) pycclean: command not found


Instruction:
How to process SIGTERM signal gracefully?

Let's assume we have such a trivial daemon written in python: def mainloop(): while True: # 1. do # 2. some # 3. important # 4. job # 5. sleep mainloop() and we daemonize it using start-stop-daemon which by default sends SIGTERM (TERM) signal on --stop. Let's suppose the current step performed is #2. And at this very moment we're sending TERM signal. What happens is that the execution terminates immediately. I've found that I can handle the signal event using signal.signal(signal.SIGTERM, handler) but the thing is that it still interrupts the current execution and passes the control to handler. So, my question is - is it possible to not interrupt the current execution but handle the TERM signal in a separated thread (?) so that I was able to set shutdown_flag = True so that mainloop() had a chance to stop gracefully?

Output: A class based clean to use solution: import signal import time class GracefulKiller: kill_now = False def __init__(self): signal.signal(signal.SIGINT, self.exit_gracefully) signal.signal(signal.SIGTERM, self.exit_gracefully) def exit_gracefully(self, signum, frame): self.kill_now = True if __name__ == '__main__': killer = GracefulKiller() while not killer.kill_now: time.sleep(1) print("doing something in a loop ...") print("End of the program. I was killed gracefully :)")


Instruction:
Mapping over values in a python dictionary

Given a dictionary { k1: v1, k2: v2 ... } I want to get { k1: f(v1), k2: f(v2) ... } provided I pass a function f. Is there any such built in function? Or do I have to do dict([(k, f(v)) for (k, v) in my_dictionary.iteritems()]) Ideally I would just write my_dictionary.map_values(f) or my_dictionary.mutate_values_with(f) That is, it doesn't matter to me if the original dictionary is mutated or a copy is created.

Output: There is no such function; the easiest way to do this is to use a dict comprehension: my_dictionary = {k: f(v) for k, v in my_dictionary.items()} Note that there is no such method on lists either; you'd have to use a list comprehension or the map() function. As such, you could use the map() function for processing your dict as well: my_dictionary = dict(map(lambda kv: (kv[0], f(kv[1])), my_dictionary.items())) but that's not that readable, really. (Note that if you're still using Python 2.7, you should use the .iteritems() method instead of .items() to save memory. Also, the dict comprehension syntax wasn't introduced until Python 2.7.)


Instruction:
How to check if a value exists in a dictionary?

I have the following dictionary in python: d = {'1': 'one', '3': 'three', '2': 'two', '5': 'five', '4': 'four'} I need a way to find if a value such as "one" or "two" exists in this dictionary. For example, if I wanted to know if the index "1" existed I would simply have to type: "1" in d And then python would tell me if that is true or false, however I need to do that same exact thing except to find if a value exists.

Output: >>> d = {'1': 'one', '3': 'three', '2': 'two', '5': 'five', '4': 'four'} >>> 'one' in d.values() True Out of curiosity, some comparative timing: >>> T(lambda : 'one' in d.itervalues()).repeat() [0.28107285499572754, 0.29107213020324707, 0.27941107749938965] >>> T(lambda : 'one' in d.values()).repeat() [0.38303399085998535, 0.37257885932922363, 0.37096405029296875] >>> T(lambda : 'one' in d.viewvalues()).repeat() [0.32004380226135254, 0.31716084480285645, 0.3171098232269287] >>> T(lambda : 'four' in d.itervalues()).repeat() [0.41178202629089355, 0.3959040641784668, 0.3970959186553955] >>> T(lambda : 'four' in d.values()).repeat() [0.4631338119506836, 0.43541407585144043, 0.4359898567199707] >>> T(lambda : 'four' in d.viewvalues()).repeat() [0.43414998054504395, 0.4213531017303467, 0.41684913635253906] The reason is that each of the above returns a different type of object, which may or may not be well suited for lookup operations: >>> type(d.viewvalues()) <type 'dict_values'> >>> type(d.values()) <type 'list'> >>> type(d.itervalues()) <type 'dictionary-valueiterator'>


Instruction:
'str' object has no attribute 'decode'. Python 3 error?

Here is my code: import imaplib from email.parser import HeaderParser conn = imaplib.IMAP4_SSL('imap.gmail.com') conn.login('example@gmail.com', 'password') conn.select() conn.search(None, 'ALL') data = conn.fetch('1', '(BODY[HEADER])') header_data = data[1][0][1].decode('utf-8') At this point I get the error message: AttributeError: 'str' object has no attribute 'decode' Python 3 doesn't have str.decode() anymore, so how can I fix this?

Output: You are trying to decode an object that is already decoded. You have a str, there is no need to decode from UTF-8 anymore. Simply drop the .decode('utf-8') part: header_data = data[1][0][1]