Instruction:
How do I create variable variables?

I know that some other languages, such as PHP, support a concept of "variable variable names" - that is, the contents of a string can be used as part of a variable name. I heard that this is a bad idea in general, but I think it would solve some problems I have in my Python code. Is it possible to do something like this in Python? What can go wrong? If you are just trying to look up an existing variable by its name, see How can I select a variable by (string) name?. However, first consider whether you can reorganize the code to avoid that need, following the advice in this question.

Output: You can use dictionaries to accomplish this. Dictionaries are stores of keys and values. >>> dct = {'x': 1, 'y': 2, 'z': 3} >>> dct {'x': 1, 'y': 2, 'z': 3} >>> dct["y"] 2 You can use variable key names to achieve the effect of variable variables without the security risk. >>> x = "spam" >>> z = {x: "eggs"} >>> z["spam"] 'eggs' For cases where you're thinking of doing something like var1 = 'foo' var2 = 'bar' var3 = 'baz' ... a list may be more appropriate than a dict. A list represents an ordered sequence of objects, with integer indices: lst = ['foo', 'bar', 'baz'] print(lst[1]) # prints bar, because indices start at 0 lst.append('potatoes') # lst is now ['foo', 'bar', 'baz', 'potatoes'] For ordered sequences, lists are more convenient than dicts with integer keys, because lists support iteration in index order, slicing, append, and other operations that would require awkward key management with a dict.


Instruction:
Pretty printing XML in Python

What is the best way (or are the various ways) to pretty print XML in Python?

Output: import xml.dom.minidom dom = xml.dom.minidom.parse(xml_fname) # or xml.dom.minidom.parseString(xml_string) pretty_xml_as_string = dom.toprettyxml()


Instruction:
How to determine if Python is running inside a virtualenv?

Is it possible to determine if the current script is running inside a virtualenv environment?

Output: The reliable and documented way is to compare sys.prefix and sys.base_prefix. If they're equal, you're not in a virtual environment, otherwise you are. Inside a venv, sys.prefix points to the directory of the virtual environment, and sys.base_prefix to the Python interpreter used to create the environment. This is documented under How venvs work: It is sufficient to check sys.prefix != sys.base_prefix to determine if the current interpreter is running from a virtual environment. This works for Python stdlib venv and for virtualenv (since version 20): def in_venv(): return sys.prefix != sys.base_prefix Older versions of virtualenv used sys.real_prefix instead of sys.base_prefix, and sys.real_prefix did not exist outside a virtual environment. In Python 3.3 and earlier sys.base_prefix did not ever exist. So a check that also handles some legacy cases could look like this: import sys def get_base_prefix_compat(): """Get base/real prefix, or sys.prefix if there is none.""" return ( getattr(sys, "base_prefix", None) or getattr(sys, "real_prefix", None) or sys.prefix ) def in_virtualenv(): return sys.prefix != get_base_prefix_compat() Using the VIRTUAL_ENV environment variable is not reliable. It is set by the virtualenv activate shell script, but a virtualenv can be used without activation by directly running an executable from the virtualenv's bin/ (or Scripts) directory, in which case $VIRTUAL_ENV will not be set. Or a non-virtualenv Python binary can be executed directly while a virtualenv is activated in the shell, in which case $VIRTUAL_ENV may be set in a Python process that is not actually running in that virtualenv.


Instruction:
Automatically creating directories with file output

Say I want to make a file: filename = "/foo/bar/baz.txt" with open(filename, "w") as f: f.write("FOOBAR") This gives an IOError, since /foo/bar does not exist. What is the most pythonic way to generate those directories automatically? Is it necessary for me explicitly call os.path.exists and os.mkdir on every single one (i.e., /foo, then /foo/bar)?

Output: In Python 3.2+, using the APIs requested by the OP, you can elegantly do the following: import os filename = "/foo/bar/baz.txt" os.makedirs(os.path.dirname(filename), exist_ok=True) with open(filename, "w") as f: f.write("FOOBAR") With the Pathlib module (introduced in Python 3.4), there is an alternate syntax (thanks David258): from pathlib import Path output_file = Path("/foo/bar/baz.txt") output_file.parent.mkdir(exist_ok=True, parents=True) output_file.write_text("FOOBAR") In older python, there is a less elegant way: The os.makedirs function does this. Try the following: import os import errno filename = "/foo/bar/baz.txt" if not os.path.exists(os.path.dirname(filename)): try: os.makedirs(os.path.dirname(filename)) except OSError as exc: # Guard against race condition if exc.errno != errno.EEXIST: raise with open(filename, "w") as f: f.write("FOOBAR") The reason to add the try-except block is to handle the case when the directory was created between the os.path.exists and the os.makedirs calls, so that to protect us from race conditions.


Instruction:
Sort a list of tuples by 2nd item (integer value)

I have a list of tuples that looks something like this: [('abc', 121),('abc', 231),('abc', 148), ('abc',221)] I want to sort this list in ascending order by the integer value inside the tuples. Is it possible?

Output: Try using the key keyword argument of sorted(), which sorts in increasing order by default: sorted( [('abc', 121), ('abc', 231), ('abc', 148), ('abc', 221)], key=lambda x: x[1] ) key should be a function that identifies how to retrieve the comparable element from your data structure. In your case, it is the second element of the tuple, so we access [1]. For optimization, see jamylak's response using operator.itemgetter(1), which is essentially a faster version of lambda x: x[1].


Instruction:
Print a list in reverse order with range()?

How can you produce the following list with range() in Python? [9, 8, 7, 6, 5, 4, 3, 2, 1, 0]

Output: Use reversed() function (efficient since range implements __reversed__): reversed(range(10)) It's much more meaningful. Update: list cast If you want it to be a list (as @btk pointed out): list(reversed(range(10))) Update: range-only solution If you want to use only range to achieve the same result, you can use all its parameters. range(start, stop, step) For example, to generate a list [3, 2, 1, 0], you can use the following: range(3, -1, -1) It may be less intuitive, but it works the same with less text. This answer by @Wolf indicates this approach is slightly faster than reversed.


Instruction:
Timeout on a function call

I'm calling a function in Python which I know may stall and force me to restart the script. How do I call the function or what do I wrap it in so that if it takes longer than 5 seconds the script cancels it and does something else?

Output: You may use the signal package if you are running on UNIX: In [1]: import signal # Register an handler for the timeout In [2]: def handler(signum, frame): ...: print("Forever is over!") ...: raise Exception("end of time") ...: # This function *may* run for an indetermined time... In [3]: def loop_forever(): ...: import time ...: while 1: ...: print("sec") ...: time.sleep(1) ...: ...: # Register the signal function handler In [4]: signal.signal(signal.SIGALRM, handler) Out[4]: 0 # Define a timeout for your function In [5]: signal.alarm(10) Out[5]: 0 In [6]: try: ...: loop_forever() ...: except Exception, exc: ...: print(exc) ....: sec sec sec sec sec sec sec sec Forever is over! end of time # Cancel the timer if the function returned before timeout # (ok, mine won't but yours maybe will :) In [7]: signal.alarm(0) Out[7]: 0 10 seconds after the call signal.alarm(10), the handler is called. This raises an exception that you can intercept from the regular Python code. This module doesn't play well with threads (but then, who does?) Note that since we raise an exception when timeout happens, it may end up caught and ignored inside the function, for example of one such function: def loop_forever(): while 1: print('sec') try: time.sleep(10) except: continue


Instruction:
How to get the Cartesian product of multiple lists

How can I get the Cartesian product (every possible combination of values) from a group of lists? For example, given somelists = [ [1, 2, 3], ['a', 'b'], [4, 5] ] How do I get this? [(1, 'a', 4), (1, 'a', 5), (1, 'b', 4), (1, 'b', 5), (2, 'a', 4), (2, 'a', 5), ...] One common application for this technique is to avoid deeply nested loops. See Avoiding nested for loops for a more specific duplicate. Similarly, this technique might be used to "explode" a dictionary with list values; see Combine Python Dictionary Permutations into List of Dictionaries . If you want a Cartesian product of the same list with itself multiple times, itertools.product can handle that elegantly. See Operation on every pair of element in a list or How can I get "permutations with repetitions" from a list (Cartesian product of a list with itself)?. Many people who already know about itertools.product struggle with the fact that it expects separate arguments for each input sequence, rather than e.g. a list of lists. The accepted answer shows how to handle this with *. However, the use of * here to unpack arguments is fundamentally not different from any other time it's used in a function call. Please see Expanding tuples into arguments for this topic (and use that instead to close duplicate questions, as appropriate).

Output: Use itertools.product, which has been available since Python 2.6. import itertools somelists = [ [1, 2, 3], ['a', 'b'], [4, 5] ] for element in itertools.product(*somelists): print(element) This is the same as: for element in itertools.product([1, 2, 3], ['a', 'b'], [4, 5]): print(element)


Instruction:
How do I determine if my python shell is executing in 32bit or 64bit?

How can I tell what mode the shell is in, from within the shell? I've tried looking at the platform module, but it seems only to tell you about "the bit architecture and the linkage format used for the executable". My binary is compiled as 64bit (I'm running on OS X 10.6), so it seems to always report 64bit even though I'm using the methods described here to force 32bit mode.

Output: One way is to look at sys.maxsize as documented here: $ python-32 -c 'import sys;print("%x" % sys.maxsize, sys.maxsize > 2**32)' ('7fffffff', False) $ python-64 -c 'import sys;print("%x" % sys.maxsize, sys.maxsize > 2**32)' ('7fffffffffffffff', True) On Windows, run the same commands formatted as follows: python -c "import sys;print(\"%x\" % sys.maxsize, sys.maxsize > 2**32)" sys.maxsize was introduced in Python 2.6. If you need a test for older systems, this slightly more complicated test should work on all Python 2 and 3 releases: $ python-32 -c 'import struct;print( 8 * struct.calcsize("P"))' 32 $ python-64 -c 'import struct;print( 8 * struct.calcsize("P"))' 64 BTW, you might be tempted to use platform.architecture() for this. Unfortunately, its results are not always reliable, particularly in the case of OS X universal binaries. $ arch -x86_64 /usr/bin/python2.6 -c 'import sys,platform; print platform.architecture()[0], sys.maxsize > 2**32' 64bit True $ arch -i386 /usr/bin/python2.6 -c 'import sys,platform; print platform.architecture()[0], sys.maxsize > 2**32' 64bit False


Instruction:
How do I disable the security certificate check in Python's Requests?

I am using import requests requests.post(url='https://foo.example', data={'bar':'baz'}) but I get a request.exceptions.SSLError. The website has an expired certificate, but I am not sending sensitive data, so it doesn't matter to me. I would imagine there is an argument like 'verifiy=False' that I could use, but I can't seem to find it.

Output: From the documentation: requests can also ignore verifying the SSL certificate if you set verify to False. >>> requests.get('https://kennethreitz.com', verify=False) <Response [200]> If you're using a third-party module and want to disable the checks, here's a context manager that monkey patches requests and changes it so that verify=False is the default and suppresses the warning. import warnings import contextlib import requests from urllib3.exceptions import InsecureRequestWarning old_merge_environment_settings = requests.Session.merge_environment_settings @contextlib.contextmanager def no_ssl_verification(): opened_adapters = set() def merge_environment_settings(self, url, proxies, stream, verify, cert): # Verification happens only once per connection so we need to close # all the opened adapters once we're done. Otherwise, the effects of # verify=False persist beyond the end of this context manager. opened_adapters.add(self.get_adapter(url)) settings = old_merge_environment_settings(self, url, proxies, stream, verify, cert) settings['verify'] = False return settings requests.Session.merge_environment_settings = merge_environment_settings try: with warnings.catch_warnings(): warnings.simplefilter('ignore', InsecureRequestWarning) yield finally: requests.Session.merge_environment_settings = old_merge_environment_settings for adapter in opened_adapters: try: adapter.close() except: pass Here's how you use it: with no_ssl_verification(): requests.get('https://wrong.host.badssl.example/') print('It works') requests.get('https://wrong.host.badssl.example/', verify=True) print('Even if you try to force it to') requests.get('https://wrong.host.badssl.example/', verify=False) print('It resets back') session = requests.Session() session.verify = True with no_ssl_verification(): session.get('https://wrong.host.badssl.example/', verify=True) print('Works even here') try: requests.get('https://wrong.host.badssl.example/') except requests.exceptions.SSLError: print('It breaks') try: session.get('https://wrong.host.badssl.example/') except requests.exceptions.SSLError: print('It breaks here again') Note that this code closes all open adapters that handled a patched request once you leave the context manager. This is because requests maintains a per-session connection pool and certificate validation happens only once per connection so unexpected things like this will happen: >>> import requests >>> session = requests.Session() >>> session.get('https://wrong.host.badssl.example/', verify=False) /usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py:857: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings InsecureRequestWarning) <Response [200]> >>> session.get('https://wrong.host.badssl.example/', verify=True) /usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py:857: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings InsecureRequestWarning) <Response [200]>


Instruction:
"TypeError: method() takes 1 positional argument but 2 were given" but I only passed one

If I have a class ... class MyClass: def method(arg): print(arg) ... which I use to create an object ... my_object = MyClass() ... on which I call method("foo") like so ... >>> my_object.method("foo") Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: method() takes exactly 1 positional argument (2 given) ... why does Python tell me I gave it two arguments, when I only gave one?

Output: In Python, this: my_object.method("foo") ... is syntactic sugar, which the interpreter translates behind the scenes into: MyClass.method(my_object, "foo") ... which, as you can see, does indeed have two arguments - it's just that the first one is implicit, from the point of view of the caller. This is because most methods do some work with the object they're called on, so there needs to be some way for that object to be referred to inside the method. By convention, this first argument is called self inside the method definition: class MyNewClass: def method(self, arg): print(self) print(arg) If you call method("foo") on an instance of MyNewClass, it works as expected: >>> my_new_object = MyNewClass() >>> my_new_object.method("foo") <__main__.MyNewClass object at 0x29045d0> foo Occasionally (but not often), you really don't care about the object that your method is bound to, and in that circumstance, you can decorate the method with the builtin staticmethod() function to say so: class MyOtherClass: @staticmethod def method(arg): print(arg) ... in which case you don't need to add a self argument to the method definition, and it still works: >>> my_other_object = MyOtherClass() >>> my_other_object.method("foo") foo


Instruction:
How do I parallelize a simple Python loop?

This is probably a trivial question, but how do I parallelize the following loop in python? # setup output lists output1 = list() output2 = list() output3 = list() for j in range(0, 10): # calc individual parameter value parameter = j * offset # call the calculation out1, out2, out3 = calc_stuff(parameter = parameter) # put results into correct output list output1.append(out1) output2.append(out2) output3.append(out3) I know how to start single threads in Python but I don't know how to "collect" the results. Multiple processes would be fine too - whatever is easiest for this case. I'm using currently Linux but the code should run on Windows and Mac as-well. What's the easiest way to parallelize this code?

Output: The CPython implementation currently has a global interpreter lock (GIL) that prevents threads of the same interpreter from concurrently executing Python code. This means CPython threads are useful for concurrent I/O-bound workloads, but usually not for CPU-bound workloads. The naming calc_stuff() indicates that your workload is CPU-bound, so you want to use multiple processes here (which is often the better solution for CPU-bound workloads anyway, regardless of the GIL). There are two easy ways of creating a process pool into the Python standard library. The first one is the multiprocessing module, which can be used like this: pool = multiprocessing.Pool(4) out1, out2, out3 = zip(*pool.map(calc_stuff, range(0, 10 * offset, offset))) Note that this won't work in the interactive interpreter due to the way multiprocessing is implemented. The second way to create a process pool is concurrent.futures.ProcessPoolExecutor: with concurrent.futures.ProcessPoolExecutor() as pool: out1, out2, out3 = zip(*pool.map(calc_stuff, range(0, 10 * offset, offset))) This uses the multiprocessing module under the hood, so it behaves identically to the first version.


Instruction:
How to avoid "RuntimeError: dictionary changed size during iteration" error?

Suppose I have a dictionary of lists: d = {'a': [1], 'b': [1, 2], 'c': [], 'd':[]} Now I want to remove key-value pairs where the values are empty lists. I tried this code: for i in d: if not d[i]: d.pop(i) but this gives an error: RuntimeError: dictionary changed size during iteration I understand that entries can't be added or removed from a dictionary while iterating through it. How can I work around this limitation in order to solve the problem? See Modifying a Python dict while iterating over it for citations that this can cause problems, and why.

Output: In Python 3.x and 2.x you can use use list to force a copy of the keys to be made: for i in list(d): In Python 2.x calling .keys made a copy of the keys that you could iterate over while modifying the dict: for i in d.keys(): but on Python 3.x, .keys returns a view object instead, so it won't fix your error.


Instruction:
How to invoke the super constructor in Python?

class A: def __init__(self): print("world") class B(A): def __init__(self): print("hello") B() # output: hello In all other languages I've worked with the super constructor is invoked implicitly. How does one invoke it in Python? I would expect super(self) but this doesn't work.

Output: In line with the other answers, there are multiple ways to call super class methods (including the constructor), however in Python 3 the process has been simplified: Python 3 class A(object): def __init__(self): print("world") class B(A): def __init__(self): print("hello") super().__init__() Python 2 In Python 2, you have to call the slightly more verbose version super(<containing classname>, self), which is equivalent to super()as per the docs. class A(object): def __init__(self): print "world" class B(A): def __init__(self): print "hello" super(B, self).__init__()


Instruction:
How to convert JSON data into a Python object?

I want to convert JSON data into a Python object. I receive JSON data objects from the Facebook API, which I want to store in my database. My current View in Django (Python) (request.POST contains the JSON): response = request.POST user = FbApiUser(user_id = response['id']) user.name = response['name'] user.username = response['username'] user.save() This works fine, but how do I handle complex JSON data objects? Wouldn't it be much better if I could somehow convert this JSON object into a Python object for easy use?

Output: UPDATE With Python3, you can do it in one line, using SimpleNamespace and object_hook: import json from types import SimpleNamespace data = '{"name": "John Smith", "hometown": {"name": "New York", "id": 123}}' # Parse JSON into an object with attributes corresponding to dict keys. x = json.loads(data, object_hook=lambda d: SimpleNamespace(**d)) # Or, in Python 3.13+: # json.loads(data, object_hook=SimpleNamespace) print(x.name, x.hometown.name, x.hometown.id) OLD ANSWER (Python2) In Python2, you can do it in one line, using namedtuple and object_hook (but it's very slow with many nested objects): import json from collections import namedtuple data = '{"name": "John Smith", "hometown": {"name": "New York", "id": 123}}' # Parse JSON into an object with attributes corresponding to dict keys. x = json.loads(data, object_hook=lambda d: namedtuple('X', d.keys())(*d.values())) print x.name, x.hometown.name, x.hometown.id or, to reuse this easily: def _json_object_hook(d): return namedtuple('X', d.keys())(*d.values()) def json2obj(data): return json.loads(data, object_hook=_json_object_hook) x = json2obj(data) If you want it to handle keys that aren't good attribute names, check out namedtuple's rename parameter.


Instruction:
Hide all warnings in IPython

I need to produce a screencast of an IPython session, and to avoid confusing viewers, I want to disable all warnings emitted by warnings.warn calls from different packages. Is there a way to configure the ipythonrc file to automatically disable all such warnings?

Output: Place: import warnings warnings.filterwarnings('ignore') inside ~/.ipython/profile_default/startup/disable-warnings.py. Quite often it is useful to see a warning once. This can be set by: warnings.filterwarnings(action='once')


Instruction:
How can I Install a Python module with Pip programmatically (from my code)?

I need to install a package from PyPI straight within my script. Is there maybe some module or distutils (distribute, pip, etc.) feature which allows me to just execute something like pypi.install('requests') and requests will be installed into my virtualenv?

Output: The officially recommended way to install packages from a script is by calling pip's command-line interface via a subprocess. Most other answers presented here are not supported by pip. Furthermore since pip 10.x, all code has been moved to pip._internal precisely in order to make it clear to users that programmatic use of pip is not supported. Use sys.executable to ensure that you will call the same pip associated with the current runtime. For example: import subprocess import sys def install(package): subprocess.check_call([sys.executable, "-m", "pip", "install", package])


Instruction:
How can I get current CPU and RAM usage in Python?

How can I get the current system status (current CPU, RAM, free disk space, etc.) in Python? Ideally, it would work for both Unix-like and Windows platforms. There seems to be a few possible ways of extracting that from my search: Using a library such as PSI (that currently seems not actively developed and not supported on multiple platforms) or something like pystatgrab (again no activity since 2007 it seems and no support for Windows). Using platform specific code such as using a os.popen("ps") or similar for the Unix-like systems and MEMORYSTATUS in ctypes.windll.kernel32 (see this recipe on ActiveState) for the Windows platform. One could put a Python class together with all those code snippets. It's not that those methods are bad, but is there already a well-supported, multi-platform way of doing the same thing?

Output: The psutil library gives you information about CPU, RAM, etc., on a variety of platforms: psutil is a module providing an interface for retrieving information on running processes and system utilization (CPU, memory) in a portable way by using Python, implementing many functionalities offered by tools like ps, top and Windows task manager. It currently supports Linux, Windows, OSX, Sun Solaris, FreeBSD, OpenBSD and NetBSD, both 32-bit and 64-bit architectures, with Python versions from 2.6 to 3.5 (users of Python 2.4 and 2.5 may use 2.1.3 version). Some examples: #!/usr/bin/env python import psutil # gives a single float value psutil.cpu_percent() # gives an object with many fields psutil.virtual_memory() # you can convert that object to a dictionary dict(psutil.virtual_memory()._asdict()) # you can have the percentage of used RAM psutil.virtual_memory().percent 79.2 # you can calculate percentage of available memory psutil.virtual_memory().available * 100 / psutil.virtual_memory().total 20.8 Here's other documentation that provides more concepts and interest concepts: psutil documentation


Instruction:
What does "nonlocal" do in Python 3?

What does nonlocal do in Python 3.x? To close debugging questions where OP needs nonlocal and doesn't realize it, please use Is it possible to modify variable in python that is in outer, but not global, scope? instead. Although Python 2 is officially unsupported as of January 1, 2020, if for some reason you are forced to maintain a Python 2.x codebase and need an equivalent to nonlocal, see nonlocal keyword in Python 2.x.

Output: Compare this, without using nonlocal: x = 0 def outer(): x = 1 def inner(): x = 2 print("inner:", x) inner() print("outer:", x) outer() print("global:", x) # inner: 2 # outer: 1 # global: 0 To this, using nonlocal, where inner()'s x is now also outer()'s x: x = 0 def outer(): x = 1 def inner(): nonlocal x x = 2 print("inner:", x) inner() print("outer:", x) outer() print("global:", x) # inner: 2 # outer: 2 # global: 0 If we were to use global, it would bind x to the properly "global" value: x = 0 def outer(): x = 1 def inner(): global x x = 2 print("inner:", x) inner() print("outer:", x) outer() print("global:", x) # inner: 2 # outer: 1 # global: 2


Instruction:
Can't subtract offset-naive and offset-aware datetimes

I have a timezone aware timestamptz field in PostgreSQL. When I pull data from the table, I then want to subtract the time right now so I can get it's age. The problem I'm having is that both datetime.datetime.now() and datetime.datetime.utcnow() seem to return timezone unaware timestamps, which results in me getting this error: TypeError: can't subtract offset-naive and offset-aware datetimes Is there a way to avoid this (preferably without a third-party module being used). EDIT: Thanks for the suggestions, however trying to adjust the timezone seems to give me errors.. so I'm just going to use timezone unaware timestamps in PG and always insert using: NOW() AT TIME ZONE 'UTC' That way all my timestamps are UTC by default (even though it's more annoying to do this).

Output: Have you tried to remove the timezone awareness? From http://pytz.sourceforge.net/ naive = dt.replace(tzinfo=None) may have to add time zone conversion as well. edit: Please be aware the age of this answer. An answer involving ADDing the timezone info instead of removing it in python 3 is below. https://stackoverflow.com/a/25662061/93380


Instruction:
How do I get a value of datetime.today() in Python that is "timezone aware"?

I am trying to subtract one date value from the value of datetime.datetime.today() to calculate how long ago something was. But it complains: TypeError: can't subtract offset-naive and offset-aware datetimes The return value from datetime.datetime.today() doesn't seem to be "timezone aware", while my other date value is. How do I get a return value from datetime.datetime.today() that is timezone aware? The ideal solution would be for it to automatically know the timezone. Right now, it's giving me the time in local time, which happens to be PST, i.e. UTC - 8 hours. Worst case, is there a way I can manually enter a timezone value into the datetime object returned by datetime.datetime.today() and set it to UTC-8?

Output: In the standard library, there is no cross-platform way to create aware timezones without creating your own timezone class. (Edit: Python 3.9 introduces zoneinfo in the standard library which does provide this functionality.) On Windows, there's win32timezone.utcnow(), but that's part of pywin32. I would rather suggest to use the pytz library, which has a constantly updated database of most timezones. Working with local timezones can be very tricky (see "Further reading" links below), so you may rather want to use UTC throughout your application, especially for arithmetic operations like calculating the difference between two time points. You can get the current date/time like so: import pytz from datetime import datetime datetime.utcnow().replace(tzinfo=pytz.utc) Mind that datetime.today() and datetime.now() return the local time, not the UTC time, so applying .replace(tzinfo=pytz.utc) to them would not be correct. Another nice way to do it is: datetime.now(pytz.utc) which is a bit shorter and does the same. Further reading/watching why to prefer UTC in many cases: pytz documentation What Every Developer Should Know About Time – development hints for many real-life use cases The Problem with Time & Timezones - Computerphile – funny, eye-opening explanation about the complexity of working with timezones (video)


Instruction:
python .replace() regex

I am trying to do a grab everything after the '' tag and delete it, but my code doesn't seem to be doing anything. Does .replace() not support regex? z.write(article.replace('.+', ''))

Output: No. Regular expressions in Python are handled by the re module. article = re.sub(r'(?is)</html>.+', '</html>', article) In general: str_output = re.sub(regex_search_term, regex_replacement, str_input)


Instruction:
How do I install the yaml package for Python?

I have a Python program that uses YAML. I attempted to install it on a new server using pip install yaml and it returns the following: $ sudo pip install yaml Downloading/unpacking yaml Could not find any downloads that satisfy the requirement yaml No distributions at all found for yaml Storing complete log in /home/pa/.pip/pip.log How do I install the yaml package for Python? I'm running Python 2.7. (OS: Debian Wheezy)

Output: You could try the search the feature on https://pypi.org/search (via a browser) and look for packages in PyPI with yaml in the short description. That reveals various packages, including PyYaml, yamltools, and PySyck, among others (Note that PySyck docs recommend using PyYaml, since syck is out of date). Now you know a specific package name, you can install it: $ pip install pyyaml If you want to install python yaml system-wide in linux, you can also use a package manager, like aptitude or yum: $ sudo apt-get install python-yaml $ sudo yum install python-yaml


Instruction:
Passing a dictionary to a function as keyword parameters

I'd like to call a function in python using a dictionary with matching key-value pairs for the parameters. Here is some code: d = dict(param='test') def f(param): print(param) f(d) This prints {'param': 'test'} but I'd like it to just print test. I'd like it to work similarly for more parameters: d = dict(p1=1, p2=2) def f2(p1, p2): print(p1, p2) f2(d) Is this possible?

Output: Figured it out for myself in the end. It is simple, I was just missing the ** operator to unpack the dictionary So my example becomes: d = dict(p1=1, p2=2) def f2(p1,p2): print(p1, p2) f2(**d)


Instruction:
How to get POSTed JSON in Flask?

I'm trying to build a simple API using Flask, in which I now want to read some POSTed JSON. I do the POST with the Postman Chrome extension, and the JSON I POST is simply {"text":"hello world"}. I try to read the JSON using the following method: @app.route('/api/add_message/<uuid>', methods=['GET', 'POST']) def add_message(uuid): content = request.json print(content) return uuid On the browser it correctly returns the UUID I put in the GET, but on the console, it just prints out None (where I expect it to print out the {"text":"hello world"}. Does anybody know how I can get the posted JSON from within the Flask method?

Output: First of all, the .json attribute is a property that delegates to the request.get_json() method, which documents why you see None here. You need to set the request content type to application/json for the .json property and .get_json() method (with no arguments) to work as either will produce None otherwise. See the Flask Request documentation: The parsed JSON data if mimetype indicates JSON (application/json, see .is_json). You can tell request.get_json() to skip the content type requirement by passing it the force=True keyword argument. Note that if an exception is raised at this point (possibly resulting in a 415 Unsupported Media Type response), your JSON data is invalid. It is in some way malformed; you may want to check it with a JSON validator.


Instruction:
How to insert newlines on argparse help text?

I'm using argparse in Python 2.7 for parsing input options. One of my options is a multiple choice. I want to make a list in its help text, e.g. from argparse import ArgumentParser parser = ArgumentParser(description='test') parser.add_argument('-g', choices=['a', 'b', 'g', 'd', 'e'], default='a', help="Some option, where\n" " a = alpha\n" " b = beta\n" " g = gamma\n" " d = delta\n" " e = epsilon") parser.parse_args() However, argparse strips all newlines and consecutive spaces. The result looks like ~/Downloads:52$ python2.7 x.py -h usage: x.py [-h] [-g {a,b,g,d,e}] test optional arguments: -h, --help show this help message and exit -g {a,b,g,d,e} Some option, where a = alpha b = beta g = gamma d = delta e = epsilon How to insert newlines in the help text?

Output: Try using RawTextHelpFormatter to preserve all of your formatting: from argparse import RawTextHelpFormatter parser = ArgumentParser(description='test', formatter_class=RawTextHelpFormatter) It's similar to RawDescriptionHelpFormatter but instead of only applying to the description and epilog, RawTextHelpFormatter also applies to all help text (including arguments).


Instruction:
join list of lists in python

Is the a short syntax for joining a list of lists into a single list( or iterator) in python? For example I have a list as follows and I want to iterate over a,b and c. x = [["a","b"], ["c"]] The best I can come up with is as follows. result = [] [ result.extend(el) for el in x] for el in result: print el

Output: import itertools a = [['a','b'], ['c']] print(list(itertools.chain.from_iterable(a))) This gives ['a', 'b', 'c']


Instruction:
Get the row(s) which have the max value in groups using groupby

How do I find all rows in a pandas DataFrame which have the max value for count column, after grouping by ['Sp','Mt'] columns? Example 1: the following DataFrame: Sp Mt Value count 0 MM1 S1 a **3** 1 MM1 S1 n 2 2 MM1 S3 cb **5** 3 MM2 S3 mk **8** 4 MM2 S4 bg **10** 5 MM2 S4 dgd 1 6 MM4 S2 rd 2 7 MM4 S2 cb 2 8 MM4 S2 uyi **7** Expected output is to get the result rows whose count is max in each group, like this: Sp Mt Value count 0 MM1 S1 a **3** 2 MM1 S3 cb **5** 3 MM2 S3 mk **8** 4 MM2 S4 bg **10** 8 MM4 S2 uyi **7** Example 2: Sp Mt Value count 4 MM2 S4 bg 10 5 MM2 S4 dgd 1 6 MM4 S2 rd 2 7 MM4 S2 cb 8 8 MM4 S2 uyi 8 Expected output: Sp Mt Value count 4 MM2 S4 bg 10 7 MM4 S2 cb 8 8 MM4 S2 uyi 8

Output: Firstly, we can get the max count for each group like this: In [1]: df Out[1]: Sp Mt Value count 0 MM1 S1 a 3 1 MM1 S1 n 2 2 MM1 S3 cb 5 3 MM2 S3 mk 8 4 MM2 S4 bg 10 5 MM2 S4 dgd 1 6 MM4 S2 rd 2 7 MM4 S2 cb 2 8 MM4 S2 uyi 7 In [2]: df.groupby(['Sp', 'Mt'])['count'].max() Out[2]: Sp Mt MM1 S1 3 S3 5 MM2 S3 8 S4 10 MM4 S2 7 Name: count, dtype: int64 To get the indices of the original DF you can do: In [3]: idx = df.groupby(['Sp', 'Mt'])['count'].transform(max) == df['count'] In [4]: df[idx] Out[4]: Sp Mt Value count 0 MM1 S1 a 3 2 MM1 S3 cb 5 3 MM2 S3 mk 8 4 MM2 S4 bg 10 8 MM4 S2 uyi 7 Note that if you have multiple max values per group, all will be returned. Update On a Hail Mary chance that this is what the OP is requesting: In [5]: df['count_max'] = df.groupby(['Sp', 'Mt'])['count'].transform(max) In [6]: df Out[6]: Sp Mt Value count count_max 0 MM1 S1 a 3 3 1 MM1 S1 n 2 3 2 MM1 S3 cb 5 5 3 MM2 S3 mk 8 8 4 MM2 S4 bg 10 10 5 MM2 S4 dgd 1 10 6 MM4 S2 rd 2 7 7 MM4 S2 cb 2 7 8 MM4 S2 uyi 7 7


Instruction:
How do I create a new column where the values are selected based on an existing column?

How do I add a color column to the following dataframe so that color='green' if Set == 'Z', and color='red' otherwise? Type Set 1 A Z 2 B Z 3 B X 4 C Y

Output: If you only have two choices to select from then use np.where: df['color'] = np.where(df['Set']=='Z', 'green', 'red') For example, import pandas as pd import numpy as np df = pd.DataFrame({'Type':list('ABBC'), 'Set':list('ZZXY')}) df['color'] = np.where(df['Set']=='Z', 'green', 'red') print(df) yields Set Type color 0 Z A green 1 Z B green 2 X B red 3 Y C red If you have more than two conditions then use np.select. For example, if you want color to be yellow when (df['Set'] == 'Z') & (df['Type'] == 'A') otherwise blue when (df['Set'] == 'Z') & (df['Type'] == 'B') otherwise purple when (df['Type'] == 'B') otherwise black, then use df = pd.DataFrame({'Type':list('ABBC'), 'Set':list('ZZXY')}) conditions = [ (df['Set'] == 'Z') & (df['Type'] == 'A'), (df['Set'] == 'Z') & (df['Type'] == 'B'), (df['Type'] == 'B')] choices = ['yellow', 'blue', 'purple'] df['color'] = np.select(conditions, choices, default='black') print(df) which yields Set Type color 0 Z A yellow 1 Z B blue 2 X B purple 3 Y C black


Instruction:
Using Pip to install packages to an Anaconda environment

On Conda 4.2.13 Mac OS X v10.12.1 (Sierra) I am trying to install packages from pip to a fresh environment (virtual) created using anaconda. In the Anaconda documentation it says this is perfectly fine. It is done the same way as for virtualenv. Activate the environment where you want to put the program, then pip install a program... I created an empty environment in Anaconda like this: conda create -n shrink_venv Activate it: source activate shrink_venv I then can see in the terminal that I am working in my environment (shrink_venv). A problem is coming up when I try to install a package using pip: pip install Pillow Output: Requirement already satisfied (use --upgrade to upgrade): Pillow in /Library/Python/2.7/site-packages So I can see it thinks the requirement is satisfied from the system-wide package. So it seems the environment is not working correctly, definitely not like it said in the documentation. Am I doing something wrong here? Just a note, I know you can use conda install for the packages, but I have had an issue with Pillow from anaconda, so I wanted to get it from pip, and since the docs say that is fine. Output of which -a pip: /usr/local/bin/pip /Users/my_user/anaconda/bin/pip I see this is pretty common issue. I have found that the Conda environment doesn't play well with the PYTHONPATH. The system seems to always look in the PYTHONPATH locations even when you're using a Conda environment. Now, I always run unset PYTHONPATH when using a Conda environment, and it works much better. I'm on a Mac.

Output: For others who run into this situation, I found this to be the most straightforward solution: Run conda create -n venv_name and conda activate venv_name, where venv_name is the name of your virtual environment. Run conda install pip. This will install pip to your venv directory. Find your anaconda directory, and find the actual venv folder. It should be somewhere like /anaconda/envs/venv_name/; or, you could also run conda activate venv_name. Install new packages by doing /anaconda/envs/venv_name/bin/pip install package_name; or, simply run pip install package_name. This should now successfully install packages using that virtual environment's pip!


Instruction:
How to repeatedly execute a function every x seconds?

I want to repeatedly execute a function in Python every 60 seconds forever (just like an NSTimer in Objective C or setTimeout in JS). This code will run as a daemon and is effectively like calling the python script every minute using a cron, but without requiring that to be set up by the user. In this question about a cron implemented in Python, the solution appears to effectively just sleep() for x seconds. I don't need such advanced functionality so perhaps something like this would work while True: # Code executed here time.sleep(60) Are there any foreseeable problems with this code?

Output: If your program doesn't have a event loop already, use the sched module, which implements a general purpose event scheduler. import sched, time def do_something(scheduler): # schedule the next call first scheduler.enter(60, 1, do_something, (scheduler,)) print("Doing stuff...") # then do your stuff my_scheduler = sched.scheduler(time.time, time.sleep) my_scheduler.enter(60, 1, do_something, (my_scheduler,)) my_scheduler.run() If you're already using an event loop library like asyncio, trio, tkinter, PyQt5, gobject, kivy, and many others - just schedule the task using your existing event loop library's methods, instead.


Instruction:
How to remove axis, legends, and white padding

I would like to apply colormap to an image, and write the resulting image, without using axes, labels, titles, or anything automatically added by matplotlib. Here is what I did: def make_image(inputname,outputname): data = mpimg.imread(inputname)[:,:,0] fig = plt.imshow(data) fig.set_cmap('hot') fig.axes.get_xaxis().set_visible(False) fig.axes.get_yaxis().set_visible(False) plt.savefig(outputname) It successfully removes the axis of the figure, but the figure saved, presents a white padding, and a frame around the actual image. How can I remove them (at least the white padding)?

Output: The axis('off') method resolves one of the problems more succinctly than separately changing each axis and border. It still leaves the white space around the border however. Adding bbox_inches='tight' to the savefig command almost gets you there; you can see in the example below that the white space left is much smaller, but still present. Newer versions of matplotlib may require bbox_inches=0 instead of the string 'tight' (via @episodeyang and @kadrach) from numpy import random import matplotlib.pyplot as plt data = random.random((5,5)) img = plt.imshow(data, interpolation='nearest') img.set_cmap('hot') plt.axis('off') plt.savefig("test.png", bbox_inches='tight')


Instruction:
What are data classes and how are they different from common classes?

PEP 557 introduces data classes into the Python standard library. It says that by applying the @dataclass decorator shown below, it will generate "among other things, an __init__()". from dataclasses import dataclass @dataclass class InventoryItem: """Class for keeping track of an item in inventory.""" name: str unit_price: float quantity_on_hand: int = 0 def total_cost(self) -> float: return self.unit_price * self.quantity_on_hand It also says dataclasses are "mutable namedtuples with default", but I don't understand what this means, nor how data classes are different from common classes. What are data classes and when is it best to use them?

Output: Data classes are just regular classes that are geared towards storing state, rather than containing a lot of logic. Every time you create a class that mostly consists of attributes, you make a data class. What the dataclasses module does is to make it easier to create data classes. It takes care of a lot of boilerplate for you. This is especially useful when your data class must be hashable; because this requires a __hash__ method as well as an __eq__ method. If you add a custom __repr__ method for ease of debugging, that can become quite verbose: class InventoryItem: '''Class for keeping track of an item in inventory.''' name: str unit_price: float quantity_on_hand: int = 0 def __init__( self, name: str, unit_price: float, quantity_on_hand: int = 0 ) -> None: self.name = name self.unit_price = unit_price self.quantity_on_hand = quantity_on_hand def total_cost(self) -> float: return self.unit_price * self.quantity_on_hand def __repr__(self) -> str: return ( 'InventoryItem(' f'name={self.name!r}, unit_price={self.unit_price!r}, ' f'quantity_on_hand={self.quantity_on_hand!r})' ) def __hash__(self) -> int: return hash((self.name, self.unit_price, self.quantity_on_hand)) def __eq__(self, other) -> bool: if not isinstance(other, InventoryItem): return NotImplemented return ( (self.name, self.unit_price, self.quantity_on_hand) == (other.name, other.unit_price, other.quantity_on_hand)) With dataclasses you can reduce it to: from dataclasses import dataclass @dataclass(unsafe_hash=True) class InventoryItem: '''Class for keeping track of an item in inventory.''' name: str unit_price: float quantity_on_hand: int = 0 def total_cost(self) -> float: return self.unit_price * self.quantity_on_hand (Example based on the PEP example). The same class decorator can also generate comparison methods (__lt__, __gt__, etc.) and handle immutability. namedtuple classes are also data classes, but are immutable by default (as well as being sequences). dataclasses are much more flexible in this regard, and can easily be structured such that they can fill the same role as a namedtuple class. The PEP was inspired by the attrs project, which can do even more (including slots, validators, converters, metadata, etc.). If you want to see some examples, I recently used dataclasses for several of my Advent of Code solutions, see the solutions for day 7, day 8, day 11 and day 20. If you want to use dataclasses module in Python versions < 3.7, then you could install the backported module (requires 3.6) or use the attrs project mentioned above.


Instruction:
Error after upgrading pip: cannot import name 'main'

Whenever I am trying to install any package using pip, I am getting this import error: guru@guru-notebook:~$ pip3 install numpy Traceback (most recent call last): File "/usr/bin/pip3", line 9, in <module> from pip import main ImportError: cannot import name 'main' guru@guru-notebook:~$ cat `which pip3` #!/usr/bin/python3 # GENERATED BY DEBIAN import sys # Run the main entry point, similarly to how setuptools does it, but because # we didn't install the actual entry point from setup.py, don't use the # pkg_resources API. from pip import main if __name__ == '__main__': sys.exit(main()) It was working fine earlier, I am not sure why it is throwing this error. I have searched about this error, but can't find anything to fix it. Please let me know if you need any further detail, I will update my question.

Output: You must have inadvertently upgraded your system pip (probably through something like sudo pip install pip --upgrade) pip 10.x adjusts where its internals are situated. The pip3 command you're seeing is one provided by your package maintainer (presumably debian based here?) and is not a file managed by pip. You can read more about this on pip's issue tracker You'll probably want to not upgrade your system pip and instead use a virtualenv. To recover the pip3 binary you'll need to sudo python3 -m pip uninstall pip && sudo apt install python3-pip --reinstall If you want to continue in "unsupported territory" (upgrading a system package outside of the system package manager), you can probably get away with python3 -m pip ... instead of pip3.


Instruction:
How do I run all Python unit tests in a directory?

I have a directory that contains my Python unit tests. Each unit test module is of the form test_*.py. I am attempting to make a file called all_test.py that will, you guessed it, run all files in the aforementioned test form and return the result. I have tried two methods so far; both have failed. I will show the two methods, and I hope someone out there knows how to actually do this correctly. For my first valiant attempt, I thought "If I just import all my testing modules in the file, and then call this unittest.main() doodad, it will work, right?" Well, turns out I was wrong. import glob import unittest testSuite = unittest.TestSuite() test_file_strings = glob.glob('test_*.py') module_strings = [str[0:len(str)-3] for str in test_file_strings] if __name__ == "__main__": unittest.main() This did not work, the result I got was: $ python all_test.py ---------------------------------------------------------------------- Ran 0 tests in 0.000s OK For my second try, I though, ok, maybe I will try to do this whole testing thing in a more "manual" fashion. So I attempted to do that below: import glob import unittest testSuite = unittest.TestSuite() test_file_strings = glob.glob('test_*.py') module_strings = [str[0:len(str)-3] for str in test_file_strings] [__import__(str) for str in module_strings] suites = [unittest.TestLoader().loadTestsFromName(str) for str in module_strings] [testSuite.addTest(suite) for suite in suites] print testSuite result = unittest.TestResult() testSuite.run(result) print result #Ok, at this point I have a result #How do I display it as the normal unit test command line output? if __name__ == "__main__": unittest.main() This also did not work, but it seems so close! $ python all_test.py <unittest.TestSuite tests=[<unittest.TestSuite tests=[<unittest.TestSuite tests=[<test_main.TestMain testMethod=test_respondes_to_get>]>]>]> <unittest.TestResult run=1 errors=0 failures=0> ---------------------------------------------------------------------- Ran 0 tests in 0.000s OK I seem to have a suite of some sort, and I can execute the result. I am a little concerned about the fact that it says I have only run=1, seems like that should be run=2, but it is progress. But how do I pass and display the result to main? Or how do I basically get it working so I can just run this file, and in doing so, run all the unit tests in this directory?

Output: With Python 2.7 and higher you don't have to write new code or use third-party tools to do this; recursive test execution via the command line is built-in. Put an __init__.py in your test directory and: python -m unittest discover <test_directory> # or python -m unittest discover -s <directory> -p '*_test.py' You can read more in the python 2.7 or python 3.x unittest documentation.


Instruction:
django order_by query set, ascending and descending

How can I order by descending my query set in django by date? Reserved.objects.all().filter(client=client_id).order_by('check_in') I just want to filter from descending all the Reserved by check_in date.

Output: Reserved.objects.filter(client=client_id).order_by('-check_in') Notice the - before check_in. - before column name mean "descending order", while without - mean "ascending". Django Documentation


Instruction:
Convert Pandas Column to DateTime

I have one field in a pandas DataFrame that was imported as string format. It should be a datetime variable. How do I convert it to a datetime column, and then filter based on date? Example: raw_data = pd.DataFrame({'Mycol': ['05SEP2014:00:00:00.000']})

Output: Use the to_datetime function, specifying a format to match your data. df['Mycol'] = pd.to_datetime(df['Mycol'], format='%d%b%Y:%H:%M:%S.%f')


Instruction:
Find nearest value in numpy array

How do I find the nearest value in a numpy array? Example: np.find_nearest(array, value)

Output: import numpy as np def find_nearest(array, value): array = np.asarray(array) idx = (np.abs(array - value)).argmin() return array[idx] Example usage: array = np.random.random(10) print(array) # [ 0.21069679 0.61290182 0.63425412 0.84635244 0.91599191 0.00213826 # 0.17104965 0.56874386 0.57319379 0.28719469] print(find_nearest(array, value=0.5)) # 0.568743859261


Instruction:
How do I convert a PIL Image into a NumPy array?

How do I convert a PIL Image back and forth to a NumPy array so that I can do faster pixel-wise transformations than PIL's PixelAccess allows? I can convert it to a NumPy array via: pic = Image.open("foo.jpg") pix = numpy.array(pic.getdata()).reshape(pic.size[0], pic.size[1], 3) But how do I load it back into the PIL Image after I've modified the array? pic.putdata() isn't working well.

Output: You're not saying how exactly putdata() is not behaving. I'm assuming you're doing >>> pic.putdata(a) Traceback (most recent call last): File "...blablabla.../PIL/Image.py", line 1185, in putdata self.im.putdata(data, scale, offset) SystemError: new style getargs format but argument is not a tuple This is because putdata expects a sequence of tuples and you're giving it a numpy array. This >>> data = list(tuple(pixel) for pixel in pix) >>> pic.putdata(data) will work but it is very slow. As of PIL 1.1.6, the "proper" way to convert between images and numpy arrays is simply >>> pix = numpy.array(pic) although the resulting array is in a different format than yours (3-d array or rows/columns/rgb in this case). Then, after you make your changes to the array, you should be able to do either pic.putdata(pix) or create a new image with Image.fromarray(pix).


Instruction:
Pretty-print a NumPy array without scientific notation and with given precision

How do I print formatted NumPy arrays in a way similar to this: x = 1.23456 print('%.3f' % x) If I want to print the numpy.ndarray of floats, it prints several decimals, often in 'scientific' format, which is rather hard to read even for low-dimensional arrays. However, numpy.ndarray apparently has to be printed as a string, i.e., with %s. Is there a solution for this?

Output: Use numpy.set_printoptions to set the precision of the output: import numpy as np x = np.random.random(10) print(x) # [ 0.07837821 0.48002108 0.41274116 0.82993414 0.77610352 0.1023732 # 0.51303098 0.4617183 0.33487207 0.71162095] np.set_printoptions(precision=3) print(x) # [ 0.078 0.48 0.413 0.83 0.776 0.102 0.513 0.462 0.335 0.712] And suppress suppresses the use of scientific notation for small numbers: y = np.array([1.5e-10, 1.5, 1500]) print(y) # [ 1.500e-10 1.500e+00 1.500e+03] np.set_printoptions(suppress=True) print(y) # [ 0. 1.5 1500. ] To apply print options locally, using NumPy 1.15.0 or later, you could use the numpy.printoptions context manager. For example, inside the with-suite precision=3 and suppress=True are set: x = np.random.random(10) with np.printoptions(precision=3, suppress=True): print(x) # [ 0.073 0.461 0.689 0.754 0.624 0.901 0.049 0.582 0.557 0.348] But outside the with-suite the print options are back to default settings: print(x) # [ 0.07334334 0.46132615 0.68935231 0.75379645 0.62424021 0.90115836 # 0.04879837 0.58207504 0.55694118 0.34768638] If you are using an earlier version of NumPy, you can create the context manager yourself. For example, import numpy as np import contextlib @contextlib.contextmanager def printoptions(*args, **kwargs): original = np.get_printoptions() np.set_printoptions(*args, **kwargs) try: yield finally: np.set_printoptions(**original) x = np.random.random(10) with printoptions(precision=3, suppress=True): print(x) # [ 0.073 0.461 0.689 0.754 0.624 0.901 0.049 0.582 0.557 0.348] To prevent zeros from being stripped from the end of floats: np.set_printoptions now has a formatter parameter which allows you to specify a format function for each type. np.set_printoptions(formatter={'float': '{: 0.3f}'.format}) print(x) which prints [ 0.078 0.480 0.413 0.830 0.776 0.102 0.513 0.462 0.335 0.712] instead of [ 0.078 0.48 0.413 0.83 0.776 0.102 0.513 0.462 0.335 0.712]


Instruction:
Why do I get AttributeError: 'NoneType' object has no attribute 'something'?

I got this error: AttributeError: 'NoneType' object has no attribute 'something' What general scenarios might cause such an AttributeError, and how can I identify the problem? This is a special case of AttributeErrors. It merits separate treatment because there are a lot of ways to get an unexpected None value from the code, so it's typically a different problem; for other AttributeErrors, the problem might just as easily be the attribute name. See also What is a None value? and What is a 'NoneType' object? for an understanding of None and its type, NoneType.

Output: NoneType means that instead of an instance of whatever Class or Object you think you're working with, you've actually got None. That usually means that an assignment or function call up above failed or returned an unexpected result.


Instruction:
Convert Django Model object to dict with all of the fields intact

How does one convert a django Model object to a dict with all of its fields? All ideally includes foreign keys and fields with editable=False. Let me elaborate. Let's say I have a django model like the following: from django.db import models class OtherModel(models.Model): pass class SomeModel(models.Model): normal_value = models.IntegerField() readonly_value = models.IntegerField(editable=False) auto_now_add = models.DateTimeField(auto_now_add=True) foreign_key = models.ForeignKey(OtherModel, related_name="ref1") many_to_many = models.ManyToManyField(OtherModel, related_name="ref2") In the terminal, I have done the following: other_model = OtherModel() other_model.save() instance = SomeModel() instance.normal_value = 1 instance.readonly_value = 2 instance.foreign_key = other_model instance.save() instance.many_to_many.add(other_model) instance.save() I want to convert this to the following dictionary: {'auto_now_add': datetime.datetime(2015, 3, 16, 21, 34, 14, 926738, tzinfo=<UTC>), 'foreign_key': 1, 'id': 1, 'many_to_many': [1], 'normal_value': 1, 'readonly_value': 2} Questions with unsatisfactory answers: Django: Converting an entire set of a Model's objects into a single dictionary How can I turn Django Model objects into a dictionary and still have their foreign keys?

Output: There are many ways to convert an instance to a dictionary, with varying degrees of corner case handling and closeness to the desired result. 1. instance.__dict__ instance.__dict__ which returns {'_foreign_key_cache': <OtherModel: OtherModel object>, '_state': <django.db.models.base.ModelState at 0x7ff0993f6908>, 'auto_now_add': datetime.datetime(2018, 12, 20, 21, 34, 29, 494827, tzinfo=<UTC>), 'foreign_key_id': 2, 'id': 1, 'normal_value': 1, 'readonly_value': 2} This is by far the simplest, but is missing many_to_many, foreign_key is misnamed, and it has two unwanted extra things in it. 2. model_to_dict from django.forms.models import model_to_dict model_to_dict(instance) which returns {'foreign_key': 2, 'id': 1, 'many_to_many': [<OtherModel: OtherModel object>], 'normal_value': 1} This is the only one with many_to_many, but is missing the uneditable fields. 3. model_to_dict(..., fields=...) from django.forms.models import model_to_dict model_to_dict(instance, fields=[field.name for field in instance._meta.fields]) which returns {'foreign_key': 2, 'id': 1, 'normal_value': 1} This is strictly worse than the standard model_to_dict invocation. 4. query_set.values() SomeModel.objects.filter(id=instance.id).values()[0] which returns {'auto_now_add': datetime.datetime(2018, 12, 20, 21, 34, 29, 494827, tzinfo=<UTC>), 'foreign_key_id': 2, 'id': 1, 'normal_value': 1, 'readonly_value': 2} This is the same output as instance.__dict__ but without the extra fields. foreign_key_id is still wrong and many_to_many is still missing. 5. Custom Function The code for django's model_to_dict had most of the answer. It explicitly removed non-editable fields, so removing that check and getting the ids of foreign keys for many to many fields results in the following code which behaves as desired: from itertools import chain def to_dict(instance): opts = instance._meta data = {} for f in chain(opts.concrete_fields, opts.private_fields): data[f.name] = f.value_from_object(instance) for f in opts.many_to_many: data[f.name] = [i.id for i in f.value_from_object(instance)] return data While this is the most complicated option, calling to_dict(instance) gives us exactly the desired result: {'auto_now_add': datetime.datetime(2018, 12, 20, 21, 34, 29, 494827, tzinfo=<UTC>), 'foreign_key': 2, 'id': 1, 'many_to_many': [2], 'normal_value': 1, 'readonly_value': 2} 6. Use Serializers Django Rest Framework's ModelSerializer allows you to build a serializer automatically from a model. from rest_framework import serializers class SomeModelSerializer(serializers.ModelSerializer): class Meta: model = SomeModel fields = "__all__" SomeModelSerializer(instance).data returns {'auto_now_add': '2018-12-20T21:34:29.494827Z', 'foreign_key': 2, 'id': 1, 'many_to_many': [2], 'normal_value': 1, 'readonly_value': 2} This is almost as good as the custom function, but auto_now_add is a string instead of a datetime object. Bonus Round: better model printing If you want a django model that has a better python command-line display, have your models child-class the following: from django.db import models from itertools import chain class PrintableModel(models.Model): def __repr__(self): return str(self.to_dict()) def to_dict(instance): opts = instance._meta data = {} for f in chain(opts.concrete_fields, opts.private_fields): data[f.name] = f.value_from_object(instance) for f in opts.many_to_many: data[f.name] = [i.id for i in f.value_from_object(instance)] return data class Meta: abstract = True So, for example, if we define our models as such: class OtherModel(PrintableModel): pass class SomeModel(PrintableModel): normal_value = models.IntegerField() readonly_value = models.IntegerField(editable=False) auto_now_add = models.DateTimeField(auto_now_add=True) foreign_key = models.ForeignKey(OtherModel, related_name="ref1") many_to_many = models.ManyToManyField(OtherModel, related_name="ref2") Calling SomeModel.objects.first() now gives output like this: {'auto_now_add': datetime.datetime(2018, 12, 20, 21, 34, 29, 494827, tzinfo=<UTC>), 'foreign_key': 2, 'id': 1, 'many_to_many': [2], 'normal_value': 1, 'readonly_value': 2}


Instruction:
Convert DataFrame column type from string to datetime

How can I convert a DataFrame column of strings (in dd/mm/yyyy format) to datetime dtype?

Output: The easiest way is to use to_datetime: df['col'] = pd.to_datetime(df['col']) It also offers a dayfirst argument for European times (but beware this isn't strict). Here it is in action: In [11]: pd.to_datetime(pd.Series(['05/23/2005'])) Out[11]: 0 2005-05-23 00:00:00 dtype: datetime64[ns] You can pass a specific format: In [12]: pd.to_datetime(pd.Series(['05/23/2005']), format="%m/%d/%Y") Out[12]: 0 2005-05-23 dtype: datetime64[ns]


Instruction:
How to reversibly store and load a Pandas dataframe to/from disk

Right now I'm importing a fairly large CSV as a dataframe every time I run the script. Is there a good solution for keeping that dataframe constantly available in between runs so I don't have to spend all that time waiting for the script to run?

Output: The easiest way is to pickle it using to_pickle: df.to_pickle(file_name) # where to save it, usually as a .pkl Then you can load it back using: df = pd.read_pickle(file_name) Note: before 0.11.1 save and load were the only way to do this (they are now deprecated in favor of to_pickle and read_pickle respectively). Another popular choice is to use HDF5 (pytables) which offers very fast access times for large datasets: import pandas as pd store = pd.HDFStore('store.h5') store['df'] = df # save it store['df'] # load it More advanced strategies are discussed in the cookbook. Since 0.13 there's also msgpack which may be be better for interoperability, as a faster alternative to JSON, or if you have python object/text-heavy data (see this question).


Instruction:
Remove pandas rows with duplicate indices

How to remove rows with duplicate index values? In the weather DataFrame below, sometimes a scientist goes back and corrects observations -- not by editing the erroneous rows, but by appending a duplicate row to the end of a file. I'm reading some automated weather data from the web (observations occur every 5 minutes, and compiled into monthly files for each weather station.) After parsing a file, the DataFrame looks like: Sta Precip1hr Precip5min Temp DewPnt WindSpd WindDir AtmPress Date 2001-01-01 00:00:00 KPDX 0 0 4 3 0 0 30.31 2001-01-01 00:05:00 KPDX 0 0 4 3 0 0 30.30 2001-01-01 00:10:00 KPDX 0 0 4 3 4 80 30.30 2001-01-01 00:15:00 KPDX 0 0 3 2 5 90 30.30 2001-01-01 00:20:00 KPDX 0 0 3 2 10 110 30.28 Example of a duplicate case: import pandas as pd import datetime startdate = datetime.datetime(2001, 1, 1, 0, 0) enddate = datetime.datetime(2001, 1, 1, 5, 0) index = pd.date_range(start=startdate, end=enddate, freq='H') data1 = {'A' : range(6), 'B' : range(6)} data2 = {'A' : [20, -30, 40], 'B' : [-50, 60, -70]} df1 = pd.DataFrame(data=data1, index=index) df2 = pd.DataFrame(data=data2, index=index[:3]) df3 = df2.append(df1) df3 A B 2001-01-01 00:00:00 20 -50 2001-01-01 01:00:00 -30 60 2001-01-01 02:00:00 40 -70 2001-01-01 03:00:00 3 3 2001-01-01 04:00:00 4 4 2001-01-01 05:00:00 5 5 2001-01-01 00:00:00 0 0 2001-01-01 01:00:00 1 1 2001-01-01 02:00:00 2 2 And so I need df3 to eventually become: A B 2001-01-01 00:00:00 0 0 2001-01-01 01:00:00 1 1 2001-01-01 02:00:00 2 2 2001-01-01 03:00:00 3 3 2001-01-01 04:00:00 4 4 2001-01-01 05:00:00 5 5 I thought that adding a column of row numbers (df3['rownum'] = range(df3.shape[0])) would help me select the bottom-most row for any value of the DatetimeIndex, but I am stuck on figuring out the group_by or pivot (or ???) statements to make that work.

Output: I would suggest using the duplicated method on the Pandas Index itself: df3 = df3.loc[~df3.index.duplicated(keep='first'), :] While all the other methods work, .drop_duplicates is by far the least performant for the provided example. Furthermore, while the groupby method is only slightly less performant, I find the duplicated method to be more readable. Using the sample data provided: >>> %timeit df3.reset_index().drop_duplicates(subset='index', keep='first').set_index('index') 1000 loops, best of 3: 1.54 ms per loop >>> %timeit df3.groupby(df3.index).first() 1000 loops, best of 3: 580 µs per loop >>> %timeit df3.loc[~df3.index.duplicated(keep='first'), :] 1000 loops, best of 3: 307 µs per loop Note that you can keep the last element by changing the keep argument to 'last'. It should also be noted that this method works with MultiIndex as well (using df1 as specified in Paul's example): >>> %timeit df1.groupby(level=df1.index.names).last() 1000 loops, best of 3: 771 µs per loop >>> %timeit df1.loc[~df1.index.duplicated(keep='last'), :] 1000 loops, best of 3: 365 µs per loop Edit: While the .loc is not necessary (per @lingjiankong's comment) I agree with @shadowtalker that being explicit rather than implicit about row selection can be helpful (especially in large codebases).


Instruction:
How can I find where Python is installed on Windows?

I want to find out my Python installation path on Windows. For example: C:\Python25 How can I find where Python is installed?

Output: In your Python interpreter, type the following commands: >>> import os, sys >>> os.path.dirname(sys.executable) 'C:\\Python25' Also, you can club all these and use a single line command. Open cmd and enter following command python -c "import os, sys; print(os.path.dirname(sys.executable))"


Instruction:
What are logits? What is the difference between softmax and softmax_cross_entropy_with_logits?

In the tensorflow API docs they use a keyword called logits. What is it? A lot of methods are written like: tf.nn.softmax(logits, name=None) If logits is just a generic Tensor input, why is it named logits? Secondly, what is the difference between the following two methods? tf.nn.softmax(logits, name=None) tf.nn.softmax_cross_entropy_with_logits(logits, labels, name=None) I know what tf.nn.softmax does, but not the other. An example would be really helpful.

Output: The softmax+logits simply means that the function operates on the unscaled output of earlier layers and that the relative scale to understand the units is linear. It means, in particular, the sum of the inputs may not equal 1, that the values are not probabilities (you might have an input of 5). Internally, it first applies softmax to the unscaled output, and then computes the cross entropy of those values vs. what they "should" be as defined by the labels. tf.nn.softmax produces the result of applying the softmax function to an input tensor. The softmax "squishes" the inputs so that sum(input) = 1, and it does the mapping by interpreting the inputs as log-probabilities (logits) and then converting them back into raw probabilities between 0 and 1. The shape of output of a softmax is the same as the input: a = tf.constant(np.array([[.1, .3, .5, .9]])) print s.run(tf.nn.softmax(a)) [[ 0.16838508 0.205666 0.25120102 0.37474789]] See this answer for more about why softmax is used extensively in DNNs. tf.nn.softmax_cross_entropy_with_logits combines the softmax step with the calculation of the cross-entropy loss after applying the softmax function, but it does it all together in a more mathematically careful way. It's similar to the result of: sm = tf.nn.softmax(x) ce = cross_entropy(sm) The cross entropy is a summary metric: it sums across the elements. The output of tf.nn.softmax_cross_entropy_with_logits on a shape [2,5] tensor is of shape [2,1] (the first dimension is treated as the batch). If you want to do optimization to minimize the cross entropy AND you're softmaxing after your last layer, you should use tf.nn.softmax_cross_entropy_with_logits instead of doing it yourself, because it covers numerically unstable corner cases in the mathematically right way. Otherwise, you'll end up hacking it by adding little epsilons here and there. Edited 2016-02-07: If you have single-class labels, where an object can only belong to one class, you might now consider using tf.nn.sparse_softmax_cross_entropy_with_logits so that you don't have to convert your labels to a dense one-hot array. This function was added after release 0.6.0.


Instruction:
Extract a subset of key-value pairs from dictionary?

I have a big dictionary object that has several key value pairs (about 16), but I am only interested in 3 of them. What is the best way (shortest/efficient/most elegant) to subset such dictionary? The best I know is: bigdict = {'a':1,'b':2,....,'z':26} subdict = {'l':bigdict['l'], 'm':bigdict['m'], 'n':bigdict['n']} I am sure there is a more elegant way than this.

Output: You could try: {k: bigdict[k] for k in ('l', 'm', 'n')} Or in Python versions before 2.7: dict((k, bigdict[k]) for k in ('l', 'm', 'n')) I'm assuming that you know the keys are going to be in the dictionary. See the answer by Håvard S if you don't. Alternatively, as timbo points out in the comments, if you want a key that's missing in bigdict to map to None, you can do: {k: bigdict.get(k, None) for k in ('l', 'm', 'n')} If you're using Python 3, and you only want keys in the new dict that actually exist in the original one, you can use the fact to view objects implement some set operations: {k: bigdict[k] for k in bigdict.keys() & {'l', 'm', 'n'}}


Instruction:
Why do many examples use `fig, ax = plt.subplots()`

I'm learning to use matplotlib by studying examples, and a lot of examples seem to include a line like the following before creating a single plot... fig, ax = plt.subplots() Here are some examples... Modify tick label text http://matplotlib.org/examples/pylab_examples/boxplot_demo2.html I see this function used a lot, even though the example is only attempting to create a single chart. Is there some other advantage? The official demo for subplots() also uses f, ax = subplots when creating a single chart, and it only ever references ax after that. This is the code they use. # Just a figure and one subplot f, ax = plt.subplots() ax.plot(x, y) ax.set_title('Simple plot')

Output: plt.subplots() is a function that returns a tuple containing a figure and axes object(s). Thus when using fig, ax = plt.subplots() you unpack this tuple into the variables fig and ax. Having fig is useful if you want to change figure-level attributes or save the figure as an image file later (e.g. with fig.savefig('yourfilename.png')). You certainly don't have to use the returned figure object but many people do use it later so it's common to see. Also, all axes objects (the objects that have plotting methods), have a parent figure object anyway, thus: fig, ax = plt.subplots() is more concise than this: fig = plt.figure() ax = fig.add_subplot(111)


Instruction:
How to generate an MD5 checksum of a file in Python?

Is there any simple way of generating (and checking) MD5 checksums of a list of files in Python? (I have a small program I'm working on, and I'd like to confirm the checksums of the files).

Output: You can use hashlib.md5() Note that sometimes you won't be able to fit the whole file in memory. In that case, you'll have to read chunks sequentially and feed them to the md5 update method. In this example I'm using a 4096-byte (4 KiB) buffer, but you can adjust this size import hashlib def md5(fname): hash_md5 = hashlib.md5() with open(fname, "rb") as f: for chunk in iter(lambda: f.read(4096), b""): hash_md5.update(chunk) return hash_md5.hexdigest() Note: hash_md5.hexdigest() will return the hex string representation for the digest, if you just need the packed bytes use return hash_md5.digest(), so you don't have to convert back.


Instruction:
How to replace multiple substrings of a string?

I use the .replace function to replace multiple strings: string.replace("condition1", "").replace("condition2", "text") although that feels like bad syntax what is the proper way? like how in grep/regex you can do \1 and \2 to replace fields to certain search strings

Output: Here is a short example that should do the trick with regular expressions: import re rep = {"condition1": "", "condition2": "text"} # define desired replacements here # use these three lines to do the replacement rep = dict((re.escape(k), v) for k, v in rep.items()) pattern = re.compile("|".join(rep.keys())) text = pattern.sub(lambda m: rep[re.escape(m.group(0))], text) For example: >>> pattern.sub(lambda m: rep[re.escape(m.group(0))], "(condition1) and --condition2--") '() and --text--'


Instruction:
How to use the timeit module?

How do I use timeit to compare the performance of my own functions such as insertion_sort and tim_sort?

Output: The way timeit works is to run setup code once and then make repeated calls to a series of statements. So, if you want to test sorting, some care is required so that one pass at an in-place sort doesn't affect the next pass with already sorted data (that, of course, would make the Timsort really shine because it performs best when the data already partially ordered). Here is an example of how to set up a test for sorting: >>> import timeit >>> setup = ''' import random random.seed('slartibartfast') s = [random.random() for i in range(1000)] timsort = list.sort ''' >>> print(min(timeit.Timer('a=s[:]; timsort(a)', setup=setup).repeat(7, 1000))) 0.04485079200821929 Note that the series of statements makes a fresh copy of the unsorted data on every pass. Also, note the timing technique of running the measurement suite seven times and keeping only the best time — this can really help reduce measurement distortions due to other processes running on your system.


Instruction:
Getting the name of a variable as a string

I already read How to get a function name as a string?. How can I do the same for a variable? As opposed to functions, Python variables do not have the __name__ attribute. In other words, if I have a variable such as: foo = dict() foo['bar'] = 2 I am looking for a function/attribute, e.g. retrieve_name, where: >>> retrieve_name(foo) 'foo' This is in order to create a DataFrame in Pandas from this list, where the column names are given by the names of the actual dictionaries: # List of dictionaries for my DataFrame list_of_dicts = [n_jobs, users, queues, priorities] columns = [retrieve_name(d) for d in list_of_dicts]

Output: TL;DR Use the Wrapper helper from python-varname: from varname.helpers import Wrapper foo = Wrapper(dict()) # foo.name == 'foo' # foo.value == {} foo.value['bar'] = 2 For list comprehension part, you can do: n_jobs = Wrapper(<original_value>) users = Wrapper(<original_value>) queues = Wrapper(<original_value>) priorities = Wrapper(<original_value>) list_of_dicts = [n_jobs, users, queues, priorities] columns = [d.name for d in list_of_dicts] # ['n_jobs', 'users', 'queues', 'priorities'] # REMEMBER that you have to access the <original_value> by d.value I am the author of the python-varname package. Please let me know if you have any questions or you can submit issues on Github. The long answer Is it even possible? Yes and No. We are retrieving the variable names at runtime, so we need a function to be called to enable us to access the previous frames to retrieve the variable names. That's why we need a Wrapper there. In that function, at runtime, we are parsing the source code/AST nodes in the previous frames to get the exact variable name. However, the source code/AST nodes in the previous frames are not always available, or they could be modified by other environments (e.g: pytest's assert statement). One simple example is that the codes run via exec(). Even though we are still able to retrieve some information from the bytecode, it needs too much effort and it is also error-prone. How to do it? First of all, we need to identify which frame the variable is given. It's not always simply the direct previous frame. For example, we may have another wrapper for the function: from varname import varname def func(): return varname() def wrapped(): return func() x = wrapped() In the above example, we have to skip the frame inside wrapped to get to the right frame x = wrapped() so that we are able to locate x. The arguments frame and ignore of varname allow us to skip some of these intermediate frames. See more details in the README file and the API docs of the package. Then we need to parse the AST node to locate where the variable is assigned value (function call) to. It's not always just a simple assignment. Sometimes there could be complex AST nodes, for example, x = [wrapped()]. We need to identify the correct assignment by traversing the AST tree. How reliable is it? Once we identify the assignment node, it is reliable. varname is all depending on executing package to look for the node. The node executing detects is ensured to be the correct one (see also this). It partially works with environments where other AST magics apply, including pytest, ipython, macropy, birdseye, reticulate with R, etc. Neither executing nor varname is 100% working with those environments. Do we need a package to do it? Well, yes and no, again. If your scenario is simple, the code provided by @juan Isaza or @scohe001 probably is enough for you to work with the case where a variable is defined at the direct previous frame and the AST node is a simple assignment. You just need to go one frame back and retrieve the information there. However, if the scenario becomes complicated, or we need to adopt different application scenarios, you probably need a package like python-varname, to handle them. These scenarios may include to: present more friendly messages when the source code is not available or AST nodes are not accessible skip intermediate frames (allows the function to be wrapped or called in other intermediate frames) automatically ignores calls from built-in functions or libraries. For example: x = str(func()) retrieve multiple variable names on the left-hand side of the assignment etc. How about the f-string? Like the answer provided by @Aivar Paalberg. It's definitely fast and reliable. However, it's not at runtime, meaning that you have to know it's foo before you print the name out. But with varname, you don't have to know that variable is coming: from varname import varname def func(): return varname() # In external uses x = func() # 'x' y = func() # 'y' Finally python-varname is not only able to detect the variable name from an assignment, but also: Retrieve variable names directly, using nameof Detect next immediate attribute name, using will Fetch argument names/sources passed to a function using argname Read more from its documentation. However, the final word I want to say is that, try to avoid using it whenever you can. Because you can't make sure that the client code will run in an environment where the source node is available or AST node is accessible. And of course, it costs resources to parse the source code, identify the environment, retrieve the AST nodes and evaluate them when needed.


Instruction:
How can I see the raw SQL queries Django is running?

Is there a way to show the SQL that Django is running while performing a query?

Output: See the docs FAQ: "How can I see the raw SQL queries Django is running?" django.db.connection.queries contains a list of the SQL queries: from django.db import connection print(connection.queries) Querysets also have a query attribute containing the query to be executed: print(MyModel.objects.filter(name="my name").query) Note that the output of the query is not valid SQL, because: "Django never actually interpolates the parameters: it sends the query and the parameters separately to the database adapter, which performs the appropriate operations." From Django bug report #17741. Because of that, you should not send query output directly to a database. If you need to reset the queries to, for example, see how many queries are running in a given period, you can use reset_queries from django.db: from django.db import reset_queries from django.db import connection reset_queries() # Run your query here print(connection.queries) >>> []


Instruction:
Can I add comments to a pip requirements file?

I'd like to add comments for a few packages in a pip requirements file. (Just to explain why that package is on the list.) Can I do this? I'm imagining something like Babel==0.9.5 # translation CherryPy==3.2.0 # web server Creoleparser==0.7.1 # wiki formatting Genshi==0.5.1 # templating

Output: Sure, you can, just use # pip docs: A line that begins with # is treated as a comment and ignored. Whitespace followed by a # causes the # and the remainder of the line to be treated as a comment.


Instruction:
Total memory used by Python process?

Is there a way for a Python program to determine how much memory it's currently using? I've seen discussions about memory usage for a single object, but what I need is total memory usage for the process, so that I can determine when it's necessary to start discarding cached data.

Output: Here is a useful solution that works for various operating systems, including Linux, Windows, etc.: import psutil process = psutil.Process() print(process.memory_info().rss) # in bytes Notes: do pip install psutil if it is not installed yet handy one-liner if you quickly want to know how many MiB your process takes: import os, psutil; print(psutil.Process(os.getpid()).memory_info().rss / 1024 ** 2) with Python 2.7 and psutil 5.6.3, it was process.memory_info()[0] instead (there was a change in the API later).


Instruction:
How do I get the picture size with PIL?

How do I get a size of a pictures sides with PIL or any other Python library?

Output: from PIL import Image im = Image.open('whatever.png') width, height = im.size According to the documentation.


Instruction:
Turn a string into a valid filename?

I have a string that I want to use as a filename, so I want to remove all characters that wouldn't be allowed in filenames, using Python. I'd rather be strict than otherwise, so let's say I want to retain only letters, digits, and a small set of other characters like "_-.() ". What's the most elegant solution? The filename needs to be valid on multiple operating systems (Windows, Linux and Mac OS) - it's an MP3 file in my library with the song title as the filename, and is shared and backed up between 3 machines.

Output: You can look at the Django framework (but take their licence into account!) for how they create a "slug" from arbitrary text. A slug is URL- and filename- friendly. The Django text utils define a function, slugify(), that's probably the gold standard for this kind of thing. Essentially, their code is the following. import unicodedata import re def slugify(value, allow_unicode=False): """ Taken from https://github.com/django/django/blob/master/django/utils/text.py Convert to ASCII if 'allow_unicode' is False. Convert spaces or repeated dashes to single dashes. Remove characters that aren't alphanumerics, underscores, or hyphens. Convert to lowercase. Also strip leading and trailing whitespace, dashes, and underscores. """ value = str(value) if allow_unicode: value = unicodedata.normalize('NFKC', value) else: value = unicodedata.normalize('NFKD', value).encode('ascii', 'ignore').decode('ascii') value = re.sub(r'[^\w\s-]', '', value.lower()) return re.sub(r'[-\s]+', '-', value).strip('-_') And the older version: def slugify(value): """ Normalizes string, converts to lowercase, removes non-alpha characters, and converts spaces to hyphens. """ import unicodedata value = unicodedata.normalize('NFKD', value).encode('ascii', 'ignore') value = unicode(re.sub('[^\w\s-]', '', value).strip().lower()) value = unicode(re.sub('[-\s]+', '-', value)) # ... return value There's more, but I left it out, since it doesn't address slugification, but escaping.


Instruction:
How do I put a variable’s value inside a string (interpolate it into the string)?

I would like to put an int into a string. This is what I am doing at the moment: num = 40 plot.savefig('hanning40.pdf') #problem line I have to run the program for several different numbers, so I'd like to do a loop. But inserting the variable like this doesn't work: plot.savefig('hanning', num, '.pdf') How do I insert a variable into a Python string? See also If you are trying to create a file path, see How can I create a full path to a file from parts (e.g. path to the folder, name and extension)? for additional techniques. It will usually be better to use code that is specific to creating paths. If you are trying to construct an URL with variable data, do not use ordinary string formatting, because it is error-prone and more difficult than necessary. Specialized tools are available. See Add params to given URL in Python. If you are trying to construct a SQL query, do not use ordinary string formatting, because it is a major security risk. This is the cause of "SQL injection" which costs real companies huge amounts of money every year. See for example How to use variables in SQL statement in Python? for proper techniques. If you just want to print (output) the string, you can prepare it this way first, or if you don't need the string for anything else, print each piece of the output individually using a single call to print. See How can I print multiple things (fixed text and/or variable values) on the same line, all at once? for details on both approaches. See How can I concatenate str and int objects? for bugs caused by trying to use + to join "strings" when one of them isn't a string.

Output: plot.savefig('hanning(%d).pdf' % num) The % operator, when following a string, allows you to insert values into that string via format codes (the %d in this case). For more details, see the Python documentation: printf-style String Formatting


Instruction:
Why do I get "TypeError: Missing 1 required positional argument: 'self'"?

I have some code like: class Pump: def __init__(self): print("init") def getPumps(self): pass p = Pump.getPumps() print(p) But I get an error like: Traceback (most recent call last): File "C:\Users\Dom\Desktop\test\test.py", line 7, in <module> p = Pump.getPumps() TypeError: getPumps() missing 1 required positional argument: 'self' Why doesn't __init__ seem to be called, and what does this exception mean? My understanding is that self is passed to the constructor and methods automatically. What am I doing wrong here? See Why do I get 'takes exactly 1 argument (2 given)' when trying to call a method? for the opposite problem.

Output: To use the class, first create an instance, like so: p = Pump() p.getPumps() A full example: >>> class TestClass: ... def __init__(self): ... print("init") ... def testFunc(self): ... print("Test Func") ... >>> testInstance = TestClass() init >>> testInstance.testFunc() Test Func


Instruction:
What does "SyntaxError: Missing parentheses in call to 'print'" mean in Python?

When I try to use a print statement in Python, it gives me this error: >>> print "Hello, World!" File "<stdin>", line 1 print "Hello, World!" ^ SyntaxError: Missing parentheses in call to 'print' What does that mean? See Getting SyntaxError for print with keyword argument end=' ' for the opposite problem. See Python 3 print without parenthesis for workarounds, and confirmation that print cannot be made to work like a statement in Python 3.

Output: The error message SyntaxError: Missing parentheses in call to 'print' occurs when you attempt to use Python 3 syntax with the Python 2 print statement. Example: print "Hello, World!" In Python 3, the print statement was replaced with a print() function, requiring parentheses around the value to be printed. Solution print("Hello, World!") In Python 3, the print statement was replaced with a print() function, requiring parentheses around the value to be printed. >>> print("Hello, World!") Hello, World! In earlier versions of Python 3, the interpreter just reports a generic syntax error, without providing any useful hints as to what might be going wrong: >>> print "Hello, World!" File "<stdin>", line 1 print "Hello, World!" ^ SyntaxError: invalid syntax As for why print became an ordinary function in Python 3, that didn't relate to the basic form of the statement, but rather to how you did more complicated things like printing multiple items to stderr with a trailing space rather than ending the line. In Python 2: >>> import sys >>> print >> sys.stderr, 1, 2, 3,; print >> sys.stderr, 4, 5, 6 1 2 3 4 5 6 In Python 3: >>> import sys >>> print(1, 2, 3, file=sys.stderr, end=" "); print(4, 5, 6, file=sys.stderr) 1 2 3 4 5 6 Starting with the Python 3.6.3 release in September 2017, some error messages related to the Python 2.x print syntax have been updated to recommend their Python 3.x counterparts: >>> print "Hello!" File "<stdin>", line 1 print "Hello!" ^ SyntaxError: Missing parentheses in call to 'print'. Did you mean print("Hello!")? Since the "Missing parentheses in call to print" case is a compile time syntax error and hence has access to the raw source code, it's able to include the full text on the rest of the line in the suggested replacement. However, it doesn't currently try to work out the appropriate quotes to place around that expression (that's not impossible, just sufficiently complicated that it hasn't been done). The TypeError raised for the right shift operator has also been customised: >>> print >> sys.stderr Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: unsupported operand type(s) for >>: 'builtin_function_or_method' and '_io.TextIOWrapper'. Did you mean "print(<message>, file=<output_stream>)"? Since this error is raised when the code runs, rather than when it is compiled, it doesn't have access to the raw source code, and hence uses meta-variables (<message> and <output_stream>) in the suggested replacement expression instead of whatever the user actually typed. Unlike the syntax error case, it's straightforward to place quotes around the Python expression in the custom right shift error message.


Instruction:
How to delete items from a dictionary while iterating over it?

Can I delete items from a dictionary in Python while iterating over it? I want to remove elements that don't meet a certain condition from the dictionary, instead of creating an entirely new dictionary. Is the following a good solution, or are there better ways? for k, v in mydict.items(): if k == val: del mydict[k]

Output: For Python 3+: >>> mydict {'four': 4, 'three': 3, 'one': 1} >>> for k in list(mydict.keys()): ... if mydict[k] == 3: ... del mydict[k] >>> mydict {'four': 4, 'one': 1} The other answers work fine with Python 2 but raise a RuntimeError for Python 3: RuntimeError: dictionary changed size during iteration. This happens because mydict.keys() returns an iterator not a list. As pointed out in comments simply convert mydict.keys() to a list by list(mydict.keys()) and it should work. For Python 2: A simple test in the console shows you cannot modify a dictionary while iterating over it: >>> mydict = {'one': 1, 'two': 2, 'three': 3, 'four': 4} >>> for k, v in mydict.iteritems(): ... if k == 'two': ... del mydict[k] ------------------------------------------------------------ Traceback (most recent call last): File "<ipython console>", line 1, in <module> RuntimeError: dictionary changed size during iteration As stated in delnan's answer, deleting entries causes problems when the iterator tries to move onto the next entry. Instead, use the keys() method to get a list of the keys and work with that: >>> for k in mydict.keys(): ... if k == 'two': ... del mydict[k] >>> mydict {'four': 4, 'three': 3, 'one': 1} If you need to delete based on the items value, use the items() method instead: >>> for k, v in mydict.items(): ... if v == 3: ... del mydict[k] >>> mydict {'four': 4, 'one': 1}


Instruction:
How should I use the Optional type hint?

I'm trying to understand how to use the Optional type hint. From PEP-484, I know I can use Optional for def test(a: int = None) either as def test(a: Union[int, None]) or def test(a: Optional[int]). But how about following examples? def test(a : dict = None): #print(a) ==> {'a': 1234} #or #print(a) ==> None def test(a : list = None): #print(a) ==> [1,2,3,4, 'a', 'b'] #or #print(a) ==> None If Optional[type] seems to mean the same thing as Union[type, None], why should I use Optional[] at all?

Output: If your code is designed to work with Python 3.10 or newer, you want to use the PEP 604 syntax, using ... | None union syntax, and not use typing.Optional: def test(a: dict[Any, Any] | None = None) -> None: #print(a) ==> {'a': 1234} #or #print(a) ==> None def test(a: list[Any] | None = None) -> None: #print(a) ==> [1, 2, 3, 4, 'a', 'b'] #or #print(a) ==> None Code that still supports older Python versions can still stick to using Optional. Optional[...] is a shorthand notation for Union[..., None], telling the type checker that either an object of the specific type is required, or None is required. ... stands for any valid type hint, including complex compound types or a Union[] of more types. Whenever you have a keyword argument with default value None, you should use Optional. So for your two examples, you have dict and list container types, but the default value for the a keyword argument shows that None is permitted too so use Optional[...]: from typing import Optional def test(a: Optional[dict] = None) -> None: #print(a) ==> {'a': 1234} #or #print(a) ==> None def test(a: Optional[list] = None) -> None: #print(a) ==> [1, 2, 3, 4, 'a', 'b'] #or #print(a) ==> None There is technically no difference between using Optional[] on a Union[], or just adding None to the Union[]. So Optional[Union[str, int]] and Union[str, int, None] are exactly the same thing. Personally, I'd stick with always using Optional[] when setting the type for a keyword argument that uses = None to set a default value, this documents the reason why None is allowed better. Moreover, it makes it easier to move the Union[...] part into a separate type alias, or to later remove the Optional[...] part if an argument becomes mandatory. For example, say you have from typing import Optional, Union def api_function(optional_argument: Optional[Union[str, int]] = None) -> None: """Frob the fooznar. If optional_argument is given, it must be an id of the fooznar subwidget to filter on. The id should be a string, or for backwards compatibility, an integer is also accepted. """ then documentation is improved by pulling out the Union[str, int] into a type alias: from typing import Optional, Union # subwidget ids used to be integers, now they are strings. Support both. SubWidgetId = Union[str, int] def api_function(optional_argument: Optional[SubWidgetId] = None) -> None: """Frob the fooznar. If optional_argument is given, it must be an id of the fooznar subwidget to filter on. The id should be a string, or for backwards compatibility, an integer is also accepted. """ The refactor to move the Union[] into an alias was made all the much easier because Optional[...] was used instead of Union[str, int, None]. The None value is not a 'subwidget id' after all, it's not part of the value, None is meant to flag the absence of a value. Side note: Unless your code only has to support Python 3.9 or newer, you want to avoid using the standard library container types in type hinting, as you can't say anything about what types they must contain. So instead of dict and list, use typing.Dict and typing.List, respectively. And when only reading from a container type, you may just as well accept any immutable abstract container type; lists and tuples are Sequence objects, while dict is a Mapping type: from typing import Mapping, Optional, Sequence, Union def test(a: Optional[Mapping[str, int]] = None) -> None: """accepts an optional map with string keys and integer values""" # print(a) ==> {'a': 1234} # or # print(a) ==> None def test(a: Optional[Sequence[Union[int, str]]] = None) -> None: """accepts an optional sequence of integers and strings # print(a) ==> [1, 2, 3, 4, 'a', 'b'] # or # print(a) ==> None In Python 3.9 and up, the standard container types have all been updated to support using them in type hints, see PEP 585. But, while you now can use dict[str, int] or list[Union[int, str]], you still may want to use the more expressive Mapping and Sequence annotations to indicate that a function won't be mutating the contents (they are treated as 'read only'), and that the functions would work with any object that works as a mapping or sequence, respectively. Python 3.10 introduces the | union operator into type hinting, see PEP 604. Instead of Union[str, int] you can write str | int. In line with other type-hinted languages, the preferred (and more concise) way to denote an optional argument in Python 3.10 and up, is now Type | None, e.g. str | None or list | None.