Instruction:
UnboundLocalError trying to use a variable (supposed to be global) that is (re)assigned (even after first use)

When I try this code: a, b, c = (1, 2, 3) def test(): print(a) print(b) print(c) c += 1 test() I get an error from the print(c) line that says: UnboundLocalError: local variable 'c' referenced before assignment or in some older versions: UnboundLocalError: 'c' not assigned If I comment out c += 1, all the prints are successful. I don't understand: why does printing a and b work, if c does not? How did c += 1 cause print(c) to fail, even when it comes later in the code? It seems like the assignment c += 1 creates a local variable c, which takes precedence over the global c. But how can a variable "steal" scope before it exists? Why is c apparently local here? See also How to use a global variable in a function? for questions that are simply about how to reassign a global variable from within a function, and Is it possible to modify a variable in python that is in an outer (enclosing), but not global, scope? for reassigning from an enclosing function (closure). See Why isn't the 'global' keyword needed to access a global variable? for cases where OP expected an error but didn't get one, from simply accessing a global without the global keyword. See How can a name be "unbound" in Python? What code can cause an `UnboundLocalError`? for cases where OP expected the variable to be local, but has a logical error that prevents assignment in every case. See How can "NameError: free variable 'var' referenced before assignment in enclosing scope" occur in real code? for a related problem caused by the del keyword.

Output: Python treats variables in functions differently depending on whether you assign values to them from inside or outside the function. If a variable is assigned within a function, it is treated by default as a local variable. Therefore, when you uncomment the line, you are trying to reference the local variable c before any value has been assigned to it. If you want the variable c to refer to the global c = 3 assigned before the function, put global c as the first line of the function. As for python 3, there is now nonlocal c that you can use to refer to the nearest enclosing function scope that has a c variable.


Instruction:
What is the difference between json.load() and json.loads() functions

In Python, what is the difference between json.load() and json.loads()? I guess that the load() function must be used with a file object (I need thus to use a context manager) while the loads() function take the path to the file as a string. It is a bit confusing. Does the letter "s" in json.loads() stand for string?

Output: Yes, s stands for string. The json.loads function does not take the file path, but the file contents as a string. Look at the documentation. Simple example: with open("file.json") as f: data = json.load(f) # ok data = json.loads(f) # not ok, f is not a string but a file text = '{"a": 1, "b": 2}' # a string with json encoded data data = json.loads(text)


Instruction:
What's the difference between dist-packages and site-packages?

I'm a bit miffed by the python package installation process. Specifically, what's the difference between packages installed in the dist-packages directory and the site-packages directory?

Output: dist-packages is a Debian-specific convention that is also present in its derivatives, like Ubuntu. Modules are installed to dist-packages when they come from the Debian package manager into this location: /usr/lib/python2.7/dist-packages Since easy_install and pip are installed from the package manager, they also use dist-packages, but they put packages here: /usr/local/lib/python2.7/dist-packages From the Debian Python Wiki: dist-packages instead of site-packages. Third party Python software installed from Debian packages goes into dist-packages, not site-packages. This is to reduce conflict between the system Python, and any from-source Python build you might install manually. This means that if you manually compile and install Python interpreter from source, it uses the site-packages directory. This allows you to keep the two installations separate, especially since Debian and Ubuntu rely on the system version of Python for many system utilities.


Instruction:
How to filter a dictionary according to an arbitrary condition function?

I have a dictionary of points, say: >>> points={'a':(3,4), 'b':(1,2), 'c':(5,5), 'd':(3,3)} I want to create a new dictionary with all the points whose x and y value is smaller than 5, i.e. points 'a', 'b' and 'd'. According to the the book, each dictionary has the items() function, which returns a list of (key, pair) tuple: >>> points.items() [('a', (3, 4)), ('c', (5, 5)), ('b', (1, 2)), ('d', (3, 3))] So I have written this: >>> for item in [i for i in points.items() if i[1][0]<5 and i[1][1]<5]: ... points_small[item[0]]=item[1] ... >>> points_small {'a': (3, 4), 'b': (1, 2), 'd': (3, 3)} Is there a more elegant way? I was expecting Python to have some super-awesome dictionary.filter(f) function...

Output: You can use a dict comprehension: {k: v for k, v in points.items() if v[0] < 5 and v[1] < 5} And in Python 2, starting from 2.7: {k: v for k, v in points.iteritems() if v[0] < 5 and v[1] < 5}


Instruction:
Iterating over every two elements in a list

How do I make a for loop or a list comprehension so that every iteration gives me two elements? l = [1,2,3,4,5,6] for i,k in ???: print str(i), '+', str(k), '=', str(i+k) Output: 1+2=3 3+4=7 5+6=11

Output: Starting with Python 3.12, you can use the batched() function provided by the itertools module: from itertools import batched for x, y in batched(l, n=2): print("%d + %d = %d" % (x, y, x + y)) Otherwise, you need a pairwise() (or grouped()) implementation. def pairwise(iterable): "s -> (s0, s1), (s2, s3), (s4, s5), ..." a = iter(iterable) return zip(a, a) for x, y in pairwise(l): print("%d + %d = %d" % (x, y, x + y)) Or, more generally: def grouped(iterable, n): "s -> (s0,s1,s2,...sn-1), (sn,sn+1,sn+2,...s2n-1), (s2n,s2n+1,s2n+2,...s3n-1), ..." return zip(*[iter(iterable)]*n) for x, y in grouped(l, 2): print("%d + %d = %d" % (x, y, x + y)) In Python 2, you should import izip as a replacement for Python 3's built-in zip() function. All credit to martineau for his answer to my question, I have found this to be very efficient as it only iterates once over the list and does not create any unnecessary lists in the process. N.B: This should not be confused with the pairwise recipe in Python's own itertools documentation, which yields s -> (s0, s1), (s1, s2), (s2, s3), ..., as pointed out by @lazyr in the comments. Little addition for those who would like to do type checking with mypy on Python 3: from typing import Iterable, Tuple, TypeVar T = TypeVar("T") def grouped(iterable: Iterable[T], n=2) -> Iterable[Tuple[T, ...]]: """s -> (s0,s1,s2,...sn-1), (sn,sn+1,sn+2,...s2n-1), ...""" return zip(*[iter(iterable)] * n)


Instruction:
Django set default form values

I have a Model as follows: class TankJournal(models.Model): user = models.ForeignKey(User) tank = models.ForeignKey(TankProfile) ts = models.IntegerField(max_length=15) title = models.CharField(max_length=50) body = models.TextField() I also have a model form for the above model as follows: class JournalForm(ModelForm): tank = forms.IntegerField(widget=forms.HiddenInput()) class Meta: model = TankJournal exclude = ('user','ts') I want to know how to set the default value for that tank hidden field. Here is my function to show/save the form so far: def addJournal(request, id=0): if not request.user.is_authenticated(): return HttpResponseRedirect('/') # checking if they own the tank from django.contrib.auth.models import User user = User.objects.get(pk=request.session['id']) if request.method == 'POST': form = JournalForm(request.POST) if form.is_valid(): obj = form.save(commit=False) # setting the user and ts from time import time obj.ts = int(time()) obj.user = user obj.tank = TankProfile.objects.get(pk=form.cleaned_data['tank_id']) # saving the test obj.save() else: form = JournalForm() try: tank = TankProfile.objects.get(user=user, id=id) except TankProfile.DoesNotExist: return HttpResponseRedirect('/error/')

Output: You can use Form.initial. You have two options either populate the value when calling form constructor: form = JournalForm(initial={'tank': 123}) or set the value in the form definition: tank = forms.IntegerField(widget=forms.HiddenInput(), initial=123)


Instruction:
How do I calculate percentiles with python/numpy?

Is there a convenient way to calculate percentiles for a sequence or single-dimensional numpy array? I am looking for something similar to Excel's percentile function.

Output: NumPy has np.percentile(). import numpy as np a = np.array([1,2,3,4,5]) p = np.percentile(a, 50) # return 50th percentile, i.e. median. >>> print(p) 3.0 SciPy has scipy.stats.scoreatpercentile(), in addition to many other statistical goodies.


Instruction:
tight_layout() doesn't take into account figure suptitle

If I add a subtitle to my matplotlib figure it gets overlaid by the subplot's titles. Does anybody know how to easily take care of that? I tried the tight_layout() function, but it only makes things worse. Example: import numpy as np import matplotlib.pyplot as plt f = np.random.random(100) g = np.random.random(100) fig = plt.figure() fig.suptitle('Long Suptitle', fontsize=24) plt.subplot(121) plt.plot(f) plt.title('Very Long Title 1', fontsize=20) plt.subplot(122) plt.plot(g) plt.title('Very Long Title 2', fontsize=20) plt.tight_layout() plt.show()

Output: You can adjust the subplot geometry in the very tight_layout call as follows: fig.tight_layout(rect=[0, 0.03, 1, 0.95]) As it's stated in the documentation (https://matplotlib.org/stable/users/explain/axes/tight_layout_guide.html): tight_layout() only considers ticklabels, axis labels, and titles. Thus, other artists may be clipped and also may overlap.


Instruction:
live output from subprocess command

I'm using a python script as a driver for a hydrodynamics code. When it comes time to run the simulation, I use subprocess.Popen to run the code, collect the output from stdout and stderr into a subprocess.PIPE --- then I can print (and save to a log-file) the output information, and check for any errors. The problem is, I have no idea how the code is progressing. If I run it directly from the command line, it gives me output about what iteration its at, what time, what the next time-step is, etc. Is there a way to both store the output (for logging and error checking), and also produce a live-streaming output? The relevant section of my code: ret_val = subprocess.Popen( run_command, stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=True ) output, errors = ret_val.communicate() log_file.write(output) print output if( ret_val.returncode ): print "RUN failed\n\n%s\n\n" % (errors) success = False if( errors ): log_file.write("\n\n%s\n\n" % errors) Originally I was piping the run_command through tee so that a copy went directly to the log-file, and the stream still output directly to the terminal -- but that way I can't store any errors (to my knowlege). My temporary solution so far: ret_val = subprocess.Popen( run_command, stdout=log_file, stderr=subprocess.PIPE, shell=True ) while not ret_val.poll(): log_file.flush() then, in another terminal, run tail -f log.txt (s.t. log_file = 'log.txt').

Output: TLDR for Python 3: import subprocess import sys with open("test.log", "wb") as f: process = subprocess.Popen(your_command, stdout=subprocess.PIPE) for c in iter(lambda: process.stdout.read(1), b""): sys.stdout.buffer.write(c) f.buffer.write(c) You have two ways of doing this, either by creating an iterator from the read or readline functions and do: import subprocess import sys # replace "w" with "wb" for Python 3 with open("test.log", "w") as f: process = subprocess.Popen(your_command, stdout=subprocess.PIPE) # replace "" with b'' for Python 3 for c in iter(lambda: process.stdout.read(1), ""): sys.stdout.write(c) f.write(c) or import subprocess import sys # replace "w" with "wb" for Python 3 with open("test.log", "w") as f: process = subprocess.Popen(your_command, stdout=subprocess.PIPE) # replace "" with b"" for Python 3 for line in iter(process.stdout.readline, ""): sys.stdout.write(line) f.write(line) Or you can create a reader and a writer file. Pass the writer to the Popen and read from the reader import io import time import subprocess import sys filename = "test.log" with io.open(filename, "wb") as writer, io.open(filename, "rb", 1) as reader: process = subprocess.Popen(command, stdout=writer) while process.poll() is None: sys.stdout.write(reader.read()) time.sleep(0.5) # Read the remaining sys.stdout.write(reader.read()) This way you will have the data written in the test.log as well as on the standard output. The only advantage of the file approach is that your code doesn't block. So you can do whatever you want in the meantime and read whenever you want from the reader in a non-blocking way. When you use PIPE, read and readline functions will block until either one character is written to the pipe or a line is written to the pipe respectively.


Instruction:
How do I loop through a list by twos?

I want to loop through a Python list and process 2 list items at a time. Something like this in another language: for(int i = 0; i < list.length(); i+=2) { // do something with list[i] and list[i + 1] } What's the best way to accomplish this?

Output: You can use a range with a step size of 2: Python 2 for i in xrange(0,10,2): print(i) Python 3 for i in range(0,10,2): print(i) Note: Use xrange in Python 2 instead of range because it is more efficient as it generates an iterable object, and not the whole list.


Instruction:
Identifying the dependency relationship for python packages installed with pip

When I do a pip freeze I see large number of Python packages that I didn't explicitly install, e.g. $ pip freeze Cheetah==2.4.3 GnuPGInterface==0.3.2 Landscape-Client==11.01 M2Crypto==0.20.1 PAM==0.4.2 PIL==1.1.7 PyYAML==3.09 Twisted-Core==10.2.0 Twisted-Web==10.2.0 (etc.) Is there a way for me to determine why pip installed these particular dependent packages? In other words, how do I determine the parent package that had these packages as dependencies? For example, I might want to use Twisted and I don't want to depend on a package until I know more about not accidentally uninstalling it or upgrading it.

Output: You could try pipdeptree, which displays dependencies as a tree structure e.g.: $ pipdeptree Lookupy==0.1 wsgiref==0.1.2 argparse==1.2.1 psycopg2==2.5.2 Flask-Script==0.6.6 - Flask [installed: 0.10.1] - Werkzeug [required: >=0.7, installed: 0.9.4] - Jinja2 [required: >=2.4, installed: 2.7.2] - MarkupSafe [installed: 0.18] - itsdangerous [required: >=0.21, installed: 0.23] alembic==0.6.2 - SQLAlchemy [required: >=0.7.3, installed: 0.9.1] - Mako [installed: 0.9.1] - MarkupSafe [required: >=0.9.2, installed: 0.18] ipython==2.0.0 slugify==0.0.1 redis==2.9.1 To install it, run: pip install pipdeptree As noted by @Esteban in the comments you can also list the tree in reverse with -r or for a single package with -p <package_name>. So to find which module(s) Werkzeug is a dependency for, you could run: $ pipdeptree -r -p Werkzeug Werkzeug==0.11.15 - Flask==0.12 [requires: Werkzeug>=0.7]


Instruction:
Anaconda export Environment file

How can I make anaconda environment file which could be use on other computers? I exported my anaconda python environment to YML using conda env export > environment.yml. The exported environment.yml contains this line prefix: /home/superdev/miniconda3/envs/juicyenv which maps to my anaconda's location which will be different on other's pcs.

Output: I can't find anything in the conda specs which allows you to export an environment file without the prefix: ... line. However, like Alex pointed out in the comments, conda doesn't seem to care about the prefix line when creating an environment from the file. With that in mind, if you want the other user to have no knowledge of your default install path, you can remove the prefix line with grep before writing to environment.yml. conda env export | grep -v "^prefix: " > environment.yml Either way, the other user then runs: conda env create -f environment.yml and the environment will get installed in their default conda environment path. If you want to specify a different install path than the default for your system (not related to 'prefix' in the environment.yml), just use the -p flag followed by the required path. conda env create -f environment.yml -p /home/user/anaconda3/envs/env_name Note that Conda recommends creating the environment.yml by hand, which is especially important if you are wanting to share your environment across platforms (Windows/Linux/Mac). In this case, you can just leave out the prefix line.


Instruction:
Catch a thread's exception in the caller thread?

I'm very new to Python and multithreaded programming in general. Basically, I have a script that will copy files to another location. I would like this to be placed in another thread so I can output .... to indicate that the script is still running. The problem that I am having is that if the files cannot be copied it will throw an exception. This is OK if running in the main thread; however, having the following code does not work: try: threadClass = TheThread(param1, param2, etc.) threadClass.start() ##### **Exception takes place here** except: print "Caught an exception" In the thread class itself, I tried to re-throw the exception, but it does not work. I have seen people on here ask similar questions, but they all seem to be doing something more specific than what I am trying to do (and I don't quite understand the solutions offered). I have seen people mention the usage of sys.exc_info(), however I do not know where or how to use it. Edit: The code for the thread class is below: class TheThread(threading.Thread): def __init__(self, sourceFolder, destFolder): threading.Thread.__init__(self) self.sourceFolder = sourceFolder self.destFolder = destFolder def run(self): try: shul.copytree(self.sourceFolder, self.destFolder) except: raise

Output: The problem is that thread_obj.start() returns immediately. The child thread that you spawned executes in its own context, with its own stack. Any exception that occurs there is in the context of the child thread, and it is in its own stack. One way I can think of right now to communicate this information to the parent thread is by using some sort of message passing, so you might look into that. Try this on for size: import sys import threading import queue class ExcThread(threading.Thread): def __init__(self, bucket): threading.Thread.__init__(self) self.bucket = bucket def run(self): try: raise Exception('An error occured here.') except Exception: self.bucket.put(sys.exc_info()) def main(): bucket = queue.Queue() thread_obj = ExcThread(bucket) thread_obj.start() while True: try: exc = bucket.get(block=False) except queue.Empty: pass else: exc_type, exc_obj, exc_trace = exc # deal with the exception print exc_type, exc_obj print exc_trace thread_obj.join(0.1) if thread_obj.isAlive(): continue else: break if __name__ == '__main__': main()


Instruction:
How can I call a shell script from Python code?

How can I call a shell script from Python code?

Output: The subprocess module will help you out. Blatantly trivial example: >>> import subprocess >>> subprocess.call(['sh', './test.sh']) # Thanks @Jim Dennis for suggesting the [] 0 Where test.sh is a simple shell script and 0 is its return value for this run.


Instruction:
How can I print the value of a Tensor object in TensorFlow?

I have been using the introductory example of matrix multiplication in TensorFlow. matrix1 = tf.constant([[3., 3.]]) matrix2 = tf.constant([[2.],[2.]]) product = tf.matmul(matrix1, matrix2) When I print the product, it is displaying it as a Tensor object: <tensorflow.python.framework.ops.Tensor object at 0x10470fcd0> But how do I know the value of product? The following doesn't help: print product Output: Tensor("MatMul:0", shape=TensorShape([Dimension(1), Dimension(1)]), dtype=float32) I know that graphs run on Sessions, but isn't there a way I can check the output of a Tensor object without running the graph in a session?

Output: The easiest[A] way to evaluate the actual value of a Tensor object is to pass it to the Session.run() method, or call Tensor.eval() when you have a default session (i.e. in a with tf.Session(): block, or see below). In general[B], you cannot print the value of a tensor without running some code in a session. If you are experimenting with the programming model, and want an easy way to evaluate tensors, the tf.InteractiveSession lets you open a session at the start of your program, and then use that session for all Tensor.eval() (and Operation.run()) calls. This can be easier in an interactive setting, such as the shell or an IPython notebook, when it's tedious to pass around a Session object everywhere. For example, the following works in a Jupyter notebook: with tf.Session() as sess: print(product.eval()) This might seem silly for such a small expression, but one of the key ideas in TensorFlow 1.x is deferred execution: it's very cheap to build a large and complex expression, and when you want to evaluate it, the back-end (to which you connect with a Session) is able to schedule its execution more efficiently (e.g., executing independent parts in parallel and using GPUs). [A]: To print the value of a tensor without returning it to your Python program, you can use the tf.print() operator, as Andrzej suggests in another answer. According to the official documentation: To make sure the operator runs, users need to pass the produced op to tf.compat.v1.Session's run method, or to use the op as a control dependency for executed ops by specifying with tf.compat.v1.control_dependencies([print_op]), which is printed to standard output. Also note that: In Jupyter notebooks and colabs, tf.print prints to the notebook cell outputs. It will not write to the notebook kernel's console logs. [B]: You might be able to use the tf.get_static_value() function to get the constant value of the given tensor if its value is efficiently calculable.


Instruction:
Difference between defining typing.Dict and dict?

I am practicing using type hints in Python 3.5. One of my colleague uses typing.Dict: import typing def change_bandwidths(new_bandwidths: typing.Dict, user_id: int, user_name: str) -> bool: print(new_bandwidths, user_id, user_name) return False def my_change_bandwidths(new_bandwidths: dict, user_id: int, user_name: str) ->bool: print(new_bandwidths, user_id, user_name) return True def main(): my_id, my_name = 23, "Tiras" simple_dict = {"Hello": "Moon"} change_bandwidths(simple_dict, my_id, my_name) new_dict = {"new": "energy source"} my_change_bandwidths(new_dict, my_id, my_name) if __name__ == "__main__": main() Both of them work just fine, there doesn't appear to be a difference. I have read the typing module documentation. Between typing.Dict or dict which one should I use in the program?

Output: Note: typing.Dict has been deprecated as of Python 3.9, because the dict type itself can be used as a generic type directly (together with other standard containers). You can do the same in Python 3.7 or 3.8 if you use a from __future__ import annotations directive. My answer was originally written for much older Python 3 releases. There is no real difference between using a plain typing.Dict and dict, no. However, typing.Dict is a Generic type that lets you specify the type of the keys and values too, making it more flexible: def change_bandwidths(new_bandwidths: typing.Dict[str, str], user_id: int, user_name: str) -> bool: As such, it could well be that at some point in your project lifetime you want to define the dictionary argument a little more precisely, at which point expanding typing.Dict to typing.Dict[key_type, value_type] is a 'smaller' change than replacing dict. You can make this even more generic by using Mapping or MutableMapping types here; since your function doesn't need to alter the mapping, I'd stick with Mapping. A dict is one mapping, but you could create other objects that also satisfy the mapping interface, and your function might well still work with those: from collections.abc import Mapping def change_bandwidths(new_bandwidths: Mapping[str, str], user_id: int, user_name: str) -> bool: Now you are clearly telling other users of this function that your code won't actually alter the new_bandwidths mapping passed in. Your actual implementation is merely expecting an object that is printable. That may be a test implementation, but as it stands your code would continue to work if you used new_bandwidths: object, because any object in Python is printable.


Instruction:
Starting python debugger automatically on error

This is a question I have wondered about for quite some time, yet I have never found a suitable solution. If I run a script and I come across, let's say an IndexError, python prints the line, location and quick description of the error and exits. Is it possible to automatically start pdb when an error is encountered? I am not against having an extra import statement at the top of the file, nor a few extra lines of code.

Output: You can use traceback.print_exc to print the exceptions traceback. Then use sys.exc_info to extract the traceback and finally call pdb.post_mortem with that traceback import pdb, traceback, sys def bombs(): a = [] print a[0] if __name__ == '__main__': try: bombs() except: extype, value, tb = sys.exc_info() traceback.print_exc() pdb.post_mortem(tb) If you want to start an interactive command line with code.interact using the locals of the frame where the exception originated you can do import traceback, sys, code def bombs(): a = [] print a[0] if __name__ == '__main__': try: bombs() except: type, value, tb = sys.exc_info() traceback.print_exc() last_frame = lambda tb=tb: last_frame(tb.tb_next) if tb.tb_next else tb frame = last_frame().tb_frame ns = dict(frame.f_globals) ns.update(frame.f_locals) code.interact(local=ns)


Instruction:
Abstract methods in Python

I am having trouble in using inheritance with Python. While the concept seems too easy for me in Java yet up till now I have been unable to understand in Python which is surprising to me at least. I have a prototype which follow: class Shape(): def __init__(self, shape_name): self.shape = shape_name class Rectangle(Shape): def __init__(self, name): self.shape = name In the above code how can I make an abstract method that would need to be implemented for all the subclasses?

Output: Something along these lines, using ABC import abc class Shape(object): __metaclass__ = abc.ABCMeta @abc.abstractmethod def method_to_implement(self, input): """Method documentation""" return Also read this good tutorial: https://pymotw.com/3/abc/ You can also check out zope.interface which was used prior to introduction of ABC in python. http://pypi.python.org/pypi/zope.interface https://zopeinterface.readthedocs.io/en/latest/README.html


Instruction:
Convert python datetime to epoch with strftime

I have a time in UTC from which I want the number of seconds since epoch. I am using strftime to convert it to the number of seconds. Taking 1st April 2012 as an example. >>>datetime.datetime(2012,04,01,0,0).strftime('%s') '1333234800' 1st of April 2012 UTC from epoch is 1333238400 but this above returns 1333234800 which is different by 1 hour. So it looks like that strftime is taking my system time into account and applies a timezone shift somewhere. I thought datetime was purely naive? How can I get around that? If possible avoiding to import other libraries unless standard. (I have portability concerns).

Output: In Python 3.3+ you can use timestamp(): >>> datetime.datetime(2012,4,1,0,0).timestamp() 1333234800.0 In Python 3.2 or earlier, you could do it explicitly: >>> (datetime.datetime(2012,4,1,0,0) - datetime.datetime(1970,1,1)).total_seconds() 1333238400.0 Why you should not use datetime.strftime('%s') Python doesn't actually support %s as an argument to strftime (if you check at http://docs.python.org/library/datetime.html#strftime-and-strptime-behavior it's not in the list), the only reason it's working is because Python is passing the information to your system's strftime, which uses your local timezone. >>> datetime.datetime(2012,04,01,0,0).strftime('%s') '1333234800'


Instruction:
Create a directly-executable cross-platform GUI app using Python

Python works on multiple platforms and can be used for desktop and web applications, thus I conclude that there is some way to compile it into an executable for Mac, Windows and Linux. The problem being I have no idea where to start or how to write a GUI with it, can anybody shed some light on this and point me in the right direction please?

Output: First you will need some GUI library with Python bindings and then (if you want) some program that will convert your python scripts into standalone executables. Cross-platform GUI libraries with Python bindings (Windows, Linux, Mac) Of course, there are many, but the most popular that I've seen in wild are: Tkinter - based on Tk GUI toolkit. De-facto standard GUI library for python, free for commercial projects. WxPython - based on WxWidgets. Popular, and free for commercial projects. Qt using the PyQt bindings or Qt for Python. The former is not free for commercial projects. The latter is less mature, but can be used for free. Qt itself supposedly supports Android and iOS as well, but achiving same with it's bindings should be tricky. Kivy written in Python for Python (update 2023). Supposedly supports Android and iOS as well. Note that users of WxWidgets (hence WxPython users), often need to use WxQt as well, because WxWidgets's own GUI is not yet at Qt's level (at time of writting). Complete list is at http://wiki.python.org/moin/GuiProgramming Stand-alone/ single executables For all platforms: PyInstaller - The most active (which could also be used with PyQt) fbs - if you chose Qt above (commercial, with free plan) For Windows: py2exe - used to be the most popular For Linux: Freeze - works the same way like py2exe but targets Linux platform For MacOS: py2app - again, works like py2exe but targets Mac OS


Instruction:
Pandas percentage of total with groupby

This is obviously simple, but as a numpy newbe I'm getting stuck. I have a CSV file that contains 3 columns, the State, the Office ID, and the Sales for that office. I want to calculate the percentage of sales per office in a given state (total of all percentages in each state is 100%). df = pd.DataFrame({'state': ['CA', 'WA', 'CO', 'AZ'] * 3, 'office_id': list(range(1, 7)) * 2, 'sales': [np.random.randint(100000, 999999) for _ in range(12)]}) df.groupby(['state', 'office_id']).agg({'sales': 'sum'}) This returns: sales state office_id AZ 2 839507 4 373917 6 347225 CA 1 798585 3 890850 5 454423 CO 1 819975 3 202969 5 614011 WA 2 163942 4 369858 6 959285 I can't seem to figure out how to "reach up" to the state level of the groupby to total up the sales for the entire state to calculate the fraction.

Output: Update 2022-03 This answer by caner using transform looks much better than my original answer! df['sales'] / df.groupby('state')['sales'].transform('sum') Thanks to this comment by Paul Rougieux for surfacing it. Original Answer (2014) Paul H's answer is right that you will have to make a second groupby object, but you can calculate the percentage in a simpler way -- just groupby the state_office and divide the sales column by its sum. Copying the beginning of Paul H's answer: # From Paul H import numpy as np import pandas as pd np.random.seed(0) df = pd.DataFrame({'state': ['CA', 'WA', 'CO', 'AZ'] * 3, 'office_id': list(range(1, 7)) * 2, 'sales': [np.random.randint(100000, 999999) for _ in range(12)]}) state_office = df.groupby(['state', 'office_id']).agg({'sales': 'sum'}) # Change: groupby state_office and divide by sum state_pcts = state_office.groupby(level=0).apply(lambda x: 100 * x / float(x.sum())) Returns: sales state office_id AZ 2 16.981365 4 19.250033 6 63.768601 CA 1 19.331879 3 33.858747 5 46.809373 CO 1 36.851857 3 19.874290 5 43.273852 WA 2 34.707233 4 35.511259 6 29.781508


Instruction:
How to unzip a list of tuples into individual lists?

I have a list of tuples l = [(1,2), (3,4), (8,9)]. How can I, succinctly and Pythonically, unzip this list into two independent lists, to get [ [1, 3, 8], [2, 4, 9] ]? In other words, how do I get the inverse of what zip does?

Output: Use zip(*list): >>> l = [(1,2), (3,4), (8,9)] >>> list(zip(*l)) [(1, 3, 8), (2, 4, 9)] The zip() function pairs up the elements from all inputs, starting with the first values, then the second, etc. By using *l you apply all tuples in l as separate arguments to the zip() function, so zip() pairs up 1 with 3 with 8 first, then 2 with 4 and 9. Those happen to correspond nicely with the columns, or the transposition of l. zip() produces tuples; if you must have mutable list objects, just map() the tuples to lists or use a list comprehension to produce a list of lists: map(list, zip(*l)) # keep it a generator [list(t) for t in zip(*l)] # consume the zip generator into a list of lists


Instruction:
Python logging: use milliseconds in time format

By default logging.Formatter('%(asctime)s') prints with the following format: 2011-06-09 10:54:40,638 where 638 is the millisecond. I need to change the comma to a dot: 2011-06-09 10:54:40.638 To format the time I can use: logging.Formatter(fmt='%(asctime)s',datestr=date_format_str) however the documentation doesn't specify how to format milliseconds. I've found this SO question which talks about microseconds, but a) I would prefer milliseconds and b) the following doesn't work on Python 2.6 (which I'm working on) due to the %f: logging.Formatter(fmt='%(asctime)s',datefmt='%Y-%m-%d,%H:%M:%S.%f')

Output: This should work too: logging.Formatter( fmt='%(asctime)s.%(msecs)03d', datefmt='%Y-%m-%d,%H:%M:%S' )


Instruction:
Logical operators for Boolean indexing in Pandas

I'm working with a Boolean index in Pandas. The question is why the statement: a[(a['some_column']==some_number) & (a['some_other_column']==some_other_number)] works fine whereas a[(a['some_column']==some_number) and (a['some_other_column']==some_other_number)] exits with error? Example: a = pd.DataFrame({'x':[1,1],'y':[10,20]}) In: a[(a['x']==1)&(a['y']==10)] Out: x y 0 1 10 In: a[(a['x']==1) and (a['y']==10)] Out: ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()

Output: When you say (a['x']==1) and (a['y']==10) You are implicitly asking Python to convert (a['x']==1) and (a['y']==10) to Boolean values. NumPy arrays (of length greater than 1) and Pandas objects such as Series do not have a Boolean value -- in other words, they raise ValueError: The truth value of an array is ambiguous. Use a.empty, a.any() or a.all(). when used as a Boolean value. That's because it's unclear when it should be True or False. Some users might assume they are True if they have non-zero length, like a Python list. Others might desire for it to be True only if all its elements are True. Others might want it to be True if any of its elements are True. Because there are so many conflicting expectations, the designers of NumPy and Pandas refuse to guess, and instead raise a ValueError. Instead, you must be explicit, by calling the empty(), all() or any() method to indicate which behavior you desire. In this case, however, it looks like you do not want Boolean evaluation, you want element-wise logical-and. That is what the & binary operator performs: (a['x']==1) & (a['y']==10) returns a boolean array. By the way, as alexpmil notes, the parentheses are mandatory since & has a higher operator precedence than ==. Without the parentheses, a['x']==1 & a['y']==10 would be evaluated as a['x'] == (1 & a['y']) == 10 which would in turn be equivalent to the chained comparison (a['x'] == (1 & a['y'])) and ((1 & a['y']) == 10) That is an expression of the form Series and Series. The use of and with two Series would again trigger the same ValueError as above. That's why the parentheses are mandatory.


Instruction:
Visual Studio Code: How debug Python script with arguments

I'm using Visual Studio Code with the inbuilt Debugger in order to debug a Python script. Following this guide, I set up the argument in the launch.json file: But when I press on Debug, it says that my argument is not recognized my shell, or rather the argparser, says: error: unrecognized arguments: --city Auckland As Visual Studio Code is using PowerShell, let's execute the same file with the same argument: So: the same file, same path, and same argument. In the terminal it is working, but not in Visual Studio Code's Debugger. Where am I wrong?

Output: I think the --City and Auckland are used as a single argument. Maybe try separating them like so... Single argument "args": ["--city","Auckland"] Multiple arguments and multiple values Such as: --key1 value1 value2 --key2 value3 value4 Just put them into the args list one by one in sequence: "args": ["--key1", "value1", "value2", "--key2", "value3", "value4"]


Instruction:
How to add multiple columns to pandas dataframe in one assignment

I'm trying to figure out how to add multiple columns to pandas simultaneously with Pandas. I would like to do this in one step rather than multiple repeated steps. import pandas as pd data = {'col_1': [0, 1, 2, 3], 'col_2': [4, 5, 6, 7]} df = pd.DataFrame(data) I thought this would work here... df[['column_new_1', 'column_new_2', 'column_new_3']] = [np.nan, 'dogs', 3]

Output: I would have expected your syntax to work too. The problem arises because when you create new columns with the column-list syntax (df[[new1, new2]] = ...), pandas requires that the right hand side be a DataFrame (note that it doesn't actually matter if the columns of the DataFrame have the same names as the columns you are creating). Your syntax works fine for assigning scalar values to existing columns, and pandas is also happy to assign scalar values to a new column using the single-column syntax (df[new1] = ...). So the solution is either to convert this into several single-column assignments, or create a suitable DataFrame for the right-hand side. Here are several approaches that will work: import pandas as pd import numpy as np df = pd.DataFrame({ 'col_1': [0, 1, 2, 3], 'col_2': [4, 5, 6, 7] }) Then one of the following: 1) Three assignments in one, using iterable unpacking df['column_new_1'], df['column_new_2'], df['column_new_3'] = np.nan, 'dogs', 3 2) Use DataFrame() to expand a single row to match the index df[['column_new_1', 'column_new_2', 'column_new_3']] = pd.DataFrame([[np.nan, 'dogs', 3]], index=df.index) 3) Combine with a temporary DataFrame using pd.concat df = pd.concat( [ df, pd.DataFrame( [[np.nan, 'dogs', 3]], index=df.index, columns=['column_new_1', 'column_new_2', 'column_new_3'] ) ], axis=1 ) 4) Combine with a temporary DataFrame using .join This is similar to 3, but may be less efficient. df = df.join(pd.DataFrame( [[np.nan, 'dogs', 3]], index=df.index, columns=['column_new_1', 'column_new_2', 'column_new_3'] )) 5) Use a dictionary instead of the lists used in 3 and 4 This is a more "natural" way to create the temporary DataFrame than the previous two. Note that in Python 3.5 or earlier, the new columns will be sorted alphabetically. df = df.join(pd.DataFrame( { 'column_new_1': np.nan, 'column_new_2': 'dogs', 'column_new_3': 3 }, index=df.index )) 6) Use .assign() with multiple column arguments This may be the winner in Python 3.6+. But like the previous one, the new columns will be sorted alphabetically in earlier versions of Python. df = df.assign(column_new_1=np.nan, column_new_2='dogs', column_new_3=3) 7) Create new columns, then assign all values at once Based on this answer. This is interesting, but I don't know when it would be worth the trouble. new_cols = ['column_new_1', 'column_new_2', 'column_new_3'] new_vals = [np.nan, 'dogs', 3] df = df.reindex(columns=df.columns.tolist() + new_cols) # add empty cols df[new_cols] = new_vals # multi-column assignment works for existing cols 8) Three separate assignments In the end, it's hard to beat this. df['column_new_1'] = np.nan df['column_new_2'] = 'dogs' df['column_new_3'] = 3 Note: many of these options have already been covered in other questions: Add multiple columns to DataFrame and set them equal to an existing column Is it possible to add several columns at once to a pandas DataFrame? Add multiple empty columns to pandas DataFrame


Instruction:
How do I raise the same Exception with a custom message in Python?

I have this try block in my code: try: do_something_that_might_raise_an_exception() except ValueError as err: errmsg = 'My custom error message.' raise ValueError(errmsg) Strictly speaking, I am actually raising another ValueError, not the ValueError thrown by do_something...(), which is referred to as err in this case. How do I attach a custom message to err? I try the following code but fails due to err, a ValueError instance, not being callable: try: do_something_that_might_raise_an_exception() except ValueError as err: errmsg = 'My custom error message.' raise err(errmsg)

Output: raise from We can chain the exceptions using raise from. try: 1 / 0 except ZeroDivisionError as e: raise Exception('Smelly socks') from e In this case, the exception your caller would catch has the line number of the place where we raise our exception. Traceback (most recent call last): File "test.py", line 2, in <module> 1 / 0 ZeroDivisionError: division by zero The above exception was the direct cause of the following exception: Traceback (most recent call last): File "test.py", line 4, in <module> raise Exception('Smelly socks') from e Exception: Smelly socks Notice the bottom exception only has the stacktrace from where we raised our exception. Your caller could still get the original exception by accessing the __cause__ attribute of the exception they catch. with_traceback Or you can use with_traceback. try: 1 / 0 except ZeroDivisionError as e: raise Exception('Smelly socks').with_traceback(e.__traceback__) Using this form, the exception your caller would catch has the traceback from where the original error occurred. Traceback (most recent call last): File "test.py", line 2, in <module> 1 / 0 ZeroDivisionError: division by zero During handling of the above exception, another exception occurred: Traceback (most recent call last): File "test.py", line 4, in <module> raise Exception('Smelly socks').with_traceback(e.__traceback__) File "test.py", line 2, in <module> 1 / 0 Exception: Smelly socks Notice the bottom exception has the line where we performed the invalid division as well as the line where we reraise the exception.


Instruction:
What is the use of join() in threading?

I was studying the python threading and came across join(). The author told that if thread is in daemon mode then i need to use join() so that thread can finish itself before main thread terminates. but I have also seen him using t.join() even though t was not daemon example code is this import threading import time import logging logging.basicConfig(level=logging.DEBUG, format='(%(threadName)-10s) %(message)s', ) def daemon(): logging.debug('Starting') time.sleep(2) logging.debug('Exiting') d = threading.Thread(name='daemon', target=daemon) d.setDaemon(True) def non_daemon(): logging.debug('Starting') logging.debug('Exiting') t = threading.Thread(name='non-daemon', target=non_daemon) d.start() t.start() d.join() t.join() i don't know what is use of t.join() as it is not daemon and i can see no change even if i remove it

Output: A somewhat clumsy ascii-art to demonstrate the mechanism: The join() is presumably called by the main-thread. It could also be called by another thread, but would needlessly complicate the diagram. join-calling should be placed in the track of the main-thread, but to express thread-relation and keep it as simple as possible, I choose to place it in the child-thread instead. without join: +---+---+------------------ main-thread | | | +........... child-thread(short) +.................................. child-thread(long) with join +---+---+------------------***********+### main-thread | | | | +...........join() | child-thread(short) +......................join()...... child-thread(long) with join and daemon thread +-+--+---+------------------***********+### parent-thread | | | | | | +...........join() | child-thread(short) | +......................join()...... child-thread(long) +,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,, child-thread(long + daemonized) '-' main-thread/parent-thread/main-program execution '.' child-thread execution '#' optional parent-thread execution after join()-blocked parent-thread could continue '*' main-thread 'sleeping' in join-method, waiting for child-thread to finish ',' daemonized thread - 'ignores' lifetime of other threads; terminates when main-programs exits; is normally meant for join-independent tasks So the reason you don't see any changes is because your main-thread does nothing after your join. You could say join is (only) relevant for the execution-flow of the main-thread. If, for example, you want to concurrently download a bunch of pages to concatenate them into a single large page, you may start concurrent downloads using threads, but need to wait until the last page/thread is finished before you start assembling a single page out of many. That's when you use join().


Instruction:
How should I write a Windows path in a Python string literal?

Suppose I need to refer to the path C:\meshes\as. If I try writing that directly, like "C:\meshes\as", I encounter problems - either some exception, or the path just doesn't work. Is this because \ is acting as an escape character? How should I write the paths?

Output: You can use always: 'C:/mydir' This works both in Linux and Windows. Another possibility is: 'C:\\mydir' If you have problems with some names you can also try raw string literals: r'C:\mydir' However, the best practice is to use the os.path module functions that always joins with the correct path separator (os.path.sep) for your OS: os.path.join(mydir, myfile) From python 3.4 you can also use the pathlib module. This is equivalent to the above: pathlib.Path(mydir, myfile) or: pathlib.Path(mydir) / myfile


Instruction:
When would the -e, --editable option be useful with pip install?

When would the -e, or --editable option be useful with pip install? For some projects the last line in requirements.txt is -e .. What does it do exactly?

Output: As the man page says it: -e,--editable <path/url> Install a project in editable mode (i.e. setuptools "develop mode") from a local project path or a VCS url. So you would use this when trying to install a package locally, most often in the case when you are developing it on your system. It will just link the package to the original location, basically meaning any changes to the original package would reflect directly in your environment. Some nuggets around the same here and here. An example run can be: pip install -e . or pip install -e ~/ultimate-utils/ultimate-utils-proj-src/ note the second is the full path to where the setup.py would be at.


Instruction:
Checking if a string can be converted to float in Python

I've got some Python code that runs through a list of strings and converts them to integers or floating point numbers if possible. Doing this for integers is pretty easy if element.isdigit(): newelement = int(element) Floating point numbers are more difficult. Right now I'm using partition('.') to split the string and checking to make sure that one or both sides are digits. partition = element.partition('.') if (partition[0].isdigit() and partition[1] == '.' and partition[2].isdigit()) or (partition[0] == '' and partition[1] == '.' and partition[2].isdigit()) or (partition[0].isdigit() and partition[1] == '.' and partition[2] == ''): newelement = float(element) This works, but obviously the if statement for that is a bit of a bear. The other solution I considered is to just wrap the conversion in a try/catch block and see if it succeeds, as described in this question. Anyone have any other ideas? Opinions on the relative merits of the partition and try/catch approaches?

Output: I would just use.. try: float(element) except ValueError: print("Not a float") ..it's simple, and it works. Note that it will still throw OverflowError if element is e.g. 1<<1024. Another option would be a regular expression: import re if re.match(r'^-?\d+(?:\.\d+)$', element) is None: print("Not float")


Instruction:
Calculating a directory's size using Python?

Before I re-invent this particular wheel, has anybody got a nice routine for calculating the size of a directory using Python? It would be very nice if the routine would format the size nicely in Mb/Gb etc.

Output: This walks all sub-directories; summing file sizes: import os def get_size(start_path = '.'): total_size = 0 for dirpath, dirnames, filenames in os.walk(start_path): for f in filenames: fp = os.path.join(dirpath, f) # skip if it is symbolic link if not os.path.islink(fp): total_size += os.path.getsize(fp) return total_size print(get_size(), 'bytes') And a oneliner for fun using os.listdir (Does not include sub-directories): import os sum(os.path.getsize(f) for f in os.listdir('.') if os.path.isfile(f)) Reference: os.path.getsize - Gives the size in bytes os.walk os.path.islink Updated To use os.path.getsize, this is clearer than using the os.stat().st_size method. Thanks to ghostdog74 for pointing this out! os.stat - st_size Gives the size in bytes. Can also be used to get file size and other file related information. import os nbytes = sum(d.stat().st_size for d in os.scandir('.') if d.is_file()) Update 2018 If you use Python 3.4 or previous then you may consider using the more efficient walk method provided by the third-party scandir package. In Python 3.5 and later, this package has been incorporated into the standard library and os.walk has received the corresponding increase in performance. Update 2019 Recently I've been using pathlib more and more, here's a pathlib solution: from pathlib import Path root_directory = Path('.') sum(f.stat().st_size for f in root_directory.glob('**/*') if f.is_file())


Instruction:
How to step through Python code to help debug issues?

In Java/C# you can easily step through code to trace what might be going wrong, and IDE's make this process very user friendly. Can you trace through python code in a similar fashion?

Output: Yes! There's a Python debugger called pdb just for doing that! You can launch a Python program through pdb via python -m pdb myscript.py. There are a few commands you can then issue, which are documented on the pdb page. Some useful ones to remember are: b: set a breakpoint c: continue debugging until you hit a breakpoint s: step through the code n: to go to next line of code l: list source code for the current file (default: 11 lines including the line being executed) u: navigate up a stack frame d: navigate down a stack frame p: to print the value of an expression in the current context If you don't want to use a command line debugger, some IDEs like Pydev, Wing IDE or PyCharm have a GUI debugger. Wing and PyCharm are commercial products, but Wing has a free "Personal" edition, and PyCharm has a free community edition.


Instruction:
Fastest way to get the first object from a queryset in django?

Often I find myself wanting to get the first object from a queryset in Django, or return None if there aren't any. There are lots of ways to do this which all work. But I'm wondering which is the most performant. qs = MyModel.objects.filter(blah = blah) if qs.count() > 0: return qs[0] else: return None Does this result in two database calls? That seems wasteful. Is this any faster? qs = MyModel.objects.filter(blah = blah) if len(qs) > 0: return qs[0] else: return None Another option would be: qs = MyModel.objects.filter(blah = blah) try: return qs[0] except IndexError: return None This generates a single database call, which is good. But requires creating an exception object a lot of the time, which is a very memory-intensive thing to do when all you really need is a trivial if-test. How can I do this with just a single database call and without churning memory with exception objects?

Output: Django 1.6 (released Nov 2013) introduced the convenience methods first() and last() which swallow the resulting exception and return None if the queryset returns no objects.


Instruction:
Merge PDF files

Is it possible, using Python, to merge separate PDF files? Assuming so, I need to extend this a little further. I am hoping to loop through folders in a directory and repeat this procedure. And I may be pushing my luck, but is it possible to exclude a page that is contained in each of the PDFs (my report generation always creates an extra blank page)?

Output: Use Pypdf or its successor PyPDF2: A Pure-Python library built as a PDF toolkit. It is capable of: splitting documents page by page, merging documents page by page, (and much more) Here's a sample program that works with both versions. #!/usr/bin/env python import sys try: from PyPDF2 import PdfReader, PdfWriter except ImportError: from pyPdf import PdfFileReader, PdfFileWriter def pdf_cat(input_files, output_stream): input_streams = [] try: # First open all the files, then produce the output file, and # finally close the input files. This is necessary because # the data isn't read from the input files until the write # operation. Thanks to # https://stackoverflow.com/questions/6773631/problem-with-closing-python-pypdf-writing-getting-a-valueerror-i-o-operation/6773733#6773733 for input_file in input_files: input_streams.append(open(input_file, 'rb')) writer = PdfWriter() for reader in map(PdfReader, input_streams): for n in range(len(reader.pages)): writer.add_page(reader.pages[n]) writer.write(output_stream) finally: for f in input_streams: f.close() output_stream.close() if __name__ == '__main__': if sys.platform == "win32": import os, msvcrt msvcrt.setmode(sys.stdout.fileno(), os.O_BINARY) pdf_cat(sys.argv[1:], sys.stdout)


Instruction:
How do I convert a Pandas series or index to a NumPy array?

How can I get the index or column of a DataFrame as a NumPy array or Python list?

Output: To get a NumPy array, you should use the values attribute: In [1]: df = pd.DataFrame({'A': [1, 2, 3], 'B': [4, 5, 6]}, index=['a', 'b', 'c']); df A B a 1 4 b 2 5 c 3 6 In [2]: df.index.values Out[2]: array(['a', 'b', 'c'], dtype=object) This accesses how the data is already stored, so there isn't any need for a conversion. Note: This attribute is also available for many other pandas objects. In [3]: df['A'].values Out[3]: Out[16]: array([1, 2, 3]) To get the index as a list, call tolist: In [4]: df.index.tolist() Out[4]: ['a', 'b', 'c'] And similarly, for columns.


Instruction:
What is the proper way to comment functions in Python?

Is there a generally accepted way to comment functions in Python? Is the following acceptable? ######################################################### # Create a new user ######################################################### def add(self):

Output: The correct way to do it is to provide a docstring. That way, help(add) will also spit out your comment. def add(self): """Create a new user. Line 2 of comment... And so on... """ That's three double quotes to open the comment and another three double quotes to end it. You can also use any valid Python string. It doesn't need to be multiline and double quotes can be replaced by single quotes. See: PEP 257


Instruction:
How do I create a numpy array of all True or all False?

In Python, how do I create a numpy array of arbitrary shape filled with all True or all False?

Output: The answer: numpy.full((2, 2), True) Explanation: numpy creates arrays of all ones or all zeros very easily: e.g. numpy.ones((2, 2)) or numpy.zeros((2, 2)) Since True and False are represented in Python as 1 and 0, respectively, we have only to specify this array should be boolean using the optional dtype parameter and we are done: numpy.ones((2, 2), dtype=bool) returns: array([[ True, True], [ True, True]], dtype=bool) UPDATE: 30 October 2013 Since numpy version 1.8, we can use full to achieve the same result with syntax that more clearly shows our intent (as fmonegaglia points out): numpy.full((2, 2), True, dtype=bool) UPDATE: 16 January 2017 Since at least numpy version 1.12, full automatically casts to the dtype of the second parameter, so we can just write: numpy.full((2, 2), True)


Instruction:
Python void return type annotation

In python 3.x, it is common to use return type annotation of a function, such as: def foo() -> str: return "bar" What is the correct annotation for the "void" type? I'm considering 3 options: def foo() -> None: not logical IMO, because None is not a type, def foo() -> type(None): using the best syntax I know for obtaining NoneType, def foo(): omit explicit return type information. Option 2. seems the most logical to me, but I've already seen some instances of 1.

Output: Use option 1 for simplicity & adherence to spec. def foo() -> None Option 1 & 2 are the 'same' as per PEP 484 -- Type Hints, ... When used in a type hint, the expression None is considered equivalent to type(None). but the type-hinting specification does not use type(...). This is why most of the examples use None as return type.


Instruction:
Set markers for individual points on a line

I have used Matplotlib to plot lines on a figure. Now I would now like to set the style, specifically the marker, for individual points on the line. How do I do this? To clarify my question, I want to be able to set the style for individual markers on a line, not every marker on said line.

Output: Specify the keyword args linestyle and/or marker in your call to plot. For example, using a dashed line and blue circle markers: plt.plot(range(10), linestyle='--', marker='o', color='b', label='line with marker') plt.legend() A shortcut call for the same thing: plt.plot(range(10), '--bo', label='line with marker') plt.legend() Here is a list of the possible line and marker styles: ================ =============================== character description ================ =============================== - solid line style -- dashed line style -. dash-dot line style : dotted line style . point marker , pixel marker o circle marker v triangle_down marker ^ triangle_up marker < triangle_left marker > triangle_right marker 1 tri_down marker 2 tri_up marker 3 tri_left marker 4 tri_right marker s square marker p pentagon marker * star marker h hexagon1 marker H hexagon2 marker + plus marker x x marker D diamond marker d thin_diamond marker | vline marker _ hline marker ================ =============================== edit: with an example of marking an arbitrary subset of points, as requested in the comments: import numpy as np import matplotlib.pyplot as plt xs = np.linspace(-np.pi, np.pi, 30) ys = np.sin(xs) markers_on = [12, 17, 18, 19] plt.plot(xs, ys, '-gD', markevery=markers_on, label='line with select markers') plt.legend() plt.show() This last example using the markevery kwarg is possible in since 1.4+, due to the merge of this feature branch. If you are stuck on an older version of matplotlib, you can still achieve the result by overlaying a scatterplot on the line plot. See the edit history for more details.


Instruction:
Python re.sub group: number after \number

How can I replace foobar with foo123bar? This doesn't work: >>> re.sub(r'(foo)', r'\1123', 'foobar') 'J3bar' This works: >>> re.sub(r'(foo)', r'\1hi', 'foobar') 'foohibar'

Output: The answer is: re.sub(r'(foo)', r'\g<1>123', 'foobar') Relevant excerpt from the documentation: In addition to character escapes and backreferences as described above, \g<name> will use the substring matched by the group named name, as defined by the (?P<name>...) syntax. \g<number> uses the corresponding group number; \g<2> is therefore equivalent to \2, but isn’t ambiguous in a replacement such as \g<2>0. \20 would be interpreted as a reference to group 20, not a reference to group 2 followed by the literal character '0'. The backreference \g<0> substitutes in the entire substring matched by the RE.


Instruction:
Moving matplotlib legend outside of the axis makes it cutoff by the figure box

I'm familiar with the following questions: Matplotlib savefig with a legend outside the plot How to put the legend out of the plot It seems that the answers in these questions have the luxury of being able to fiddle with the exact shrinking of the axis so that the legend fits. Shrinking the axes, however, is not an ideal solution because it makes the data smaller making it actually more difficult to interpret; particularly when its complex and there are lots of things going on ... hence needing a large legend The example of a complex legend in the documentation demonstrates the need for this because the legend in their plot actually completely obscures multiple data points. http://matplotlib.sourceforge.net/users/legend_guide.html#legend-of-complex-plots What I would like to be able to do is dynamically expand the size of the figure box to accommodate the expanding figure legend. import matplotlib.pyplot as plt import numpy as np x = np.arange(-2*np.pi, 2*np.pi, 0.1) fig = plt.figure(1) ax = fig.add_subplot(111) ax.plot(x, np.sin(x), label='Sine') ax.plot(x, np.cos(x), label='Cosine') ax.plot(x, np.arctan(x), label='Inverse tan') lgd = ax.legend(loc=9, bbox_to_anchor=(0.5,0)) ax.grid('on') Notice how the final label 'Inverse tan' is actually outside the figure box (and looks badly cutoff - not publication quality!) Finally, I've been told that this is normal behaviour in R and LaTeX, so I'm a little confused why this is so difficult in python... Is there a historical reason? Is Matlab equally poor on this matter? I have the (only slightly) longer version of this code on pastebin http://pastebin.com/grVjc007

Output: [EDIT - 25th Feb 2025] My day job is no longer Python, so I'm not following the recent matplotlib developments. Please read all the newer answers here as there look to be some excellent modern suggestions compared to this solution from the ancient history of 2012. Sorry EMS, but I actually just got another response from the matplotlib mailling list (Thanks goes out to Benjamin Root). The code I am looking for is adjusting the savefig call to: fig.savefig('samplefigure', bbox_extra_artists=(lgd,), bbox_inches='tight') #Note that the bbox_extra_artists must be an iterable This is apparently similar to calling tight_layout, but instead you allow savefig to consider extra artists in the calculation. This did in fact resize the figure box as desired. import matplotlib.pyplot as plt import numpy as np plt.gcf().clear() x = np.arange(-2*np.pi, 2*np.pi, 0.1) fig = plt.figure(1) ax = fig.add_subplot(111) ax.plot(x, np.sin(x), label='Sine') ax.plot(x, np.cos(x), label='Cosine') ax.plot(x, np.arctan(x), label='Inverse tan') handles, labels = ax.get_legend_handles_labels() lgd = ax.legend(handles, labels, loc='upper center', bbox_to_anchor=(0.5,-0.1)) text = ax.text(-0.2,1.05, "Aribitrary text", transform=ax.transAxes) ax.set_title("Trigonometry") ax.grid('on') fig.savefig('samplefigure', bbox_extra_artists=(lgd,text), bbox_inches='tight') This produces: [edit] The intent of this question was to completely avoid the use of arbitrary coordinate placements of arbitrary text as was the traditional solution to these problems. Despite this, numerous edits recently have insisted on putting these in, often in ways that led to the code raising an error. I have now fixed the issues and tidied the arbitrary text to show how these are also considered within the bbox_extra_artists algorithm. [edit] Some of the comments below note that since 2019, the command has been simplified. plt.savefig('x.png', bbox_inches='tight') was sufficient. Thanks for sharing. – mateuszb Jun 27, 2019


Instruction:
Is False == 0 and True == 1 an implementation detail or is it guaranteed by the language?

Is it guaranteed that False == 0 and True == 1, in Python (assuming that they are not reassigned by the user)? For instance, is it in any way guaranteed that the following code will always produce the same results, whatever the version of Python (both existing and, likely, future ones)? 0 == False # True 1 == True # True ['zero', 'one'][False] # is 'zero' Any reference to the official documentation would be much appreciated! As noted in many answers, bool inherits from int. The question can therefore be recast as: "Does the documentation officially say that programmers can rely on booleans inheriting from integers, with the values 0 and 1?". This question is relevant for writing robust code that won't fail because of implementation details!

Output: In Python 2.x this is not guaranteed as it is possible for True and False to be reassigned. However, even if this happens, boolean True and boolean False are still properly returned for comparisons. In Python 3.x True and False are keywords and will always be equal to 1 and 0. Under normal circumstances in Python 2, and always in Python 3: False object is of type bool which is a subclass of int: object | int | bool It is the only reason why in your example, ['zero', 'one'][False] does work. It would not work with an object which is not a subclass of integer, because list indexing only works with integers, or objects that define a __index__ method (thanks mark-dickinson). Edit: It is true of the current python version, and of that of Python 3. The docs for python 2 and the docs for Python 3 both say: There are two types of integers: [...] Integers (int) [...] Booleans (bool) and in the boolean subsection: Booleans: These represent the truth values False and True [...] Boolean values behave like the values 0 and 1, respectively, in almost all contexts, the exception being that when converted to a string, the strings "False" or "True" are returned, respectively. There is also, for Python 2: In numeric contexts (for example when used as the argument to an arithmetic operator), they [False and True] behave like the integers 0 and 1, respectively. So booleans are explicitly considered as integers in Python 2 and 3. So you're safe until Python 4 comes along. ;-)


Instruction:
How to get string objects instead of Unicode from JSON

I'm using Python 2 to parse JSON from ASCII encoded text files. When loading these files with either json or simplejson, all my string values are cast to Unicode objects instead of string objects. The problem is, I have to use the data with some libraries that only accept string objects. I can't change the libraries nor update them. Is it possible to get string objects instead of Unicode ones? Example >>> import json >>> original_list = ['a', 'b'] >>> json_list = json.dumps(original_list) >>> json_list '["a", "b"]' >>> new_list = json.loads(json_list) >>> new_list [u'a', u'b'] # I want these to be of type `str`, not `unicode` (One easy and clean solution for 2017 is to use a recent version of Python — i.e. Python 3 and forward.)

Output: A solution with object_hook It works for both Python 2.7 and 3.x. import json def json_load_byteified(file_handle): return _byteify( json.load(file_handle, object_hook=_byteify), ignore_dicts=True ) def json_loads_byteified(json_text): return _byteify( json.loads(json_text, object_hook=_byteify), ignore_dicts=True ) def _byteify(data, ignore_dicts = False): if isinstance(data, str): return data # If this is a list of values, return list of byteified values if isinstance(data, list): return [ _byteify(item, ignore_dicts=True) for item in data ] # If this is a dictionary, return dictionary of byteified keys and values # but only if we haven't already byteified it if isinstance(data, dict) and not ignore_dicts: return { _byteify(key, ignore_dicts=True): _byteify(value, ignore_dicts=True) for key, value in data.items() # changed to .items() for Python 2.7/3 } # Python 3 compatible duck-typing # If this is a Unicode string, return its string representation if str(type(data)) == "<type 'unicode'>": return data.encode('utf-8') # If it's anything else, return it in its original form return data Example usage: >>> json_loads_byteified('{"Hello": "World"}') {'Hello': 'World'} >>> json_loads_byteified('"I am a top-level string"') 'I am a top-level string' >>> json_loads_byteified('7') 7 >>> json_loads_byteified('["I am inside a list"]') ['I am inside a list'] >>> json_loads_byteified('[[[[[[[["I am inside a big nest of lists"]]]]]]]]') [[[[[[[['I am inside a big nest of lists']]]]]]]] >>> json_loads_byteified('{"foo": "bar", "things": [7, {"qux": "baz", "moo": {"cow": ["milk"]}}]}') {'things': [7, {'qux': 'baz', 'moo': {'cow': ['milk']}}], 'foo': 'bar'} >>> json_load_byteified(open('somefile.json')) {'more json': 'from a file'} How does this work and why would I use it? Mark Amery's function is shorter and clearer than these ones, so what's the point of them? Why would you want to use them? Purely for performance. Mark's answer decodes the JSON text fully first with Unicode strings, then recurses through the entire decoded value to convert all strings to byte strings. This has a couple of undesirable effects: A copy of the entire decoded structure gets created in memory If your JSON object is really deeply nested (500 levels or more) then you'll hit Python's maximum recursion depth This answer mitigates both of those performance issues by using the object_hook parameter of json.load and json.loads. From the documentation: object_hook is an optional function that will be called with the result of any object literal decoded (a dict). The return value of object_hook will be used instead of the dict. This feature can be used to implement custom decoders Since dictionaries nested many levels deep in other dictionaries get passed to object_hook as they're decoded, we can byteify any strings or lists inside them at that point and avoid the need for deep recursion later. Mark's answer isn't suitable for use as an object_hook as it stands, because it recurses into nested dictionaries. We prevent that recursion in this answer with the ignore_dicts parameter to _byteify, which gets passed to it at all times except when object_hook passes it a new dict to byteify. The ignore_dicts flag tells _byteify to ignore dicts since they already been byteified. Finally, our implementations of json_load_byteified and json_loads_byteified call _byteify (with ignore_dicts=True) on the result returned from json.load or json.loads to handle the case where the JSON text being decoded doesn't have a dict at the top level.


Instruction:
How to create module-wide variables in Python?

Is there a way to set up a global variable inside of a module? When I tried to do it the most obvious way as appears below, the Python interpreter said the variable __DBNAME__ did not exist. ... __DBNAME__ = None def initDB(name): if not __DBNAME__: __DBNAME__ = name else: raise RuntimeError("Database name has already been set.") ... And after importing the module in a different file ... import mymodule mymodule.initDB('mydb.sqlite') ... And the traceback was: UnboundLocalError: local variable '__DBNAME__' referenced before assignment Any ideas? I'm trying to set up a singleton by using a module, as per this fellow's recommendation.

Output: Here is what is going on. First, the only global variables Python really has are module-scoped variables. You cannot make a variable that is truly global; all you can do is make a variable in a particular scope. (If you make a variable inside the Python interpreter, and then import other modules, your variable is in the outermost scope and thus global within your Python session.) All you have to do to make a module-global variable is just assign to a name. Imagine a file called foo.py, containing this single line: X = 1 Now imagine you import it. import foo print(foo.X) # prints 1 However, let's suppose you want to use one of your module-scope variables as a global inside a function, as in your example. Python's default is to assume that function variables are local. You simply add a global declaration in your function, before you try to use the global. def initDB(name): global __DBNAME__ # add this line! if __DBNAME__ is None: # see notes below; explicit test for None __DBNAME__ = name else: raise RuntimeError("Database name has already been set.") By the way, for this example, the simple if not __DBNAME__ test is adequate, because any string value other than an empty string will evaluate true, so any actual database name will evaluate true. But for variables that might contain a number value that might be 0, you can't just say if not variablename; in that case, you should explicitly test for None using the is operator. I modified the example to add an explicit None test. The explicit test for None is never wrong, so I default to using it. Finally, as others have noted on this page, two leading underscores signals to Python that you want the variable to be "private" to the module. If you ever do an from mymodule import *, Python will not import names with two leading underscores into your name space. But if you just do a simple import mymodule and then say dir(mymodule) you will see the "private" variables in the list, and if you explicitly refer to mymodule.__DBNAME__ Python won't care, it will just let you refer to it. The double leading underscores are a major clue to users of your module that you don't want them rebinding that name to some value of their own. It is considered best practice in Python not to do import *, but to minimize the coupling and maximize explicitness by either using mymodule.something or by explicitly doing an import like from mymodule import something. EDIT: If, for some reason, you need to do something like this in a very old version of Python that doesn't have the global keyword, there is an easy workaround. Instead of setting a module global variable directly, use a mutable type at the module global level, and store your values inside it. In your functions, the global variable name will be read-only; you won't be able to rebind the actual global variable name. (If you assign to that variable name inside your function it will only affect the local variable name inside the function.) But you can use that local variable name to access the actual global object, and store data inside it. You can use a list but your code will be ugly: __DBNAME__ = [None] # use length-1 list as a mutable # later, in code: if __DBNAME__[0] is None: __DBNAME__[0] = name A dict is better. But the most convenient is a class instance, and you can just use a trivial class: class Box: pass __m = Box() # m will contain all module-level values __m.dbname = None # database name global in module # later, in code: if __m.dbname is None: __m.dbname = name (You don't really need to capitalize the database name variable.) I like the syntactic sugar of just using __m.dbname rather than __m["DBNAME"]; it seems the most convenient solution in my opinion. But the dict solution works fine also. With a dict you can use any hashable value as a key, but when you are happy with names that are valid identifiers, you can use a trivial class like Box in the above.


Instruction:
How to print a percentage value?

Given a float between 0 and 1, how to print it as a percentage? For example, 1/3 should print as 33%.

Output: Since Python 3.0, str.format and format support a percentage presentation type: >>> f"{1/3:.0%}" '33%' >>> "{:.0%}".format(1/3) '33%' >>> format(1/3, ".0%") '33%' Percentage. Multiplies the number by 100 and displays in fixed ('f') format, followed by a percent sign. The .0 part of the format spec .0% indicates that you want zero digits of precision after the decimal point, because with f"{1/3:%}" you would get the string '33.333333%'. It works with integers, floats, and decimals. See PEP 3101.


Instruction:
How can mypy ignore a single line in a source file?

I'm using mypy in my python project for type checking. I'm also using PyYAML for reading and writing the project configuration files. Unfortunately, when using the recommended import mechanism from the PyYAML documentation this generates a spurious error in a try/except clause that attempts to import native libraries: from yaml import load, dump try: from yaml import CLoader as Loader, CDumper as Dumper except ImportError: from yaml import Loader, Dumper On my system CLoader and CDumper aren't present, which results in the errors error: Module 'yaml' has no attribute 'CLoader' and error: Module 'yaml' has no attribute 'CDumper'. Is there a way to have mypy ignore errors on this line? I was hoping that I could do something like this to have mypy skip that line: from yaml import load, dump try: from yaml import CLoader as Loader, CDumper as Dumper # nomypy except ImportError: from yaml import Loader, Dumper

Output: You can ignore type errors with # type: ignore as of version 0.2 (see issue #500, Ignore specific lines): PEP 484 uses # type: ignore for ignoring type errors on particular lines ... Also, using # type: ignore close to the top of a file [skips] checking that file altogether. Source: mypy#500. See also the mypy documentation.


Instruction:
What does model.eval() do in pytorch?

When should I use .eval()? I understand it is supposed to allow me to "evaluate my model". How do I turn it back off for training? Example training code using .eval().

Output: model.eval() is a kind of switch for some specific layers/parts of the model that behave differently during training and inference (evaluating) time. For example, Dropouts Layers, BatchNorm Layers etc. You need to turn them off during model evaluation, and .eval() will do it for you. In addition, the common practice for evaluating/validation is using torch.no_grad() in pair with model.eval() to turn off gradients computation: # evaluate model: model.eval() with torch.no_grad(): ... out_data = model(data) ... BUT, don't forget to turn back to training mode after eval step: # training step ... model.train() ...


Instruction:
How do I disable a test using pytest?

Let's say I have a bunch of tests: def test_func_one(): ... def test_func_two(): ... def test_func_three(): ... Is there a decorator or something similar that I could add to the functions to prevent pytest from running just that test? The result might look something like... @pytest.disable() def test_func_one(): ... def test_func_two(): ... def test_func_three(): ...

Output: Pytest has the skip and skipif decorators, similar to the Python unittest module (which uses skip and skipIf), which can be found in the documentation here. Examples from the link can be found here: @pytest.mark.skip(reason="no way of currently testing this") def test_the_unknown(): ... import sys @pytest.mark.skipif(sys.version_info < (3,3), reason="requires python3.3") def test_function(): ... The first example always skips the test, the second example allows you to conditionally skip tests (great when tests depend on the platform, executable version, or optional libraries). For example, if I want to check if someone has the library pandas installed for a test. @pytest.mark.skipif( not importlib.util.find_spec("pandas"), reason="requires the Pandas library" ) def test_pandas_function(): import pandas ...


Instruction:
How to avoid .pyc files?

Can I run the python interpreter without generating the compiled .pyc files?

Output: From "What’s New in Python 2.6 - Interpreter Changes": Python can now be prevented from writing .pyc or .pyo files by supplying the -B switch to the Python interpreter, or by setting the PYTHONDONTWRITEBYTECODE environment variable before running the interpreter. This setting is available to Python programs as the sys.dont_write_bytecode variable, and Python code can change the value to modify the interpreter’s behaviour. So run your program as python -B prog.py. Update 2010-11-27: Python 3.2 addresses the issue of cluttering source folders with .pyc files by introducing a special __pycache__ subfolder, see What's New in Python 3.2 - PYC Repository Directories. NOTE: The default behavior is to generate the bytecode and is done for "performance" reasons (for more information see here for python2 and see here for python3). The generation of bytecode .pyc files is a form of caching (i.e. greatly improves average performance). Configuring python with PYTHONDONTWRITEBYTECODE=1 can be bad for python performance (for python2 see https://www.python.org/dev/peps/pep-0304/ and for python3 see https://www.python.org/dev/peps/pep-3147/ ). If you are interested in the performance impact please see here https://github.com/python/cpython .


Instruction:
Getting distance between two points based on latitude/longitude

I tried implementing the formula in Finding distances based on Latitude and Longitude. The applet does good for the two points I am testing: Yet my code is not working. from math import sin, cos, sqrt, atan2 R = 6373.0 lat1 = 52.2296756 lon1 = 21.0122287 lat2 = 52.406374 lon2 = 16.9251681 dlon = lon2 - lon1 dlat = lat2 - lat1 a = (sin(dlat/2))**2 + cos(lat1) * cos(lat2) * (sin(dlon/2))**2 c = 2 * atan2(sqrt(a), sqrt(1-a)) distance = R * c print "Result", distance print "Should be", 278.546 It returns the distance 5447.05546147. Why?

Output: Just as a note, if you just need a quick and easy way of finding the distance between two points, I strongly recommend using the approach described in Kurt's answer instead of reimplementing Haversine—see his post for rationale. This answer focuses just on answering the specific bug the OP ran into. It's because in Python, all the trigonometry functions use radians, not degrees. You can either convert the numbers manually to radians, or use the radians function from the math module: from math import sin, cos, sqrt, atan2, radians # Approximate radius of earth in km R = 6373.0 lat1 = radians(52.2296756) lon1 = radians(21.0122287) lat2 = radians(52.406374) lon2 = radians(16.9251681) dlon = lon2 - lon1 dlat = lat2 - lat1 a = sin(dlat / 2)**2 + cos(lat1) * cos(lat2) * sin(dlon / 2)**2 c = 2 * atan2(sqrt(a), sqrt(1 - a)) distance = R * c print("Result: ", distance) print("Should be: ", 278.546, "km") The distance is now returning the correct value of 278.545589351 km.


Instruction:
Replacing Pandas or Numpy Nan with a None to use with MysqlDB

I am trying to write a Pandas dataframe (or can use a numpy array) to a mysql database using MysqlDB . MysqlDB doesn't seem understand 'nan' and my database throws out an error saying nan is not in the field list. I need to find a way to convert the 'nan' into a NoneType. Any ideas?

Output: For pandas > 1.3.0 see this answer. @bogatron has it right, you can use where, it's worth noting that you can do this natively in pandas: df1 = df.where(pd.notnull(df), None) Note: this changes the dtype of all columns to object. Example: In [1]: df = pd.DataFrame([1, np.nan]) In [2]: df Out[2]: 0 0 1 1 NaN In [3]: df1 = df.where(pd.notnull(df), None) In [4]: df1 Out[4]: 0 0 1 1 None Note: what you cannot do recast the DataFrames dtype to allow all datatypes types, using astype, and then the DataFrame fillna method: df1 = df.astype(object).replace(np.nan, 'None') Unfortunately neither this, nor using replace, works with None see this (closed) issue. As an aside, it's worth noting that for most use cases you don't need to replace NaN with None, see this question about the difference between NaN and None in pandas. However, in this specific case it seems you do (at least at the time of this answer).


Instruction:
What do *args and **kwargs mean?

What exactly do *args and **kwargs mean? According to the Python documentation, from what it seems, it passes in a tuple of arguments. def foo(hello, *args): print(hello) for each in args: print(each) if __name__ == '__main__': foo("LOVE", ["lol", "lololol"]) This prints out: LOVE ['lol', 'lololol'] How do you effectively use them?

Output: Putting *args and/or **kwargs as the last items in your function definition’s argument list allows that function to accept an arbitrary number of arguments and/or keyword arguments. For example, if you wanted to write a function that returned the sum of all its arguments, no matter how many you supply, you could write it like this: def my_sum(*args): return sum(args) It’s probably more commonly used in object-oriented programming, when you’re overriding a function, and want to call the original function with whatever arguments the user passes in. You don’t actually have to call them args and kwargs, that’s just a convention. It’s the * and ** that do the magic. There's a more in-depth look in the official Python documentation on arbitrary argument lists.


Instruction:
How do I copy a string to the clipboard?

I'm trying to make a basic Windows application that builds a string out of user input and then adds it to the clipboard. How do I copy a string to the clipboard using Python?

Output: Actually, pywin32 and ctypes seem to be an overkill for this simple task. tkinter is a cross-platform GUI framework, which ships with Python by default and has clipboard accessing methods along with other cool stuff. If all you need is to put some text to system clipboard, this will do it: from tkinter import Tk # in Python 2, use "Tkinter" instead r = Tk() r.withdraw() r.clipboard_clear() r.clipboard_append('i can has clipboardz?') r.update() # now it stays on the clipboard after the window is closed r.destroy() And that's all, no need to mess around with platform-specific third-party libraries. If you are using Python 2, replace tkinter with Tkinter.


Instruction:
Multiple aggregations of the same column using pandas GroupBy.agg()

Is there a pandas built-in way to apply two different aggregating functions f1, f2 to the same column df["returns"], without having to call agg() multiple times? Example dataframe: import pandas as pd import datetime as dt import numpy as np pd.np.random.seed(0) df = pd.DataFrame({ "date" : [dt.date(2012, x, 1) for x in range(1, 11)], "returns" : 0.05 * np.random.randn(10), "dummy" : np.repeat(1, 10) }) The syntactically wrong, but intuitively right, way to do it would be: # Assume `f1` and `f2` are defined for aggregating. df.groupby("dummy").agg({"returns": f1, "returns": f2}) Obviously, Python doesn't allow duplicate keys. Is there any other manner for expressing the input to agg()? Perhaps a list of tuples [(column, function)] would work better, to allow multiple functions applied to the same column? But agg() seems like it only accepts a dictionary. Is there a workaround for this besides defining an auxiliary function that just applies both of the functions inside of it? (How would this work with aggregation anyway?)

Output: As of 2022-06-20, the below is the accepted practice for aggregations: df.groupby('dummy').agg( Mean=('returns', np.mean), Sum=('returns', np.sum)) see this answer for more information. Below the fold included for historical versions of pandas. You can simply pass the functions as a list: In [20]: df.groupby("dummy").agg({"returns": [np.mean, np.sum]}) Out[20]: mean sum dummy 1 0.036901 0.369012 or as a dictionary: In [21]: df.groupby('dummy').agg({'returns': {'Mean': np.mean, 'Sum': np.sum}}) Out[21]: returns Mean Sum dummy 1 0.036901 0.369012


Instruction:
Simplest async/await example possible in Python

I've read many examples, blog posts, questions/answers about asyncio / async / await in Python 3.5+, many were complex, the simplest I found was probably this one. Still it uses ensure_future, and for learning purposes about asynchronous programming in Python, I would like to see an even more minimal example, and what are the minimal tools necessary to do a basic async / await example. Question: is it possible to give a simple example showing how async / await works, by using only these two keywords + code to run the async loop + other Python code but no other asyncio functions? Example: something like this: import asyncio async def async_foo(): print("async_foo started") await asyncio.sleep(5) print("async_foo done") async def main(): asyncio.ensure_future(async_foo()) # fire and forget async_foo() print('Do some actions 1') await asyncio.sleep(5) print('Do some actions 2') loop = asyncio.get_event_loop() loop.run_until_complete(main()) but without ensure_future, and still demonstrates how await / async works.

Output: To answer your questions, I will provide three different solutions to the same problem. Case 1: just normal Python import time def sleep(): print(f'Time: {time.time() - start:.2f}') time.sleep(1) def sum_(name, numbers): total = 0 for number in numbers: print(f'Task {name}: Computing {total}+{number}') sleep() total += number print(f'Task {name}: Sum = {total}\n') start = time.time() tasks = [ sum_("A", [1, 2]), sum_("B", [1, 2, 3]), ] end = time.time() print(f'Time: {end-start:.2f} sec') Output: Task A: Computing 0+1 Time: 0.00 Task A: Computing 1+2 Time: 1.00 Task A: Sum = 3 Task B: Computing 0+1 Time: 2.00 Task B: Computing 1+2 Time: 3.00 Task B: Computing 3+3 Time: 4.00 Task B: Sum = 6 Time: 5.00 sec Case 2: async/await done wrong import asyncio import time async def sleep(): print(f'Time: {time.time() - start:.2f}') time.sleep(1) async def sum_(name, numbers): total = 0 for number in numbers: print(f'Task {name}: Computing {total}+{number}') await sleep() total += number print(f'Task {name}: Sum = {total}\n') start = time.time() loop = asyncio.new_event_loop() tasks = [ loop.create_task(sum_("A", [1, 2])), loop.create_task(sum_("B", [1, 2, 3])), ] loop.run_until_complete(asyncio.wait(tasks)) loop.close() end = time.time() print(f'Time: {end-start:.2f} sec') Output: Task A: Computing 0+1 Time: 0.00 Task A: Computing 1+2 Time: 1.00 Task A: Sum = 3 Task B: Computing 0+1 Time: 2.00 Task B: Computing 1+2 Time: 3.00 Task B: Computing 3+3 Time: 4.00 Task B: Sum = 6 Time: 5.00 sec Case 3: async/await done right The same as case 2, except the sleep function: async def sleep(): print(f'Time: {time.time() - start:.2f}') await asyncio.sleep(1) Output: Task A: Computing 0+1 Time: 0.00 Task B: Computing 0+1 Time: 0.00 Task A: Computing 1+2 Time: 1.01 Task B: Computing 1+2 Time: 1.01 Task A: Sum = 3 Task B: Computing 3+3 Time: 2.01 Task B: Sum = 6 Time: 3.02 sec Case 1 and case 2 give the same 5 seconds, whereas case 3 just 3 seconds. So the async/await done right is faster. The reason for the difference is within the implementation of the sleep function. # Case 1 def sleep(): ... time.sleep(1) # Case 2 async def sleep(): ... time.sleep(1) # Case 3 async def sleep(): ... await asyncio.sleep(1) In case 1 and case 2, they are the "same": they "sleep" without allowing others to use the resources. Whereas in case 3, it allows access to the resources when it is asleep. In case 2, we added async to the normal function. However the event loop will run it without interruption. Why? Because we didn't say where the loop is allowed to interrupt your function to run another task. In case 3, we told the event loop exactly where to interrupt the function to run another task. Where exactly? Right here! await asyncio.sleep(1) For more on this, read here. Consider reading A Hitchhikers Guide to Asynchronous Programming Asyncio Futures and Coroutines


Instruction:
What do square brackets mean in pip install?

$ pip install "splinter[django]" What do these square brackets do?

Output: The syntax that you are using is: pip install "project[extra]" In your case, you are installing the splinter package which has the added support for django. • pip install splinter django would install two packages named splinter and django. • pip install splinter[django], on the other hand, installs splinter, but it also installs optional dependencies defined by splinter using the keyword in the brackets. In this case, as of 2024-05-15 it's Django, lxml and cssselect. Note that the keyword in brackets has nothing to do with the django package itself, but is just a string defined by the splinter package for a particular set of dependencies that also get installed. How the argument django is interpreted depends on the build system, but any setuptools-based build system (including most instances of setup.py) will likely just use them as a hook for optional dependencies. It's worth noting that the syntax supports using multiple keywords, e.g.: pip install "splinter[django,flask,selenium]" Kudos to @chepner for adding context in the comments.


Instruction:
How to draw a rectangle on image

How to draw a rectangle on an image, like this: import matplotlib.pyplot as plt from PIL import Image import numpy as np im = np.array(Image.open('dog.png'), dtype=np.uint8) plt.imshow(im) To make it clear, I meant to draw a rectangle on top of the image for visualization, not to change the image data. So using matplotlib.patches.Patch would be the best option.

Output: You can add a Rectangle patch to the matplotlib Axes. For example (using the image from the tutorial here): import matplotlib.pyplot as plt import matplotlib.patches as patches from PIL import Image im = Image.open('stinkbug.png') # Create figure and axes fig, ax = plt.subplots() # Display the image ax.imshow(im) # Create a Rectangle patch rect = patches.Rectangle((50, 100), 40, 30, linewidth=1, edgecolor='r', facecolor='none') # Add the patch to the Axes ax.add_patch(rect) plt.show()


Instruction:
pandas get column average/mean

I can't get the average or mean of a column in pandas. A have a dataframe. Neither of things I tried below gives me the average of the column weight >>> allDF ID birthyear weight 0 619040 1962 0.1231231 1 600161 1963 0.981742 2 25602033 1963 1.3123124 3 624870 1987 0.94212 The following returns several values, not one: allDF[['weight']].mean(axis=1) So does this: allDF.groupby('weight').mean()

Output: If you only want the mean of the weight column, select the column (which is a Series) and call .mean(): In [479]: df Out[479]: ID birthyear weight 0 619040 1962 0.123123 1 600161 1963 0.981742 2 25602033 1963 1.312312 3 624870 1987 0.942120 In [480]: df.loc[:, 'weight'].mean() Out[480]: 0.83982437500000007


Instruction:
Set Matplotlib colorbar size to match graph

I cannot get the colorbar on imshow graphs like this one to be the same height as the graph, short of using Photoshop after the fact. How do I get the heights to match?

Output: You can do this easily with a matplotlib AxisDivider. The example from the linked page also works without using subplots: import matplotlib.pyplot as plt from mpl_toolkits.axes_grid1 import make_axes_locatable import numpy as np plt.figure() ax = plt.gca() im = ax.imshow(np.arange(100).reshape((10,10))) # create an axes on the right side of ax. The width of cax will be 5% # of ax and the padding between cax and ax will be fixed at 0.05 inch. divider = make_axes_locatable(ax) cax = divider.append_axes("right", size="5%", pad=0.05) plt.colorbar(im, cax=cax)


Instruction:
How to output loop.counter in python jinja template?

I want to be able to output the current loop iteration to my template. According to the docs, there is a loop.counter variable that I am trying to use: <ul> {% for user in userlist %} <li> {{ user }} {{loop.counter}} </li> {% if loop.counter == 1 %} This is the First user {% endif %} {% endfor %} </ul> But is being outputed to my template. What is the correct syntax?

Output: The counter variable inside the loop is called loop.index in Jinja2. >>> from jinja2 import Template >>> s = "{% for element in elements %}{{loop.index}} {% endfor %}" >>> Template(s).render(elements=["a", "b", "c", "d"]) 1 2 3 4 In addition to loop.index, there is also loop.index0 (index starting at 0) loop.revindex (reverse index; ending at 1) loop.revindex0 (reverse index; ending at 0) Even more at http://jinja.pocoo.org/docs/templates/.


Instruction:
How can I create a full path to a file from parts (e.g. path to the folder, name and extension)?

I need to pass a file path name to a module. How do I build the file path from a directory name, base filename, and a file format string? The directory may or may not exist at the time of call. For example: dir_name='/home/me/dev/my_reports' base_filename='daily_report' format = 'pdf' I need to create a string '/home/me/dev/my_reports/daily_report.pdf' Concatenating the pieces manually doesn't seem to be a good way. I tried os.path.join: join(dir_name,base_filename,format) but it gives /home/me/dev/my_reports/daily_report/pdf

Output: This works fine: os.path.join(dir_name, base_filename + '.' + filename_suffix) Keep in mind that os.path.join() exists only because different operating systems use different path separator characters. It smooths over that difference so cross-platform code doesn't have to be cluttered with special cases for each OS. There is no need to do this for file name "extensions" (see footnote) because they are always preceded by a dot character, on every OS that implements them. If using a function anyway makes you feel better (and you like needlessly complicating your code), you can do this: os.path.join(dir_name, '.'.join((base_filename, filename_suffix))) If you prefer to keep your code clean, simply include the dot in the suffix: suffix = '.pdf' os.path.join(dir_name, base_filename + suffix) That approach also happens to be compatible with the suffix conventions in pathlib, which was introduced in python 3.4 a few years after this question was asked. New code that doesn't require backward compatibility can do this: suffix = '.pdf' pathlib.PurePath(dir_name, base_filename + suffix) You might be tempted to use the shorter Path() instead of PurePath() if you're only handling paths for the local OS. I would question that choice, given the cross-platform issues behind the original question. Warning: Do not use pathlib's with_suffix() for this purpose. That method will corrupt base_filename if it ever contains a dot. Footnote: Outside of Microsoft operating systems, there is no such thing as a file name "extension". Its presence on Windows comes from MS-DOS and FAT, which borrowed it from CP/M, which has been dead for decades. That dot-plus-three-letters that many of us are accustomed to seeing is just part of the file name on every other modern OS, where it has no built-in meaning.


Instruction:
How can I replace (or strip) an extension from a filename in Python?

Is there a built-in function in Python that would replace (or remove, whatever) the extension of a filename (if it has one)? Example: print replace_extension('/home/user/somefile.txt', '.jpg') In my example: /home/user/somefile.txt would become /home/user/somefile.jpg I don't know if it matters, but I need this for a SCons module I'm writing. (So perhaps there is some SCons specific function I can use ?) I'd like something clean. Doing a simple string replacement of all occurrences of .txt within the string is obviously not clean. (This would fail if my filename is somefile.txt.txt.txt)

Output: Try os.path.splitext it should do what you want. import os print os.path.splitext('/home/user/somefile.txt')[0]+'.jpg' # /home/user/somefile.jpg os.path.splitext('/home/user/somefile.txt') # returns ('/home/user/somefile', '.txt')