Instruction:
Round number to nearest integer

I've been trying to round long float numbers like: 32.268907563; 32.268907563; 31.2396694215; 33.6206896552; ... With no success so far. I tried math.ceil(x), math.floor(x) (although that would round up or down, which is not what I'm looking for) and round(x) which didn't work either (still float numbers). What could I do? Code: for i in widthRange: for j in heightRange: r, g, b = rgb_im.getpixel((i, j)) h, s, v = colorsys.rgb_to_hsv(r/255.0, g/255.0, b/255.0) h = h * 360 int(round(h)) print(h)

Output: TL;DR: round(x) will round it and change it to integer. You are not assigning round(h) to any variable. When you call round(h), it returns the integer number but does nothing else; you have to change that line for: h = round(h) to assign the new value to h. As @plowman said in the comments, Python's round() doesn't work as one would normally expect, and that's because the way the number is stored as a variable is usually not the way you see it on screen. There are lots of answers that explain this behavior. One way to avoid this problem is to use the Decimal as stated by this answer. In order for this answer to work properly without using extra libraries it would be convenient to use a custom rounding function. I came up with the following solution, that as far as I tested avoided all the storing issues. It is based on using the string representation, obtained with repr() (NOT str()!). It looks hacky but it was the only way I found to solve all the cases. It works with both Python2 and Python3. def proper_round(num, dec=0): num = str(num)[:str(num).index('.')+dec+2] if num[-1]>='5': return float(num[:-2-(not dec)]+str(int(num[-2-(not dec)])+1)) return float(num[:-1]) Tests: >>> print(proper_round(1.0005,3)) 1.001 >>> print(proper_round(2.0005,3)) 2.001 >>> print(proper_round(3.0005,3)) 3.001 >>> print(proper_round(4.0005,3)) 4.001 >>> print(proper_round(5.0005,3)) 5.001 >>> print(proper_round(1.005,2)) 1.01 >>> print(proper_round(2.005,2)) 2.01 >>> print(proper_round(3.005,2)) 3.01 >>> print(proper_round(4.005,2)) 4.01 >>> print(proper_round(5.005,2)) 5.01 >>> print(proper_round(1.05,1)) 1.1 >>> print(proper_round(2.05,1)) 2.1 >>> print(proper_round(3.05,1)) 3.1 >>> print(proper_round(4.05,1)) 4.1 >>> print(proper_round(5.05,1)) 5.1 >>> print(proper_round(1.5)) 2.0 >>> print(proper_round(2.5)) 3.0 >>> print(proper_round(3.5)) 4.0 >>> print(proper_round(4.5)) 5.0 >>> print(proper_round(5.5)) 6.0 >>> >>> print(proper_round(1.000499999999,3)) 1.0 >>> print(proper_round(2.000499999999,3)) 2.0 >>> print(proper_round(3.000499999999,3)) 3.0 >>> print(proper_round(4.000499999999,3)) 4.0 >>> print(proper_round(5.000499999999,3)) 5.0 >>> print(proper_round(1.00499999999,2)) 1.0 >>> print(proper_round(2.00499999999,2)) 2.0 >>> print(proper_round(3.00499999999,2)) 3.0 >>> print(proper_round(4.00499999999,2)) 4.0 >>> print(proper_round(5.00499999999,2)) 5.0 >>> print(proper_round(1.0499999999,1)) 1.0 >>> print(proper_round(2.0499999999,1)) 2.0 >>> print(proper_round(3.0499999999,1)) 3.0 >>> print(proper_round(4.0499999999,1)) 4.0 >>> print(proper_round(5.0499999999,1)) 5.0 >>> print(proper_round(1.499999999)) 1.0 >>> print(proper_round(2.499999999)) 2.0 >>> print(proper_round(3.499999999)) 3.0 >>> print(proper_round(4.499999999)) 4.0 >>> print(proper_round(5.499999999)) 5.0 Finally, the corrected answer would be: # Having proper_round defined as previously stated h = int(proper_round(h)) Tests: >>> proper_round(6.39764125, 2) 6.31 # should be 6.4 >>> proper_round(6.9764125, 1) 6.1 # should be 7 The gotcha here is that the dec-th decimal can be 9 and if the dec+1-th digit >=5 the 9 will become a 0 and a 1 should be carried to the dec-1-th digit. If we take this into consideration, we get: def proper_round(num, dec=0): num = str(num)[:str(num).index('.')+dec+2] if num[-1]>='5': a = num[:-2-(not dec)] # integer part b = int(num[-2-(not dec)])+1 # decimal part return float(a)+b**(-dec+1) if a and b == 10 else float(a+str(b)) return float(num[:-1]) In the situation described above b = 10 and the previous version would just concatenate a and b which would result in a concatenation of 10 where the trailing 0 would disappear. This version transforms b to the right decimal place based on dec, as a proper carry.


Instruction:
Convert array of indices to one-hot encoded array in NumPy

Given a 1D array of indices: a = array([1, 0, 3]) I want to one-hot encode this as a 2D array: b = array([[0,1,0,0], [1,0,0,0], [0,0,0,1]])

Output: Create a zeroed array b with enough columns, i.e. a.max() + 1. Then, for each row i, set the a[i]th column to 1. >>> a = np.array([1, 0, 3]) >>> b = np.zeros((a.size, a.max() + 1)) >>> b[np.arange(a.size), a] = 1 >>> b array([[ 0., 1., 0., 0.], [ 1., 0., 0., 0.], [ 0., 0., 0., 1.]])


Instruction:
How do I print the model summary in PyTorch?

How do I print the summary of a model in PyTorch like what model.summary() does in Keras: Model Summary: ____________________________________________________________________________________________________ Layer (type) Output Shape Param # Connected to ==================================================================================================== input_1 (InputLayer) (None, 1, 15, 27) 0 ____________________________________________________________________________________________________ convolution2d_1 (Convolution2D) (None, 8, 15, 27) 872 input_1[0][0] ____________________________________________________________________________________________________ maxpooling2d_1 (MaxPooling2D) (None, 8, 7, 27) 0 convolution2d_1[0][0] ____________________________________________________________________________________________________ flatten_1 (Flatten) (None, 1512) 0 maxpooling2d_1[0][0] ____________________________________________________________________________________________________ dense_1 (Dense) (None, 1) 1513 flatten_1[0][0] ==================================================================================================== Total params: 2,385 Trainable params: 2,385 Non-trainable params: 0

Output: While you will not get as detailed information about the model as in Keras' model.summary, simply printing the model will give you some idea about the different layers involved and their specifications. For instance: from torchvision import models model = models.vgg16() print(model) The output in this case would be something as follows: VGG ( (features): Sequential ( (0): Conv2d(3, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)) (1): ReLU (inplace) (2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)) (3): ReLU (inplace) (4): MaxPool2d (size=(2, 2), stride=(2, 2), dilation=(1, 1)) (5): Conv2d(64, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)) (6): ReLU (inplace) (7): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)) (8): ReLU (inplace) (9): MaxPool2d (size=(2, 2), stride=(2, 2), dilation=(1, 1)) (10): Conv2d(128, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)) (11): ReLU (inplace) (12): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)) (13): ReLU (inplace) (14): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)) (15): ReLU (inplace) (16): MaxPool2d (size=(2, 2), stride=(2, 2), dilation=(1, 1)) (17): Conv2d(256, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)) (18): ReLU (inplace) (19): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)) (20): ReLU (inplace) (21): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)) (22): ReLU (inplace) (23): MaxPool2d (size=(2, 2), stride=(2, 2), dilation=(1, 1)) (24): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)) (25): ReLU (inplace) (26): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)) (27): ReLU (inplace) (28): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)) (29): ReLU (inplace) (30): MaxPool2d (size=(2, 2), stride=(2, 2), dilation=(1, 1)) ) (classifier): Sequential ( (0): Dropout (p = 0.5) (1): Linear (25088 -> 4096) (2): ReLU (inplace) (3): Dropout (p = 0.5) (4): Linear (4096 -> 4096) (5): ReLU (inplace) (6): Linear (4096 -> 1000) ) ) Now you could, as mentioned by Kashyap, use the state_dict method to get the weights of the different layers. But using this listing of the layers would perhaps provide more direction is creating a helper function to get that Keras like model summary!


Instruction:
Split a string by spaces -- preserving quoted substrings -- in Python

I have a string which is like this: this is "a test" I'm trying to write something in Python to split it up by space while ignoring spaces within quotes. The result I'm looking for is: ['this', 'is', 'a test'] PS. I know you are going to ask "what happens if there are quotes within the quotes, well, in my application, that will never happen.

Output: You want split, from the built-in shlex module. >>> import shlex >>> shlex.split('this is "a test"') ['this', 'is', 'a test'] This should do exactly what you want. If you want to preserve the quotation marks, then you can pass the posix=False kwarg. >>> shlex.split('this is "a test"', posix=False) ['this', 'is', '"a test"']


Instruction:
Sending "User-agent" using Requests library in Python

I want to send a value for "User-agent" while requesting a webpage using Python Requests. I am not sure is if it is okay to send this as a part of the header, as in the code below: debug = {'verbose': sys.stderr} user_agent = {'User-agent': 'Mozilla/5.0'} response = requests.get(url, headers = user_agent, config=debug) The debug information isn't showing the headers being sent during the request. Is it acceptable to send this information in the header? If not, how can I send it?

Output: The user-agent should be specified as a field in the header. Here is a list of HTTP header fields, and you'd probably be interested in request-specific fields, which includes User-Agent. If you're using requests v2.13 and newer The simplest way to do what you want is to create a dictionary and specify your headers directly, like so: import requests url = 'SOME URL' headers = { 'User-Agent': 'My User Agent 1.0', 'From': 'youremail@domain.example' # This is another valid field } response = requests.get(url, headers=headers) If you're using requests v2.12.x and older Older versions of requests clobbered default headers, so you'd want to do the following to preserve default headers and then add your own to them. import requests url = 'SOME URL' # Get a copy of the default headers that requests would use headers = requests.utils.default_headers() # Update the headers with your custom ones # You don't have to worry about case-sensitivity with # the dictionary keys, because default_headers uses a custom # CaseInsensitiveDict implementation within requests' source code. headers.update( { 'User-Agent': 'My User Agent 1.0', } ) response = requests.get(url, headers=headers)


Instruction:
Does uninstalling a package with "pip" also remove the dependent packages?

When you use pip to install a package, all the required packages will also be installed with it (dependencies). Does uninstalling that package also remove the dependent packages?

Output: No, it doesn't uninstall the dependencies packages. It only removes the specified package: $ pip install specloud $ pip freeze # all the packages here are dependencies of specloud package figleaf==0.6.1 nose==1.1.2 pinocchio==0.3 specloud==0.4.5 $ pip uninstall specloud $ pip freeze figleaf==0.6.1 nose==1.1.2 pinocchio==0.3 As you can see those packages are dependencies from specloud and they're still there, but not the specloud package itself. As mentioned below, you can install and use the pip-autoremove utility to remove a package plus unused dependencies.


Instruction:
How do I update a Python package?

I'm running Ubuntu 9:10 and a package called M2Crypto is installed (version is 0.19.1). I need to download, build and install the latest version of the M2Crypto package (0.20.2). The 0.19.1 package has files in a number of locations including (/usr/share/pyshared and /usr/lib/pymodules.python2.6). How can I completely uninstall version 0.19.1 from my system before installing 0.20.2?

Output: You might want to look into a Python package manager like pip. If you don't want to use a Python package manager, you should be able to download M2Crypto and build/compile/install over the old installation.


Instruction:
Install tkinter for Python

I am trying to import Tkinter. However, I get an error stating that Tkinter has not been installed: ImportError: No module named _tkinter, please install the python-tk package I could probably install it using synaptic manager (can I?), however, I would have to install it on every machine I program on. Would it be possible to add the Tkinter library into my workspace and reference it from there?

Output: It is not very easy to install Tkinter locally to use with system-provided Python. You may build it from sources, but this is usually not the best idea with a binary package-based distro you're apparently running. It's safer to apt-get install python3-tk on your machine(s). (Works on Debian-derived distributions like for Ubuntu; refer to your package manager and package list on other distributions.)


Instruction:
How can I install packages offline?

What's the best way to download a Python package and its dependencies from PyPI for offline installation on another machine? Is there an easy way to do this with pip or easy_install? I'm trying to install the Requests library on a FreeBSD box that is not connected to the Internet_.

Output: If the package is on PyPI, download it and its dependencies to some local directory. E.g., mkdir /pypi && cd /pypi ls -la Output: -rw-r--r-- 1 pavel staff 237954 Apr 19 11:31 Flask-WTF-0.6.tar.gz -rw-r--r-- 1 pavel staff 389741 Feb 22 17:10 Jinja2-2.6.tar.gz -rw-r--r-- 1 pavel staff 70305 Apr 11 00:28 MySQL-python-1.2.3.tar.gz -rw-r--r-- 1 pavel staff 2597214 Apr 10 18:26 SQLAlchemy-0.7.6.tar.gz -rw-r--r-- 1 pavel staff 1108056 Feb 22 17:10 Werkzeug-0.8.2.tar.gz -rw-r--r-- 1 pavel staff 488207 Apr 10 18:26 boto-2.3.0.tar.gz -rw-r--r-- 1 pavel staff 490192 Apr 16 12:00 flask-0.9-dev-2a6c80a.tar.gz Some packages may have to be archived into similar looking tarballs by hand. I do it a lot when I want a more recent (less stable) version of something. Some packages aren't on PyPI, so same applies to them. Suppose you have a properly formed Python application in ~/src/myapp. ~/src/myapp/setup.py will have install_requires list that mentions one or more things that you have in your /pypi directory. Like so: install_requires = [ 'boto', 'Flask', 'Werkzeug', # And so on If you want to be able to run your application with all the necessary dependencies while still hacking on it, you'll do something like this: cd ~/src/myapp python setup.py develop --always-unzip --allow-hosts=None --find-links=/pypi This way, your application will be executed straight from your source directory. You can hack on things, and then rerun the application without rebuilding anything. If you want to install your application and its dependencies into the current python environment, you'll do something like this: cd ~/src/myapp easy_install --always-unzip --allow-hosts=None --find-links=/pypi . In both cases, the build will fail if one or more dependencies aren't present in the /pypi directory. It won't attempt to promiscuously install missing things from Internet. I highly recommend to invoke setup.py develop ... and easy_install ... within an active virtual environment to avoid contaminating your global Python environment. It is (virtualenv that is) pretty much the way to go. Never install anything into global Python environment. If the machine that you've built your application has same architecture as the machine on which you want to deploy it, you can simply tarball the entire virtual environment directory into which you easy_install-ed everything. Just before tarballing though, you must make the virtual environment directory relocatable (see the --relocatable option). Note: the destination machine needs to have the same version of Python installed, and also any C-based dependencies your application may have must be preinstalled there too (e.g., say if you depend on PIL, then libpng, libjpeg, etc. must be preinstalled).


Instruction:
Merge two dataframes by index

I have the following dataframes: > df1 id begin conditional confidence discoveryTechnique 0 278 56 false 0.0 1 1 421 18 false 0.0 1 > df2 concept 0 A 1 B How do I merge on the indices to get: id begin conditional confidence discoveryTechnique concept 0 278 56 false 0.0 1 A 1 421 18 false 0.0 1 B I ask because it is my understanding that merge() i.e. df1.merge(df2) uses columns to do the matching. In fact, doing this I get: Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/usr/local/lib/python2.7/dist-packages/pandas/core/frame.py", line 4618, in merge copy=copy, indicator=indicator) File "/usr/local/lib/python2.7/dist-packages/pandas/tools/merge.py", line 58, in merge copy=copy, indicator=indicator) File "/usr/local/lib/python2.7/dist-packages/pandas/tools/merge.py", line 491, in __init__ self._validate_specification() File "/usr/local/lib/python2.7/dist-packages/pandas/tools/merge.py", line 812, in _validate_specification raise MergeError('No common columns to perform merge on') pandas.tools.merge.MergeError: No common columns to perform merge on Is it bad practice to merge on index? Is it impossible? If so, how can I shift the index into a new column called "index"?

Output: Use merge, which is an inner join by default: pd.merge(df1, df2, left_index=True, right_index=True) Or join, which is a left join by default: df1.join(df2) Or concat, which is an outer join by default: pd.concat([df1, df2], axis=1) Samples: df1 = pd.DataFrame({'a':range(6), 'b':[5,3,6,9,2,4]}, index=list('abcdef')) print (df1) a b a 0 5 b 1 3 c 2 6 d 3 9 e 4 2 f 5 4 df2 = pd.DataFrame({'c':range(4), 'd':[10,20,30, 40]}, index=list('abhi')) print (df2) c d a 0 10 b 1 20 h 2 30 i 3 40 # Default inner join df3 = pd.merge(df1, df2, left_index=True, right_index=True) print (df3) a b c d a 0 5 0 10 b 1 3 1 20 # Default left join df4 = df1.join(df2) print (df4) a b c d a 0 5 0.0 10.0 b 1 3 1.0 20.0 c 2 6 NaN NaN d 3 9 NaN NaN e 4 2 NaN NaN f 5 4 NaN NaN # Default outer join df5 = pd.concat([df1, df2], axis=1) print (df5) a b c d a 0.0 5.0 0.0 10.0 b 1.0 3.0 1.0 20.0 c 2.0 6.0 NaN NaN d 3.0 9.0 NaN NaN e 4.0 2.0 NaN NaN f 5.0 4.0 NaN NaN h NaN NaN 2.0 30.0 i NaN NaN 3.0 40.0


Instruction:
How do I make a single legend for many subplots?

I am plotting the same type of information, but for different countries, with multiple subplots with Matplotlib. That is, I have nine plots on a 3x3 grid, all with the same for lines (of course, different values per line). However, I have not figured out how to put a single legend (since all nine subplots have the same lines) on the figure just once. How do I do that?

Output: There is also a nice function get_legend_handles_labels() you can call on the last axis (if you iterate over them) that would collect everything you need from label= arguments: handles, labels = ax.get_legend_handles_labels() fig.legend(handles, labels, loc='upper center') If the pyplot interface is being used instead of the Axes interface, use: handles, labels = plt.gca().get_legend_handles_labels() To remove legends from subplots, see Remove the legend on a matplotlib figure. To merge twinx legends, see Secondary axis with twinx(): how to add to legend.


Instruction:
What does "script xyz.py returned exit code 0" mean in Python?

I got a message saying script xyz.py returned exit code 0. What does this mean? What do the exit codes in Python mean? How many are there? Which ones are important?

Output: You're looking for calls to sys.exit(...) (exit(...) calls sys.exit(...)) in the script. The argument to that method is returned to the environment as the exit code. It's fairly likely that the script is never calling the exit(...) method, and that 0 is the default exit code.


Instruction:
What does `view()` do in PyTorch?

What does view() do to the tensor x? What do negative values mean? x = x.view(-1, 16 * 5 * 5)

Output: view() reshapes the tensor without copying memory, similar to numpy's reshape(). Given a tensor a with 16 elements: import torch a = torch.range(1, 16) To reshape this tensor to make it a 4 x 4 tensor, use: a = a.view(4, 4) Now a will be a 4 x 4 tensor. Note that after the reshape the total number of elements need to remain the same. Reshaping the tensor a to a 3 x 5 tensor would not be appropriate. What is the meaning of parameter -1? If there is any situation that you don't know how many rows you want but are sure of the number of columns, then you can specify this with a -1. (Note that you can extend this to tensors with more dimensions. Only one of the axis value can be -1). This is a way of telling the library: "give me a tensor that has these many columns and you compute the appropriate number of rows that is necessary to make this happen". This can be seen in this model definition code. After the line x = self.pool(F.relu(self.conv2(x))) in the forward function, you will have a 16 depth feature map. You have to flatten this to give it to the fully connected layer. So you tell PyTorch to reshape the tensor you obtained to have specific number of columns and tell it to decide the number of rows by itself.


Instruction:
pytest: assert almost equal

How to do assert almost equal with pytest for floats without resorting to something like: assert x - 0.00001 <= y <= x + 0.00001 More specifically it will be useful to know a neat solution for quickly comparing pairs of float, without unpacking them: assert (1.32, 2.4) == i_return_tuple_of_two_floats()

Output: I noticed that this question specifically asked about pytest. pytest 3.0 includes an approx() function (well, really class) that is very useful for this purpose. import pytest assert 2.2 == pytest.approx(2.3) # fails, default is ± 2.3e-06 assert 2.2 == pytest.approx(2.3, 0.1) # passes # also works the other way, in case you were worried: assert pytest.approx(2.3, 0.1) == 2.2 # passes


Instruction:
Check if a string contains a number

Most of the questions I've found are biased on the fact they're looking for letters in their numbers, whereas I'm looking for numbers in what I'd like to be a numberless string. I need to enter a string and check to see if it contains any numbers and if it does reject it. The function isdigit() only returns True if ALL of the characters are numbers. I just want to see if the user has entered a number so a sentence like "I own 1 dog" or something. Any ideas?

Output: You can use any function, with the str.isdigit function, like this def has_numbers(inputString): return any(char.isdigit() for char in inputString) has_numbers("I own 1 dog") # True has_numbers("I own no dog") # False Alternatively you can use a Regular Expression, like this import re def has_numbers(inputString): return bool(re.search(r'\d', inputString)) has_numbers("I own 1 dog") # True has_numbers("I own no dog") # False


Instruction:
How to write to a file, using the logging Python module?

How can I use the logging module in Python to write to a file? Every time I try to use it, it just prints out the message.

Output: An example of using logging.basicConfig rather than logging.fileHandler() logging.basicConfig(filename=logname, filemode='a', format='%(asctime)s,%(msecs)03d %(name)s %(levelname)s %(message)s', datefmt='%Y-%m-%d %H:%M:%S', level=logging.DEBUG) logging.info("Running Urban Planning") logger = logging.getLogger('urbanGUI') In order, the five parts do the following: set the output file (filename=logname) set it to append (filemode='a') rather than overwrite (filemode='w') determine the format of the output message (format=...) determine the format of the output date and time (datefmt='%Y-%m-%d %H:%M:%S') and determine the minimum message level it will accept (level=logging.DEBUG).


Instruction:
Convert a tensor to numpy array in Tensorflow?

How to convert a tensor into a numpy array when using Tensorflow with Python bindings?

Output: TensorFlow 2.x Eager Execution is enabled by default, so just call .numpy() on the Tensor object. import tensorflow as tf a = tf.constant([[1, 2], [3, 4]]) b = tf.add(a, 1) a.numpy() # array([[1, 2], # [3, 4]], dtype=int32) b.numpy() # array([[2, 3], # [4, 5]], dtype=int32) tf.multiply(a, b).numpy() # array([[ 2, 6], # [12, 20]], dtype=int32) See NumPy Compatibility for more. It is worth noting (from the docs), Numpy array may share a memory with the Tensor object. Any changes to one may be reflected in the other. Bold emphasis mine. A copy may or may not be returned, and this is an implementation detail based on whether the data is in CPU or GPU (in the latter case, a copy has to be made from GPU to host memory). But why am I getting the AttributeError: 'Tensor' object has no attribute 'numpy'?. A lot of folks have commented about this issue, there are a couple of possible reasons: TF 2.0 is not correctly installed (in which case, try re-installing), or TF 2.0 is installed, but eager execution is disabled for some reason. In such cases, call tf.compat.v1.enable_eager_execution() to enable it, or see below. If Eager Execution is disabled, you can build a graph and then run it through tf.compat.v1.Session: a = tf.constant([[1, 2], [3, 4]]) b = tf.add(a, 1) out = tf.multiply(a, b) out.eval(session=tf.compat.v1.Session()) # array([[ 2, 6], # [12, 20]], dtype=int32) See also TF 2.0 Symbols Map for a mapping of the old API to the new one.


Instruction:
Defining private module functions in python

According to http://www.faqs.org/docs/diveintopython/fileinfo_private.html: Like most languages, Python has the concept of private elements: Private functions, which can't be called from outside their module However, if I define two files: #a.py __num=1 and: #b.py import a print a.__num when i run b.py it prints out 1 without giving any exception. Is diveintopython wrong, or did I misunderstand something? And is there some way to do define a module's function as private?

Output: In Python, "privacy" depends on "consenting adults'" levels of agreement - you can't force it. A single leading underscore means you're not supposed to access it "from the outside" -- two leading underscores (w/o trailing underscores) carry the message even more forcefully... but, in the end, it still depends on social convention and consensus: Python's introspection is forceful enough that you can't handcuff every other programmer in the world to respect your wishes. ((Btw, though it's a closely held secret, much the same holds for C++: with most compilers, a simple #define private public line before #includeing your .h file is all it takes for wily coders to make hash of your "privacy"...!-))


Instruction:
Is it possible to ignore one single specific line with Pylint?

I have the following line in my header: import config.logging_settings This actually changes my Python logging settings, but Pylint thinks it is an unused import. I do not want to remove unused-import warnings in general, so is it possible to just ignore this one specific line? I wouldn't mind having a .pylintrc for this project, so answers changing a configuration file will be accepted. Otherwise, something like this will also be appreciated: import config.logging_settings # pylint: disable-this-line-in-some-way

Output: Message control is documented in the pylint FAQ: Is it possible to locally disable a particular message? Yes, this feature has been added in Pylint 0.11. This may be done by adding "# pylint: disable=some-message,another-one" at the desired block level or at the end of the desired line of code. You can use the message code or the symbolic names. For example, def test(): # Disable all the no-member violations in this function # pylint: disable=no-member ... # pylint: enable=no-member apply to a specific line only: global VAR # pylint: disable=global-statement or for less verbosity, disable the ONLY following line (pylint 2.10+): # pylint: disable-next=global-statement global VAR Pylint's manual also has further examples. There is a wiki that documents all pylint messages and their codes.


Instruction:
Overriding default values when using inheritance in dataclasses

I'm currently trying my hands on the dataclass constructions. I am currently stuck on trying to do some inheritance of a parent class. It looks like the order of the arguments are botched by my current approach such that the bool parameter in the child class is passed before the other parameters. This is causing a type error. from dataclasses import dataclass @dataclass class Parent: name: str age: int ugly: bool = False def print_name(self): print(self.name) def print_age(self): print(self.age) def print_id(self): print(f'The Name is {self.name} and {self.name} is {self.age} year old') @dataclass class Child(Parent): school: str ugly: bool = True jack = Parent('jack snr', 32, ugly=True) jack_son = Child('jack jnr', 12, school='havard', ugly=True) jack.print_id() jack_son.print_id() When I run this code I get this TypeError: TypeError: non-default argument 'school' follows default argument How do I fix this?

Output: The way dataclasses combines attributes prevents you from being able to use attributes with defaults in a base class and then use attributes without a default (positional attributes) in a subclass. That's because the attributes are combined by starting from the bottom of the MRO, and building up an ordered list of the attributes in first-seen order; overrides are kept in their original location. So Parent starts out with ['name', 'age', 'ugly'], where ugly has a default, and then Child adds ['school'] to the end of that list (with ugly already in the list). This means you end up with ['name', 'age', 'ugly', 'school'] and because school doesn't have a default, this results in an invalid argument listing for __init__. This is documented in PEP-557 Dataclasses, under inheritance: When the Data Class is being created by the @dataclass decorator, it looks through all of the class's base classes in reverse MRO (that is, starting at object) and, for each Data Class that it finds, adds the fields from that base class to an ordered mapping of fields. After all of the base class fields are added, it adds its own fields to the ordered mapping. All of the generated methods will use this combined, calculated ordered mapping of fields. Because the fields are in insertion order, derived classes override base classes. and under Specification: TypeError will be raised if a field without a default value follows a field with a default value. This is true either when this occurs in a single class, or as a result of class inheritance. You do have a few options here to avoid this issue. The first option is to use separate base classes to force fields with defaults into a later position in the MRO order. At all cost, avoid setting fields directly on classes that are to be used as base classes, such as Parent. The following class hierarchy works: # base classes with fields; fields without defaults separate from fields with. @dataclass class _ParentBase: name: str age: int @dataclass class _ParentDefaultsBase: ugly: bool = False @dataclass class _ChildBase(_ParentBase): school: str @dataclass class _ChildDefaultsBase(_ParentDefaultsBase): ugly: bool = True # public classes, deriving from base-with, base-without field classes # subclasses of public classes should put the public base class up front. @dataclass class Parent(_ParentDefaultsBase, _ParentBase): def print_name(self): print(self.name) def print_age(self): print(self.age) def print_id(self): print(f"The Name is {self.name} and {self.name} is {self.age} year old") @dataclass class Child(_ChildDefaultsBase, Parent, _ChildBase): pass By pulling out fields into separate base classes with fields without defaults and fields with defaults, and a carefully selected inheritance order, you can produce an MRO that puts all fields without defaults before those with defaults. The reversed MRO (ignoring object) for Child is: _ParentBase _ChildBase _ParentDefaultsBase Parent _ChildDefaultsBase Note that while Parent doesn't set any new fields, it does inherit the fields from _ParentDefaultsBase and should not end up 'last' in the field listing order; the above order puts _ChildDefaultsBase last so its fields 'win'. The dataclass rules are also satisfied; the classes with fields without defaults (_ParentBase and _ChildBase) precede the classes with fields with defaults (_ParentDefaultsBase and _ChildDefaultsBase). The result is Parent and Child classes with a sane field older, while Child is still a subclass of Parent: >>> from inspect import signature >>> signature(Parent) <Signature (name: str, age: int, ugly: bool = False) -> None> >>> signature(Child) <Signature (name: str, age: int, school: str, ugly: bool = True) -> None> >>> issubclass(Child, Parent) True and so you can create instances of both classes: >>> jack = Parent('jack snr', 32, ugly=True) >>> jack_son = Child('jack jnr', 12, school='havard', ugly=True) >>> jack Parent(name='jack snr', age=32, ugly=True) >>> jack_son Child(name='jack jnr', age=12, school='havard', ugly=True) Another option is to only use fields with defaults; you can still make in an error to not supply a school value, by raising one in __post_init__: _no_default = object() @dataclass class Child(Parent): school: str = _no_default ugly: bool = True def __post_init__(self): if self.school is _no_default: raise TypeError("__init__ missing 1 required argument: 'school'") but this does alter the field order; school ends up after ugly: <Signature (name: str, age: int, ugly: bool = True, school: str = <object object at 0x1101d1210>) -> None> and a type hint checker will complain about _no_default not being a string. You can also use the attrs project, which was the project that inspired dataclasses. It uses a different inheritance merging strategy; it pulls overridden fields in a subclass to the end of the fields list, so ['name', 'age', 'ugly'] in the Parent class becomes ['name', 'age', 'school', 'ugly'] in the Child class; by overriding the field with a default, attrs allows the override without needing to do a MRO dance. attrs supports defining fields without type hints, but lets stick to the supported type hinting mode by setting auto_attribs=True: import attr @attr.s(auto_attribs=True) class Parent: name: str age: int ugly: bool = False def print_name(self): print(self.name) def print_age(self): print(self.age) def print_id(self): print(f"The Name is {self.name} and {self.name} is {self.age} year old") @attr.s(auto_attribs=True) class Child(Parent): school: str ugly: bool = True


Instruction:
How can I display an image from a file in Jupyter Notebook?

I would like to use an IPython notebook as a way to interactively analyze some genome charts I am making with Biopython's GenomeDiagram module. While there is extensive documentation on how to use matplotlib to get graphs inline in IPython notebook, GenomeDiagram uses the ReportLab toolkit which I don't think is supported for inline graphing in IPython. I was thinking, however, that a way around this would be to write out the plot/genome diagram to a file and then open the image inline which would have the same result with something like this: gd_diagram.write("test.png", "PNG") display(file="test.png") However, I can't figure out how to do this - or know if it's possible. So does anyone know if images can be opened/displayed in IPython?

Output: Courtesy of this post, you can do the following: from IPython.display import Image Image(filename='test.png')


Instruction:
How to execute raw SQL in Flask-SQLAlchemy app

How do you execute raw SQL in SQLAlchemy? I have a python web app that runs on flask and interfaces to the database through SQLAlchemy. I need a way to run the raw SQL. The query involves multiple table joins along with Inline views. I've tried: connection = db.session.connection() connection.execute( <sql here> ) But I keep getting gateway errors.

Output: SQLAlchemy 2.0: with engine.connect() as connection: result = connection.execute(text('SELECT * FROM your_table')) # do something with the result.. SQLAlchemy 1.x: from sqlalchemy import text sql = text('select name from penguins') result = db.engine.execute(sql) names = [row[0] for row in result] print names Note that db.engine.execute() is "connectionless", which is deprecated in SQLAlchemy 2.0.


Instruction:
What is the pythonic way to detect the last element in a 'for' loop?

How can I treat the last element of the input specially, when iterating with a for loop? In particular, if there is code that should only occur "between" elements (and not "after" the last one), how can I structure the code? Currently, I write code like so: for i, data in enumerate(data_list): code_that_is_done_for_every_element if i != len(data_list) - 1: code_that_is_done_between_elements How can I simplify or improve this?

Output: Most of the times it is easier (and cheaper) to make the first iteration the special case instead of the last one: first = True for data in data_list: if first: first = False else: between_items() item() This will work for any iterable, even for those that have no len(): file = open('/path/to/file') for line in file: process_line(line) # No way of telling if this is the last line! Apart from that, I don't think there is a generally superior solution as it depends on what you are trying to do. For example, if you are building a string from a list, it's naturally better to use str.join() than using a for loop “with special case”. Using the same principle but more compact: for i, line in enumerate(data_list): if i > 0: between_items() item() Looks familiar, doesn't it? :) For @ofko, and others who really need to find out if the current value of an iterable without len() is the last one, you will need to look ahead: def lookahead(iterable): """Pass through all values from the given iterable, augmented by the information if there are more values to come after the current one (True), or if it is the last value (False). """ # Get an iterator and pull the first value. it = iter(iterable) try: last = next(it) except StopIteration: return # Run the iterator to exhaustion (starting from the second value). for val in it: # Report the *previous* value (more to come). yield last, True last = val # Report the last value. yield last, False Then you can use it like this: >>> for i, has_more in lookahead(range(3)): ... print(i, has_more) 0 True 1 True 2 False


Instruction:
remove None value from a list without removing the 0 value

This was my source I started with. My List L = [0, 23, 234, 89, None, 0, 35, 9] When I run this : L = filter(None, L) I get this results [23, 234, 89, 35, 9] But this is not what I need, what I really need is : [0, 23, 234, 89, 0, 35, 9] Because I'm calculating percentile of the data and the 0 make a lot of difference. How to remove the None value from a list without removing 0 value?

Output: >>> L = [0, 23, 234, 89, None, 0, 35, 9] >>> [x for x in L if x is not None] [0, 23, 234, 89, 0, 35, 9] Just for fun, here's how you can adapt filter to do this without using a lambda, (I wouldn't recommend this code - it's just for scientific purposes) >>> from operator import is_not >>> from functools import partial >>> L = [0, 23, 234, 89, None, 0, 35, 9] >>> list(filter(partial(is_not, None), L)) [0, 23, 234, 89, 0, 35, 9]


Instruction:
asyncio.gather vs asyncio.wait (vs asyncio.TaskGroup)

asyncio.gather and asyncio.wait seem to have similar uses: I have a bunch of async things that I want to execute/wait for (not necessarily waiting for one to finish before the next one starts). Since Python 3.11 there is yet another similar feature, asyncio.TaskGroup. They use a different syntax, and differ in some details, but it seems very un-pythonic to me to have several functions that have such a huge overlap in functionality. What am I missing?

Output: Although similar in general cases ("run and get results for many tasks"), each function has some specific functionality for other cases (and see also TaskGroup for Python 3.11+ below): asyncio.gather() Returns a Future instance, allowing high level grouping of tasks: import asyncio from pprint import pprint import random async def coro(tag): print(">", tag) await asyncio.sleep(random.uniform(1, 3)) print("<", tag) return tag loop = asyncio.get_event_loop() group1 = asyncio.gather(*[coro("group 1.{}".format(i)) for i in range(1, 6)]) group2 = asyncio.gather(*[coro("group 2.{}".format(i)) for i in range(1, 4)]) group3 = asyncio.gather(*[coro("group 3.{}".format(i)) for i in range(1, 10)]) all_groups = asyncio.gather(group1, group2, group3) results = loop.run_until_complete(all_groups) loop.close() pprint(results) All tasks in a group can be cancelled by calling group2.cancel() or even all_groups.cancel(). See also .gather(..., return_exceptions=True), asyncio.wait() Supports waiting to be stopped after the first task is done, or after a specified timeout, allowing lower level precision of operations: import asyncio import random async def coro(tag): print(">", tag) await asyncio.sleep(random.uniform(0.5, 5)) print("<", tag) return tag loop = asyncio.get_event_loop() tasks = [coro(i) for i in range(1, 11)] print("Get first result:") finished, unfinished = loop.run_until_complete( asyncio.wait(tasks, return_when=asyncio.FIRST_COMPLETED)) for task in finished: print(task.result()) print("unfinished:", len(unfinished)) print("Get more results in 2 seconds:") finished2, unfinished2 = loop.run_until_complete( asyncio.wait(unfinished, timeout=2)) for task in finished2: print(task.result()) print("unfinished2:", len(unfinished2)) print("Get all other results:") finished3, unfinished3 = loop.run_until_complete(asyncio.wait(unfinished2)) for task in finished3: print(task.result()) loop.close() TaskGroup (Python 3.11+) Update: Python 3.11 introduces TaskGroups which can "automatically" await more than one task without gather() or await(): # Python 3.11+ ONLY! async def main(): async with asyncio.TaskGroup() as tg: task1 = tg.create_task(some_coro(...)) task2 = tg.create_task(another_coro(...)) print("Both tasks have completed now.")


Instruction:
How to check if a user is logged in (how to properly use user.is_authenticated)?

I am looking over this website but just can't seem to figure out how to do this as it's not working. I need to check if the current site user is logged in (authenticated), and am trying: request.user.is_authenticated despite being sure that the user is logged in, it returns just: > I'm able to do other requests (from the first section in the url above), such as: request.user.is_active which returns a successful response.

Output: Update for Django 2.0 and later is_authenticated is a read-only attribute: if request.user.is_authenticated: # do something if the user is authenticated For Django 1.9 and older is_authenticated() was a function. Called like: if request.user.is_authenticated(): # do something if the user is authenticated As Peter Rowell pointed out, what may be tripping you up is that in the default Django template language, you don't tack on parenthesis to call functions. So you may have seen something like this in template code: {% if user.is_authenticated %} However, in Python code, it is indeed a method in the User class. NB: The method was removed in Django 2.0.


Instruction:
How to get Linux console window width in Python

Is there a way in python to programmatically determine the width of the console? I mean the number of characters that fits in one line without wrapping, not the pixel width of the window. Edit Looking for a solution that works on Linux

Output: Not sure why it is in the module shutil, but it landed there in Python 3.3. See: Querying the size of the output terminal >>> import shutil >>> shutil.get_terminal_size((80, 20)) # pass fallback os.terminal_size(columns=87, lines=23) # returns a named-tuple A low-level implementation is in the os module. Cross-platform—works under Linux, Mac OS, and Windows, probably other Unix-likes. There's a backport as well, though no longer relevant.


Instruction:
Why use Abstract Base Classes in Python?

Because I am used to the old ways of duck typing in Python, I fail to understand the need for ABC (abstract base classes). The help is good on how to use them. I tried to read the rationale in the PEP, but it went over my head. If I was looking for a mutable sequence container, I would check for __setitem__, or more likely try to use it (EAFP). I haven't come across a real life use for the numbers module, which does use ABCs, but that is the closest I have to understanding. Can anyone explain the rationale to me, please?

Output: Short version ABCs offer a higher level of semantic contract between clients and the implemented classes. Long version There is a contract between a class and its callers. The class promises to do certain things and have certain properties. There are different levels to the contract. At a very low level, the contract might include the name of a method or its number of parameters. In a staticly-typed language, that contract would actually be enforced by the compiler. In Python, you can use EAFP or type introspection to confirm that the unknown object meets this expected contract. But there are also higher-level, semantic promises in the contract. For example, if there is a __str__() method, it is expected to return a string representation of the object. It could delete all contents of the object, commit the transaction and spit a blank page out of the printer... but there is a common understanding of what it should do, described in the Python manual. That's a special case, where the semantic contract is described in the manual. What should the print() method do? Should it write the object to a printer or a line to the screen, or something else? It depends - you need to read the comments to understand the full contract here. A piece of client code that simply checks that the print() method exists has confirmed part of the contract - that a method call can be made, but not that there is agreement on the higher level semantics of the call. Defining an Abstract Base Class (ABC) is a way of producing a contract between the class implementers and the callers. It isn't just a list of method names, but a shared understanding of what those methods should do. If you inherit from this ABC, you are promising to follow all the rules described in the comments, including the semantics of the print() method. Python's duck-typing has many advantages in flexibility over static-typing, but it doesn't solve all the problems. ABCs offer an intermediate solution between the free-form of Python and the bondage-and-discipline of a staticly-typed language.


Instruction:
List files ONLY in the current directory

In Python, I only want to list all the files in the current directory ONLY. I do not want files listed from any sub directory or parent. There do seem to be similar solutions out there, but they don't seem to work for me. Here's my code snippet: import os for subdir, dirs, files in os.walk('./'): for file in files: do some stuff print file Let's suppose I have 2 files, holygrail.py and Tim inside my current directory. I have a folder as well and it contains two files - let's call them Arthur and Lancelot - inside it. When I run the script, this is what I get: holygrail.py Tim Arthur Lancelot I am happy with holygrail.py and Tim. But the two files, Arthur and Lancelot, I do not want listed.

Output: Just use os.listdir and os.path.isfile instead of os.walk. Example: import os files = [f for f in os.listdir('.') if os.path.isfile(f)] for f in files: # do something But be careful while applying this to other directory, like files = [f for f in os.listdir(somedir) if os.path.isfile(f)] which would not work because f is not a full path but relative to the current directory. Therefore, for filtering on another directory, do os.path.isfile(os.path.join(somedir, f)) (Thanks Causality for the hint)


Instruction:
Seaborn plots not showing up

I'm sure I'm forgetting something very simple, but I cannot get certain plots to work with Seaborn. If I do: import seaborn as sns Then any plots that I create as usual with matplotlib get the Seaborn styling (with the grey grid in the background). However, if I try to do one of the examples, such as: In [1]: import seaborn as sns In [2]: sns.set() In [3]: df = sns.load_dataset('iris') In [4]: sns.pairplot(df, hue='species', size=2.5) Out[4]: <seaborn.axisgrid.PairGrid at 0x3e59150> The pairplot function returns a PairGrid object, but the plot doesn't show up. I'm a little confused because matplotlib seems to be functioning properly, and the Seaborn styles are applied to other matplotlib plots, but the Seaborn functions don't seem to do anything. Does anybody have any idea what might be the problem?

Output: Plots created using seaborn need to be displayed like ordinary matplotlib plots. This can be done using the plt.show() function from matplotlib. Originally I posted the solution to use the already imported matplotlib object from seaborn (sns.plt.show()) however this is considered to be a bad practice. Therefore, simply directly import the _matplotlib.pyplot_ module and show your plots with import matplotlib.pyplot as plt plt.show() If the IPython notebook is used the inline backend can be invoked to remove the necessity of calling show after each plot. The respective magic is %matplotlib inline


Instruction:
How can I read large text files line by line, without loading them into memory?

I want to read a large file (>5GB), line by line, without loading its entire contents into memory. I cannot use readlines() since it creates a very large list in memory.

Output: Use a for loop on a file object to read it line-by-line. Use with open(...) to let a context manager ensure that the file is closed after reading: with open("log.txt") as infile: for line in infile: print(line)


Instruction:
Python/Json:Expecting property name enclosed in double quotes

I've been trying to figure out a good way to load JSON objects in Python. I send this json data: {'http://example.org/about': {'http://purl.org/dc/terms/title': [{'type': 'literal', 'value': "Anna's Homepage"}]}} to the backend where it will be received as a string then I used json.loads(data) to parse it. But each time I got the same exception : ValueError: Expecting property name enclosed in double quotes: line 1 column 2 (char 1) I googled it but nothing seems to work besides this solution json.loads(json.dumps(data)) which personally seems for me not that efficient since it accept any kind of data even the ones that are not in json format. Any suggestions will be much appreciated.

Output: This is not JSON: { 'http://example.org/about': { 'http://purl.org/dc/terms/title': [ {'type': 'literal', 'value': "Anna's Homepage"} ] } } This is JSON: { "http://example.org/about": { "http://purl.org/dc/terms/title": [ {"type": "literal", "value": "Anna's Homepage"} ] } } JSON specification - RFC7159 states that a string begins and ends with quotation mark. That is ". Single quote ' has no semantic meaning in JSON and is allowed only inside a string.


Instruction:
Using Python's os.path, how do I go up one directory?

I recently upgrade Django from v1.3.1 to v1.4. In my old settings.py I have TEMPLATE_DIRS = ( os.path.join(os.path.dirname( __file__ ), 'templates').replace('\\', '/'), # Put strings here, like "/home/html/django_templates" or "C:/www/django/templates". # Always use forward slashes, even on Windows. # Don't forget to use absolute paths, not relative paths. ) This will point to /Users/hobbes3/Sites/mysite/templates, but because Django v1.4 moved the project folder to the same level as the app folders, my settings.py file is now in /Users/hobbes3/Sites/mysite/mysite/ instead of /Users/hobbes3/Sites/mysite/. How do I use os.path to look at a directory one level above from __file__. In other words, I want /Users/hobbes3/Sites/mysite/mysite/settings.py to find /Users/hobbes3/Sites/mysite/templates using relative paths.

Output: os.path.abspath(os.path.join(os.path.dirname( __file__ ), '..', 'templates')) You can also use normpath to clean up the path, rather than abspath. However, in this situation, Django expects an absolute path rather than a relative path. For cross platform compatability, use os.pardir instead of '..'.


Instruction:
ValueError: setting an array element with a sequence

Why do the following code samples: np.array([[1, 2], [2, 3, 4]]) np.array([1.2, "abc"], dtype=float) all give the following error? ValueError: setting an array element with a sequence.

Output: Possible reason 1: trying to create a jagged array You may be creating an array from a list that isn't shaped like a multi-dimensional array: numpy.array([[1, 2], [2, 3, 4]]) # wrong! numpy.array([[1, 2], [2, [3, 4]]]) # wrong! In these examples, the argument to numpy.array contains sequences of different lengths. Those will yield this error message because the input list is not shaped like a "box" that can be turned into a multidimensional array. Possible reason 2: providing elements of incompatible types For example, providing a string as an element in an array of type float: numpy.array([1.2, "abc"], dtype=float) # wrong! If you really want to have a NumPy array containing both strings and floats, you could use the dtype object, which allows the array to hold arbitrary Python objects: numpy.array([1.2, "abc"], dtype=object)


Instruction:
How to redirect 'print' output to a file?

I want to redirect the print to a .txt file using Python. I have a for loop, which will print the output for each of my .bam file while I want to redirect all output to one file. So I tried to put: f = open('output.txt','w') sys.stdout = f at the beginning of my script. However I get nothing in the .txt file. My script is: #!/usr/bin/python import os,sys import subprocess import glob from os import path f = open('output.txt','w') sys.stdout = f path= '/home/xxx/nearline/bamfiles' bamfiles = glob.glob(path + '/*.bam') for bamfile in bamfiles: filename = bamfile.split('/')[-1] print 'Filename:', filename samtoolsin = subprocess.Popen(["/share/bin/samtools/samtools","view",bamfile], stdout=subprocess.PIPE,bufsize=1) linelist= samtoolsin.stdout.readlines() print 'Readlines finished!' So what's the problem? Any other way besides this sys.stdout? I need my result look like: Filename: ERR001268.bam Readlines finished! Mean: 233 SD: 10 Interval is: (213, 252)

Output: The most obvious way to do this would be to print to a file object: with open('out.txt', 'w') as f: print('Filename:', filename, file=f) # Python 3.x # print >> f, 'Filename:', filename # Python 2.x However, redirecting stdout also works for me. It is probably fine for a one-off script such as this: import sys orig_stdout = sys.stdout f = open('out.txt', 'w') sys.stdout = f for i in range(2): print('i = ', i) sys.stdout = orig_stdout f.close() Since Python 3.4 there's a simple context manager available to do this in the standard library: from contextlib import redirect_stdout with open('out.txt', 'w') as f: with redirect_stdout(f): print('data') Redirecting externally from the shell itself is another option, and often preferable: ./script.py > out.txt Other questions: What is the first filename in your script? I don't see it initialized. My first guess is that glob doesn't find any bamfiles, and therefore the for loop doesn't run. Check that the folder exists, and print out bamfiles in your script. Also, use os.path.join and os.path.basename to manipulate paths and filenames.


Instruction:
Generating a PNG with matplotlib when DISPLAY is undefined

I am trying to use networkx with Python. When I run this program it get this error. Is there anything missing? #!/usr/bin/env python import networkx as nx import matplotlib import matplotlib.pyplot import matplotlib.pyplot as plt G=nx.Graph() G.add_node(1) G.add_nodes_from([2,3,4,5,6,7,8,9,10]) #nx.draw_graphviz(G) #nx_write_dot(G, 'node.png') nx.draw(G) plt.savefig("/var/www/node.png") Traceback (most recent call last): File "graph.py", line 13, in <module> nx.draw(G) File "/usr/lib/pymodules/python2.5/networkx/drawing/nx_pylab.py", line 124, in draw cf=pylab.gcf() File "/usr/lib/pymodules/python2.5/matplotlib/pyplot.py", line 276, in gcf return figure() File "/usr/lib/pymodules/python2.5/matplotlib/pyplot.py", line 254, in figure **kwargs) File "/usr/lib/pymodules/python2.5/matplotlib/backends/backend_tkagg.py", line 90, in new_figure_manager window = Tk.Tk() File "/usr/lib/python2.5/lib-tk/Tkinter.py", line 1650, in __init__ self.tk = _tkinter.create(screenName, baseName, className, interactive, wantobjects, useTk, sync, use) _tkinter.TclError: no display name and no $DISPLAY environment variable I get a different error now: #!/usr/bin/env python import networkx as nx import matplotlib import matplotlib.pyplot import matplotlib.pyplot as plt matplotlib.use('Agg') G=nx.Graph() G.add_node(1) G.add_nodes_from([2,3,4,5,6,7,8,9,10]) #nx.draw_graphviz(G) #nx_write_dot(G, 'node.png') nx.draw(G) plt.savefig("/var/www/node.png") /usr/lib/pymodules/python2.5/matplotlib/__init__.py:835: UserWarning: This call to matplotlib.use() has no effect because the the backend has already been chosen; matplotlib.use() must be called *before* pylab, matplotlib.pyplot, or matplotlib.backends is imported for the first time. if warn: warnings.warn(_use_error_msg) Traceback (most recent call last): File "graph.py", line 15, in <module> nx.draw(G) File "/usr/lib/python2.5/site-packages/networkx-1.2.dev-py2.5.egg/networkx/drawing/nx_pylab.py", line 124, in draw cf=pylab.gcf() File "/usr/lib/pymodules/python2.5/matplotlib/pyplot.py", line 276, in gcf return figure() File "/usr/lib/pymodules/python2.5/matplotlib/pyplot.py", line 254, in figure **kwargs) File "/usr/lib/pymodules/python2.5/matplotlib/backends/backend_tkagg.py", line 90, in new_figure_manager window = Tk.Tk() File "/usr/lib/python2.5/lib-tk/Tkinter.py", line 1650, in __init__ self.tk = _tkinter.create(screenName, baseName, className, interactive, wantobjects, useTk, sync, use) _tkinter.TclError: no display name and no $DISPLAY environment variable I get a different error now: #!/usr/bin/env python import networkx as nx import matplotlib import matplotlib.pyplot import matplotlib.pyplot as plt matplotlib.use('Agg') G=nx.Graph() G.add_node(1) G.add_nodes_from([2,3,4,5,6,7,8,9,10]) #nx.draw_graphviz(G) #nx_write_dot(G, 'node.png') nx.draw(G) plt.savefig("/var/www/node.png") /usr/lib/pymodules/python2.5/matplotlib/__init__.py:835: UserWarning: This call to matplotlib.use() has no effect because the the backend has already been chosen; matplotlib.use() must be called *before* pylab, matplotlib.pyplot, or matplotlib.backends is imported for the first time. if warn: warnings.warn(_use_error_msg) Traceback (most recent call last): File "graph.py", line 15, in <module> nx.draw(G) File "/usr/lib/python2.5/site-packages/networkx-1.2.dev-py2.5.egg/networkx/drawing/nx_pylab.py", line 124, in draw cf=pylab.gcf() File "/usr/lib/pymodules/python2.5/matplotlib/pyplot.py", line 276, in gcf return figure() File "/usr/lib/pymodules/python2.5/matplotlib/pyplot.py", line 254, in figure **kwargs) File "/usr/lib/pymodules/python2.5/matplotlib/backends/backend_tkagg.py", line 90, in new_figure_manager window = Tk.Tk() File "/usr/lib/python2.5/lib-tk/Tkinter.py", line 1650, in __init__ self.tk = _tkinter.create(screenName, baseName, className, interactive, wantobjects, useTk, sync, use) _tkinter.TclError: no display name and no $DISPLAY environment variable

Output: The main problem is that (on your system) matplotlib chooses an x-using backend by default. I just had the same problem on one of my servers. The solution for me was to add the following code in a place that gets read before any other pylab/matplotlib/pyplot import: import matplotlib # Force matplotlib to not use any Xwindows backend. matplotlib.use('Agg') The alternative is to set it in your .matplotlibrc


Instruction:
Get IP address of visitors using Flask for Python

I'm making a website where users can log on and download files, using the Flask micro-framework (based on Werkzeug) which uses Python (2.6 in my case). I need to get the IP address of users when they log on (for logging purposes). Does anyone know how to do this? Surely there is a way to do it with Python?

Output: See the documentation on how to access the Request object and then get from this same Request object, the attribute remote_addr. Code example from flask import request from flask import jsonify @app.route("/get_my_ip", methods=["GET"]) def get_my_ip(): return jsonify({'ip': request.remote_addr}), 200 For more information see the Werkzeug documentation.


Instruction:
How to skip the headers when processing a csv file using Python?

I am using below referred code to edit a csv using Python. Functions called in the code form upper part of the code. Problem: I want the below referred code to start editing the csv from 2nd row, I want it to exclude 1st row which contains headers. Right now it is applying the functions on 1st row only and my header row is getting changed. in_file = open("tmob_notcleaned.csv", "rb") reader = csv.reader(in_file) out_file = open("tmob_cleaned.csv", "wb") writer = csv.writer(out_file) row = 1 for row in reader: row[13] = handle_color(row[10])[1].replace(" - ","").strip() row[10] = handle_color(row[10])[0].replace("-","").replace("(","").replace(")","").strip() row[14] = handle_gb(row[10])[1].replace("-","").replace(" ","").replace("GB","").strip() row[10] = handle_gb(row[10])[0].strip() row[9] = handle_oem(row[10])[1].replace("Blackberry","RIM").replace("TMobile","T-Mobile").strip() row[15] = handle_addon(row[10])[1].strip() row[10] = handle_addon(row[10])[0].replace(" by","").replace("FREE","").strip() writer.writerow(row) in_file.close() out_file.close() I tried to solve this problem by initializing row variable to 1 but it didn't work. Please help me in solving this issue.

Output: Your reader variable is an iterable, by looping over it you retrieve the rows. To make it skip one item before your loop, simply call next(reader, None) and ignore the return value. You can also simplify your code a little; use the opened files as context managers to have them closed automatically: with open("tmob_notcleaned.csv", "rb") as infile, open("tmob_cleaned.csv", "wb") as outfile: reader = csv.reader(infile) next(reader, None) # skip the headers writer = csv.writer(outfile) for row in reader: # process each row writer.writerow(row) # no need to close, the files are closed automatically when you get to this point. If you wanted to write the header to the output file unprocessed, that's easy too, pass the output of next() to writer.writerow(): headers = next(reader, None) # returns the headers or `None` if the input is empty if headers: writer.writerow(headers)


Instruction:
How to handle errors with boto3?

I am trying to figure how to do proper error handling with boto3. I am trying to create an IAM user: def create_user(username, iam_conn): try: user = iam_conn.create_user(UserName=username) return user except Exception as e: return e When the call to create_user succeeds, I get a neat object that contains the http status code of the API call and the data of the newly created user. Example: {'ResponseMetadata': {'HTTPStatusCode': 200, 'RequestId': 'omitted' }, u'User': {u'Arn': 'arn:aws:iam::omitted:user/omitted', u'CreateDate': datetime.datetime(2015, 10, 11, 17, 13, 5, 882000, tzinfo=tzutc()), u'Path': '/', u'UserId': 'omitted', u'UserName': 'omitted' } } This works great. But when this fails (like if the user already exists), I just get an object of type botocore.exceptions.ClientError with only text to tell me what went wrong. Example: ClientError('An error occurred (EntityAlreadyExists) when calling the CreateUser operation: User with name omitted already exists.',) This (AFAIK) makes error handling very hard because I can't just switch on the resulting http status code (409 for user already exists according to the AWS API docs for IAM). This makes me think that I must be doing something the wrong way. The optimal way would be for boto3 to never throw exceptions, but juts always return an object that reflects how the API call went. Can anyone enlighten me on this issue or point me in the right direction?

Output: Use the response contained within the exception. Here is an example: import boto3 from botocore.exceptions import ClientError try: iam = boto3.client('iam') user = iam.create_user(UserName='fred') print("Created user: %s" % user) except ClientError as e: if e.response['Error']['Code'] == 'EntityAlreadyExists': print("User already exists") else: print("Unexpected error: %s" % e) The response dict in the exception will contain the following: ['Error']['Code'] e.g. 'EntityAlreadyExists' or 'ValidationError' ['ResponseMetadata']['HTTPStatusCode'] e.g. 400 ['ResponseMetadata']['RequestId'] e.g. 'd2b06652-88d7-11e5-99d0-812348583a35' ['Error']['Message'] e.g. "An error occurred (EntityAlreadyExists) ..." ['Error']['Type'] e.g. 'Sender' For more information see: boto3 error handling botocore error handling [Updated: 2018-03-07] The AWS Python SDK has begun to expose service exceptions on clients (though not on resources) that you can explicitly catch, so it is now possible to write that code like this: import botocore import boto3 try: iam = boto3.client('iam') user = iam.create_user(UserName='fred') print("Created user: %s" % user) except iam.exceptions.EntityAlreadyExistsException: print("User already exists") except botocore.exceptions.ParamValidationError as e: print("Parameter validation error: %s" % e) except botocore.exceptions.ClientError as e: print("Unexpected error: %s" % e) Unfortunately, there is currently no documentation for these errors/exceptions but you can get a list of the core errors as follows: import botocore import boto3 [e for e in dir(botocore.exceptions) if e.endswith('Error')] Note that you must import both botocore and boto3. If you only import botocore then you will find that botocore has no attribute named exceptions. This is because the exceptions are dynamically populated into botocore by boto3. You can get a list of service-specific exceptions as follows (replace iam with the relevant service as needed): import boto3 iam = boto3.client('iam') [e for e in dir(iam.exceptions) if e.endswith('Exception')] [Updated: 2021-09-07] In addition to the aforementioned client exception method, there is also a third-party helper package named aws-error-utils.


Instruction:
python pandas remove duplicate columns

What is the easiest way to remove duplicate columns from a dataframe? I am reading a text file that has duplicate columns via: import pandas as pd df=pd.read_table(fname) The column names are: Time, Time Relative, N2, Time, Time Relative, H2, etc... All the Time and Time Relative columns contain the same data. I want: Time, Time Relative, N2, H2 All my attempts at dropping, deleting, etc such as: df=df.T.drop_duplicates().T Result in uniquely valued index errors: Reindexing only valid with uniquely valued index objects Sorry for being a Pandas noob. Any Suggestions would be appreciated. Additional Details Pandas version: 0.9.0 Python Version: 2.7.3 Windows 7 (installed via Pythonxy 2.7.3.0) data file (note: in the real file, columns are separated by tabs, here they are separated by 4 spaces): Time Time Relative [s] N2[%] Time Time Relative [s] H2[ppm] 2/12/2013 9:20:55 AM 6.177 9.99268e+001 2/12/2013 9:20:55 AM 6.177 3.216293e-005 2/12/2013 9:21:06 AM 17.689 9.99296e+001 2/12/2013 9:21:06 AM 17.689 3.841667e-005 2/12/2013 9:21:18 AM 29.186 9.992954e+001 2/12/2013 9:21:18 AM 29.186 3.880365e-005 ... etc ... 2/12/2013 2:12:44 PM 17515.269 9.991756+001 2/12/2013 2:12:44 PM 17515.269 2.800279e-005 2/12/2013 2:12:55 PM 17526.769 9.991754e+001 2/12/2013 2:12:55 PM 17526.769 2.880386e-005 2/12/2013 2:13:07 PM 17538.273 9.991797e+001 2/12/2013 2:13:07 PM 17538.273 3.131447e-005

Output: Here's a one line solution to remove columns based on duplicate column names: df = df.loc[:,~df.columns.duplicated()].copy() How it works: Suppose the columns of the data frame are ['alpha','beta','alpha'] df.columns.duplicated() returns a boolean array: a True or False for each column. If it is False then the column name is unique up to that point, if it is True then the column name is duplicated earlier. For example, using the given example, the returned value would be [False,False,True]. Pandas allows one to index using boolean values whereby it selects only the True values. Since we want to keep the unduplicated columns, we need the above boolean array to be flipped (ie [True, True, False] = ~[False,False,True]) Finally, df.loc[:,[True,True,False]] selects only the non-duplicated columns using the aforementioned indexing capability. The final .copy() is there to copy the dataframe to (mostly) avoid getting errors about trying to modify an existing dataframe later down the line. Note: the above only checks columns names, not column values. To remove duplicated indexes Since it is similar enough, do the same thing on the index: df = df.loc[~df.index.duplicated(),:].copy() To remove duplicates by checking values without transposing Update and caveat: please be careful in applying this. Per the counter-example provided by DrWhat in the comments, this solution may not have the desired outcome in all cases. df = df.loc[:,~df.apply(lambda x: x.duplicated(),axis=1).all()].copy() This avoids the issue of transposing. Is it fast? No. Does it work? In some cases. Here, try it on this: # create a large(ish) dataframe ldf = pd.DataFrame(np.random.randint(0,100,size= (736334,1312))) #to see size in gigs #ldf.memory_usage().sum()/1e9 #it's about 3 gigs # duplicate a column ldf.loc[:,'dup'] = ldf.loc[:,101] # take out duplicated columns by values ldf = ldf.loc[:,~ldf.apply(lambda x: x.duplicated(),axis=1).all()].copy()


Instruction:
How to use "raise" keyword in Python

I have read the official definition of "raise", but I still don't quite understand what it does. In simplest terms, what is "raise"? Example usage would help.

Output: It has two purposes. jackcogdill has given the first one: It's used for raising your own errors. if something: raise Exception('My error!') The second is to reraise the current exception in an exception handler, so that it can be handled further up the call stack. try: generate_exception() except SomeException as e: if not can_handle(e): raise handle_exception(e)


Instruction:
Appending to an empty DataFrame in Pandas?

Is it possible to append to an empty data frame that doesn't contain any indices or columns? I have tried to do this, but keep getting an empty dataframe at the end. e.g. import pandas as pd df = pd.DataFrame() data = ['some kind of data here' --> I have checked the type already, and it is a dataframe] df.append(data) The result looks like this: Empty DataFrame Columns: [] Index: []

Output: The answers are very useful, but since pandas.DataFrame.append was deprecated (as already mentioned by various users), and the answers using pandas.concat are not "Runnable Code Snippets" I would like to add the following snippet: import pandas as pd df = pd.DataFrame(columns =['name','age']) row_to_append = pd.DataFrame([{'name':"Alice", 'age':"25"},{'name':"Bob", 'age':"32"}]) df = pd.concat([df,row_to_append]) So df is now: name age 0 Alice 25 1 Bob 32


Instruction:
Are nested try/except blocks in Python a good programming practice?

I'm writing my own container, which needs to give access to a dictionary inside by attribute calls. The typical use of the container would be like this: dict_container = DictContainer() dict_container['foo'] = bar ... print dict_container.foo I know that it might be stupid to write something like this, but that's the functionality I need to provide. I was thinking about implementing this in a following way: def __getattribute__(self, item): try: return object.__getattribute__(item) except AttributeError: try: return self.dict[item] except KeyError: print "The object doesn't have such attribute" I'm not sure whether nested try/except blocks are a good practice, so another way would be to use hasattr() and has_key(): def __getattribute__(self, item): if hasattr(self, item): return object.__getattribute__(item) else: if self.dict.has_key(item): return self.dict[item] else: raise AttributeError("some customised error") Or to use one of them and one try catch block like this: def __getattribute__(self, item): if hasattr(self, item): return object.__getattribute__(item) else: try: return self.dict[item] except KeyError: raise AttributeError("some customised error") Which option is most Pythonic and elegant?

Output: Your first example is perfectly fine. Even the official Python documentation recommends this style known as EAFP. Personally, I prefer to avoid nesting when it's not necessary: def __getattribute__(self, item): try: return object.__getattribute__(self, item) except AttributeError: pass # Fallback to dict try: return self.dict[item] except KeyError: raise AttributeError("The object doesn't have such attribute") from None PS. has_key() has been deprecated for a long time in Python 2. Use item in self.dict instead.


Instruction:
Elegant Python function to convert CamelCase to snake_case?

Example: >>> convert('CamelCase') 'camel_case'

Output: Camel case to snake case import re name = 'CamelCaseName' name = re.sub(r'(?<!^)(?=[A-Z])', '_', name).lower() print(name) # camel_case_name If you do this many times and the above is slow, compile the regex beforehand: pattern = re.compile(r'(?<!^)(?=[A-Z])') name = pattern.sub('_', name).lower() Note that this and immediately following regex use a zero-width match, which is not handled correctly by Python 3.6 or earlier. See further below for alternatives that don't use lookahead/lookbehind if you need to support older EOL Python. If you want to avoid converting "HTTPHeader" into "h_t_t_p_header", you can use this variant with regex alternation: pattern = re.compile(r"(?<=[a-z])(?=[A-Z])|(?<=[A-Z])(?=[A-Z][a-z])") name = pattern.sub('_', name).lower() See Regex101.com for test cases (that don't include final lowercase). You can improve readability with ?x or re.X: pattern = re.compile( r""" (?<=[a-z]) # preceded by lowercase (?=[A-Z]) # followed by uppercase | # OR (?<=[A-Z]) # preceded by lowercase (?=[A-Z][a-z]) # followed by uppercase, then lowercase """, re.X, ) If you use the regex module instead of re, you can use the more readable POSIX character classes (which are not limited to ASCII). pattern = re.compile( r""" (?<=[[:lower:]]) # preceded by lowercase (?=[[:upper:]]) # followed by uppercase | # OR (?<[[:upper:]]) # preceded by lower (?=[[:upper:]][[:lower:]]) # followed by upper then lower """, re.X, ) Another way to handle more advanced cases without relying on lookahead/lookbehind, using two substitution passes: def camel_to_snake(name): name = re.sub('(.)([A-Z][a-z]+)', r'\1_\2', name) return re.sub('([a-z0-9])([A-Z])', r'\1_\2', name).lower() print(camel_to_snake('camel2_camel2_case')) # camel2_camel2_case print(camel_to_snake('getHTTPResponseCode')) # get_http_response_code print(camel_to_snake('HTTPResponseCodeXYZ')) # http_response_code_xyz To add also cases with two underscores or more: def to_snake_case(name): name = re.sub('(.)([A-Z][a-z]+)', r'\1_\2', name) name = re.sub('__([A-Z])', r'_\1', name) name = re.sub('([a-z0-9])([A-Z])', r'\1_\2', name) return name.lower() Snake case to pascal case name = 'snake_case_name' name = ''.join(word.title() for word in name.split('_')) print(name) # SnakeCaseName


Instruction:
Convert UTC datetime string to local datetime

I've never had to convert time to and from UTC. Recently had a request to have my app be timezone aware, and I've been running myself in circles. Lots of information on converting local time to UTC, which I found fairly elementary (maybe I'm doing that wrong as well), but I can not find any information on easily converting the UTC time to the end-users timezone. In a nutshell, and android app sends me (appengine app) data and within that data is a timestamp. To store that timestamp to utc time I am using: datetime.utcfromtimestamp(timestamp) That seems to be working. When my app stores the data, it is being store as 5 hours ahead (I am EST -5) The data is being stored on appengine's BigTable, and when retrieved it comes out as a string like so: "2011-01-21 02:37:21" How do I convert this string to a DateTime in the users correct time zone? Also, what is the recommended storage for a users timezone information? (How do you typically store tz info ie: "-5:00" or "EST" etc etc ?) I'm sure the answer to my first question might contain a parameter the answers the second.

Output: If you don't want to provide your own tzinfo objects, check out the python-dateutil library. It provides tzinfo implementations on top of a zoneinfo (Olson) database such that you can refer to time zone rules by a somewhat canonical name. from datetime import datetime from dateutil import tz # METHOD 1: Hardcode zones: from_zone = tz.gettz('UTC') to_zone = tz.gettz('America/New_York') # METHOD 2: Auto-detect zones: from_zone = tz.tzutc() to_zone = tz.tzlocal() # Since datetime.utcnow() is deprecated since version 3.12 use datetime.now() # utc = datetime.now() utc = datetime.strptime('2011-01-21 02:37:21', '%Y-%m-%d %H:%M:%S') # Tell the datetime object that it's in UTC time zone since # datetime objects are 'naive' by default utc = utc.replace(tzinfo=from_zone) # Convert time zone central = utc.astimezone(to_zone) Edit Expanded example to show strptime usage Edit 2 Fixed API usage to show better entry point method Edit 3 Included auto-detect methods for timezones (Yarin)


Instruction:
How to get the parents of a Python class?

How can I get the parent class(es) of a Python class?

Output: Use the following attribute: cls.__bases__ From the docs: The tuple of base classes of a class object. Example: >>> str.__bases__ (<class 'object'>,) Another example: >>> class A(object): ... pass ... >>> class B(object): ... pass ... >>> class C(A, B): ... pass ... >>> C.__bases__ (<class '__main__.A'>, <class '__main__.B'>)


Instruction:
How do I get a list of all the duplicate items using pandas in python?

I have a list of items that likely has some export issues. I would like to get a list of the duplicate items so I can manually compare them. When I try to use pandas duplicated method, it only returns the first duplicate. Is there a a way to get all of the duplicates and not just the first one? A small subsection of my dataset looks like this: ID,ENROLLMENT_DATE,TRAINER_MANAGING,TRAINER_OPERATOR,FIRST_VISIT_DATE 1536D,12-Feb-12,"06DA1B3-Lebanon NH",,15-Feb-12 F15D,18-May-12,"06405B2-Lebanon NH",,25-Jul-12 8096,8-Aug-12,"0643D38-Hanover NH","0643D38-Hanover NH",25-Jun-12 A036,1-Apr-12,"06CB8CF-Hanover NH","06CB8CF-Hanover NH",9-Aug-12 8944,19-Feb-12,"06D26AD-Hanover NH",,4-Feb-12 1004E,8-Jun-12,"06388B2-Lebanon NH",,24-Dec-11 11795,3-Jul-12,"0649597-White River VT","0649597-White River VT",30-Mar-12 30D7,11-Nov-12,"06D95A3-Hanover NH","06D95A3-Hanover NH",30-Nov-11 3AE2,21-Feb-12,"06405B2-Lebanon NH",,26-Oct-12 B0FE,17-Feb-12,"06D1B9D-Hartland VT",,16-Feb-12 127A1,11-Dec-11,"064456E-Hanover NH","064456E-Hanover NH",11-Nov-12 161FF,20-Feb-12,"0643D38-Hanover NH","0643D38-Hanover NH",3-Jul-12 A036,30-Nov-11,"063B208-Randolph VT","063B208-Randolph VT", 475B,25-Sep-12,"06D26AD-Hanover NH",,5-Nov-12 151A3,7-Mar-12,"06388B2-Lebanon NH",,16-Nov-12 CA62,3-Jan-12,,, D31B,18-Dec-11,"06405B2-Lebanon NH",,9-Jan-12 20F5,8-Jul-12,"0669C50-Randolph VT",,3-Feb-12 8096,19-Dec-11,"0649597-White River VT","0649597-White River VT",9-Apr-12 14E48,1-Aug-12,"06D3206-Hanover NH",, 177F8,20-Aug-12,"063B208-Randolph VT","063B208-Randolph VT",5-May-12 553E,11-Oct-12,"06D95A3-Hanover NH","06D95A3-Hanover NH",8-Mar-12 12D5F,18-Jul-12,"0649597-White River VT","0649597-White River VT",2-Nov-12 C6DC,13-Apr-12,"06388B2-Lebanon NH",, 11795,27-Feb-12,"0643D38-Hanover NH","0643D38-Hanover NH",19-Jun-12 17B43,11-Aug-12,,,22-Oct-12 A036,11-Aug-12,"06D3206-Hanover NH",,19-Jun-12 My code looks like this currently: df_bigdata_duplicates = df_bigdata[df_bigdata.duplicated(cols='ID')] There area a couple duplicate items. But, when I use the above code, I only get the first item. In the API reference, I see how I can get the last item, but I would like to have all of them so I can visually inspect them to see why I am getting the discrepancy. So, in this example I would like to get all three A036 entries and both 11795 entries and any other duplicated entries, instead of the just first one. Any help is most appreciated.

Output: Method #1: print all rows where the ID is one of the IDs in duplicated: >>> import pandas as pd >>> df = pd.read_csv("dup.csv") >>> ids = df["ID"] >>> df[ids.isin(ids[ids.duplicated()])].sort_values("ID") ID ENROLLMENT_DATE TRAINER_MANAGING TRAINER_OPERATOR FIRST_VISIT_DATE 24 11795 27-Feb-12 0643D38-Hanover NH 0643D38-Hanover NH 19-Jun-12 6 11795 3-Jul-12 0649597-White River VT 0649597-White River VT 30-Mar-12 18 8096 19-Dec-11 0649597-White River VT 0649597-White River VT 9-Apr-12 2 8096 8-Aug-12 0643D38-Hanover NH 0643D38-Hanover NH 25-Jun-12 12 A036 30-Nov-11 063B208-Randolph VT 063B208-Randolph VT NaN 3 A036 1-Apr-12 06CB8CF-Hanover NH 06CB8CF-Hanover NH 9-Aug-12 26 A036 11-Aug-12 06D3206-Hanover NH NaN 19-Jun-12 but I couldn't think of a nice way to prevent repeating ids so many times. I prefer method #2: groupby on the ID. >>> pd.concat(g for _, g in df.groupby("ID") if len(g) > 1) ID ENROLLMENT_DATE TRAINER_MANAGING TRAINER_OPERATOR FIRST_VISIT_DATE 6 11795 3-Jul-12 0649597-White River VT 0649597-White River VT 30-Mar-12 24 11795 27-Feb-12 0643D38-Hanover NH 0643D38-Hanover NH 19-Jun-12 2 8096 8-Aug-12 0643D38-Hanover NH 0643D38-Hanover NH 25-Jun-12 18 8096 19-Dec-11 0649597-White River VT 0649597-White River VT 9-Apr-12 3 A036 1-Apr-12 06CB8CF-Hanover NH 06CB8CF-Hanover NH 9-Aug-12 12 A036 30-Nov-11 063B208-Randolph VT 063B208-Randolph VT NaN 26 A036 11-Aug-12 06D3206-Hanover NH NaN 19-Jun-12


Instruction:
Find out how much memory is being used by an object in Python

How would you go about finding out how much memory is being used by an object? I know it is possible to find out how much is used by a block of code, but not by an instantiated object (anytime during its life), which is what I want.

Output: There's no easy way to find out the memory size of a python object. One of the problems you may find is that Python objects - like lists and dicts - may have references to other python objects (in this case, what would your size be? The size containing the size of each object or not?). There are some pointers overhead and internal structures related to object types and garbage collection. Finally, some python objects have non-obvious behaviors. For instance, lists reserve space for more objects than they have, most of the time; dicts are even more complicated since they can operate in different ways (they have a different implementation for small number of keys and sometimes they over allocate entries). There is a big chunk of code (and an updated big chunk of code) out there to try to best approximate the size of a python object in memory. You may also want to check some old description about PyObject (the internal C struct that represents virtually all python objects).


Instruction:
pip installs packages successfully, but executables are not found from the command line

I installed Python 2.7 and pip following these instructions, on Mac OS X v10.10.3 (Yosemite). I can successfully install packages and import them inside my Python environment and Python scripts. However, any executable associated with a package that can be called from the command line in the terminal are not found. For example, I tried to install a package called "rosdep" as instructed here. I can run sudo pip install -U rosdep which installs without errors and the corresponding files are located in /opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages. However, sudo rosdep init subsequently reports that the rosdep command is not found. I even tried adding the above site-packages path to my $PATH, but the executables are still not found on the command line, even though the packages work perfectly from within python. Why does this happen, and how can I fix it? See also: Unable to import a module that is definitely installed - these two problems often have the same root cause.

Output: Check your $PATH environment variable. tox has a command line mode: $ pip list | grep tox tox (2.3.1) Where is it? (The 2.7 stuff doesn't matter much here, sub in any 3.x and pip's behaving pretty much the same way) $ which tox /opt/local/Library/Frameworks/Python.framework/Versions/2.7/bin/tox And what's in my $PATH? $ echo $PATH /opt/chefdk/bin:/opt/chefdk/embedded/bin:/opt/local/bin:..../opt/local/Library/Frameworks/Python.framework/Versions/2.7/bin... Notice the /opt/local/Library/Frameworks/Python.framework/Versions/2.7/bin? That's what allows finding my pip-installed stuff Now, to see where things are from Python, try doing this (substitute rosdep for tox). $python >>> import tox >>> tox.__file__ that prints out: '/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/tox/__init__.pyc' Now, cd to the directory right above lib in the above. Do you see a bin directory? Do you see rosdep in that bin? If so, try adding the bin to your $PATH. $ cd /opt/local/Library/Frameworks/Python.framework/Versions/2.7 $ ls -1 Output: Headers Python Resources bin include lib man share


Instruction:
How can I convert an RGB image into grayscale in Python?

I'm trying to use matplotlib to read in an RGB image and convert it to grayscale. In matlab I use this: img = rgb2gray(imread('image.png')); In the matplotlib tutorial they don't cover it. They just read in the image import matplotlib.image as mpimg img = mpimg.imread('image.png') and then they slice the array, but that's not the same thing as converting RGB to grayscale from what I understand. lum_img = img[:,:,0] I find it hard to believe that numpy or matplotlib doesn't have a built-in function to convert from rgb to gray. Isn't this a common operation in image processing? I wrote a very simple function that works with the image imported using imread in 5 minutes. It's horribly inefficient, but that's why I was hoping for a professional implementation built-in. Sebastian has improved my function, but I'm still hoping to find the built-in one. matlab's (NTSC/PAL) implementation: import numpy as np def rgb2gray(rgb): r, g, b = rgb[:,:,0], rgb[:,:,1], rgb[:,:,2] gray = 0.2989 * r + 0.5870 * g + 0.1140 * b return gray

Output: How about doing it with Pillow: from PIL import Image img = Image.open('image.png').convert('L') img.save('greyscale.png') If an alpha (transparency) channel is present in the input image and should be preserved, use mode LA: img = Image.open('image.png').convert('LA') Using matplotlib and the formula Y' = 0.2989 R + 0.5870 G + 0.1140 B you could do: import numpy as np import matplotlib.pyplot as plt import matplotlib.image as mpimg def rgb2gray(rgb): return np.dot(rgb[...,:3], [0.2989, 0.5870, 0.1140]) img = mpimg.imread('image.png') gray = rgb2gray(img) plt.imshow(gray, cmap=plt.get_cmap('gray'), vmin=0, vmax=1) plt.show()


Instruction:
Find the unique values in a column and then sort them

I have a pandas dataframe. I want to print the unique values of one of its columns in ascending order. This is how I am doing it: import pandas as pd df = pd.DataFrame({'A':[1,1,3,2,6,2,8]}) a = df['A'].unique() print a.sort() The problem is that I am getting a None for the output.

Output: sorted(iterable): Return a new sorted list from the items in iterable. CODE import pandas as pd df = pd.DataFrame({'A':[1,1,3,2,6,2,8]}) a = df['A'].unique() print(sorted(a)) OUTPUT [1, 2, 3, 6, 8]


Instruction:
Simplify Chained Comparison

I have an integer value x, and I need to check if it is between a start and end values, so I write the following statements: if x >= start and x <= end: # do stuff This statement gets underlined, and the tooltip tells me that I must simplify chained comparison As far as I can tell, that comparison is about as simple as they come. What have I missed here?

Output: In Python you can "chain" comparison operations which just means they are "and"ed together. In your case, it'd be like this: if start <= x <= end: Reference: https://docs.python.org/3/reference/expressions.html#comparisons


Instruction:
Check if string ends with one of the strings from a list

What is the pythonic way of writing the following code? extensions = ['.mp3','.avi'] file_name = 'test.mp3' for extension in extensions: if file_name.endswith(extension): #do stuff I have a vague memory that the explicit declaration of the for loop can be avoided and be written in the if condition. Is this true?

Output: str.endswith also accepts a tuple: >>> 'test.mp3'.endswith(('.mp3', '.avi')) True


Instruction:
How can I verify if one list is a subset of another?

I need to verify if a list is a subset of another - a boolean return is all I seek. Is testing equality on the smaller list after an intersection the fastest way to do this? Performance is of utmost importance given the number of datasets that need to be compared. Adding further facts based on discussions: Will either of the lists be the same for many tests? It does as one of them is a static lookup table. Does it need to be a list? It does not - the static lookup table can be anything that performs best. The dynamic one is a dict from which we extract the keys to perform a static lookup on. What would be the optimal solution given the scenario?

Output: Use set.issubset Example: a = {1,2} b = {1,2,3} a.issubset(b) # True a = {1,2,4} b = {1,2,3} a.issubset(b) # False The performant function Python provides for this is set.issubset. It does have a few restrictions that make it unclear if it's the answer to your question, however. A list may contain items multiple times and has a specific order. A set does not. Additionally, sets only work on hashable objects. Are you asking about subset or subsequence (which means you'll want a string search algorithm)? Will either of the lists be the same for many tests? What are the datatypes contained in the list? And for that matter, does it need to be a list? Your other post intersect a dict and list made the types clearer and did get a recommendation to use dictionary key views for their set-like functionality. In that case it was known to work because dictionary keys behave like a set (so much so that before we had sets in Python we used dictionaries). One wonders how the issue got less specific in three hours.


Instruction:
How do I read image data from a URL in Python?

What I'm trying to do is fairly simple when we're dealing with a local file, but the problem comes when I try to do this with a remote URL. Basically, I'm trying to create a PIL image object from a file pulled from a URL. Sure, I could always just fetch the URL and store it in a temp file, then open it into an image object, but that feels very inefficient. Here's what I have: Image.open(urlopen(url)) It flakes out complaining that seek() isn't available, so then I tried this: Image.open(urlopen(url).read()) But that didn't work either. Is there a Better Way to do this, or is writing to a temporary file the accepted way of doing this sort of thing?

Output: The following works for Python 3: from PIL import Image import requests im = Image.open(requests.get(url, stream=True).raw) References: https://github.com/python-pillow/Pillow/pull/1151 https://github.com/python-pillow/Pillow/blob/master/CHANGES.rst#280-2015-04-01


Instruction:
How to iterate over columns of a pandas dataframe

I have this code using Pandas in Python: all_data = {} for ticker in ['FIUIX', 'FSAIX', 'FSAVX', 'FSTMX']: all_data[ticker] = web.get_data_yahoo(ticker, '1/1/2010', '1/1/2015') prices = DataFrame({tic: data['Adj Close'] for tic, data in all_data.iteritems()}) returns = prices.pct_change() I know I can run a regression like this: regs = sm.OLS(returns.FIUIX,returns.FSTMX).fit() but how can I do this for each column in the dataframe? Specifically, how can I iterate over columns, in order to run the regression on each? Specifically, I want to regress each other ticker symbol (FIUIX, FSAIX and FSAVX) on FSTMX, and store the residuals for each regression. I've tried various versions of the following, but nothing I've tried gives the desired result: resids = {} for k in returns.keys(): reg = sm.OLS(returns[k],returns.FSTMX).fit() resids[k] = reg.resid Is there something wrong with the returns[k] part of the code? How can I use the k value to access a column? Or else is there a simpler approach?

Output: Old answer: for column in df: print(df[column]) The previous answer still works, but was added around the time of pandas 0.16.0. Better versions are available. Now you can do: for series_name, series in df.items(): print(series_name) print(series)


Instruction:
How do I check that multiple keys are in a dict in a single pass?

I want to do something like: foo = { 'foo': 1, 'zip': 2, 'zam': 3, 'bar': 4 } if ("foo", "bar") in foo: #do stuff How do I check whether both foo and bar are in dict foo?

Output: Well, you could do this: >>> if all(k in foo for k in ("foo","bar")): ... print "They're there!" ... They're there!


Instruction:
read subprocess stdout line by line

My python script uses subprocess to call a linux utility that is very noisy. I want to store all of the output to a log file and show some of it to the user. I thought the following would work, but the output doesn't show up in my application until the utility has produced a significant amount of output. # fake_utility.py, just generates lots of output over time import time i = 0 while True: print(hex(i)*512) i += 1 time.sleep(0.5) In the parent process: import subprocess proc = subprocess.Popen(['python', 'fake_utility.py'], stdout=subprocess.PIPE) for line in proc.stdout: # the real code does filtering here print("test:", line.rstrip()) The behavior I really want is for the filter script to print each line as it is received from the subprocess, like tee does but within Python code. What am I missing? Is this even possible?

Output: I think the problem is with the statement for line in proc.stdout, which reads the entire input before iterating over it. The solution is to use readline() instead: #filters output import subprocess proc = subprocess.Popen(['python','fake_utility.py'],stdout=subprocess.PIPE) while True: line = proc.stdout.readline() if not line: break #the real code does filtering here print "test:", line.rstrip() Of course you still have to deal with the subprocess' buffering. Note: according to the documentation the solution with an iterator should be equivalent to using readline(), except for the read-ahead buffer, but (or exactly because of this) the proposed change did produce different results for me (Python 2.5 on Windows XP).


Instruction:
pandas three-way joining multiple dataframes on columns

I have 3 CSV files. Each has the first column as the (string) names of people, while all the other columns in each dataframe are attributes of that person. How can I "join" together all three CSV documents to create a single CSV with each row having all the attributes for each unique value of the person's string name? The join() function in pandas specifies that I need a multiindex, but I'm confused about what a hierarchical indexing scheme has to do with making a join based on a single index.

Output: Zero's answer is basically a reduce operation. If I had more than a handful of dataframes, I'd put them in a list like this (generated via list comprehensions or loops or whatnot): dfs = [df0, df1, df2, ..., dfN] Assuming they have a common column, like name in your example, I'd do the following: import functools as ft df_final = ft.reduce(lambda left, right: pd.merge(left, right, on='name'), dfs) That way, your code should work with whatever number of dataframes you want to merge.


Instruction:
Is there a math nCr function in Python?

Is there a built-in nCr (n choose r) function included in the Python math library like the one shown below? I understand that the computation can be programmed, but I thought I'd check to see if it's built-in before I do.

Output: On Python 3.8+, use math.comb: >>> from math import comb >>> comb(10, 3) 120 For older versions of Python, you can use the following program: import operator as op from functools import reduce def ncr(n, r): r = min(r, n-r) numer = reduce(op.mul, range(n, n-r, -1), 1) denom = reduce(op.mul, range(1, r+1), 1) return numer // denom # or / in Python 2


Instruction:
How do I upgrade to Python 3.6 with Conda?

I want to get the latest version of Python to use f-strings in my code. Currently my version is (python -V): Python 3.5.2 :: Anaconda 4.2.0 (x86_64) How would I upgrade to Python 3.6?

Output: Anaconda had not updated Python internally to 3.6, but later versions of Anaconda has a Python 3.6 version here. a) Method 1 If you wanted to update, you will type conda update python To update Anaconda, type conda update conda If you want to upgrade between major python versions, like 3.5 to 3.6, you'll have to do conda install python=$pythonversion$ b) Method 2 - Create a new environment (the better method) conda create --name py36 python=3.6 c) To get the absolute latest Python (3.6.5 at time of writing) conda create --name py365 python=3.6.5 --channel conda-forge You can see all this from here. Also, refer to this for force upgrading.


Instruction:
How can I use the variables from "views.py" in JavasScript, "<script></script>" in a Django template?

When I render a page using the Django template renderer, I can pass in a dictionary variable containing various values to manipulate them in the page using {{ myVar }}. Is there a way to access the same variable in JavaScript, <script></script> (perhaps using the DOM; I don't know how Django makes the variables accessible)? I want to be able to look up details using an Ajax lookup based on the values contained in the variables passed in.

Output: The {{variable}} is substituted directly into the HTML. Do a view source; it isn't a "variable" or anything like it. It's just rendered text. Having said that, you can put this kind of substitution into your JavaScript. <script type="text/javascript"> var a = "{{someDjangoVariable}}"; </script> This gives you "dynamic" JavaScript code.